Geometric Structure and Polynomial-time Algorithm of Game Equilibria

Whether game equilibria can be efficiently solved has been an open question since Nash equilibrium[1] is proposed. The absence of such a polynomial-time algorithm has indications and consequences in three different fields.

• •

  In algorithmic game theory, the most widely used algorithms for approximating equilibria are no-regret[2, 3], self-play[4, 5], and their variants.

  • –

    Neither of them converges to Nash equilibria in general static games. No-regret converges to coarse correlated equilibria, and self-play converges to strict Nash equilibria in static games that satisfy the fictitious play property.

  • –

    When applied to dynamic games, neither of them guarantees the approximated equilibria to be perfect, namely, being a Nash equilibrium at every stage of the game, and thus non-perfect equilibria are not optimal in dynamic games.

• •

  In MARL, there are problems known as non-stationarity and curse of multiagency[6], and these two problems are related to problems in algorithmic game theory, such that there is currently no algorithm that can converge to Nash equilibria or perfect equilibria in polynomial time for general games.

  • –

    Non-stationarity means that the policies are hard to converge when reinforcement learning agents simultaneously maximize their utilities.

  • –

    Curse of multiagency means that the computation needed for the policies of all agents to achieve optimal is exponential to the number of agents.

• •

  In computational complexity theory, there are the following results about approximating Nash equilibria.

  • –

    There are two different approximations: weak approximation and strong approximation[7].

    • *

      Weak approximation is the approximation to an $\epsilon$-equilibrium, namely, a policy profile such that every player is at most $\epsilon$ away from its maximum utility.

    • *

      Strong approximation is the approximation to an $\epsilon$-neighborhood of an exact equilibrium, namely, a policy profile whose distance from an exact equilibrium is less than $\epsilon$.

    • *

      A weak approximation of an exact equilibrium could be far from the exact equilibrium.

  • –

    For static games with any number of players, computing a weak approximation of Nash equilibria is PPAD-complete[8], if $\epsilon$ is inversely proportional to a polynomial in the game size[9], indicating that an FPTAS does not exist for weak approximation of Nash equilibria unless PPAD=FP.

  • –

    For static games with any number of players, computing a weak approximation of Nash equilibria for fixed $\epsilon$ is also PPAD-complete[10], indicating that an PTAS does not exist for weak approximation of Nash equilibria unless PPAD=FP.

  • –

    For static games with two players, computing an exact Nash equilibrium is PPAD-complete[11].

  • –

    For static games with three or more players, computing an exact Nash equilibrium and computing a strong approximation of Nash equilibria are both FIXP-complete[12].

In this paper, we deal with fully observable dynamic games, and all the results of dynamic games hold for static games as single-state degenerations. Our results eventually lead to an FPTAS for weak approximation of any perfect equilibrium of any dynamic game, implying PPAD=FP.

PTAS, FPTAS, PPAD, FP are all general concepts in computational complexity theory. A PTAS (polynomial-time approximation scheme) for a class of problems is an algorithm that approximates an answer to precision $\epsilon$ in polynomial time with respect to the problem size. An FPTAS (fully polynomial-time approximation scheme) is a PTAS whose running time is not only a polynomial with respect to the problem size, but also a polynomial with respect to $1/\epsilon$. FP is a complexity class that consists of function problems that can be solved in polynomial time, corresponding to the well-known complexity class P, which consists of decision problems that can be solved in polynomial time. Decision problems are the computational problems for which the answer is simply yes or no, while function problems are the computational problems for which the answer can be anything. PPAD is a complexity class defined as all the function problems that reduce to an End-Of-The-Line problem in polynomial time[13], and End-Of-The-Line is pretty convincingly intractable in polynomial time, making PPAD believed to contain hard problems, that is, it is believed to be unlikely that PPAD=FP.

In End-Of-The-Line, there is a directed graph $DG$, and a polynomial-time computable function $f$ given by a boolean circuit. $DG$ consists of vertices that are connected by arrows, where each vertex has at most one predecessor and at most one successor. Each vertex is encoded in $n$ bits, and $f$ takes the bits of a vertex as input to output its predecessor and successor, either of which may be none. A vertex is called an unbalanced vertex if exactly one of its predecessor and successor is none. End-Of-The-Line is the problem that: given an unbalanced vertex, find another unbalanced vertex. In $DG$, every vertex is either on a chain or isolated, and the problem seems to let us follow a potentially exponentially long chain from a given source to a sink, making End-Of-The-Line pretty convincingly intractable in polynomial time.

PPAD stands for Polynomial Parity Arguments on Directed graphs, where the parity argument refers to that given an unbalanced vertex, there exist an odd number of other unbalanced vertices. The oddness theorem[14] of Nash equilibria stating that there are an odd number of Nash equilibria for almost all¹¹ 1 Almost all means other cases form a null set in all the cases, that is, the probability of encountering them is zero. static games is also a parity argument. The Sperner’s lemma in combinatorics, which is equivalent to Brouwer’s fixed point theorem that is used to prove the existence[15] of Nash equilibria, is another parity argument. All these three parity arguments are based on the fact that the solutions are connected in pairs.

Notations in this paper generally follow Einstein summation convention since the operations involved are all tensor operations. In mathematics and physics, Einstein summation convention is a widely used notation to achieve brevity in denoting tensor operations. In computer science, Einstein summation is also one of the basic functions implemented in tensor computing platforms such as NumPy and PyTorch. In this paper, we have attached an experimental code implementing our algorithm, which is based on the einsum function in NumPy and PyTorch, and thus a good illustration of Einstein summation convention can be found in their documentation.

First, a tensor is denoted as a symbol with superscripts and subscripts such as $\mu_{a}^{i}$, where the superscripts and subscripts represent the indices of the tensor, and the range of an index such as $a\in\mathcal{A}$ is implicitly inferred from context. A tensor can be denoted using different index symbols, such as $\mu_{a}^{i}$ and $\mu_{a^{\prime\prime}}^{k}$ represent the same tensor. In particular, $\mathbf{1}_{s}$ represents the tensor whose elements are all $1$, and $I_{xy}$, $I^{xy}$, or $I_{x}^{y}$ is the identity matrix, which is a second-order tensor with two indices.

Second, Einstein summation in this paper is an operation that first takes element-wise product and then sums over vanishing indices, which follows the einsum function in NumPy and PyTorch, instead of the classic Einstein summation. For example, the Einstein summation of $Y_{a}^{s}$ and $Z_{a}^{si}$ is denoted as $Y_{a}^{s}Z_{a}^{si}$, and the indices it sums over is inferred from context, such as the equation

where indices $s$ and $i$ appear in other additives of the equation, while index $a$ vanishes. Thus, the Einstein summation $Y_{a}^{s}Z_{a}^{si}$ is the summation over index $a$, where the range of $a$ is implicitly inferred from outer context. In particular, when an Einstein summation appears in the objective of an optimization problem where there is no equation, such as $w^{s}\pi_{a}^{si}r_{a}^{si}$, the summation over all the indices is implied since the objective is a scalar by definition such that all the indices vanish. In addition, element-wise product with no summation over any index is explicitly denoted using Hadamard product, such as $\pi_{a}^{si}\circ r_{a}^{si}$, though it can be implicitly inferred from context.

Third, there are a few unconventional notations to further simplify the expressions, which make sense after the problem definition in the next subsection. For policy $\pi_{a}^{si}$, let

where $A$ represents $(a_{i})_{i\in N}$ and $[\cdot]$ is the Iverson bracket that outputs $1$ if the input is true and outputs $0$ otherwise. Then, let $\pi_{A}^{s}U_{\pi A}^{si}:=(\pi_{A}^{s}U_{\pi A}^{si})^{si}$, $\pi_{Aa}^{si-}U_{\pi A}^{si}:=(\pi_{Aa}^{si-}U_{\pi A}^{si})_{a}^{si}$, and $\pi_{Aaa^{\prime}}^{sij-}U_{\pi A}^{si}:=(\pi_{Aaa^{\prime}}^{sij-}U_{\pi A}^{si})_{aa^{\prime}}^{sij}$. Let $T_{\pi s^{\prime}}^{s}:=\pi_{a}^{s}T_{s^{\prime}a}^{s}$ and $u_{\pi}^{s}:=\pi_{a}^{s}u_{a}^{s}$. Let $\max_{a}$ represent the maximum with respect to index $a$ for every other indices. Finally, in dealing with static games, we may drop the index $s$ when one state is referred and put it on when all states are referred, such as $\pi_{a}^{i}$ represents $\pi_{a}^{si}(x)$ for some given state $x$.

Dynamic game and perfect equilibrium inherit the major characteristics of stochastic game and Markov perfect equilibrium[16, 17] in existing research, and they are defined in the most intuitive and general way. Then, static games and Nash equilibria are the single-state degenerations of dynamic games and perfect equilibria.

Dynamic game generally follows the state-space representation. First, on a given state $s\in\mathcal{S}$, a set $N$ of players choose their actions in the action space $\mathcal{A}$ simultaneously. Second, according to the action combination $(a_{i})_{i\in N}$, the utility function $u$ specifies the utility of each player, and the transition function $T$ specifies the next state $s^{\prime}\in\mathcal{S}$ of the game. Third, the process is repeated, and the goal of each player $i\in N$ is to maximize its cumulative utility discounted by factor $\gamma$ on every initial state. We denote the utility function as tensor $u_{A}^{si}$ and the transition function as tensor $T_{s^{\prime}A}^{s}$, where $A:=(a_{i})_{i\in N}$ is the index of action combination.

The policy $\pi$ specifies a probability distribution $\Delta(\mathcal{A})$ of actions for each player $i\in N$ on every state $s\in\mathcal{S}$. The value function $V$ measures the value of each state $s\in\mathcal{S}$ to each player $i\in N$. Denote $\mathcal{P}=\{\pi:\mathcal{S}\to\prod_{i\in N}\Delta(\mathcal{A})\}$ as the policy space, and $\mathcal{V}=\{V:\mathcal{S}\to\prod_{i\in N}\mathbb{R}\}$ as the value function space. We denote the policy as tensor $\pi_{a}^{si}$ and the value function as tensor $V_{s}^{i}$. Note that for any policy $\pi_{a}^{si}$, there is a unique value function satisfying $V_{s}^{i}=\pi_{A}^{s}(u_{A}^{si}+\gamma T_{s^{\prime}A}^{s}V_{s^{\prime}}^{i})$, which we denote as $V_{\pi s}^{i}$, because $u_{\pi}^{s}=(I_{s^{\prime}}^{s}-\gamma T_{\pi s^{\prime}}^{s})V_{\pi s^{\prime}}$ and $I_{s^{\prime}}^{s}-\gamma T_{\pi s^{\prime}}^{s}$ is invertible by Lemma 16 (ii), where $T_{\pi s^{\prime}}^{s}:=\pi_{a}^{s}T_{s^{\prime}a}^{s}$ and $u_{\pi}^{s}:=\pi_{a}^{s}u_{a}^{s}$. In addition, we call $V_{\pi s}^{i}$ the value function of policy $\pi_{a}^{si}$ because, intuitively, for any state $x\in\mathcal{S}$, $V_{\pi s}^{i}(x)$ is the discounted cumulative utility generated by $\pi_{a}^{si}$ on initial state $x$.

A perfect equilibrium is a policy where all the players simultaneously earn their maximum discounted cumulative utilities on every initial state, which involves two equations. First, $V_{s}^{i}=\pi_{A}^{s}(u_{A}^{si}+\gamma T_{s^{\prime}A}^{s}V_{s^{\prime}}^{i})$ is to ensure that value function $V_{s}^{i}$ is the discounted cumulative utility generated by policy $\pi_{a}^{si}$ on every initial state. Second, $V_{s}^{i}=\max_{a}\pi_{Aa}^{si-}(u_{A}^{si}+\gamma T_{s^{\prime}A}^{s}V_{s^{\prime}}^{i})$ is to ensure that $V_{s}^{i}$ is the maximum discounted cumulative utilities of each player on every initial state. A weak approximation of a perfect equilibrium is an $\epsilon$-perfect equilibrium, which is a policy that every player is at most $\epsilon$ away from its maximum utility. There are many equivalent definitions of $\epsilon$-Nash equilibrium, but we use equation (24) as the definition of $\epsilon$-perfect equilibrium, and its single-state degeneration can be shown equivalent to existing $\epsilon$-Nash equilibrium definitions.

In this paper, we introduce an equation called unbiased KKT conditions and a geometric object called equilibrium bundle, along with unbiased barrier problem and Brouwer function. The major results are centered around the unbiased KKT conditions and the equilibrium bundle, which are respectively introduced as we go through the following sections, but we put their definitions here only to describe our major results.

The equilibrium bundle is a geometric object called fiber bundle²² 2 We only use the basic idea that the total space is the disjoint union of the fibers over the base space to formalize a geometric structure, no deeper theories are involved. in differential geometry, its base space is policy space $\mathcal{P}$, its total space is $E$ consisting of pairs $(\pi_{a}^{si},\mu_{a}^{si})$, its fiber over $\pi_{a}^{si}\in\mathcal{P}$ is $\alpha^{-1}(\pi_{a}^{si})=\{\pi_{a}^{si}\}\times B(\pi_{a}^{si})$, and canonical section $\bar{\mu}_{a}^{si}$ is its section that maps each $\pi_{a}^{si}\in\mathcal{P}$ to the least element in $B(\pi_{a}^{si})$. Conventionally, we can also call $E$ as the equilibrium bundle, and call $B(\pi_{a}^{si})$ as its fiber over $\pi_{a}^{si}$. The major results are several facts about the unbiased KKT conditions and the equilibrium bundle, which we show as we go through the following sections.

• •

  There are many equivalent characterizations of perfect equilibria of dynamic games centered around the unbiased KKT conditions.

  • –

    For every barrier parameter, unbiased KKT conditions derive a Brouwer function that maps a primal policy to a dual policy, such that the Brouwer’s fixed point theorem applies, leading to the existence theorem of perfect equilibrium as an extension of that of Nash equilibrium[15].

    • *

      There is an unbiased barrier problem depicting the approximation of the fixed points of the Brouwer function, such that the fixed points are its global optimal points, or equivalently its zero points.

  • –

    Unbiased KKT conditions derive a polynomial function that maps a policy to a barrier parameter, such that the Newton-Puiseux theorem applies, leading to the oddness theorem of perfect equilibrium as an extension of that of Nash equilibrium[14].

  • –

    The solution space of unbiased KKT conditions can be structured as an equilibrium bundle, such that perfect equilibria are the zero points of the canonical section of the equilibrium bundle, where the canonical section depicts the global distribution of perfect equilibria in policy space.

• •

  There is a hybrid iteration of dynamic programming and interior point method formalized as a line search method on the equilibrium bundle.

  • –

    The method consists of two levels of iteration: updating onto the equilibrium bundle by alternating the steps of projected gradient (of the unbiased barrier problem) descent and dynamic programming, and hopping across the fibers of the equilibrium bundle to a zero point of the canonical section.

  • –

    For any perfect equilibrium of any dynamic game, there are infinite many path on the equilibrium bundle that lead exactly to it. The method achieves a weak approximation, namely, the approximation to an $\epsilon$-approximate perfect equilibrium, in fully polynomial time, and the time complexity of achieving a strong approximation, namely, the approximation to an $\epsilon$-neighborhood of an actual perfect equilibrium, depends on the gradient of the canonical section near the actual perfect equilibrium.

  • –

    The computation of the algorithm is based on a variant of the expected utility instead of a specific game model, and thus the algorithm works with any game model that has a polynomial-time algorithm for the expected utility problem, and also with model-free cases.

  • –

    As for intermediate results, the game equilibrium problem is formalized as an optimization problem, which splits into two subproblems with respect to policy and value function.

    • *

      The subproblem with respect to policy is equivalent to the Nash equilibrium problem of static games. We introduce two equivalent concepts called unbiased barrier problem and unbiased KKT conditions, making the interior point method to satisfy a so-called primal-dual unbiased condition using these two concepts, so that it can approximate Nash equilibria by local optimization.

    • *

      The subproblem with respect to value function is equivalent to the convergence problem of dynamic programming in dynamic games. We introduce a concept called policy cone, giving the iteration properties of dynamic programming and the equivalent conditions of equilibria in the context of policy cone, and then use them to give the sufficient and necessary condition for dynamic programming to converge to perfect equilibria.

Considering the research status and our major contributions, there are three direct impacts of our discovery.

• •

  Our discovery leads to a fundamental leap in the understanding of game equilibria through the equilibrium bundle.

• •

  There is previously no efficient algorithms solving general game equilibrium problem, but now there is, and it is general enough to derive an efficient algorithm for any problem involving strategic interactions.

• •

  PPAD=FP, where PPAD class is previously considered to contain hard problems, but our discovery shows that it actually does not, which means there is an efficient algorithm for every problem in the PPAD class to be discovered.

The road to our FPTAS for perfect equilibria of dynamic games starts from the regret minimization problem (3) of dynamic games. The global optimal points of problem (3) are equivalently the perfect equilibria and the optimal value is zero as shown in Theorem 13.

MDP (Markov decision process) is the model for single-player decision problem[18], and an optimal policy of an MDP is the single-player degeneration of a perfect equilibrium of a dynamic game. Value iteration $V_{s,k+1}=\max_{a}(u_{a}^{s}+\gamma T_{s^{\prime}a}^{s}V_{s^{\prime},k})$ and policy iteration $\pi_{a,k+1}^{s}=\arg\max_{a}(u_{a}^{s}+\gamma T_{s^{\prime}a}^{s}V_{\pi_{k}s^{\prime}})$ are two dynamic programming methods to compute the optimal policies of MDPs in existing studies. Proposition 1 (i) restates an existing result about the relation between dynamic programming and linear programming. In addition, the two dynamic programming methods correspond to the two linear programmings, such that value iteration solves linear programming problem (4), and policy iteration solves linear programming problem (5), where the iteration variables correspond to the optimization variables.

Proposition 1 (ii) shows that the single-player case of problem (3) is in the form of the duality gap of a pair of dual linear programming problems, making alternately updating $\pi_{a}^{si}$ and $V_{s}^{i}$ feasible. However, the general case of problem (3) is a multilinear programming problem with joint constraints such that $\pi_{a}^{si}$ and $V_{s}^{i}$ simultaneously appear in a constraint, making it not possible to be decomposed with respect to $\pi_{a}^{si}$ and $V_{s}^{i}$. Although decomposition with respect to $\pi_{a}^{si}$ and $V_{s}^{i}$ is unfeasible, we can still try to alternately update $\pi_{a}^{si}$ and $V_{s}^{i}$ by taking one of $\pi_{a}^{si}$ and $V_{s}^{i}$ as variable while the other as constant, splitting problem (3) into two subproblems.

• •

  Regret minimization problem (6) of static games with respect to $\pi_{a}^{si}$, which is equivalent to solving Nash equilibria of static games.

• •

  Dynamic programming problem (17) with respect to $V_{s}^{i}$, which is equivalent to the convergence problem of dynamic programming in dynamic games.

Then, the perfect equilibrium problem is transformed to a set of Nash equilibrium problems indexed by states plus a dynamic programming problem. In the next subsection, we introduce the technical routes of this paper to construct polynomial-time algorithms solving the two subproblems and combine them to a polynomial-time algorithm solving problem (3), which equivalently approximates perfect equilibria.

In section 3, we study the subproblem with respect to $\pi_{a}^{si}$ of problem (3) and develope an algorithm solving it to update $\pi_{a}^{si}$. The subproblem is regret minimization problem (6), and it is equivalent to the problem of computing Nash equilibria of static games.

• •

  Solving Nash equilibria is equivalent to solving the global optimal points of problem (6).

  • –

    Existing methods like no-regret and self-play lack the ability to converge to Nash equilibria.

    • *

      According to the purification theorem[19], a policy is a Nash equilibrium if and only if for every player, every non-zero weighted action has the same utility.

    • *

      As a consequence, mixed Nash equilibria are not stable equilibrium points for any method based on optimizing the utility of the updated mixed strategy, such as no-regret and self-play.

    • *

      In no-regret[2], regret is a vector of differences between the utility of the updated mixed strategy and the utilities of every action, and the regret is minimized.

    • *

      In self-play[4], the utility of the updated mixed strategy is maximized.

    • *

      In regret minimization problem (6), the regret is a vector of differences between the maximum utility and the utilities of every action, plus a scalar.

  • –

    Solving Nash equilibria is equivalent to solving the primal-dual unbiased local extreme points of problem (6). We introduce a primal-dual unbiased condition here, so that we can use local optimization methods that are polynomial-time.

  • –

    Using the interior point method to perform local optimization. The interior point method approximate a local extreme point along the central path through two levels of iteration: updating onto the central path by locally optimizating the barrier problem, and updating along the central path to a local extreme point by gradually reducing the barrier parameter to zero, where the central path consists of solutions of the perturbed KKT conditions.

• •

  Using interior point method to approximate primal-dual unbiased local extreme points of problem (6). We want the interior point method to always lead to a primal-dual unbiased local extreme point of problem (6), instead of just an ordinary local extreme point.

  • –

    Equivalently characterizing primal-dual unbiased local extreme points of the barrier problem of problem (6) through unbiased barrier problem (8) or unbiased KKT conditions (9). We introduce (primal-dual) unbiased barrier problem (8), (primal-dual) unbiased KKT conditions (9), and a Brouwer function here, so that we can make the following transformation.

  • –

    Transforming the interior point method into a primal-dual unbiased interior point method. The primal-dual unbiased interior point method approximate a primal-dual unbiased local extreme point along the (primal-dual) unbiased central path through two levels of iteration, where the unbiased central path consists of solutions of unbiased KKT conditions (9).

    • *

      In the first iteration level, usually, the barrier problem is numerically optimized by SQP (sequential quadratic programming)[20], which is a numerical second-order method. But for unbiased barrier problem (8), there exists an analytical expression of its projected gradient. So we use projected gradient (10) to update onto the unbiased central path, and this analytical first-order method is much more simple.

    • *

      In the second iteration level, we reduce the barrier parameter to update along the unbiased central path.

  • –

    There are three assumptions for the primal-dual unbiased interior point method to work.

    • *

      Starting point: there has to be a starting point sufficiently close to the unbiased central path.

    • *

      Differentiability: the updated point has to move an infinitesimal step along the unbiased central path with an infinitesimal reduction of the barrier parameter.

    • *

      Convexity: unbiased barrier problem has to be strictly locally convex for projected gradient descent (10) to update onto the unbiased central path.

• •

  Introducing the equilibrium bundle to settle the three assumptions.

  • –

    We introduce a geometric object called equilibrium bundle, along with its canonical section and singular point.

    • *

      Equilibrium bundle is a fiber bundle in differential geometry, which consists of disjoint fibers over every policy in the policy space.

    • *

      Equilibrium bundle is the structured solution space of unbiased KKT conditions (9).

    • *

      Nash equilibria are equivalently the zero points of the canonical section of the equilibrium bundle.

  • –

    Settling the three assumptions on the equilibrium bundle. Equilibrium bundle corresponds to the previous unbiased central path in the following statements.

    • *

      Starting point: for any given policy, a large enough barrier parameter on its fiber is nearly on the equilibrium bundle.

    • *

      Differentiability: on non-singular points of the equilibrium bundle, policy moves an infinitesimal step with an infinitesimal step of barrier parameter.

    • *

      Convexity: on non-singular points of the equilibrium bundle, unbiased barrier problem (8) is strictly locally convex.

    • *

      Singularity: for any given policy, a large enough barrier parameter on its fiber is guaranteed to be a non-singular point pairing with the policy. Note that moving along a fiber changes neither the policy nor the canonical section, so this singular avoidance does not affect the line search to Nash equilibria.

  • –

    Transforming the primal-dual unbiased interior point method into a line search method on the equilibrium bundle. This method approximates any Nash equilibrium of any static game through two levels of iteration.

    • *

      In the first iteration level, we update onto the equilibrium bundle by projected gradient descent (10).

    • *

      In the second iteration level, singular points must be avoided as we update along the equilibrium bundle, which is done by canonical section descent (16) leveraging the structure of the equilibrium bundle. So we hop across the fibers of the equilibrium bundle to a zero point of its canonical section by canonical section descent (16).

• •

  To show the significance of our results, we additionally extend the existence and oddness theorems of Nash equilibria to the unbiased KKT conditions, such that the two theorems of Nash equilibria are implied by degenerating the two theorems of the unbiased KKT conditions.

  • –

    Existence theorem[15]: for any barrier parameter, there is always a solution of the unbiased KKT conditions. Because unbiased KKT conditions derive a Brouwer function mapping primal policies to dual policies for any barrier parameter, then a primal-dual unbiased policy always exists by Brouwer’s fixed point theorem.

  • –

    Oddness theorem[14]: for any barrier parameter, there are almost always an odd number of solutions on the unbiased KKT conditions. Because unbiased KKT conditions derive a polynomial function mapping policies to barrier parameters, then the equilibria and a unique starting point are almost always connected in pairs by Newton-Puiseux theorem.

In section 4, we study the subproblem with respect to $V_{s}^{i}$ of problem (3) and develope an algorithm solving it to update $V_{s}^{i}$. The subproblem is dynamic programming problem (17), and it is equivalent to the convergence problem of dynamic programming in dynamic games.

• •

  Solving dynamic programming problem (17) is equivalent to making the convergence point of dynamic programming to lie within the constraint of problem (17).

  • –

    Existing methods that directly use the Bellman operator in dynamic games lack the ability to converge at all.

    • *

      Value iteration is a dynamic programming method that uses the Bellman operator to iterate the value function, and it is a polynomial-time exact algorithm for computing optimal policies of MDPs[21]. The iterative convergence of Bellman operator is contraction mapping convergence.

    • *

      The Bellman operator is also directly used to solve perfect equilibria of dynamic games in early attempts[22, 23], showing that it generally do not converge.

  • –

    Settling for monotone convergence instead of contraction mapping convergence.

    • *

      We introduce two dynamic programming operators respectively given by the objective and constraint of problem (17), and introduce the policy cone and the best response cone respectively based on the two dynamic programming operators.

    • *

      It is discovered that our dynamic programming operator achieves monotone convergence within the policy cone as shown in the following.

    • *

      Contraction mapping convergence can be considered as a convergence in every direction, and monotone convergence can be considered as a convergence in only a certain direction.

    • *

      The Bellman operator is a special case of our dynamic programming operator, allowing us to analyze why Bellman operator does not converge in dynamic games later.

  • –

    For a fixed policy, dynamic programming operator achieves monotone convergence to the apex within the policy cone, and adding a large enough scalar keeps the iteration within the best response cone.

    • *

      Policy cone is the monotonic and closed domain of dynamic programming operator such that monotone convergence theorem applies.

    • *

      Every value function plus a large enough scalar is in the best response cone, and the best response cone lies inside the policy cone.

    • *

      Policy cone has an apex that is the value function of its policy, and the apex is the limit point of the dynamic programming operator iteration, and plusing scalars during the iteration does not affect the convergence.

  • –

    A policy is a perfect equilibrium if and only if the apex of its policy cone lies in its best response cone, and if and only if the apex is the global optimal point of dynamic programming problem (17).

• •

  Making dynamic programming operator to monotonically converge to a policy cone apex that lies in the best response cone.

  • –

    Explaining why Bellman operator cannot be simply generalized from MDPs to dynamic games. Bellman operator achieves both contraction mapping convergence and monotone convergence in single-player case, but it achieves neither the convergence in multi-player case.

  • –

    The sufficient and necessary condition for dynamic programming iteration (22) to converges to a perfect equilibrium value function.

    • *

      Not only is the policy cone bounded below given a fixed policy, but the apex of it is also bounded below with respect to a varying policy. Thus, monotone convergence with a varying policy is maintained by adding a large enough scalar so that the iteration lies within the policy cone and letting this scalar converge to zero.

    • *

      The limit point lies in the best response cone if the iteration is kept within the best response cone. Thus, we further require the scalar large enough so that the iteration lies within the best response cone.

In section 5, we combine the above two algorithms by alternating the steps updating $\pi_{a}^{si}$ and $V_{s}^{i}$ such that the resulting algorithm solves problem (3), which is equivalent to the problem of computing perfect equilibria of dynamic games.

• •

  Perfect equilibria are equivalently the global optimal points of regret minimization problem (3) of dynamic games.

  • –

    Existing methods that directly use variants of the static game method in dynamic games lack the ability to converge to perfect equilibria.

    • *

      For every dynamic game, there is a corresponding static game, in which the behavioral strategies of the dynamic game serve as the static strategies of the corresponding static game.

    • *

      When the variants of no-regret[3] and self-play[5] are used in dynamic games, they deal with interactions of behavioral strategies, instead of static strategies at each stage of the dynamic game. This means that their resulting equilibria are at most Nash equilibria of the corresponding static game of behavioral strategy interactions, while not guaranteed to be a Nash equilibrium at every stage of the dynamic game.

  • –

    Combining the line search on equilibrium bundle and the dynamic programming within policy cone.

    • *

      A policy is a perfect equilibrium if and only if it is the global optimal point of problem (3). There is a relation between canonical section and the two dynamic programming operators, which then establishs a relation between the results in the last two sections.

    • *

      Determining the nesting order of the three iteration formulas of the two sub-algorithms. Dynamic programming (22) has to be at the same level of iteration as the projected gradient descent (10), because dynamic programming moves more than an infinitesimal step every iteration, which would cause the updated point to be too far away from the equilibrium bundle to update back, except for when dynamic programming is nearly converged.

    • *

      Defining the unbiased KKT conditions and the equilibrium bundle of dynamic games. By setting the value function as the convergence point of dynamic programming, which is the value function of a specific policy, there are still unbiased KKT conditions and the equilibrium bundle of dynamic games, such that those of static games are the single-state degenerations. Furthermore, the existence and oddness theorems generalize to dynamic games.

    • *

      The combined method is still formalized as a line search on the equilibrium bundle, which approximates any perfect equilibrium of any dynamic game through two levels of iteration: updating onto the equilibrium bundle by projected gradient descent (10) and dynamic programming (22), and hopping across the fibers of the equilibrium bundle to a zero point of its canonical section by canonical section descent (16).

  • –

    Finally, we show that the resulting algorithm achieves a weak approximation in fully polynomial time by showing the convergence rates of the three iterations (10), (22), and (16).

In this section, we deal with the subproblem with respect to $\pi_{a}^{si}$ of problem (3), which is the regret minimization problem (6) with $\mu_{a}^{i}=0$ and $U_{A}^{i}=(u_{A}^{si}+\gamma T_{s^{\prime}A}^{s}V_{s^{\prime}}^{i})(x)$. The global optimal points of problem (6) are equivalently Nash equilibria, and thus the goal of this section is to find a polynomial-time algorithm for Nash equilibria. We first use the purification theorem to explain why existing approximation methods for Nash equilibrium lack convergence guarantee while methods based on regret minimization problem (6) do. Then we give Theorem 2 stating that a Nash equilibrium is equivalently a global optimal point of problem (6), and is also equivalently a local extreme point that also satisfies a condition called primal-dual unbiased condition. Finally, the second equivalent condition would allow us to approximate Nash equilibria by local optimization, which is chosen as the interior point method. In addition, this section mainly deals with static games, so we drop the state index $s$ as we state before.

According to the purification theorem[19], $\pi_{a}^{i}$ is a Nash equilibrium if and only if for every player $i$, every action $a$ that satisfies $\pi_{a}^{i}>0$ has the same utility. Thus, mixed Nash equilibria are not stable equilibrium points for any method based on optimizing the utility of the updated mixed strategy, such as no-regret[2, 3] and self-play[4, 5]. Note that the meanings of regret are different in no-regret methods and in this paper. The regret in no-regret methods is a vector of differences between the utility of the updated mixed strategy and the utilities of every action, while the regret in regret minimization problem (6) is a vector of differences between the maximum utility and the utilities of every action plus a scalar. The difference upon the quantity being optimized is why our method has convergence guarantee while existing methods based on no-regret or self-play do not, and consequently exhibit non-stationarity when used in MARL.

Theorem 2 points out that the Nash equilibrium is not only equivalent to the global optimal point of regret minimization problem (6), but also equivalent to its local extreme point that satisfies the primal-dual unbiased condition, where the local extreme point is namely the point that satisfies the KKT conditions[24] in (iii). In previous research[25, 26], linear programming, quadratic programming, linear complementarity problem, and Nash equilibrium of bimatrix games have been unified, and the unification is actually a degeneration of $(i)\Leftrightarrow(ii)$ into two-player static games. $(i)\Leftrightarrow(ii)$ points out the connection between our method and existing methods, but it is not necessary in constructing our method. $(i)\Leftrightarrow(iii)$ makes it possible to use local optimization that leads to local extreme points where primal-dual bias $\pi_{a}^{i}-\hat{\pi}_{a}^{i}$ is $0$ to compute global optimal points of (6), and equivalently Nash equilibria.

We choose interior point method to perform the local optimization, where central path is a path in the solution space of perturbed KKT conditions (7) with the barrier parameter $\mu_{a}^{i}=\mathbf{1}_{a}^{i}\mu$, the first iteration level is to update onto the central path by locally optimizating the barrier problem (6), the second iteration level is to update along the central path by reducing the scalar barrier parameter $\mu$, and the central path leads to a local extreme point of the orginal problem as $\mu$ reduces to $0$. In our case, we additionally intend to keep the update on a particular central path on which primal-dual bias $\pi_{a}^{i}-\hat{\pi}_{a}^{i}$ is $0$.

Note that Theorem 2 (iii) only applies to the $\mu_{a}^{i}=0$ case, but we have to study the $\mu_{a}^{i}>0$ case to find a way to keep the update on the central path where primal-dual bias is $0$. Thus, we introduce two concepts called unbiased barrier problem and unbiased KKT conditions, and give Theorem 3 to extend our equivalent result about the $\mu_{a}^{i}=0$ case in Theorem 2 (iii) onto the subject of interior point method where $\mu_{a}^{i}>0$, so that we can construct a primal-dual unbiased interior point method. In addition, we use these two concepts to give an extension of the existence theorem of Nash equilibria.

The Brouwer function connects unbiased barrier problem and unbiased KKT conditions. The unbiased KKT conditions derive the Brouwer function mapping primal policies to dual policies, such that the fixed points of the Brouwer function are solutions of the unbiased KKT conditions. The unbiased barrier problem depicts the approximation of fixed points of the Brouwer function, such that the fixed points are the global optimal points, or zero points, of the unbiased barrier problem.

There are two equivalence formulas in Theorem 3. The first one $(i)\Leftrightarrow(ii)\Leftrightarrow(iii)\Leftrightarrow((iv)\wedge\pi_{a}^{i}=\hat{\pi}_{a}^{i})$ establishs the equivalence between the unbiased barrier problem, Brouwer function, unbiased KKT conditions, and the $\mu_{a}^{i}>0$ case in Theorem 2 (iii), allowing us the construct a primal-dual unbiased interior point method. Actually, we only need the first formula to make subsequent constructions, while the second formula $(iii)\Leftrightarrow(iv)\Leftrightarrow(v)$ only illustrates the connection between our method and interior point method, that is, a path that consists of solutions of the unbiased KKT conditions really is a central path in interior point methods on which the primal-dual bias is $0$, ignoring the difference that the barrier parameter in our method is actually a vector instead of a scalar.

Theorem 4 (i) shows two facts. First, (i) shows that there is a unique $v^{i}$ to let dual policy $\hat{\pi}_{a}^{i}$ satisfy constraint $\mathbf{1}_{a}\pi_{a}^{i}=\mathbf{1}^{i}$. As a consequence, (ii) shows that when $\mathbf{1}_{a}\hat{\pi}_{a}^{i}=\mathbf{1}^{i}$, $\pi_{a}^{i}-\hat{\pi}_{a}^{i}$ and $dv^{i}$ are orthogonal, and the projected gradient of unbiased barrier problem (8) has the form of an analytical expression as equation (10) shows. Second, (i) shows that for every policy $\pi_{a}^{i}$, there is a unique dual policy $\hat{\pi}_{a}^{i}$ that satisfies (9a). As a consequence, the Brouwer function $\hat{\pi}_{a}^{i}=M(\mu_{a}^{i})(\pi_{a}^{i})$ is indeed a map, then by Brouwer’s fixed point theorem, (iii) asserts the existence of a solution of unbiased KKT conditions (9) for every $\mu_{a}^{i}$, as an extension of the existence of Nash equilibria[15].

Note that equation (10) specifies the gradient with respect to $\ln\pi_{a}^{i}$, and it needs to be projected orthogonally to the hyperplane of $\mathbf{1}_{a}\pi_{a}^{i}=\mathbf{1}^{i}$. However, we use $\pi_{a}^{i}=\exp(\sigma_{a}^{i})/(\mathbf{1}_{a}\exp(\sigma_{a}^{i}))$ as the representation of policy in our experiments, and the gradient with respect to $\sigma_{a}^{i}$ is ${\rm pg}_{a^{\prime\prime}}^{j}$ in the following equation, which naturally satisfies constraint $\mathbf{1}_{a}\pi_{a}^{i}=\mathbf{1}^{i}$ without projection. In general, the gradient may be different for different representation of $\pi_{a}^{i}$, and it generally needs to be projected.

In summary, we’ve transformed the interior point method into a primal-dual unbiased interior point method, where (primal-dual) unbiased central path is a path in the solution space of unbiased KKT conditions (9), the first iteration level is to update onto the unbiased central path by projected gradient descent (10), the second iteration level is to update along the unbiased central path by reducing barrier parameter $\mu_{a}^{i}$, and the unbiased central path leads to a Nash equilibrium as $\mu_{a}^{i}$ reduces to $0$.

However, there are three assumptions for the primal-dual unbiased interior point method to work. Starting point: the iteration has to start from a point that is sufficiently close to a known point on the unbiased central path. Differentiability: in the second iteration level, the policy $\pi_{a}^{i}$ has to move an infinitesimal step along the unbiased central path with an infinitesimal reduction of the barrier parameter $\mu_{a}^{i}$, so that the first iteration level can update back onto the unbiased central path. Convexity: in the first iteration level, unbiased barrier problem (8) has to be strictly locally convex for projected gradient descent (10) to update onto the unbiased central path.

Settling the three assumptions is beyond the capacity of mathematical optimization, it turns out to be a geometric problem regarding the solution space of unbiased KKT conditions (9) where unbiased central paths lie. Thus, we first introduce a geometric object called equilibrium bundle to structure the solution space of unbiased KKT conditions (9), so that we can study the equilibrium bundle instead of the unbiased central path. Then we settle the three assumptions using the geometric properties of the equilibrium bundle, transforming our primal-dual unbiased interior point method into a line search method on the equilibrium bundle. Finally, we give the oddness theorem of the equilibrium bundle as an extension of that of Nash equilibria.

Definition 10 and Theorem 5 show the relation between the equilibrium bundle, the solution space of unbiased KKT conditions (9), and the joint space $\mathcal{P}\times\{\mu_{a}^{i}|\mu_{a}^{i}\geq 0\}$ of policy and barrier parameter. First, the equilibrium bundle is the solution space of unbiased KKT conditions (9), and a fiber $B(\pi_{a}^{i})$ is the solution subspace relating to a given $\pi_{a}^{i}$. Second, $B(\pi_{a}^{i})$ is the intersection between $\{\mu_{a}^{i}|\mu_{a}^{i}\geq 0\}$ and a $\left\lvert N\right\rvert$-dimensional affine space spanned by $v^{i}$, and the union of all the $B(\pi_{a}^{i})$ is exactly $\{\mu_{a}^{i}|\mu_{a}^{i}\geq 0\}$ by the existence theorem. Third, the canonical section $\bar{\mu}_{a}^{i}(\pi_{a}^{i})$ is the least element in $B(\pi_{a}^{i})$ such that for any $\mu_{a}^{i}\in B(\pi_{a}^{i})$ there is $\mu_{a}^{i}\geq\bar{\mu}_{a}^{i}(\pi_{a}^{i})$, and $\bar{\mu}_{a}^{i}(\pi_{a}^{i})$ is on the boundary of $\{\mu_{a}^{i}|\mu_{a}^{i}\geq 0\}$. Finally, Theorem 5 (iii) shows that Nash equilibria are exactly the zero points of the canonical section $\bar{\mu}_{a}^{i}$.

Theorem 5 (iii) has two implications. First, unlike the primal-dual unbiased interior point method, $\mu_{a}^{i}$ does not have to decrease to $0$ to reach a Nash equilibrium, instead, $\mu_{a}^{i}$ can be any value on the fiber as long as the canonical section $\bar{\mu}_{a}^{i}(\pi_{a}^{i})$ decreases to $0$. Second, the canonical section $\bar{\mu}_{a}^{i}(\pi_{a}^{i})$ depicts the global distribution of Nash equilibria, which can be used to search the policy space globally for the entire set of Nash equilibria.

In summary, the equilibrium bundle is exactly the structured solution space of unbiased KKT conditions (9) where the unbiased central paths lie, making it possible to leverage the geometric properties of the equilibrium bundle to study the three assumptions left to settle in the primal-dual unbiased interior point method, which gives the following theorem.