acceptodds
Under review as a conference paper at ICLR 2027

Learning Dynamics in Continuous-Time Mean Field Games with Congestion

Abstract

We study stationary equilibria and learning dynamics in a finite-state continuous-time mean field game with population-dependent action-completion rates generated by a smooth convex congestion potential. In this game, stationary equilibrium populations correspond one-to-one with optimizers of an Eisenberg–Gale-type convex program. We show that this variational structure also yields global convergence of the state–action dynamics under any fixed unichain policy, even though these dynamics are governed by a nonlinear Markov process. We then formulate an equivalent population game over deterministic unichain policies, where a concave ordinal potential gives convergence of best-response dynamics based on stationary long-run payoffs to the policy-equilibrium set. Finally, we study state–policy dynamics that track agents' physical states and committed policies under two revision mechanisms: exogenous opportunities arriving at a Poisson rate, and event-driven opportunities arising at action completions with a specified probability. Under both mechanisms, policy marginals converge to the policy-equilibrium set for every fixed admissible positive revision parameter. As the revision rate or probability tends to zero, the state–action populations associated with the omega-limit points of these dynamics approach the stationary mean field equilibrium populations. These results reveal how equilibrium emerges through repeated policy revisions and give our notion of stationary mean field equilibrium a dynamic foundation.

Primary area:
learning theory
Keywords:
mean field game, learning dynamics, Eisenberg--Gale program, non-linear Markov process, Lyapunov function

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_pBmnOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a finite-state continuous-time mean field game with congestion-dependent action-completion rates. Under a convex congestion potential, it derives an Eisenberg-Gale-type equilibrium characterization and analyzes several exact best-response dynamics. The key equilibrium-recovery result for the physical population is mainly a rare-revision limit result.

Critical strengths
  1. The paper is technically substantial, and the equilibrium characterization is developed carefully under the stated assumptions.
  2. The event-driven formulation is meaningful, and the paper correctly distinguishes policy convergence from convergence of the physical state-policy population.
Critical weaknesses
  1. The central 'learning' mechanism assumes exact stationary payoffs H(ν) for all policies from the current policy distribution, even while the actual population is not stationary. This oracle assumption removes payoff estimation and exploration, which are central to learning in MFGs. The paper itself leaves model-free learning from observed transitions and rewards as future work. I therefore view the main result as convergence of exact best-response dynamics rather than a convincing learning result for ICLR.
  2. For fixed positive κ or ε, the paper generally proves convergence only of the policy marginal, not of the full state-policy population to a stationary MFG equilibrium. Recovery of the physical equilibrium is obtained mainly as κ or ε tends to zero, and event-driven dynamics can be nonconvergent at ties. The restrictive potential/unichain assumptions and a single illustrative Manhattan experiment without learning baselines further limit the contribution's generality and empirical support.
Questions for authors
  1. What guarantees survive when H(ν) must be estimated from trajectories rather than computed exactly from the known model?
  2. For fixed nonzero κ or ε, when does the full state-policy population converge to an MFG equilibrium beyond the strict-pure-equilibrium case?
3 comments
AuthorsOpenReview ↗

Response to Reviewer pBmn Part 1

Thank you for your comments on the information needed for learning, population convergence, and empirical support. Our point-by-point responses are as follows.

Q1

Q1. What guarantees survive when H(ν) must be estimated from trajectories rather than computed exactly from the known model?

Thank you for raising this question. Our key convergence results remain valid when agents choose best responses to increasingly accurate estimates of H(ν)H(\nu) obtained from trajectories. At any fixed positive revision rate or probability, the policy marginal converges to the equilibrium set as the estimation errors tend to zero uniformly across policies. With the diminishing revision schedules described below, the state–policy population also converges to the stationary MFG equilibrium set.

To make this precise, let H~π(t)\widetilde H_\pi(t) estimate the stationary payoff Hπ(ν(t))H_\pi(\nu(t)), and suppose that, for almost every tt,

\le c_{\rm err}(t),$$ where $c_{\rm err}$ is nonnegative, measurable, and locally bounded. Agents choose policies maximizing these estimates, using a common measurable selection $\gamma(t)\in\mathcal M_m(\Pi)$ supported on $\arg\max_\pi\widetilde H_\pi(t)$ and revision probabilities $\gamma(t)/m$. Under Assumptions 1 and 4, consider a global absolutely continuous trajectory with a continuous revision rate $\kappa(t)>0$ or revision probability $\varepsilon(t)\in(0,1]$, in physical time. The key observation is that a best response to the estimates loses at most $2c_{\rm err}(t)$ in true payoff: one estimation error concerns the chosen policy, and the other concerns a true best response. Since the true payoffs have a uniform positive lower bound $c_{H,\min}$, the corresponding loss in logarithmic payoff is at most $2c_{\rm err}(t)/c_{H,\min}$. This gives a direct perturbation of the policy-potential argument. The potential gap $\Delta(\nu)=\Phi^{\ast}-\Phi(\nu)$ measures how far the policy potential $\Phi(\nu)$ is below its maximum $\Phi^{\ast}$. Since the maximizers of $\Phi$ are exactly the equilibrium policy distributions, this gap is zero precisely at policy equilibrium. We obtain $$\begin{aligned} \dot\Delta(\nu(t)) &\le-\kappa(t)\Delta(\nu(t)) +\frac{2m\kappa(t)}{c_{H,\min}}c_{\rm err}(t), &&\text{exogenous revision},\\\\ \dot\Delta(\nu(t)) &\le-\varepsilon(t)\underline\lambda\Delta(\nu(t)) +\frac{2m\varepsilon(t)\overline\lambda}{c_{H,\min}}c_{\rm err}(t), &&\text{event-driven revision}. \end{aligned}$$ Here $m$ is the population mass, and $\underline\lambda,\overline\lambda$ are the uniform lower and upper bounds on individual completion rates. The first term gives the exact-payoff decrease of the potential gap; the second quantifies the effect of payoff-estimation errors. If $\int_0^\infty\kappa(t')\\,dt'=\infty$ or $\int_0^\infty\varepsilon(t')\\,dt'=\infty$, respectively, integrating these inequalities yields $$\limsup_{t\to\infty}\Delta(\nu(t))\le \begin{cases} \displaystyle\frac{2m}{c_{H,\min}} \limsup_{t\to\infty}c_{\rm err}(t),&\text{exogenous revision},\\\\[4pt] \displaystyle\frac{2m\overline\lambda}{\underline\lambda c_{H,\min}} \limsup_{t\to\infty}c_{\rm err}(t),&\text{event-driven revision}. \end{cases}$$ The bounds quantify the effect of imperfect payoff estimates: halving the long-run payoff-error bound halves the corresponding upper bound on the potential's shortfall from its equilibrium value. When $c_{\rm err}(t)\to0$, the bounds give $\Delta(\nu(t))\to0$, so the policy marginal approaches $\mathcal E_\Pi^{\ast}$, including at fixed positive revision parameters. If the respective revision rate or probability also tends to zero under the same integral condition, the population tracks $\bar\eta^{\nu(t)}$ and approaches $\mathcal E_{\mathcal S\times\Pi}^{\ast}$. Thus vanishing payoff errors give policy optimality, while diminishing revision allows the state–policy population to approach stationarity. We have added these direct payoff-error bounds and convergence conclusions as Proposition 41, “Payoff-estimation errors,” in Appendix F.2.
AuthorsOpenReview ↗

Response to Reviewer pBmn Part 2

Q2&W2

Q2. For fixed nonzero κ or ε, when does the full state-policy population converge to an MFG equilibrium beyond the strict-pure-equilibrium case?

W2. For fixed positive κ or ε, the paper generally proves convergence only of the policy marginal, not of the full state-policy population to a stationary MFG equilibrium. Recovery of the physical equilibrium is obtained mainly as κ or ε tends to zero, and event-driven dynamics can be nonconvergent at ties.

Thank you for asking about convergence beyond strict pure equilibria. For the original best-response revision rules, an agent can switch between equally good policies even at a policy equilibrium. Such switches can change the state–policy population and disturb state balance, so convergence of the policy marginal alone does not ensure population convergence. Remark 37 gives a periodic event-driven trajectory illustrating this issue. This motivates making acceptance of a revision opportunity depend on the payoff improvement. Tempered best-response acceptance and our Brown–von Neumann–Nash (BNN) protocols give state–policy equilibrium-set convergence at fixed positive opportunity parameters. Both mechanisms combine policy improvement with a revision term that vanishes as the incentive to switch disappears.

Specifically, fix an exogenous opportunity rate κ>0\kappa>0 or an event-driven opportunity probability ε∈(0,1]\varepsilon\in(0,1]. An agent using policy π\pi accepts a revision opportunity with probability

1−Hπ(ν)max⁡π′Hπ′(ν),1-\frac{H_\pi(\nu)}{\max_{\pi'}H_{\pi'}(\nu)},

and, upon acceptance, draws a best response with probability distribution γ/m\gamma/m, where γ(t)∈Γ(ν(t))\gamma(t)\in\Gamma(\nu(t)) is a common measurable best-response selection. In the event-driven model, this decision is made at the arrival state after action completion. This is a normalized tempered best-response rule [1]. An optimal policy is kept, while a small payoff improvement leads to a small acceptance probability.

For exogenous revision, the policy potential satisfies

=\kappa\sum_\pi\nu_\pi \left(1-\frac{H_\pi(\nu)}{\max_{\pi'}H_{\pi'}(\nu)}\right) \log\frac{\max_{\pi'}H_{\pi'}(\nu)}{H_\pi(\nu)}\ge0.$$ For event-driven revision, replace $\kappa\nu_\pi$ by $\varepsilon\chi_\pi$, where $\chi_\pi$ is the policy completion rate. The uniform payoff and completion-rate bounds imply that the rate of increase of $\Phi$ is at least a positive constant times the squared potential gap. Since $\Phi$ is bounded above by $\Phi^{\ast}$, the gap must tend to zero. At a policy equilibrium, every policy with positive mass is optimal. Consequently, the total revision term tends to zero even though the opportunity parameters remain fixed. The frozen state–policy convergence argument then gives, from every initial state–policy population, $$\\|\eta(t)-\bar\eta^{\nu(t)}\\|_1\to0, \qquad \operatorname{dist}\bigl(\eta(t),\mathcal E_{\mathcal S\times\Pi}^{\ast}\bigr)\to0.$$ The same principle applies to the BNN protocols. For exogenous BNN, policy improvement is measured by the positive payoff excess $$\left[H_\pi(\nu)-\frac1m\sum_{\pi'}\nu_{\pi'}H_{\pi'}(\nu)\right]^+.$$ The event-driven version uses an average payoff weighted by policy completion shares. These excesses vanish at policy equilibrium, so the revision term also vanishes. Propositions 33(i) and 39(i), in Appendices D.5 and E.6, establish convergence to the state–policy equilibrium set for every fixed $\kappa>0$ or $\varepsilon\in(0,1]$, respectively. If the equilibrium policy distribution is unique, convergence is to its single associated stationary population. We have added the tempered best-response acceptance argument in Appendix F.1, following the reciprocal revision schedules. ### **W1** > **W1.** The central 'learning' mechanism assumes exact stationary payoffs H(ν) for all policies from the current policy distribution, even while the actual population is not stationary. This oracle assumption removes payoff estimation and exploration, which are central to learning in MFGs. The paper itself leaves model-free learning from observed transitions and rewards as future work. I therefore view the main result as convergence of exact best-response dynamics rather than a convincing learning result for ICLR. Thank you for raising this point. Model-free multi-agent learning with congestion-dependent dynamics is a key motivation for this study, and we agree that estimation and exploration are central to that goal. The oracle assumption allows us to take a foundational first step: understanding the population and policy dynamics when the model is known and stationary payoffs can be evaluated accurately. We believe our model-based analysis provides the convergence framework for studying learning from observations.

W2 Cont.

W2. The restrictive potential/unichain assumptions and a single illustrative Manhattan experiment without learning baselines further limit the contribution's generality and empirical support.

Thank you for these comments. We clarify the model assumptions and their scope, and explain the purpose of the numerical study below.

Potential assumptions. The potential-congestion assumption applies to a broad class of games with shared-resource congestion. In traffic routing, an action uses a route and its mean travel time is the sum of the delays on its roads. Two routes sharing a road experience the same marginal effect from an increase in that road's flow. Integrating each road's smooth, nondecreasing delay over its flow and summing over roads produces the required convex potential. The same construction applies to packet routes sharing communication links and processing actions sharing service pools. It permits overlapping resource use and nonseparable congestion between different actions and states. This shared-resource structure is classical in congestion games and traffic models [2, 3], and throughput-dependent sojourn times also appear in approximations for closed queueing networks [4].

Unichain assumptions. The unichain restriction applies only to the policy space explored by the revision process. A policy is unichain when its induced Markov chain has exactly one recurrent class; transient states are allowed. This gives each policy a unique invariant probability distribution and a stationary payoff independent of the agent's initial state, as in standard average-reward MDP analysis [5]. We take Π\Pi to contain all deterministic unichain policies. Even with this restriction, agents can have a very large policy space to explore. For example, when every deterministic policy is unichain, Π\Pi contains all ∣A∣∣S∣|\mathcal A|^{|\mathcal S|} deterministic policies, so the number of available policies grows exponentially with the number of states. The assumption on the underlying MDP is only weak communication. Apart from states that are transient under every stationary policy, each state can reach any other under a suitable deterministic stationary policy. This condition also allows multichain policies in the underlying model. Weakly communicating MDPs form a standard, general class in average-reward reinforcement learning, including the analyses of Wan, Yu and Sutton [6], REGAL [7], and Wei et al. [8]. Under this condition, Lemma 22 guarantees that Π\Pi contains an optimal policy for every positive sojourn-time vector. Thus restricting policy revision to this family preserves the optimal average reward in every stationary environment.

Empirical support. The purpose of the numerical study is to demonstrate convergence and provide numerical support for the paper's theoretical results in a concrete congestion-dependent model. The four dynamics illustrate convergence toward population stationarity under a fixed policy, decreasing exploitability under best-response policy dynamics, and improvement in both quantities under the two state–policy systems. The revision-frequency comparison further shows that less frequent revision reduces residual flow imbalance, consistent with the rare-revision results; this comparison uses a fixed slow-time window.

Thank you again for your constructive comments. We have revised the paper as described above, with the manuscript changes highlighted in blue. We hope our responses address your questions and concerns. We welcome any further questions.

References

  1. Dai Zusai. 2018. Tempered best response dynamics. International Journal of Game Theory 47, 1–34.
  2. Robert W. Rosenthal. 1973. A class of games possessing pure-strategy Nash equilibria. International Journal of Game Theory 2, 65–67.
  3. Tim Roughgarden and Éva Tardos. 2002. How Bad Is Selfish Routing? Journal of the ACM 49(2), 236–259.
  4. Frank P. Kelly. 1989. On a Class of Approximations for Closed Queueing Networks. Queueing Systems 4, 69–76.
  5. Martin L. Puterman. 1994. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons.
  6. Yi Wan, Huizhen Yu, and Richard S. Sutton. 2024. On Convergence of Average-Reward Q-Learning in Weakly Communicating Markov Decision Processes. arXiv preprint arXiv:2408.16262.
  7. Peter L. Bartlett and Ambuj Tewari. 2009. REGAL: A Regularization Based Algorithm for Reinforcement Learning in Weakly Communicating MDPs. In Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence (UAI '09). AUAI Press, 35–42.
  8. Chen-Yu Wei, Mehdi Jafarnia Jahromi, Haipeng Luo, Hiteshi Sharma, and Rahul Jain. 2020. Model-free Reinforcement Learning in Infinite-horizon Average-reward Markov Decision Processes. In Proceedings of the 37th International Conference on Machine Learning (ICML 2020). Proceedings of Machine Learning Research 119, 10170–10180.
Reviewer_5jS2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies a type of mean field games (MFGs) which are games for infinitely many players. Here the paper considers a continuous time, finite state model and a stationary equilibrium notion. Under specific structural conditions, the authors identify a characterization of the equilibrium, and analyze a learning dynamics.

Critical strengths

(1) The paper seems relatively precise mathematically

(2) There is an effort to present one numerical result.

Critical weaknesses

(1) The assumptions are very restrictive; it is not clear how to check them in practice and hence it is not clear whether the results will be of interest for a broad community.

(2) The discussion on the numerical example is extremely brief and it’s not possible, based on the main text, to understand what is really solved and what to learn from the results.

(3) Last but not least, the contributions are mostly theoretical but the proofs are in supplementary material, which does not have to be checked during the review process. The supplementary material is very long and I can’t personally certify the correctness of the results. I would not be surprised if the results were true under some assumptions (and the ones made are so strong that they might indeed be sufficient).

Questions for authors
  1. Could you please provide a few different examples satisfying all the assumptions you require?

  2. Could you please explain how your assumptions compare with more classical assumptions in the literature (e.g., Lasry-Lions monotonicity etc.)?

  3. Could you please explain, for each assumption whether you expect it to be necessary or not?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 5jS2 Part 1

Thank you for the comments on the assumptions, numerical exposition, and presentation of the proofs. Our point-by-point responses are as follows.

Q1&W1

Q1. Could you please provide a few different examples satisfying all the assumptions you require?

W1. The assumptions are very restrictive; it is not clear how to check them in practice and hence it is not clear whether the results will be of interest for a broad community.

Thank you for asking for concrete examples and practical ways to check the assumptions. In Section 2, we have added a traffic-routing example and explained how symmetry and positive semidefiniteness of the sojourn-time Jacobian can be checked. Below we give three explicit models: route choice in a road network, packet routing in a communication network, and service-pool selection in a processing network. Their common structure is shared-resource congestion: an action uses a collection of resources, and its duration depends on the total completion rate through those resources. This structure appears in congestion games (Rosenthal [1]) and traffic-flow models (Roughgarden and Tardos [2]).

For a state–action pair ii, let Ri\mathcal R_i be the set of resources it uses. Given state–action completion rates x=(xi)x=(x_i), the total completion rate through resource kk is yk(x)=∑i:k∈Rixiy_k(x)=\sum_{i:k\in\mathcal R_i}x_i. With a unit baseline time and a linear congestion delay at each resource, set

\qquad G(x)=\sum_i x_i+\frac12\sum_k y_k(x)^2.$$ The potential is obtained by integrating each resource's delay over its completion rate and summing over resources. Positive baseline times and nonnegative congestion coefficients can also be chosen separately for each action and resource. 1. **Road routes with a shared bottleneck.** There are two terminals, $s\in\\{0,1\\}$. At either terminal, an agent chooses one of two routes, $a\in\\{0,1\\}$, to the other terminal. The two routes in each direction share a bottleneck and each has its own additional road segment. Their sojourn times are $\tau_{sa}(x)=1+x_{sa}+x_{s0}+x_{s1}$. Completing either route moves the agent to terminal $1-s$. Every stationary policy therefore has one recurrent state class. The shared bottleneck gives nonseparable congestion: increasing one route's flow increases the other route's duration. 2. **Packet routing on a ring.** Packets circulate among four routers, $s\in\\{0,1,2,3\\}$. The two actions send a packet to the clockwise or counterclockwise neighbor, with indices interpreted modulo four. Each directed link is a resource, with $\tau_{sa}(x)=1+x_{sa}$. The network is communicating because successive clockwise choices reach every router. It also admits multichain policies: choosing $0\to1$, $1\to0$, $2\to3$, and $3\to2$ creates two recurrent classes. This example satisfies weak communication without requiring every policy to be unichain. 3. **Processing stages sharing service pools.** Jobs alternate between two stages, $s\in\\{0,1\\}$, and at each stage choose one of two service pools, $a\in\\{0,1\\}$. Pool $a$ is shared by jobs from both stages, giving $\tau_{sa}(x)=1+x_{0a}+x_{1a}$. After service, the job enters stage $1-s$, so every stationary policy is unichain. Here congestion couples different states through the shared pool, and the potential is convex without being strongly convex. Modeling sojourn times as functions of throughput has a classical precedent in approximations for closed queueing networks (Kelly [3]). **The assumptions can be checked directly as follows.** 1. **Potential structure.** In the road example, each route has its own resource with load $x_{sa}$ and shares a bottleneck with load $x_{s0}+x_{s1}$. In the packet example, each directed link is a separate resource with load $x_{sa}$. In the service-pool example, pool $a$ has load $x_{0a}+x_{1a}$. Substituting these resource loads into the displayed formula for $G$ and differentiating gives the stated sojourn times in each example, so $\tau=\nabla G$. Each potential is a quadratic polynomial on the entire space, and each $\tau_{sa}\ge1$ on the nonnegative orthant. Thus smoothness and positivity also hold. 2. **Convexity.** The Hessian is a sum of positive semidefinite matrices, one for each resource. Explicitly, for every real vector $v$, $$v^\top\nabla^2G(x)v= \begin{cases} \displaystyle\sum_{s,a}v_{sa}^2+\sum_s(v_{s0}+v_{s1})^2, &\text{road routes},\\\\[3pt] \displaystyle\sum_{s,a}v_{sa}^2, &\text{packet routing},\\\\[3pt] \displaystyle\sum_a(v_{0a}+v_{1a})^2, &\text{service pools}. \end{cases}$$ Each expression is nonnegative. Together with the preceding smoothness and positivity checks, this verifies Assumption 1 for all three examples.
AuthorsOpenReview ↗

Response to Reviewer 5jS2 Part 2

Q1&W1(Cont.)

  1. Unichain policies. In the road and service-pool examples, every stationary policy alternates between the two states, which form a single recurrent class. In the packet example, the policy that always moves clockwise has one recurrent class containing all four routers. Theorem 7 therefore applies to this policy and to any other stationary unichain policy. For policy revision, Π\Pi consists of all deterministic unichain policies: it includes every deterministic policy in the road and service-pool examples, and excludes the multichain policies in the packet example.

  2. Weak communication. In the road and service-pool examples, either state is reached from the other after one action completion. In the packet example, successive clockwise moves reach every router from every other router. All three controlled transition models are therefore communicating, and hence satisfy weak communication (Assumption 4). Lemma 22 then guarantees that Π\Pi contains an optimal policy for every positive sojourn-time vector in each example.

Q2

Q2. Could you please explain how your assumptions compare with more classical assumptions in the literature (e.g., Lasry-Lions monotonicity etc.)?

Thank you for asking for this comparison. Our potential assumption concerns how sojourn times depend on completion rates. We compare it below with monotonicity of reward rates and contraction of policy and population updates, using the examples from Q1.

Lasry–Lions monotonicity. This condition requires a nonpositive pairing between changes in population and changes in its reward coupling, as in Perrin et al. [4, Section 2]. Their transition probabilities are independent of the population. In our continuous-time model, an agent holding state–action pair ii earns reward at instantaneous expected rate riλi(μ)r_i\lambda_i(\mu), and congestion also changes its transition rate. Using the held pair as the agent's state, the corresponding reward-rate monotonicity condition would require

\bigl[r_i\lambda_i(\mu)-r_i\lambda_i(\mu')\bigr]\le0.$$ Take the shared-service-pool example in Q1 with congestion coefficient eight and rewards $(r_{00},r_{01},r_{10},r_{11})=(6,1,1,1)$: $$\tau_{sa}(x)=1+8(x_{0a}+x_{1a}), \qquad G(x)=\sum_{s,a}x_{sa}+4\sum_a(x_{0a}+x_{1a})^2.$$ This has a smooth convex potential and a unichain two-stage transition cycle. In coordinate order $(00,01,10,11)$, consider two populations of mass one, $$\mu=\left(\frac14,\frac38,0,\frac38\right), \qquad \mu'=\left(0,\frac18,\frac34,\frac18\right).$$ Little's law gives sojourn-time vectors $(2,3,2,3)$ and $(3,2,3,2)$, respectively. Substitution yields $$\sum_i(\mu_i-\mu'_i) \bigl[r_i\lambda_i(\mu)-r_i\lambda_i(\mu')\bigr] =\frac1{24}>0.$$ Thus convexity of $G$ as a function of completion rates does not imply monotonicity of reward rates as a function of the state–action population. Our Lyapunov argument uses the potential and the relation $\mu_i=x_i\partial_iG(x)$ to analyze the congestion-dependent state–action dynamics. **Contraction.** Guo et al. [5, Assumptions 1–2 and Theorem 1] obtain contraction from a Lipschitz optimal-policy selection and Lipschitz population updates. Writing their constants as $c_1,c_2,c_3$, the required bound is $c_1c_2+c_3<1$: $c_1$ controls the optimal policy's dependence on population, while $c_2,c_3$ control the population update's dependence on policy and population. This gives a unique fixed point in their discounted MFG framework. The optimal-policy Lipschitz condition already fails in the road example of Q1. With mass one, take the two populations $$\mu^\pm=\left(\frac14\pm h,\frac14\mp h, \frac14\pm h,\frac14\mp h\right), \qquad 0<h<\frac14.$$ At either terminal, Little's law and the specified sojourn times give $$\mu_{s0}-\mu_{s1} =(x_{s0}-x_{s1})[1+2(x_{s0}+x_{s1})], \qquad \tau_{s0}-\tau_{s1}=x_{s0}-x_{s1}.$$ Hence route 1 is uniquely optimal at both terminals under $\mu^+$, while route 0 is uniquely optimal under $\mu^-$. Denote these best responses by $\pi^+$ and $\pi^-$. With the discrete metric, the Wasserstein distance between the two populations is $W_1(\mu^+,\mu^-)=4h$, whereas the maximum over states of the Wasserstein distance between their action probability distributions is $D(\pi^+,\pi^-)=1$. The required Lipschitz inequality would therefore give $$1\le4c_1h\qquad\left(0<h<\frac14\right),$$ where the same constant $c_1$ must work for every $0<h<1/4$. This requires $c_1\ge1/(4h)$ for all such $h$, but $1/(4h)\to\infty$ as $h\downarrow0$. Hence no finite $c_1$ can satisfy the Lipschitz inequality. Our convergence analysis accommodates these discontinuous best responses through a differential inclusion and the policy potential.
AuthorsOpenReview ↗

Response to Reviewer 5jS2 Part 3

Q3

Q3. Could you please explain, for each assumption whether you expect it to be necessary or not?

Thank you for asking us to clarify which assumptions are needed for each result. The table below shows where the four structural conditions enter the main results in Sections 3–6.

Main resultPotential structureConvexityUnichain conditionWeak communication
State–Action Dynamics under a Fixed Policy (Section 3): Lyapunov function in Proposition 6 and global convergence in Theorem 7Used to construct the Lyapunov function.Used to obtain well-defined completion rates and a Lyapunov function minimized at the stationary population.Required for the chosen fixed policy.Not required.
Policy Dynamics in an Equivalent Population Game (Section 4): potential structure and convergence in Theorem 8; policy equilibria and stationary MFG equilibria in Theorem 9Used to construct the policy potential.Used to establish concavity of the policy potential and convergence to its maximizer set.Required for every policy in Π\Pi, to define its unique invariant probability distribution and stationary payoff.Used for optimal-policy coverage and the correspondence with stationary MFG equilibria.
State–Policy Dynamics with Exogenous Revision (Section 5): frozen state–policy convergence in Proposition 10 and convergence as policy revisions become rare in Theorem 11Used in the frozen Lyapunov function and the policy potential.Used in both frozen state–policy convergence and policy convergence.Required for every policy in Π\Pi, so each fixed policy marginal has a unique stationary state–policy population.Used to identify the limiting populations as stationary MFG equilibria through the correspondence in Section 4.
Event-Driven State–Policy Dynamics (Section 6): policy-potential identity in Proposition 12 and event-driven convergence as policy revisions become rare in Theorem 13Used in the policy-potential derivative and the frozen Lyapunov argument.Used to bound the potential gap by payoff improvement and to establish frozen state–policy convergence.Required for every policy in Π\Pi, as in the exogenous-revision model.Used to identify the limiting populations as stationary MFG equilibria through the correspondence in Section 4.

Potential structure and convexity are used in all four analyses. Convexity is also important for the completion rates to be well defined. For example, if τ(x)=2−x\tau(x)=2^{-x}, then xτ(x)=1/2x\tau(x)=1/2 at both x=1x=1 and x=2x=2, so the same population can correspond to different completion rates when convexity is removed.

The unichain condition concerns the policies used in the dynamics. Section 3 requires it only for the chosen fixed policy; Sections 4–6 require it for the policies in Π\Pi. None of these results requires every policy of the underlying MDP to be unichain, and transient states are allowed. For state–action dynamics under a fixed policy, multiple recurrent classes conserve their separate masses, so different initial allocations cannot all approach the same stationary population. The same issue arises in the frozen state–policy dynamics if a policy has several recurrent classes.

Weak communication is not required for state–action convergence under a fixed unichain policy. The results in Sections 4–6 are stated under Assumptions 1 and 4. Under Assumption 1, their policy-potential and frozen-dynamics convergence arguments apply to a prescribed nonempty family of unichain policies without using weak communication. Weak communication guarantees that Π\Pi contains an optimal policy for every positive sojourn-time vector (Lemma 22) and supports the correspondence between policy equilibria and stationary MFG equilibria (Theorem 9). This is the step that gives the convergence results their MFG interpretation. Without weak communication, separate absorbing states can have different optimal average rewards, so this correspondence can fail.

Reviewer_8diXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The manuscript discusses stationary equilibria in a finite-state, continuous-time mean field game. Based on assumptions of convex potential function and weak communication, the manuscript establishes an equilibrium-optimizer correspondence for an Eisenberg-Gale type program, global convergence of the nonlinear population dynamics under a fixed unichain policy, and a concave ordinal potential for a population game whose strategies are deterministic unichain policies. In Sections 5 and 6, it addresses joint state-policy dynamics and strategic adjustment under exogenous revision or revision at action completions.

Critical strengths

The manuscript provides a useful convergence theory for nonlinear physical dynamics. The population dependence of completion rates prevents the physical system from being a linear Markov process. The Lyapunov function addresses this dependence directly.

Critical weaknesses
  1. Courcoubetis and Dimakis (2023)[1] already study convex-potential congestion-dependent sojourn times under weak communication. The optimization program in equation (3) in the manuscript is recovered from their Theorem 1 by taking one population type and zero resource requirements. Their Lemma 1 and Corollary 1 also establish a continuous Little’s-law map, equilibrium existence, and common equilibrium average rewards. This work is absent from the manuscript’s references. The authors should clarify the incremental content of contribution (i).
  2. The paper does not give a finite-time population error bound for selecting a revision parameter. Such a bound would help interpret the rare-revision results and guide parameter selection.
  3. The main contribution needs a clearer separation from prior work. The main text contains four propositions and six theorems. However, it does not give sufficient priority to the main new results. A clearer hierarchy would improve the readability.

Reference:

  1. Costas Courcoubetis and Antonis Dimakis. 2023. Stationary Equilibrium of Mean Field Games with Congestion-dependent Sojourn Times. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems (AAMAS '23). International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 2913–2915.
Questions for authors
  1. How do proposition 2 and each part of theorem 5 differ from the prior work? Please revise contribution (i) accordingly and identify the precise new claims.
  2. For the reported 66 Manhattan TLC zones runs, what are the maximum number of stored policies, the number of best-response LP solves, and elapsed time on specified hardware?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 8diX Part 1

Thank you for recognizing the convergence result for the nonlinear state–action dynamics and for suggesting that we clarify our contributions and their relation to prior work. Our point-by-point responses are as follows.

Q1&W1:

Q1. How do proposition 2 and each part of theorem 5 differ from the prior work? Please revise contribution (i) accordingly and identify the precise new claims.

W1. Courcoubetis and Dimakis (2023)[1] already study convex-potential congestion-dependent sojourn times under weak communication. The optimization program in equation (3) in the manuscript is recovered from their Theorem 1 by taking one population type and zero resource requirements. Their Lemma 1 and Corollary 1 also establish a continuous Little’s-law map, equilibrium existence, and common equilibrium average rewards. This work is absent from the manuscript’s references. The authors should clarify the incremental content of contribution (i).

Thank you for asking us to clarify the relationship to prior work. We agree that the stationary variational framework and proof approach in Proposition 2 and Theorem 5 follow [1], and we now cite [1] explicitly. We build on [1]'s formulation of congestion-dependent sojourn times to study a different dynamical model: an MFG with population-dependent continuous-time Markov transition rates. The modeling distinction concerns the evolution away from stationarity. The continuous-time semi-Markov model in [1] describes stationary average rewards and flow balance through transition probabilities and mean sojourn times. Different holding-time distributions with the same means can produce different action completion rates over time, since these rates may depend on the time already spent on each action. Thus mean sojourn times alone do not specify the transient state–action population dynamics. We specify this evolution through a population-dependent Markov generator and analyze the convergence of the population dynamics, including their coupling to policy revision. The differences of Proposition 2 and each part of Theorem 5 from [1] are listed in the table below.

ResultDifference from [1]
Proposition 2Under our C2C^2 assumptions, Appendix A.1 establishes local C1C^1 extensions at boundary populations and Lipschitz continuity of x(μ)x(\mu) and λ(μ)\lambda(\mu) on the state–action simplex. These regularity properties support the differential equations describing the continuous-time Markov population evolution.
Theorem 5(i)A generator-invariance formulation of the equilibrium–optimizer correspondence and explicit reconstruction of an optimal stationary policy, including its choices at states with zero completion rate.
Theorem 5(ii)The whole sojourn-time vector τ∗=∇G(x∗)\tau^{\ast}=\nabla G(x^{\ast}) is common to all optimizers, and hence the individual completion rates λi(μ∗)=1/τi∗\lambda_i(\mu^{\ast})=1/\tau_i^{\ast} are common to all equilibria.
Theorem 5(iii)Compactness and convexity of the equilibrium state–action population set, using EI∗=diag⁡(τ∗)X∗\mathcal E_{\mathcal I}^{\ast}=\operatorname{diag}(\tau^{\ast})\mathcal X^{\ast}.

These properties support the subsequent dynamics analysis. We have also revised contribution (i) accordingly:

(i) We formulate the congestion-dependent continuous-time MFG and prove a one-to-one correspondence between stationary MFG equilibrium populations and the optimal solutions of a finite-dimensional Eisenberg–Gale-type convex program, following the stationary framework of Courcoubetis & Dimakis (2023; 2025). This correspondence yields equilibrium existence and a direct computational method. We further characterize the regularity of the completion-rate map and the convex structure of the equilibrium population set, which are needed for the study of the learning dynamics.

Q2:

Q2. For the reported 66 Manhattan TLC zones runs, what are the maximum number of stored policies, the number of best-response LP solves, and elapsed time on specified hardware?

Thank you for asking about the computational cost. For the four runs underlying Figure 1, the measurements are:

Figure 1 runMaximum stored policiesBest-response LP solvesElapsed time (s)
(a) Fixed policy13500.99
(b) Policy best response4924,46640.74
(c) Exogenous revision5234,50071.34
(d) Event-driven revision4935,99078.18

These measurements were obtained on an Apple M4 CPU with 16 GB RAM.

AuthorsOpenReview ↗

Response to Reviewer 8diX Part 2

W2

W2. The paper does not give a finite-time population error bound for selecting a revision parameter. Such a bound would help interpret the rare-revision results and guide parameter selection.

Thank you for raising this point. We agree that the paper does not provide an explicit finite-time MFG population-error bound for general GG. The difficulty is to quantify the distance of the stationary state–policy population to the MFG equilibrium set in terms of the potential gap. Continuity and compactness guarantee that this distance vanishes with the gap, but give no explicit rate; the smoothness and convexity assumptions on GG do not specify how rapidly the policy potential Φ\Phi decreases away from its maximizers. For quadratic GG, as in the Manhattan numerical model, we can prove the following population-error bounds from every initial state–policy population. For exogenous revision at Poisson rate κ,\kappa,

\operatorname{dist}(\eta(t),\mathcal E_{\mathcal S\times\Pi}^{\ast}) \le c_{\rm init} e^{-c_{\rm dec}t}\\|\eta(0)-\bar\eta^{\nu(0)}\\| +c_{\rm rev,exo}\kappa +c_{\rm eq} e^{-\kappa t/2}\sqrt{\Delta(\nu(0))}. \end{aligned}$$ For event-driven revision with probability $\varepsilon$ at each action completion, $$\begin{aligned} \operatorname{dist}(\eta(t),\mathcal E_{\mathcal S\times\Pi}^{\ast}) \le c_{\rm init} e^{-c_{\rm dec}t}\\|\eta(0)-\bar\eta^{\nu(0)}\\| +c_{\rm rev,evt}\varepsilon +c_{\rm eq} e^{-\varepsilon\underline\lambda t/2} \sqrt{\Delta(\nu(0))}. \end{aligned}$$ Here $\mathcal E_{\mathcal S\times\Pi}^{\ast}$ is the stationary MFG state–policy population set, $\nu(t)$ is the policy marginal of $\eta(t)$, and $\bar\eta^{\nu(t)}$ is its associated stationary population. As in Section 4, the potential gap is $\Delta(\nu):=\Phi^{\ast}-\Phi(\nu)$, where $\Phi^{\ast}=\max_{\nu'\in\mathcal M_m(\Pi)}\Phi(\nu')$. Both bounds use physical time $t$ and Euclidean distance; $\underline\lambda$ is the uniform lower bound on individual completion rates. The positive constants $c_{\rm init},c_{\rm dec},c_{\rm rev,exo},c_{\rm rev,evt},c_{\rm eq}$ depend on the population mass $m$, transition probabilities, rewards, coefficients of $G$, and the finite policy family $\Pi$. They are independent of the initial population, time, revision parameters, and the measurable best-response selection. The constants $c_{\rm init}$ and $c_{\rm dec}$ control the initial transient; $c_{\rm rev,exo}$ and $c_{\rm rev,evt}$ bound the disturbance due to revision; and $c_{\rm eq}$ converts the potential gap into stationary population distance. The bounds combine two estimates. Exponential convergence of the frozen state–policy dynamics and a perturbation estimate give the initial-transient and revision terms. For quadratic $G$, the equilibrium conditions and regularity of the stationary-population map yield $$\operatorname{dist}(\bar\eta^\nu,\mathcal E_{\mathcal S\times\Pi}^{\ast}) \le c_{\rm eq}\sqrt{\Delta(\nu)}.$$ Combining this estimate with the proved exponential decay of $\Delta(\nu(t))$ gives the last term in each population-error bound. These bounds indeed guide parameter selection. For a run of duration $T$, we choose the revision parameters as follows: $$\kappa=\frac{2}{T}\log(1+\underline\lambda T), \qquad \varepsilon=\frac{2}{\underline\lambda T} \log(1+\underline\lambda T),$$ with $T$ large enough that $\varepsilon\le1$. These parameters are held constant during the run. In both bounds, the policy-error term at $T$ is then at most $c_{\rm eq}\sqrt{\Delta(\nu(0))}/(1+\underline\lambda T)$. Together with the revision term and the decaying initial transient, this gives a population-error bound of order $\log(1+\underline\lambda T)/T$ for the fixed model.
AuthorsOpenReview ↗

Response to Reviewer 8diX Part 3

W3

W3. The main contribution needs a clearer separation from prior work. The main text contains four propositions and six theorems. However, it does not give sufficient priority to the main new results. A clearer hierarchy would improve the readability.

Thank you for this suggestion. We agree that the contribution hierarchy should be clearer. We have revised the contribution paragraph in Section 1 (p. 2) to distinguish the stationary framework of [1, 2] from the main convergence results. Item (i), quoted above, now cites [1, 2] for that framework and explains that the further regularity and equilibrium-set properties support the study of learning dynamics. Items (ii)–(iv) now identify the convergence theorems directly: Theorem 7 for state–action dynamics under a fixed policy, Theorem 8 for best-response policy dynamics, and Theorems 11 and 13 for state–policy dynamics with exogenous and event-driven revision. This makes the progression from a fixed policy to policy revision and then to their coupling explicit.

Thank you again for your constructive comments. We have revised the paper as described above, with the manuscript changes highlighted in blue. We hope our responses address your questions and concerns. We welcome any further questions.

References

  1. Costas Courcoubetis and Antonis Dimakis. 2023. Stationary Equilibrium of Mean Field Games with Congestion-dependent Sojourn Times. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems (AAMAS '23), 2913–2915.
  2. Costas Courcoubetis and Antonis Dimakis. 2025. Selfish Behavior and Resource Competition in Multi-Agent Systems. In Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2025), 529–537.

Thank you for the response. I will maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.