acceptodds
Under review as a conference paper at ICLR 2027

Reinforcement Learning for Quantal Stackelberg Equilibrium in Mean-Field Games

Abstract

We study online reinforcement learning in episodic Stackelberg mean-field games, where a leader interacts with an infinite population of boundedly rational, myopic followers governed by a quantal-response model. The leader knows neither the followers' reward function nor their realized rewards and must infer their response model from behavioral observations. We analyze the complete observation (CO) setting, where the leader observes the exact joint state-action distribution of the population, and then extend the analysis to finite-sample observation (FSO), where only followers are observed per step and the true population distributions remain unobserved. We propose an algorithm combining kernelized optimistic least-squares value iteration with a confidence set based on maximum likelihood estimation for the followers' reward parameter. Under our stated assumptions, for a leader kernel of rank and a follower feature dimension , the algorithm achieves high-probability regret under FSO relative to the optimal CO policy, where is the number of episodes and is the horizon length. With fixed response and regularization parameters, this rate holds when , , and , where is the followers' state-action space. We further establish a CO leader-side lower bound , showing that the dependence is unavoidable. For the FSO commitment-rule class in this paper, we prove a fixed-dimensional lower bound ; hence is necessary for uniform expected regret within this class.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, Stackelberg games, mean-field games

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AE4KOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies solving episodic Stackelberg mean-field games with online samples in an RL manner. The setting considered here is that the leader does not know the followers' reward function (which is assumed to be linear with known feature but an unknown parameter) and only observes the followers' states and actions. The authors introduces an algorithm that combines kernelized optimistic least-squares value iteration with an MLE-type method for the estimation of the unknown reward parameter. The regret of the algorithm in complete-observation and finite-sample-observation settings is derived relative to the complete-observation optimal policy. The authors further establish a lower bound, showing that the T\sqrt{T} dependency is the best possible.

Critical strengths

To my best understanding, the paper establishes a significant, previously unknown result -- the online RL-type algorithm for solving a Stackelberg mean-field game and its regret analysis.

Critical weaknesses

Overall I feel the paper is more like a technical documentation for the authors themselves to refer to, rather than a research paper written for a broader audience: important technical results, including algorithm design, assumptions, theoretical results are stated without proper explanation. This makes it very challenging for the readers to parse the paper and understand the contributions. Some specific comments below:

  1. This assumption is difficult to make sense of. I struggle thinking of a system where the mean field eolves deterministically. Can the authors clarify how restrictive this assumption is and under what circumstances it is guaranteed to hold?

  2. Some choices in the problem setup are unexplained. For example, why do the authors consider the quantal response on the followers' side (which means the followers are solving a entropy-regularized problem) while the leader's objective is not regularized? Additionally, the follower's reward is assumed to have a linear structure, but the authors did not explain what dictates this choice and how/whether the algorithm design and analysis becomes harder without the linearity assumption.

  3. I can see the authors' efforts to introduce the algorithm step by step, first introducing the CO baseline, followed by ways to handle uncertainty in the FSO setting. However, Section 3 overall remains difficult to read. In particular, Section 3.3 introduces dense equations without sufficient interpretation. I strongly suggest a major revision here.

  4. The bound in Theorem 4.1 looks very distracting in the presence of the unexplained constants. If the authors would like to explicitly show the dependency on these constants, consider explaining them immediately before/after the theorem in the main paper. Otherwise I think it is fine to hide the constants and just present the dependency of the bound on the most important structural parameters.

  5. Regarding line 143-145 -- after reading through the math I understand νhπ\nu_h^\pi is the follower's (quantal) best response to π\pi. That could have been explained in plain English for clarity. This comment applies to other notations as well. As the paper is notation-heavy, the authors should consider adding proper explanatory remarks and/or a table of notations.

Questions for authors

See weaknesses above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer AE4K (1)

Dear Reviewer AE4K,

Thank you for your review and for recognizing the significance of our online RL algorithm and regret analysis. We address your questions and concerns below.

W1: Conditional mean-field evolution and model scope

Assump. 2.2 allows STOCHASTIC population trajectories: its determinism is CONDITIONAL on (sh,ah,μh,νh)(s_h,a_h,\mu_h,\nu_h). Eq. (2.1) fixes the infinite-population mean field μh+1=Φh(sh,ah,μh,νh)\mu_{h+1}=\Phi_h(s_h,a_h,\mu_h,\nu_h) once these variables are given. The conditioning variables can themselves be random, so different realizations can yield different population trajectories. The restriction excludes residual population-level shocks after conditioning on these variables.

Individual randomness averages out. Lines 156–159 already give Φh(s,a,μ,ν)(x′)=∑x,bμ(x)ν(b∣x)PhX(x′∣s,a,μ,x,b)\Phi_h(s,a,\mu,\nu)(x')=\sum_{x,b}\mu(x)\nu(b\mid x)P_h^{\mathcal X}(x'\mid s,a,\mu,x,b). For example, conditionally i.i.d. state–action pairs with law μ(x)ν(b∣x)\mu(x)\nu(b\mid x) and conditionally independent next-state draws under PhXP_h^{\mathcal X} yield this population evolution map by a conditional law of large numbers in the infinite-population limit. Individual transitions remain random. This mechanism is established in Lasry & Lions, already cited at lines 156–159, and Huang et al. In discrete time, Subramanian & Mahajan, Eqs. (6)-(7) explicitly use Zt+1=Φ(Zt,πt)Z_{t+1}=\Phi(Z_t,\pi_t) despite stochastic individual transitions.

Our new binary example (H≥3H\ge3) has μh+1(1)=(1+2sh+ah)/5\mu_{h+1}(1)=(1+2s_h+a_h)/5 and TL,h(1∣s,a,P)=1/4+P(1,1)/2T_{L,h}(1\mid s,a,\mathcal P)=1/4+\mathcal P(1,1)/2. With ah+1=0a_{h+1}=0, μh+2(1)\mu_{h+2}(1) takes 1/51/5 or 3/53/5, both with positive probability, and E[μh+2(1)∣sh,ah,μh,νh]=3/10+Ph(1,1)/5\mathbb E[\mu_{h+2}(1)\mid s_h,a_h,\mu_h,\nu_h]=3/10+\mathcal P_h(1,1)/5. Thus follower behavior affects later population evolution through stochastic leader states. The example also satisfies Bellman closure: its rank-eight full-system representation meets (H.9), so Prop. H.2 applies. We will include its verification in the appendix.

This scope isolates the core learning problem: the FSO leader still sees only NN paired observations per step, must infer unknown follower responses, and learns population-dependent rewards and stochastic transitions. Conditional determinism does not reveal the population to the learner. Our upper bound controls these uncertainties; the CO and class-specific FSO lower bounds establish leader-learning and observation costs. We will foreground this microscopic explanation and add the example beside Assump. 2.2.

W2: Modeling choices and their rationale

We respectfully disagree with “Some choices in the problem setup are unexplained.” The regularization asymmetry follows the QSE definition established in the literature, with its rationale discussed in the Introduction and App. C.13.1. The rationale for linear follower rewards is explicitly given in Apps. C.13.2 & C.13.5. Within the 9-page main text, we present established conventions concisely and provide extended explanations in the appendix.

W2(a): Quantal followers and an unregularized leader

The solution concept studied in this paper is QSE, and the asymmetry follows the QSE definition. Here, the “Q” stands for “quantal,” which refers to the follower's quantal response, not the leader's commitment. Accordingly, the entropy term is only added to the follower's objective, not the leader's. The leader optimizes her unregularized cumulative return J(π)J(\pi). This QSE definition follows Černý et al., Chen et al. Adding leader entropy changes this definition.

The entropy term in (C.5) models followers' bounded rationality; the leader maximizes the cumulative return J(π)J(\pi). The Introduction and App. C.13.1 already distinguish these roles. For the followers, finite-η\eta QR is unique and smooth in expected rewards at each fixed commitment and follower state. Best responses can be set-valued at ties; selections may jump as rewards change, and tie-breaking can change the leader's objective. QR therefore complements best response on behavioral grounds and provides the smooth likelihood for our MLE/Fisher analysis. For the leader, a maximizing commitment suffices even if it is not unique. Unregularized objectives allow mixed commitments.

QR has behavioral and empirical support (McKelvey & Palfrey; Yang et al.). The deployed PROTECT system optimizes the defender's strategy against a QR attacker (Shieh et al.; An et al.). We will place this rationale beside the response definition.

AuthorsOpenReview ↗

Response to Reviewer AE4K (2)

W2(b): Linear follower rewards

Apps. C.13.2 & C.13.5 already explain this choice and the separated complexity profile. Linearity is in the unknown parameter, rh(x,b;s,a)=⟨ϕ(s,a,x,b),θh∗⟩r_h(x,b;s,a)=\langle\phi(s,a,x,b),\theta_h^*\rangle; known features may be nonlinear in states/actions. Finite local spaces (X,B)(\mathcal X,\mathcal B) give ∣X∣∣B∣|\mathcal X||\mathcal B| reward entries at each leader context, but the leader state space may be countably infinite. Shared dFd_F-dimensional features pool information across contexts, replacing separate reward tables. This separates response estimation from leader-side RKHS value learning and finite-population sampling. Linearity in θ\theta makes the conditional negative log-likelihood convex, with a covariance Hessian underlying the Fisher confidence sets in (3.3)–(3.5). For rewards nonlinear in θ\theta, additional curvature can destroy convexity; confidence sets and predictive-error control would need new guarantees for optimistic planning (3.7).

Linearity simplifies response estimation while retaining the mean-field learning difficulties. With one-hot features on finite domains and suitable parameter bounds, the tabular subclass retains latent responses, population-dependent leader rewards/transitions and empirical Bellman inputs. For population observation, Thm. C.46 proves an Ω(T/N)\Omega(T/\sqrt N) expected-regret lower bound already for dF=1d_F=1, even with reward, transition and response primitives known, for the specified FSO commitment-rule class. We use Chen et al.'s linear-myopic parameterization to isolate mean-field coupling and finite-observation costs. We will place the existing appendix rationale beside the reward specification.

W3–W5: Restoring main-text explanations

Our original manuscript already included the relevant interpretations and explanations in the main text. Some were moved to the appendix to meet the 9-page limit. We will use the 1 extra page in revision to restore them beside the definitions, algorithms and results.

W3: Algorithm

Building on the CO-to-FSO progression you noted, we will clarify how §3.3 combines the response confidence sets (3.3)–(3.5) with empirical-design KRR and joint optimism. Lines 335–340 explain the two error corrections; Alg. 1 and the text after (3.8) distinguish episode-start planning from rollout without replanning.

In §3.3, KRR estimates reward plus optimistic continuation; planning uses the response confidence sets from paired samples. In (3.6), the enlarged radius absorbs sampling error in historical Bellman targets, while the additive correction protects evaluation at the current empirical population. The planner then maximizes the expected optimistic action value in (3.7) jointly over commitments pp and confidence-set parameters θ\theta, coupled through μ^⊗νp,θ\widehat\mu\otimes\nu^{p,\theta}. The optimum (3.8) supplies the preceding step's continuation term. The sweep fixes the full rule before rollout; execution evaluates it at the observed state and empirical population. We will place this roadmap before (3.6), explain joint optimism before (3.7), and link these steps to the proof overview in §4.2.

W4: Regret bound

The main text already provides explanations of the constants and error terms in Thm. 4.1. “Complexity and calibration” (lines 378–392) explains the complexity and calibration quantities and points to the remaining constants' definitions. Rem. 4.2, immediately after the theorem, identifies the confidence radii and explains the contributions of the terms in (4.1). We will make these explanations easier to consult by adding a concise coefficient glossary beside the theorem.

Your suggestion to suppress constants and emphasize the main structural dependencies is helpful. The compact RKHS rate (4.2) and finite-rank rate (4.4) already provide such expressions. We will foreground these rates and their conditions and, depending on the available space, simplify the theorem display by suppressing auxiliary constants.

W5: Notation

We will add a plain-English explanation at first use: “At step hh, νhπ\nu_h^\pi is the followers' quantal-response policy induced by π\pi; it gives action probabilities at each follower state under the mixture committed at the current leader state and population.”

App. A already provides NOTATION tables for the model, estimators, and constants, etc. Table 2 distinguishes policy-level νπ\nu^\pi from local νp,θ\nu^{p,\theta}; lines 139–145 define νhπ\nu_h^\pi and specify response to the committed mixture, not the sampled action. We will cite App. A, explain recurrent symbols at first use, and use one compact guide to distinguish π,g,p\pi,g,p, true/empirical populations and joint distributions, and the latent parameter, coordinated with W4's coefficient guide.

We hope our responses resolve your questions and concerns. If so, we would appreciate your reconsideration of the overall evaluation. If any remain, please let us know so that we can address it fully.

Reviewer_TEw5OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces a new framework for mean field games called the quantal Stackelberg equilibrium. In this setting, a leader seeks to optimize its reward, which depends on the evolution of its state and action, and on the evolution of a population of agents described by a probability measure. These agents react to the leader’s policy by following a population-level policy determined by the leader’s state, their own state, the mean field, a specific feature, and a parameter θ\theta. This parameter θ\theta is unknown to the leader. To get information on the parameter, the leader takes actions in a multi period setting. Two settings are distinguished, the complete observation (C0), where the leader takes action with respect to the exact mean field. The finite sample observation, where the leader only as access to the empirical distribution. The paper gives a high-probability upper bound on the regret in the finite sample observation (FSO) setting relative to the complete observation (CO) setting. It also establishes a lower bound depending on the number of periods for both the CO and FSO settings.

Critical strengths

S1. The paper introduces a new framework for mean field games called the quantal Stackelberg equilibrium. It simplifies the classical setting, since the followers policy is closed form given the leader policy.

S2. It gives a high-probability upper bound on the regret in the finite sample observation (FSO) setting relative to the complete observation (CO) setting. The paper also establishes a lower bound depending on the number of periods for both the CO and FSO settings. To my understanding, if the results hold, the contribution is real and worth reading for the mean field game and multi-agent community.

Critical weaknesses

W1. The paper relies heavily on a 103-page appendix. On page 8, for example, the paper cites the appendix 19 times, even for crucial points such as the assumptions of the main result (assumptions that are sometimes not even discussed in the main body of the paper). The results are hard to follow without reading the appendix. To check the coherence of the assumptions of Theorem 4.1, the readers need to study dozens of pages. I want to highlight, that the assumptions are not trivial. For example Assumption C.3 on some measurability argument. The appendix is not filled with figures but with dense mathematical text. The paper can not be correctly reviewed by a human. The authors should consider an other venue, that accept longer contributions.

W2. I see the paper as theoretical in nature, introducing a new concept for mean field games. However, I find that the paper lacks mathematical rigor, which often makes it hard to follow. The definitions and assumptions use conditioning on random variables, but the filtrations involved are never defined, nor is the way the random variables of interest are generated. For some properties, it is unclear whether they are claims or assumptions. Many variables and mappings are not properly specified (domain of definition, nature of the object, etc.).

W3. The main results rely on many quantities that are not defined in the main text, which discourages the reader from engaging with them.

W4. No numerical results are provided in the main body. I think it is crucial to illustrate concepts when they are newly introduced. Stackelberg mean field games are not new, but to my knowledge, this is the first time I have seen this quantal framework. I would have expected a deeper discussion of the differences between classical Stackelberg mean field games and quantal Stackelberg mean field games. The absence of numerical results and examples makes the results and the setting hard to digest.

Questions for authors

Q1. I do not think that Assumptions 2.2 and 2.3 should be stated as assumptions. They are rather definitions of the model, and presenting them as assumptions is confusing. Could you discuss this choice?

Q2. I do not understand why you chose the name “equilibrium” for the notion introduced on line 166. An equilibrium usually refers to a fixed point in some sense, but here the population of agents does not describe an equilibrium. I find the word “equilibrium” somewhat misleading, since the problem described here consists of maximizing an objective that depends on agents who react in closed form to the leader’s policy. Could you discuss this choice?

Q3. Could you rigorously define the random variables describing the states and the actions, as well as the filtrations of interest? You conditioning is in an informal way (“conditional on the past trajectory”, line 190). This makes the paper hard to follow, and it is also not rigorous.

Q4. Is the sentence on line 196 a claim or an assumption? Could you discuss when it holds? More generally, could you discuss when the assumptions are satisfied, and give examples (in the main body of the paper)?

Q5. Many variables are not properly introduced. What is πt\pi^t on line 164? Could you introduce TT in Section 2? Could you give a mathematical definition of Fobst−1\mathcal{F}^{t-1}_{obs} on line 182? Many other variables are also not clearly introduced.

Q6. Could the authors add numerical experiments to the main body that illustrate the theoretical results and compare the proposed framework with the classical Stackelberg mean-field game setting? To make room, the main results could be stated in a compact form, such as

RegFSO(T)≤C1(M)T+C2(M) TN+C3(M),\mathrm{Reg}_{\mathrm{FSO}}(T) \le C_1(\mathcal{M})\sqrt{T} + C_2(\mathcal{M})\,\frac{T}{\sqrt{N}} + C_3(\mathcal{M}),

where the constants Ci(M)C_i(\mathcal{M}) depend only on the model M\mathcal{M} and are given explicitly in the appendix. This form shows directly how the regret depends on TT and NN, and it would also address W3.

Reviewer_APzeOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies reinforcement learning for a leader interacting with a large population of myopic followers whose responses follow a softmax model with unknown rewards. It proposes an optimistic algorithm that learns follower responses from observed behavior while estimating the leader’s value functions. The regret bounds separate leader learning, response estimation, and population-sampling error. Lower bounds identify unavoidable learning and observation costs. The guarantees rely on strong structural assumptions and, for the simplified square-root regret rate, a per-step observation budget growing with the number of episodes.

Critical strengths
  • Explicit error decomposition: the analysis separates leader learning, follower-response estimation, and population-observation error, clarifying their contributions to regret.
  • Potentially useful technical tools: the population-weighted Fisher analysis and stability of the optimistic value envelope avoid an extra state-count factor and a Lipschitz assumption on the selected policy.
  • Complementary lower bounds: the paper distinguishes exploration costs from the information loss caused by finite population observations, while explicitly restricting the latter result to its commitment-rule class.
Critical weaknesses
  • The assumptions may exclude meaningful population dynamics. The combination of deterministic population evolution, Lipschitz kernel features, and Bellman completeness for arbitrary bounded Borel envelopes appears to force the next population distribution to be independent of the current population and follower response, conditional on the leader’s state and action.

  • The observation lower bound depends critically on restricted memory. Its construction reveals information through an earlier leader state that the permitted commitment rule cannot subsequently use. The bound therefore does not establish the same limitation for general history-dependent learners.

  • Computational feasibility remains unresolved. The algorithm assumes exact solutions to generally nonconvex optimistic planning problems.

  • The experiments provide limited validation of the theorem. They use pure-action, point-estimator planning rather than the full optimistic algorithm, so they mainly illustrate related behavior.

Questions for authors
  • Compatibility of assumptions: Does the combination of Assumptions 2.2, 2.4, and 2.5 force population evolution to be constant in the population input for fixed s,as,a?

  • Nontrivial examples: Can you provide an example satisfying all assumptions in which the next population distribution genuinely depends on the current population or follower response?

  • Experimental relevance: Which parts of the theoretical algorithm are implemented in the experiments, and which guarantees survive the replacement of joint optimistic planning by pure-action, point-estimator planning?

  • The numerical experiments currently provide limited evidence beyond a small synthetic setting. Could you evaluate the method on a spatial crowd-modeling benchmark, such as the grid-based or Two Islands environments used in Finite-Sample Convergence Bounds for Trust Region Policy Optimization in Mean-Field Games (Ocello et al., 2025), adapted to your leader–follower setting?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.