- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies solving episodic Stackelberg mean-field games with online samples in an RL manner. The setting considered here is that the leader does not know the followers' reward function (which is assumed to be linear with known feature but an unknown parameter) and only observes the followers' states and actions. The authors introduces an algorithm that combines kernelized optimistic least-squares value iteration with an MLE-type method for the estimation of the unknown reward parameter. The regret of the algorithm in complete-observation and finite-sample-observation settings is derived relative to the complete-observation optimal policy. The authors further establish a lower bound, showing that the dependency is the best possible.
Critical strengths
To my best understanding, the paper establishes a significant, previously unknown result -- the online RL-type algorithm for solving a Stackelberg mean-field game and its regret analysis.
Critical weaknesses
Overall I feel the paper is more like a technical documentation for the authors themselves to refer to, rather than a research paper written for a broader audience: important technical results, including algorithm design, assumptions, theoretical results are stated without proper explanation. This makes it very challenging for the readers to parse the paper and understand the contributions. Some specific comments below:
-
This assumption is difficult to make sense of. I struggle thinking of a system where the mean field eolves deterministically. Can the authors clarify how restrictive this assumption is and under what circumstances it is guaranteed to hold?
-
Some choices in the problem setup are unexplained. For example, why do the authors consider the quantal response on the followers' side (which means the followers are solving a entropy-regularized problem) while the leader's objective is not regularized? Additionally, the follower's reward is assumed to have a linear structure, but the authors did not explain what dictates this choice and how/whether the algorithm design and analysis becomes harder without the linearity assumption.
-
I can see the authors' efforts to introduce the algorithm step by step, first introducing the CO baseline, followed by ways to handle uncertainty in the FSO setting. However, Section 3 overall remains difficult to read. In particular, Section 3.3 introduces dense equations without sufficient interpretation. I strongly suggest a major revision here.
-
The bound in Theorem 4.1 looks very distracting in the presence of the unexplained constants. If the authors would like to explicitly show the dependency on these constants, consider explaining them immediately before/after the theorem in the main paper. Otherwise I think it is fine to hide the constants and just present the dependency of the bound on the most important structural parameters.
-
Regarding line 143-145 -- after reading through the math I understand is the follower's (quantal) best response to . That could have been explained in plain English for clarity. This comment applies to other notations as well. As the paper is notation-heavy, the authors should consider adding proper explanatory remarks and/or a table of notations.
Questions for authors
See weaknesses above.
2 comments
Response to Reviewer AE4K (2)
W2(b): Linear follower rewards
Apps. C.13.2 & C.13.5 already explain this choice and the separated complexity profile. Linearity is in the unknown parameter, ; known features may be nonlinear in states/actions. Finite local spaces give reward entries at each leader context, but the leader state space may be countably infinite. Shared -dimensional features pool information across contexts, replacing separate reward tables. This separates response estimation from leader-side RKHS value learning and finite-population sampling. Linearity in makes the conditional negative log-likelihood convex, with a covariance Hessian underlying the Fisher confidence sets in (3.3)–(3.5). For rewards nonlinear in , additional curvature can destroy convexity; confidence sets and predictive-error control would need new guarantees for optimistic planning (3.7).
Linearity simplifies response estimation while retaining the mean-field learning difficulties. With one-hot features on finite domains and suitable parameter bounds, the tabular subclass retains latent responses, population-dependent leader rewards/transitions and empirical Bellman inputs. For population observation, Thm. C.46 proves an expected-regret lower bound already for , even with reward, transition and response primitives known, for the specified FSO commitment-rule class. We use Chen et al.'s linear-myopic parameterization to isolate mean-field coupling and finite-observation costs. We will place the existing appendix rationale beside the reward specification.
W3–W5: Restoring main-text explanations
Our original manuscript already included the relevant interpretations and explanations in the main text. Some were moved to the appendix to meet the 9-page limit. We will use the 1 extra page in revision to restore them beside the definitions, algorithms and results.
W3: Algorithm
Building on the CO-to-FSO progression you noted, we will clarify how §3.3 combines the response confidence sets (3.3)–(3.5) with empirical-design KRR and joint optimism. Lines 335–340 explain the two error corrections; Alg. 1 and the text after (3.8) distinguish episode-start planning from rollout without replanning.
In §3.3, KRR estimates reward plus optimistic continuation; planning uses the response confidence sets from paired samples. In (3.6), the enlarged radius absorbs sampling error in historical Bellman targets, while the additive correction protects evaluation at the current empirical population. The planner then maximizes the expected optimistic action value in (3.7) jointly over commitments and confidence-set parameters , coupled through . The optimum (3.8) supplies the preceding step's continuation term. The sweep fixes the full rule before rollout; execution evaluates it at the observed state and empirical population. We will place this roadmap before (3.6), explain joint optimism before (3.7), and link these steps to the proof overview in §4.2.
W4: Regret bound
The main text already provides explanations of the constants and error terms in Thm. 4.1. “Complexity and calibration” (lines 378–392) explains the complexity and calibration quantities and points to the remaining constants' definitions. Rem. 4.2, immediately after the theorem, identifies the confidence radii and explains the contributions of the terms in (4.1). We will make these explanations easier to consult by adding a concise coefficient glossary beside the theorem.
Your suggestion to suppress constants and emphasize the main structural dependencies is helpful. The compact RKHS rate (4.2) and finite-rank rate (4.4) already provide such expressions. We will foreground these rates and their conditions and, depending on the available space, simplify the theorem display by suppressing auxiliary constants.
W5: Notation
We will add a plain-English explanation at first use: “At step , is the followers' quantal-response policy induced by ; it gives action probabilities at each follower state under the mixture committed at the current leader state and population.”
App. A already provides NOTATION tables for the model, estimators, and constants, etc. Table 2 distinguishes policy-level from local ; lines 139–145 define and specify response to the committed mixture, not the sampled action. We will cite App. A, explain recurrent symbols at first use, and use one compact guide to distinguish , true/empirical populations and joint distributions, and the latent parameter, coordinated with W4's coefficient guide.
We hope our responses resolve your questions and concerns. If so, we would appreciate your reconsideration of the overall evaluation. If any remain, please let us know so that we can address it fully.
Response to Reviewer AE4K (1)
Dear Reviewer AE4K,
Thank you for your review and for recognizing the significance of our online RL algorithm and regret analysis. We address your questions and concerns below.
W1: Conditional mean-field evolution and model scope
Assump. 2.2 allows STOCHASTIC population trajectories: its determinism is CONDITIONAL on (sh,ah,μh,νh). Eq. (2.1) fixes the infinite-population mean field μh+1=Φh(sh,ah,μh,νh) once these variables are given. The conditioning variables can themselves be random, so different realizations can yield different population trajectories. The restriction excludes residual population-level shocks after conditioning on these variables.
Individual randomness averages out. Lines 156–159 already give Φh(s,a,μ,ν)(x′)=∑x,bμ(x)ν(b∣x)PhX(x′∣s,a,μ,x,b). For example, conditionally i.i.d. state–action pairs with law μ(x)ν(b∣x) and conditionally independent next-state draws under PhX yield this population evolution map by a conditional law of large numbers in the infinite-population limit. Individual transitions remain random. This mechanism is established in Lasry & Lions, already cited at lines 156–159, and Huang et al. In discrete time, Subramanian & Mahajan, Eqs. (6)-(7) explicitly use Zt+1=Φ(Zt,πt) despite stochastic individual transitions.
Our new binary example (H≥3) has μh+1(1)=(1+2sh+ah)/5 and TL,h(1∣s,a,P)=1/4+P(1,1)/2. With ah+1=0, μh+2(1) takes 1/5 or 3/5, both with positive probability, and E[μh+2(1)∣sh,ah,μh,νh]=3/10+Ph(1,1)/5. Thus follower behavior affects later population evolution through stochastic leader states. The example also satisfies Bellman closure: its rank-eight full-system representation meets (H.9), so Prop. H.2 applies. We will include its verification in the appendix.
This scope isolates the core learning problem: the FSO leader still sees only N paired observations per step, must infer unknown follower responses, and learns population-dependent rewards and stochastic transitions. Conditional determinism does not reveal the population to the learner. Our upper bound controls these uncertainties; the CO and class-specific FSO lower bounds establish leader-learning and observation costs. We will foreground this microscopic explanation and add the example beside Assump. 2.2.
W2: Modeling choices and their rationale
We respectfully disagree with “Some choices in the problem setup are unexplained.” The regularization asymmetry follows the QSE definition established in the literature, with its rationale discussed in the Introduction and App. C.13.1. The rationale for linear follower rewards is explicitly given in Apps. C.13.2 & C.13.5. Within the 9-page main text, we present established conventions concisely and provide extended explanations in the appendix.
W2(a): Quantal followers and an unregularized leader
The solution concept studied in this paper is QSE, and the asymmetry follows the QSE definition. Here, the “Q” stands for “quantal,” which refers to the follower's quantal response, not the leader's commitment. Accordingly, the entropy term is only added to the follower's objective, not the leader's. The leader optimizes her unregularized cumulative return J(π). This QSE definition follows Černý et al., Chen et al. Adding leader entropy changes this definition.
The entropy term in (C.5) models followers' bounded rationality; the leader maximizes the cumulative return J(π). The Introduction and App. C.13.1 already distinguish these roles. For the followers, finite-η QR is unique and smooth in expected rewards at each fixed commitment and follower state. Best responses can be set-valued at ties; selections may jump as rewards change, and tie-breaking can change the leader's objective. QR therefore complements best response on behavioral grounds and provides the smooth likelihood for our MLE/Fisher analysis. For the leader, a maximizing commitment suffices even if it is not unique. Unregularized objectives allow mixed commitments.
QR has behavioral and empirical support (McKelvey & Palfrey; Yang et al.). The deployed PROTECT system optimizes the defender's strategy against a QR attacker (Shieh et al.; An et al.). We will place this rationale beside the response definition.