- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces a new notion of mean field game where the cost of the agents depends on a provided state-decision probability measure and on the state-decision probability measure of the population of agents. It defines the notion of a stable equilibrium, where the provided state-decision probability measure is equal to its induced mean field game probability measure. The remainder of the paper consists of establishing assumptions under which such an equilibrium exists and can be computed via an algorithm. Lastly, one numerical experiment is performed on three home-made environments.
Critical strengths
Strength 1: Introducing distributional shift in mean field games is a natural and timely contribution.
Strength 2: The paper tries to be complete in its presentation. It introduces a new definition of mean field game, establishes assumptions under which an equilibrium exists, and provides an algorithm that is tested numerically. It also provides in appendix proofs for every statement.
Critical weaknesses
Weakness 1: The objective of the paper is to define a new notion of equilibrium, called 'Stable Performative Equilibrium'. Yet, I have the impression that this notion of equilibrium is simply a classical Nash equilibrium in an extended mean field game (a mean field game where the dynamics and reward depend on the joint state-action probability measure of the population of agents). Define the mean field game where is the state space, is the action space, and is the cost. Using the notation of the paper, let for any . Let . We call a policy a measurable mapping from to . We call the set of best responses with respect to any the set of policies minimizing , where has probability distribution and is a random variable distributed according to , . Let us also introduce for any initial state distribution . We call a Nash equilibrium for the extended mean field game and initialization any policy such that is a best response to .
My claim is the following: if is a Nash equilibrium for the extended mean field game above, with respect to some initialization , then is a stable equilibrium in the sense you define. Conversely, any stable Nash equilibrium , as you define it, induces a Nash equilibrium for the extended mean field game for some initial distribution.
If I understood correctly and I am right, your notion of stable equilibrium does not define anything new. Extended mean field games have already been studied in the literature, and you should clearly state what is different in your paper, and what has been newly introduced and proven.
Weakness 2: I am not convinced that the numerical experiment proves that the algorithm converges toward a stable Nash equilibrium. Why would showing that the averaged payoff converges prove that the algorithm converges to 1) a stable Nash equilibrium under the real loss, or 2) a stable Nash equilibrium under the regularized loss? You do not report any values, such as exploitability, that would show that a deviating agent cannot obtain a better loss than the one obtained under the state-decision probability measure output by the algorithm.
Weakness 3: Definitions 2 is central to this paper. Since I read it as a theoretically oriented paper, I believe the definition would benefit from additional care and precision. In their current form, they left me with several questions, which I detail in the Questions section below.
Weakness 4: From my point of view, the second contribution of the paper, "Equilibrium Convergence Analysis", is too similar to existing results on mean field games to be a real contribution for a theoretically oriented paper. Also, the third claimed contribution, "This algorithm ensures convergence to a informatively stable mean-field equilibrium by enforcing the convergence conditions to be satisfied during a repeated performative mean field game.", is a bit misleading because the algorithm does not converge toward the stable equilibrium of the original game, but toward that of the regularized game.
Questions for authors
Question 1-2: Could you discuss weakness 1,2 ?
Question 3: In definition 2, could you clearly state how is taken ? Is it any Nash equilibrium ? Is it a Nash equilibrium for some fixed ? But then, I have the impression the set should also depend on the initialization. You are taking the closure of a set. Under which topology ? You do not discuss why, could you ? What happen if is empty ?
7 comments
Regarding the repeated game definition. We thank the reviewer for the careful observation and the chance to clarify these points.
in .
- In the experiments, is a type-decision distribution with zero average decision. With this , the induced data is the raw data from the dataset.
- can be any type-decision distribution. is not required to be a Nash equilibrium, nor a Nash equilibrium for some fixed .
Dependence on initialization. Yes. is the closure of the sequence , which starts at . Thus, different initializations produce different sets . Specifically, is determined by:
- , the initial type-decision distribution,
- the game structure, determining the strong monotonicity parameter ,
- the data distribution maps, determining the sensitivity parameter .
Once is determined, our theoretical results hold.
Topology of . The closure in the repeated game definition is taken in the weak$ ^{\star}$ topology of .
- In this topology, the expected loss is continuous in for bounded continuous . By Berge's Maximum Theorem, the best-response correspondence is continuous with respect to .
- In this topology, the type-decision distribution set is compact in the weak$ ^{\star}$ topology, which ensures that the equilibrium set is compact in the weak$ ^{\star}$ topology.
Emptiness of . We clarify that cannot be empty under our conditions. By Lemma 1, under -strong monotonicity and joint continuity of loss function , exists and is unique for every .
Revision.
- We have clarified the weak$ ^{\star}$ topology in which is closed in Section 2.2 (lines 207-208).
Regarding our contributions. We thank the reviewer for the comment and the opportunity to clarify our contributions.
Contribution 2. We emphasize that the convergence analysis differs from existing results on mean field games.
-
Setting. The data distribution maps are unknown, so the loss function cannot be minimized directly. A detailed comparison with existing MFG methods is in our reply to the first point.
-
Iteration dynamics. The iteration dynamics in our work differ from those in existing MFG work, as detailed below.
| Our work | Existing results | |
|---|---|---|
| Each iteration | Equilibrium under the data distribution | Best response to the previous type-decision distribution |
| Lipschitz constant | with respect to | with respect to |
Contribution 3. We clarify that our algorithm converges to the equilibrium of a regularized game, as stated in Section 4.
We agree that the description of contribution 3 in Section 1.1 could be clearer, and we have updated this point.
Revision.
- We have clarified that our algorithm converges to the equilibrium of a regularized game in Section 1.1 (line 116).
I thank the authors for their answers and the additional experiments. However, many of my concerns remain.
1. Relation to extended MFG.
Firstly, the rebuttal agrees with the link with extended MFG (by the way, I think you must explicitly give the mathematical link as I made, in the appendix, for clarity). The claimed novelty is that is unknown. To my understanding, this is still not a novelty. It seems that it is possible to write this problem under a standard learning model-free mean field games. Guo et al, 2019 already study MFGs with unknown rewards and dynamics. More precisely, the performative MFG looks to be a special case of the general MFG (GMFG) in which the reward depends on the joint population state-action distribution and is unknown a priori. One can compltetly write the performative MFG as a GMFG, and apply Guo et al, 2019 results. Therefore, my concern remains.
Secondly, the paper's conditions make the extended game strongly monotone. Let $F(m) := \mathcal{L}_ {\theta}(m, m) $. The convergence condition is a sufficient condition for -strong Lasry–Lions monotonicity of the extended game. Uniqueness within MRPMFG then follows from the classical monotonicity argument. Moreover, fictitious play (Perrin et al., 2020) only require the cost at the current population , which can be estimated from samples of since is known. Their convergence guarantees are stated for exact cost evaluation, which is the setting of Theorem 2 of the paper; a stochastic version would correspond to Theorem 3, for which Guo et al., 2019 provides a sample-based guarantee.
2. Numerical Experiments.
The reported exploitability includes , with a different for each method ( for , for fixed regularization, data-driven for and ). Each curve therefore measures the distance to the equilibrium of a different game, and the curves are not comparable. More importantly, my question was whether a deviating agent can improve its loss in the game defined in the paper, i.e. under , not under . I do not think your new experiments answer my question.
The main text still reports average payoff, which is not an equilibrium criterion: a higher payoff does not indicate an equilibrium. It is not evidence of convergence to an equilibrium and should not be the main experiment
3. Definitions.
The rebuttal confirms that $\mathcal{M} _ {RPMFG} $ depends on . In Algorithm 1 it also depends on the sequence of , since the trajectory, and hence the set, changes with . This dependence should appear in the notation, e.g. .
The uniqueness argument (Eqs. 31–32) invokes -strong monotonicity for two arbitrary equilibria . However, strong monotonicity is only assumed on , which is itself defined through the (assumed unique) sequence of equilibria. As written, uniqueness is not established. The rebuttal's answer to the emptiness question relies on this lemma.
Summary.
I appreciate the clarification that Algorithm 1 converges to the stable equilibrium of a regularized game. But the regularized equilibrium can differ substantially from the original one, and its distance to the original equilibrium is neither bounded theoretically nor measured empirically. By contrast, Guo et al., 2019 has a convergence guarantee to an -Nash equilibrium of the original game under the same type of oracle. In my view, this limits the significance of contribution 3. I also still consider contribution 2 to be close to known results on monotone mean field games.
References
- X. Guo, A. Hu, R. Xu, and J. Zhang. Learning mean-field games. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
- S. Perrin, J. Pérolat, M. Laurière, M. Geist, R. Élie, and O. Pietquin. Fictitious play for mean field games: continuous time analysis and applications. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
We are sincerely grateful for the reviewer's commitment of valuable time in carefully evaluating our manuscript.
Below, we respond to the reviewer's valuable feedback. All revisions made in the manuscript are highlighted in blue for easy identification.
Regarding the relation to extended mean-field games. We thank the reviewer for the detailed reading and for the opportunity to clarify the relation to extended mean-field games.
Setting. We clarify that the data distribution map Dθ in c(θ,x,m)=Ez∼Dθ(m)[ℓ(x,z,θ,m)] is unknown in our setting, although the construction with cost c(θ,x,m)=Lθ(m,m)(x) is mathematically correct.
As a result, V(π;m)=E[c(θ~,X~,m)] cannot be minimized directly, whereas existing extended-MFG methods minimize V(π;m) directly.
The novelty. We clarify that the main difference lies in the learning problem, where the data distribution maps Dθ are unknown, so the cost c(θ,x,m) cannot be evaluated directly.
We compare our setting with existing mean-field game literature:
Due to these differences, we introduce the following to solve the problem:
Revision.
Regarding experimental evidence. We thank the reviewer for the comment and for the chance to strengthen our experimental evidence.
Regularized equilibrium. The algorithm targets the regularized game and converges to its performatively stable mean-field equilibrium.
Additional experimental results. We conducted additional experiments to verify that our algorithm converges to a performatively stable mean-field equilibrium, which are detailed as follows.
Exploitability. In Appendix J.4, we report the exploitability, i.e., the gap between the loss of the deployed decision and the lowest loss that a deviating agent can obtain given the decision distribution m(t) at each iteration:
Eθ∼μ[EZθ∼Dθ(m(t))[ℓ(xθ⋆,Zθ,θ,m(t))+γC(xθ⋆,θ,m(t))]]−Eθ∼μ[minx∈X(EZθ∼Dθ(m(t))[ℓ(x,Zθ,θ,m(t))+γC(x,θ,m(t))])], where xθ⋆ is the deployed decision of a player with type θ under Zθ∼Dθ(m(t)) given decision distribution m(t).
Figs. 12–14 in Appendix J.4 show that the exploitability approaches 0 for our method with b=1.1,10 within 3 iterations, indicating that our algorithm converges to the mean-field equilibrium for the regularized game.
Equilibrium residual. In Appendix J.5, we further report the equilibrium residual ∥m⋆(t)−m(t)∥H−s, where m⋆(t) is the mean-field equilibrium under the data distribution induced by m(t).
Figs. 15–17 in Appendix J.5 show that the equilibrium residual approaches 0 for our method with b=1.1,10 within 3 iterations, showing that our algorithm converges to a mean-field equilibrium of the game it induces.
Revision.