- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper uses machine learning methods, namely neural ODEs and score learning, to solve mean-field games (MFGs). The mean field game can be solved iteratively: An individual agent’s movements is described via an SDE with a drift that changes in response to a cost function that depends on the entire population. Hence, given the distribution of the population an agent minimises the cost to find the drift. Given the best drift, this defines a new distribution of the population (via the marginal distribution of the SDE). The aim is to find an equilibrium This paper reformulates the SDE/Fokker-Planck point of view with probability flows/continuity equations and then learns a vector field for an ODE to have the same marginals as the SDE. It then learns the score of the marginals of the ODE/SDE, so that the drift is given by . This is used to define the new distribution, and this is iterated over until convergence.
Critical strengths
I find using machine learning methods to solve MFG problems to be an interesting research direction. Moreover, upon checking the mathematical details everything seemed correct and in that sense, sound. The method applies to both potential and non-potential MFGs and is therefore quite general.
Critical weaknesses
I found this paper very hard to read. For context, I have background in SDEs, score matching, flow matching and stochastic optimal control but not in mean-field games (MFGs).
-
Background on MFGs: a self-contained section on the MFG would be useful. Although some background is given in the introduction and key equations are defined in Section 2, they are defined without much motivation or explanation for what they are. I believe adding this would make the paper much clearer. Currently I would argue it is inaccessible to an ML audience who have an appropriate background to understand MFGs but are unfamiliar with them.
-
Structure: I find the structure of the paper quite unclear in places, making it hard to follow. Some terms are not defined, others are used and then defined later in the text. A non-exhaustive list of examples includes:
- non-potential MFGs are defined before MFGs. Moreover, Eq. (2) seems to come under the heading of non-potential MFGs making it unclear whether this is a general definition or only meant for non-potential MFGs.
- In line 114 it’s stated “The best response formulation applies to…”, but the best response is not defined until line 135.
- In line 142 the paper states “…the same fixed flow problem…” however, it is not clear which fixed flow problem it is referring to; include a reference to Eq. (2) and state that fixed-flow means is fixed.
- Admissible feedback in line 137 is undefined (it is mentioned in line 126 that they must be square integrable, but it’s not clear whether this is the definition).
-
Motivation: Somewhat related to the structure, I found that a lot of the definitions and propositions lacked an explanation for why they were introduced or why they were interesting. For example, in Section 3, some explanation around would be useful. Currently, it is just defined as , but explaining why it is being defined would be of value, e.g. by adding a sentence about the marginals of the ODE with vector field .
-
Probability flows: It is not clear to me the advantage of using probability flows instead of directly using the SDE parametrisation given in Eq (2). Introducing the flow formulation means both a velocity field and the score of the corresponding ODE need to be learned. However, instead one could parametrise the drift as directly, negating the need to learn a score. Can the authors explain what is gained in using the probability flow formulation?
Minor: I saw some spelling mistakes throughout and from page 6 onwards, the paper is in italics.
Summary: I think the paper presentation needs significant improvement in order to be published. Although sometimes paper presentation issues can be more minor, here I find it significantly impacts the readability of the paper to a point where it is hard to evaluate the method. For this reason I have given both 1 for presentation and informativeness. As for soundness, I did not find any issues in the proofs. However, given the lack of motivation for using probability flows over directly using the SDE (see point 4.) leads me to question the utility of the method, therefore I also give a 2 for soundness. If this is addressed I am willing to raise the soundness.
Questions for authors
What is the advantage of using the probability flow formulation and learning over directly learning a drift ?
We thank the reviewer 9eBy for the carefully reading and presentation comments. We introduce the non-potential-MFG first because it is the general MFG formulation, and then we present potential MFGs as a special case. We plan to define the representative-agent's optimal problem w.r.t. the population, the best response, admissible feedback, and population consistency before introducing potential MFGs. We will also clarify that Eq. (2) describes the fixed-population control problem and correct the overall typography.
The motivation for defining v=b−ν∇logρ is to express the controlled response law through deterministic trajectories. Here, ρ denotes the response density induced by the controlled drift b, while the population flow entering the agent's cost remains fixed during each best response. With this definition, the Fokker-Planck equation becomes ∂tρ+∇⋅(ρv)=0. Under our regularity assumptions and with the exact response score, the deterministic characteristics of v have the same time marginals as the controlled SDE. This allows us to optimize the best-response objective through deterministic particle trajectories. The continuity-equation viewpoint also connects our formulation to dynamic optimal transport [K]. [D] in the response to reviewer 1 also uses SDE as dynamcs, but it introduces a saddle-point structure and need the adverserial training. We can solve the problem directly using optimization structure. We will add this motivation immediately before the definition of v.
The reviewer is correct that the SDE drift can also be parameterized and trained directly. Direct SDE policy training [I] and FBSDE-based training [J] are valid alternatives, however, these methods are restricted at low dimensions. We will add the comparisons in the supplementary. Our choice provides a deterministic particle representation of the evolving response law: the inner policy solve requires no Brownian increments or backward network, but it requires response-score estimation and ODE backpropagation. Thus, the benefit comes with a computational trade-off.
To examine this trade-off, we compared probability flow with pathwise SDE training on an 8D frozen Gaussian LQ best response: dXt=αtdt+2νdWt, with ν=0.2, X0∼N(0,0.752I), mt=ty, and y=(2,−2,…). Per coordinate, the running cost was 0.5α2+0.1(X−mt)2, and the terminal cost was 4(X1−y)2. Both methods used 384-parameter controls, 512 training particles, 24 time steps, and 150 updates. In this single-seed experiment with an exact Gaussian score, probability flow achieved a held-out original-SDE objective of 2.820, compared with 3.323 for pathwise SDE training, evaluated using 2,048 common particles. CPU training times were 1.132 and 0.795 seconds, respectively. Probability flow therefore achieved a lower objective at greater training cost in this instance. This comparison covers one frozen best response and excludes ScoreNet training and the outer MFG iteration.
[I] Ming Min and Ruimeng Hu. Signatured Deep Fictitious Play for Mean Field Games with Common Noise. ICML, 2021.
[J] Jiequn Han, Ruimeng Hu, and Jihao Long. Learning High-Dimensional McKean-Vlasov Forward-Backward Stochastic Differential Equations with General Distribution Dependence. SIAM Journal on Numerical Analysis, 2024.
[K] Jean-David Benamou and Yann Brenier. A computational fluid mechanics solution to the Monge-Kantorovich mass transfer problem. Numerische Mathematik, 2000.