acceptodds
Under review as a conference paper at ICLR 2027

Towards Scaling Reinforcement Learning to Massive Populations: Learning Mean-Field Representations

Abstract

Modern multi-agent systems are increasingly deployed at scale over large populations of agents in settings such as ad-auctions, traffic routing, and recommendation systems. The dominant approach in such settings is to optimize each agent's policy independently, treating the other agents as part of a fixed single-agent environment rather than modeling the population dynamics. In many large-population systems, the dynamics depend on an aggregate summary of the population rather than the identity of any individual. Mean-field RL exploits such structure, providing a principled framework that models each agent's environment as an explicit function of the population distribution. However, in large state-action spaces or high-dimensional control problems, modeling the population distribution is itself intractable. How can we design a scalable framework for high-dimensional control problems with large populations? This work explores this question from the perspective of representation learning. We introduce a mean-field RL framework in which the rewards and transition dynamics depend on the population only through an unknown low-dimensional aggregate statistic. We then study this framework in the offline setting and design a provable approach that learns a near-optimal policy by learning a low-dimensional representation. Motivated by real-life supply-chain optimization problems, we design a one-step routing game to test the hypothesis that learning a low-dimensional population representation improves reward prediction and Nash gap estimation relative to baselines that don't exploit this structure. We show that under a fixed neural-network parameter count and optimization budget, learning a low-dimensional population representation improves reward prediction and the equilibrium quality of the resulting policies.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, representation learning, multi-agent systems

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EZeAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper discusses learning a low-dimensional population representation rather than directly modeling the full state-action distribution. Overall, the paper is clearly written and makes a certain theoretical contribution.

Critical strengths

This paper discusses learning a low-dimensional population representation rather than directly modeling the full state-action distribution. Overall, the paper is clearly written and makes a certain theoretical contribution.

Critical weaknesses
  1. Despite its theoretical contribution, I doubt the practical means really work far better than alternative representation learning methods. Meanwhile, the environment seems to require only that the learner estimate rewards and cannot fully support the paper's statements.

  2. The authors are suggested to clarify their theoretical contribution. The current methodology and theoretical proof follows largely what has been done in the era of offline RL. Is there any relaxed assumption or different insight?

Questions for authors

Clarify the contributions from both theoretical uniqueness and methodological comparisons.

3 comments

Scope of contributions

The theoretical contributions of this paper are providing a language to study low-dimensional representation for large population multi-agent systems and a statistical guarantee for offline equilibrium learning with an unknown low-dimensional representation. We also perform experiments to provide evidence for the claim that learning the population representation can improve both prediction and the equilibrium quality of the resulting policies. This is shown using a simple experiment for two reasons: first, a one-step experiment cleanly isolates the representation learning mechanism and is easy to understand, and second, there is no computationally efficient algorithm in the paper for complex multi-step settings.

Experiment baselines

Section 5 already compares the learned representation with a single-agent model, a monolithic raw-mean model, and a full-population-law model, under approximately matched parameter counts. We also include two oracle baselines. Figure 3 reports both prediction error and the exact true Nash gaps of the resulting policies.

The experiments support the claim that learning the population representation can improve prediction and equilibrium quality in the tested setting. The guarantees for multi-step problems with unknown transitions are theoretical results under the stated assumptions. We don’t claim to provide either a computationally efficient algorithm for this problem or empirical implementation of the main algorithm in the paper. This is very clearly mentioned in the paper; see, for example, the beginning of section 5 where we discuss experiments or the conclusion.

Novelty

It is common in theory to build on top of other works. That, by itself, cannot be a weakness of a paper. We are very clear that we build on Zhang et al.’s technique of minimizing an estimated equilibrium gap based on optimistic deviation values and pessimistic policy values. However, this isn’t the extent of our theoretical analysis. Our setting introduces two additional statistical issues relative to Zhang et al.: first, the population is observed through finite snapshots, and the empirical representation computed from each snapshot depends on the candidate feature map; second, planning under a candidate model uses that model’s own induced population flow. Also, unlike Zhang et al., we don’t impose Bellman-completeness conditions on a value function class.

Thanks for your comment. I will maintain the score due to the remaining concern on novelty.

Can the reviewer provide more details about their concerns? We will try to alleviate them.

Reviewer_oyA4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies offline mean-field reinforcement learning when population interactions depend on an unknown low-dimensional aggregate representation. It proposes a model-based learning procedure and derives finite-sample Nash-gap guarantees under realizability, regularity, and coverage conditions, accounting for both trajectory data and finite population snapshots. Experiments investigate learned population representations for reward prediction and equilibrium computation in a one-step congestion-routing game.

Critical strengths
  • The paper provides a clear formulation of learning unknown population representations and connects model estimation to approximate equilibrium computation.
  • The analysis distinguishes errors arising from limited offline trajectories and finite population snapshots, providing useful insight into these two sources of uncertainty.
  • The manuscript clearly acknowledges the computational limitations of its theoretical procedure and the narrower scope of its experiments.
Critical weaknesses
  • All experiments use a single fixed, one-step routing network, so varying demand distributions, behavior policies, and data sizes provides limited evidence of applicability across environments.
  • The empirical evaluation is confined to a single fixed, one-step routing environment, leaving the framework’s applicability to different network structures and broader mean-field RL settings insufficiently demonstrated.
  • The statistical advantage of low-dimensional representations remains implicit in model-class complexity and coverage coefficients, without a concrete multi-step example demonstrating the claimed scaling benefit.
Questions for authors
  • Does the observed advantage persist when models are trained and evaluated on additional network topologies or congestion-cost configurations?
  • What evidence supports extending the findings from one-step reward learning to multi-step problems involving learned transitions and evolving population distributions?
  • Can the authors provide a concrete multi-step model class with controlled coverage coefficients that demonstrates the statistical benefit of learning a low-dimensional population representation?
  • Have the authors evaluated the approach in other multi-agent environments with mean-field interactions, and do the benefits of learned population representations persist beyond the specific routing environment studied here?
3 comments

Thank you for the helpful review.

Scope of the experiments

The point of experiments is to provide evidence for the claim that learning the population representation can improve both prediction and the equilibrium quality of the resulting policies. This is shown using a simple experiment for two reasons: first, a one-step experiment cleanly isolates the representation learning mechanism and is easy to understand, and second, there is no computationally efficient algorithm in the paper for complex multi-step settings.

We have not evaluated additional network topologies, or other multi-agent environments. So it’s fair to judge the empirical support within that scope. But we like to think that the main contributions in the paper are of theoretical in nature and the empirical support is for a smaller claim that learning the population representation can improve both prediction and the equilibrium quality of the resulting policies in an interesting setting.

Even though it is clear to the reviewer, we still want to emphasize that the statistical results themselves are not restricted to one-step problems. They allow arbitrary finite horizons, unknown transitions, and evolving population distributions. Thus, the experiments don’t say anything about these additional complications.

Statistical advantage

Yes, the statistical advantage of low-dimensional representations is indeed implicit in model-class complexity and coverage coefficient. But this is to be expected. Our results make these dependencies explicit. To be complete, we do not claim that replacing a full population law by a low-dimensional representation necessarily improves statistical efficiency. This improvement would be a function of the environment and how population structure affects the coverage coefficient. But if there is an improvement, then our analysis may capture it. And this improvement can be substantial as the following example demonstrates.

Let H=2H=2, X={−1,1}n\mathcal X = \{-1, 1\}^n and A={−1,1}\mathcal A = \{-1, 1\}. The transition dynamics are known and are X1=A0X0X_1 = A_0 X_0, i.e., the action either preserves the initial state or reverses all its signs. The initial state is sampled from a uniform distribution. There is no reward at the first stage. An unknown feature f ⁣:X→{−1,1}f \colon \mathcal X \to \{-1, 1\} determines teh population representation m1=E[f(X1)]m_1 = \mathbb E[f(X_1)] and the reward at the second (last) stage r1f(x,a,m)=12+a6(m−2f(x))r_1^f(x,a,m) = \frac12 + \frac a6(m - 2f(x)). Suppose the model class has the following structure: f(x)=xjf(x) = x_j for some fixed j∈[n]j \in [n]. There are 2n2^n states but only nn candidate models. A not too difficult calculation reveals that, under uniformly random actions, the coverage coefficient satisfies κ≤18\kappa \le 18, giving a uniform upper bound. Our main theorem therefore gives Gap(η^)≤εopt+O(log⁡(n/δ)N+1K)\text{Gap}(\hat \eta) \le \varepsilon_{\text{opt}} + O\left(\sqrt{\frac{\log (n/\delta)}{N}} + \frac{1}{\sqrt K}\right). This is a exponential improvement over the alternative where the model class is as big as possible because ff can be an arbitrary binary function because then we have log⁡22n/N\sqrt{\log 2^{2^n}/N} instead of log⁡n/N\sqrt{\log n / N} scaling.

Thank you for the clarification. However, I remain concerned about the limited experimental scope. While I understand the paper’s theoretical focus, the current experiment leaves it unclear whether the observed benefits extend beyond this particular setup. Could you evaluate the method in at least one additional environment, even another simple one-step setting?

AuthorsOpenReview ↗

More experiments

Thank you for the reply. We created and tested on one more one-step environment.

The new env

It's a portfolio game where the state xx encodes the investor type and the actions encode going long or short a portfolio. In our instantiation we use 32 equally likely investor types, μ(x)=1/32\mu(x) = 1/32, and 6 different actions a∈±e1,±e2,±e3⊂R3a \in \\{\pm e_1,\pm e_2,\pm e_3\\} \subset \mathbb R^3.

Each type has a fixed, hidden orthogonal matrix QxQ_x that maps its chosen position to an exposure f⋆(x,a)=Qxaf^\star(x,a)=Q_xa to three common market factors. Half the types have private factor valuations αx=(0.15,0.20,0)\alpha_x=(0.15,0.20,0), and the other half have αx=(0.15,−0.20,0)\alpha_x=(0.15,-0.20,0). Within each group, types come in pairs with matrices QQ and −Q-Q, so the same action gives paired investors opposite exposures.

The population law ν\nu has 32×6=19232\times6=192 entries, and its effect on rewards is summarized by the three-dimensional mean-field (intuitively, the net exposure) m⋆(ν)=∑x,aν(x,a)f⋆(x,a)m^\star(\nu)=\sum_{x,a}\nu(x,a)f^\star(x,a). The normalized reward is r⋆(x,a,m⋆(ν))=12+f⋆(x,a)⊤αx−f⋆(x,a)⊤m⋆(ν)2.5∈[0,1]r^\star(x,a,m^\star(\nu))=\tfrac12+\tfrac{f^\star(x,a)^\top\alpha_x-f^\star(x,a)^\top m^\star(\nu)}{2.5}\in[0,1]. The first inner product gives the investor’s private value for the position. The second imposes a cost for adding to the population’s net exposure and gives a credit for offsetting it. Therefore, the 192-entry population law affects rewards only through the three-dimensional statistic m⋆(ν)m^\star(\nu).

The dense nature of the rewards differs from the sparse and binary features used in the routing game in the paper.

Modeling

The offline data is sampled using a similar method as for the routing game in the paper.

Let q(x,a)∈R192q(x,a)\in\mathbb R^{192} be the joint one-hot encoding, and let z(x,a)∈R38z(x,a)\in\mathbb R^{38} concatenate the type and action one-hot encodings (38 comes from 32 + 6). Write ν^n=1K∑k=1Kq(Xn(k),An(k)).\widehat\nu_n=\frac1K\sum_{k=1}^K q(X_n^{(k)},A_n^{(k)}). The learned mean-field model learns a three-dimensional feature for each joint state-action category:

\qquad \widehat m_n^\theta=W^\top\widehat\nu_n, \qquad W\in\mathbb R^{192\times3}.$$ ## Results We used $N \in \\{1000, 5000, 25000, 100000, 200000\\}$ and $K \in \\{1,2,4,8,16,32,64,256,512\\}$. Since we cannot attach images, we provide a table below for $N=200000$ and $K=512$ where the values are most easily read due to the large data set. We have a full N vs. K plot in the paper in Appendix C. | Model | Population RMSE | True Nash gap | |---|---:|---:| | Single-agent | 0.0961 ± 0.0001 | 0.1379 ± 0.0209 | | Raw mean | 0.0939 ± 0.0003 | 0.1372 ± 0.0280 | | **Learned mean-field** | 0.0694 ± 0.0004 | 0.0187 ± 0.0069 | | Finite-$K$ oracle | 0.0427 ± 0.0226 | 0.0108 ± 0.0132 | | Exact-population-input oracle | 0.0484 ± 0.0122 | 0.0123 ± 0.0116 | **This environment also verifies the core claim of the paper that the learned mean-field model improves both prediction and equilibrium quality relative to single-agent and raw-mean models.**
Reviewer_uWYrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies offline Nash equilibrium learning in mean-field games in which the dynamics depend on an aggregate summary of the population rather than the identity of any individual. The authors explore this problem from the perspective of representation learning. Based on this, they propose a mean-field RL framework in which the rewards and transition dynamics depend on the population only through an unknown low-dimensional aggregate statistic. Then they develop a model-based method that learns representation-dependent reward and transition models from a finite class and selects a policy by minimizing an optimistic-pessimistic upper bound on its Nash gap over all models consistent with the data. Since the model-based method is not practical, they evaluate the representation-learning mechanism in a one-step congestion-routing game and show that learning a low-dimensional population representation improves reward prediction and the equilibrium quality of the resulting policies.

Critical strengths

This paper is well-motivated and provides careful analysis of challenges specific to learning mean-field representations. It also provides a principled statistical foundation for representation learning in mean-field games.

Critical weaknesses
  1. The paper’s framing seems to overstate its practical scope, and the main contribution seems not to be empirically validated. For my understanding, Algorithm 1 and its Nash-gap guarantee are the main contributions of this paper. However, this algorithm is not practical since it needs to enumerate all candidate models and policies. In the experiment part, it only shows the low-dimensional representation helps reward prediction, not that the proposed offline learning method works or not.

  2. The theoretical contribution seems to be an extension of existing analysis and the assumptions seem to be too strong to be applied in practical. The authors state that they adopt the adaptive-bound strategy of Zhang et al. (2023b) for offline Markov games, and statistical guarantees for model-based mean-field RL already exist.

Questions for authors
  1. This paper seems to be an extension of previous works. Could the authors discuss the differences from Zhang et al. (2023b), and which steps would fail if their results were applied directly with a known representation or a full-law representation?
  2. Do the authors consider whether there is a computationally tractable approximation of Algorithm 1 that could be simply evaluated in a multi-step mean-field game? If not, what is the biggest challenge?
  3. In real-world applications, such as ad auctions or recommendation systems, how would one obtain K population snapshot samples at every stage alongside each representative trajectory? And how would the guarantee change if the population snapshots were biased or not independent of the representative trajectory?
4 comments

Scope of contributions

Our main contributions are providing a language to study low-dimensional representation for large population multi-agent systems and a statistical guarantee for offline equilibrium learning with an unknown low-dimensional representation. We don’t claim to provide either a computationally efficient algorithm for this problem or empirical implementation of the main algorithm in the paper. This is very clearly mentioned in the paper; see, for example, the beginning of section 5 where we discuss experiments or the conclusion. So the paper should be judged for what it does provide — a new framework and a statistical analysis.

We would like to correct the reviewer’s claim that the experiments “only show the low-dimensional representation helps reward prediction”. We also calculate the exact true Nash gap of policies computed from each learned reward model; see equation (7). And the experiments show that our algorithm helps with reducing the Nash gap as well. The point of experiments is to provide evidence for the claim that learning the population representation can improve both prediction and the equilibrium quality of the resulting policies. This is shown using a simple experiment for two reasons: first, a one-step experiment cleanly isolates the representation learning mechanism and is easy to understand, and second, there is no computationally efficient algorithm in the paper for complex multi-step settings as mentioned above.

Comparison with Zhang et al. (2023b)

It is common in theory to build on top of other works. That, by itself, cannot be a weakness of a paper. We are very clear that we build on Zhang et al.’s technique of minimizing an estimated equilibrium gap based on optimistic deviation values and pessimistic policy values. However, this our theoretical analysis goes beyond Zhang et al.’s in non-trivial ways. Our setting introduces two additional statistical issues relative to Zhang et al.: first, the population is observed through finite snapshots, and the empirical representation computed from each snapshot depends on the candidate feature map; second, planning under a candidate model uses that model’s own induced population flow. These issues prevent a direct application of Zhang et al.’s analysis because we must control the noise in the candidate-dependent population inputs and show how accuracy on logged data transfers to value predictions along the population flows generated by a candidate model. Also, unlike Zhang et al., we don’t impose Bellman-completeness conditions on a value function class since we use a model-based analysis.

Assumptions

We agree that the assumptions rule out some settings, but disagree with the assessment that they are “too strong”. For example, the transition dynamics allow for deterministic dynamics. Even for stochastic dynamics, in practice, it is the case that there is a uniform lower bound on the transition probabilities, and our assumptions thus allow for this.

Computationally efficient algorithm

As discussed above and in the paper, coming up with a computationally efficient algorithm is not the goal of this paper. However, it is instructive to think why coming up with such an algorithm is non-trivial. Changing the population policy changes the population flow even if the candidate model is fixed. This will in general produce a non-convex optimization problem. For a given implementation of the model class, say using neural networks, obtaining the confidence set is another challenge. A practical approximation for both of these challenges is required; interesting follow-on work could investigate how to handle these approximation challenges. One could also try circumventing these challenges by coming up a new clever algorithm that is computationally efficient.

Population snapshots

The snapshots may be dependent across stages. However, conditional on the population policy, the analysis assumes that the snapshots are independent of the representative trajectory and that each stage contains K iid observations from the population’s state-action law. But note that unconditional independence is not assumed, since the snapshots and the representative trajectory share the same random behavior population policy. These could be obtained by simply sampling K times from the large population of the agents. The analysis in the paper can be adapted to account for biased or dependent observations within a snapshot. For example, if at each stage we have a bound on the expected squared Euclidean error of the empirical representation, uniformly over candidate feature maps, then the analysis remains valid when its original snapshot-error bound is replaced by this more general mean-squared-error bound. The confidence-set tolerance and prediction-error radius are adjusted accordingly, after which the Nash-gap theorem applies.

Thanks for the detailed response. I acknowledge that this paper makes a theoretical contribution. However, the analysis seems to mainly extend an existing framework to a new setting. Furthermore, the strong assumption limits the practical significance of the contribution. I therefore keep my score.

Thank you for the reply.

As we explained in our reply above, the analysis doesn't simply extend an existing framework to a new setting. Our setting introduces two additional statistical issues relative to Zhang et al.: first, the population is observed through finite snapshots, and the empirical representation computed from each snapshot depends on the candidate feature map; second, planning under a candidate model uses that model’s own induced population flow. These issues prevent a direct application of Zhang et al.’s analysis because we must control the noise in the candidate-dependent population inputs and show how accuracy on logged data transfers to value predictions along the population flows generated by a candidate model. Also, unlike Zhang et al., we don’t impose Bellman-completeness conditions on a value function class since we use a model-based analysis.

About the assumptions, we don’t claim that they are minimal, but some restrictions are necessary for a vanishing Nash-gap guarantee. For example, without regularity in population dependence, even a known game can have every policy’s Nash gap bounded below by a positive constant. Which specific assumptions do you consider too restrictive, and which practical settings did you have in mind?

Thanks to the authors' further clarification. The remaining concern about assumptions is that the model class is assumed to be finite and to contain the true model. In practice, the model class would usually be a neural network class, which is infinite and may not contain the true model. Regarding population snapshots, I appreciate the earlier explanation that the analysis can be adapted to biased or dependent snapshots. However, “sampling K times from the population” assumes access to the population’s states and actions at each stage, which may not be recorded in typical logged data, for example, in ad auctions or recommendation systems.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.