acceptodds
Under review as a conference paper at ICLR 2027

Mean-Field Q-Learning for Heterogeneous Multi-Agent Systems with Convergence Guarantees

Abstract

We study cooperative reinforcement learning in large populations of heterogeneous agents subject to common noise. The finite-agent control problem is computationally challenging because the joint state and action spaces grow with the population size. To overcome this difficulty, we introduce a non-exchangeable mean-field approximation that retains agent heterogeneity while replacing the high-dimensional finite-agent system with a tractable mean-field control problem. For this limiting problem, we define an integrated Q-function and develop a finite-net Q-learning algorithm. We establish convergence of the learning algorithm and quantify the errors arising from policy regularization, state-action discretization, and the mean-field approximation, yielding an end-to-end performance guarantee for policies transferred to the original finite-agent system. Numerical experiments on linear-quadratic and cybersecurity benchmarks with continuous state and action spaces illustrate the behavior of the method and its effectiveness in finite populations.

Primary area:
reinforcement learning
Keywords:
Multi-agent reinforcement learning, mean-field control, heterogeneous agents, non-exchangeability, Q-learning, common noise

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eDaCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work studies cooperative heterogeneous MARL with agents subject to common noise. It proposes a finite-net Q-learning scheme and shows that restricting relaxed policies to Lipschitz kernels is uniformly consistent. Finally, it presents numerical simulations to validate the theory in settings of congestion and cybersecurity.

Critical strengths

None

Critical weaknesses
  1. It is quite hard to read the paper due to the notation. For instance, almost all works in RL use γ\gamma for the discount parameter in the objective, but this work uses β\beta. Sections 2.1 and 2.2 also read a bit strangely. Ideally, they should be combined and written in a more cohesive way, as Section 2.2 is the infinite limit of 2.1.

  2. A bigger concern I have is with the novelty of the work: for instance, as the authors point out, Mekkaoui & Pham (2026) do almost the same thing in the control-theoretic framework (so its only distinction is that it doesn't have the finite-net Q-learning scheme), and the works of Carmona, Laurrier, Tan (2023) presents a Q-learning scheme (but for finite spaces), and the works of Gu, Guo, Wei, Xu (2021) explicitly studies epsilon-nets of lifted state/action spaces. So my concern here is that this work seems to be spanned by the works above: in particular, I don't think any of the results here are particularly surprising, but maybe I'm missing some critical detail here and the authors could comment further on this (I'm certainly open to raising my score).

Questions for authors
  • As mentioned above, mean-field approximation with finite-net Q-learning and finite-agent performance bound has been done a lot in the literature. I don't think prior work has shown that restricting relaxed policies to Lipschitz kernels is uniformly consistent, but I'm not sure how interesting the uniform convergence of restricted optimal values is, compared to (for instance) a uniform convergence of, say, every measurable kernel. Can the authors comment on this? Specifically, why is this the right restriction to study?
  • I think a lot of the motivation for the mean-field's necessity comes from References 1,2,3, which was among the first works to establish that it suffices to look at your neighbors (or a local neighborhood) rather than the full population. Mean-field then said that it suffices to look at a representative agent, building on this. Can the authors provide a discussion of this?
  • I think the empirical sampling ideas in MNM_N (line 1081 of the manuscript) has been studied a lot in the mean-field sampling works. For this, references 4,5,6 should really be cited, and perhaps some of the analyses can be abstracted by referencing results from 4-6 directly?

References:

  1. Scalable reinforcement learning for multiagent networked systems. Qu et. al. (2022)

  2. Multi-agent reinforcement learning in stochastic networked systems. Qu et. al. (2020)

  3. Scalable Multi-Agent Reinforcement Learning for Networked Systems with Average Reward. Lin et. al. (2021)

  4. Mean-field sampling for cooperative multi-agent reinforcement learning. Anand et. al. (2025)

  5. Graphon Mean-Field Subsampling for Cooperative Heterogeneous Multi-Agent Reinforcement Learning. Anand et. al. (2026)

  6. Learning Approximate Nash Equilibria in Cooperative Multi-Agent Reinforcement Learning via Mean-Field Subsampling. Anand et. al. (2026)

1 comment
AuthorsOpenReview ↗

response to Reviewer eDaC

We thank the reviewer for the careful review. Changes are in the revised PDF in blue

1. Notation and presentation.

Section 2.2 now explicitly introduces the N→∞N\to\infty limit of Section 2.1. We now use γ\gamma for the discount factor and have renamed the exponent in Corollary 11 to αMP\alpha_{\mathrm{MP}}. Appendix N summarises the notation.

2. Novelty.

We agree that mean-field control, finite-net Q-learning, and finite-population approximation have each been studied previously. However, our contribution is not simply their combination: applying existing Q-learning results to the non-exchangeable mean-field model does not directly yield a convergent, near-optimal policy that can be deployed in finite heterogeneous populations. The difficulty is making the policy approximation, learning, and finite-population transfer guarantees compatible.

In the continuously labelled setting, relaxed actions are kernels on I×XI\times\mathcal X, even when X\mathcal X and AA are finite. Unlike the finite-dimensional lifted action spaces of Carmona et al. (2023) and Gu et al.(2021), the unrestricted Borel-kernel class is not totally bounded under d∞d_\infty, so it cannot be directly approximated by finite nets. Restricting to LL-Lipschitz kernels restores compactness, but introduces a potential loss of optimality. Theorem 7 resolves this obstruction by proving that the restriction loss gL→0g_L\to0 uniformly over all population laws. This is essential because the learned policy must be evaluated at population laws that are not known in advance.

Theorem 9 and Corollary 10 then quantify how state and kernel discretization, together with finite-table learning error, affect the greedy policy's Bellman residual. Crucially, this policy is can be discontinuous in the population law because of nearest neighbour projection. Its deployment therefore requires a comparison argument that does not assume such continuity. Proposition 23 establishes this directly, using regularity in the label–state variables and controlling label approximation, empirical sampling, and model mismatch.

Thus, whereas Mekkaoui and Pham (2026, Thm 3.6) establish finite-population transfer given an ε\varepsilon-optimal label–state-Lipschitz feedback, our results establish how such a feedback can be approximated learned and transferred with a quantified end-to-end loss. The uniform Lipschitz consistency, finite-net approximation analysis, and deployment bound are linked: each addresses an obstruction that prevents a direct application of the preceding literature. Section 1 compares these works

3. Why Lipschitz kernels and uniform convergence?

We use Lipschitz kernels because they make the problem manageable for learning and allow us to transfer the learned policy to finite populations. For a fixed LL, the class of Lipschitz kernels is compact (Lemma 16), so finite nets exist, unlike for arbitrary measurable kernels, which can change abruptly. Lipschitz regularity also ensures that nearby population laws produce nearby state–action distributions (Lemma 14), helping us control disretisation errors. It is also important for finite-population transfer: two kernels can behave identically in the mean-field model but differently at the exact labels of finite agents. For example, a kernel that chooses action 1 at rational labels and 0 elsewhere is equivalent to always choosing 0 under a uniform label distribution, but not when finite-agent labels are i/Ni/N. Lipschtz regularity prevents this issue, and Proposition 23 quantifies the resulting label approximation error. Finally, our goal is not to approximate every measurable kernel, which is generally impossible uniformly, but to achieve nearly optimal performance. Theorem 7 shows that the loss from restricting to Lipschitz kernels goes to zero uniformly over population laws.

  1. Networked MARL.

We now discuss Qu et al. (2020, 2022) and Lin et al. (2021) in Section 1 (related literature). These works exploit the decay of influence with graph distance, allowing agents to use local neighbourhoods. Our mean-field approach addresses the complementary dense interaction setting, where the population law summarize global interactions while continuous labels preserve heterogeneity.

  1. Mean-field subsampling.

We now cite Anand et al. (2025, 2026) and Anand and Karmarkar (2026) in Section 1 and Appendix G.1. Following the suggestion, we relate our Wasserstein sampling estimate to finite-space total-variation bounds, including the approach of Anand et al. (2025, Thm. D.2). Our setting differs because the samples are independent but not identically distributed, with action laws depending on agent labels and states. A short variance argument gives M‾N(Y)≤2r+diam⁡(Y)2mN,\overline M_N(Y)\le 2r+\frac{\operatorname{diam}(Y)}{2}\sqrt{\frac{m}{N}}, where YY is covered by mm sets of diameter at most rr. This is an empirical approximation estimate used in Proposition 23, not a new subsampling algorithm.

Reviewer_LC9JOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors lift heterogeneous cooperative MARL with common noise to a mean-field MDP on label-state laws, discretize it and run tabular Q-learning with asymptotic guarantees. The main takeaway appears to be that discretization (rather than learning) is the bottleneck.

Critical strengths
  • Clean separation of the error sources and fairly honest experiments (predeclared tests in Table 1; PPO reported 0.08 ahead in C3)
  • I trust the negative takeaway (and think a revision rather than a rewrite is needed)
Critical weaknesses
  • My main concern is that the distinguishing features are not shown to matter
  • The law-conditioned tables roughly match (Fig. 3) or modestly exceed (Fig. 8) the best static ramp (a 625-member grid of (u, x) controls blind to the population law)
  • Lines 1275-1277 concede that no material advantage from label or law information is established (as far as I can tell the setting's premise is untested)
Questions for authors

Can the authors state Corollary 11's hypotheses and define M_N, epsilon^f_N, epsilon^F_N, gamma and C here and confirm the deterministic labels satisfy them under the lift (19)?

1 comment
AuthorsOpenReview ↗

response to Reviewer LC9J

Thank you for the review. We answer both points below. The code and experimental scripts are available in our anonymous repository.

1. Do label and population-law information matter?

We added DFP2, a linear–quadratic benchmark derived from De Crescenzo et al. (2026), in which a parameter h∈[0,1]h\in[0,1] controls the strength of label dependence in the costs, dynamics, interactions, and initial states (Appendix J).

With matched net sizes and training budgets, label-aware nearest-neighbour Q-learning, using a label-aware representation and label-dependent actions, reduces cost relative to label-blind NN by 20.9% at h=0.75h=0.75 and 16.0% at h=1h=1. At low heterogeneity, label information provides little benefit and can hurt because the larger representation is harder to approximate with a fixed table size. This allows us to identify settings where label information is useful rather than assuming it always is.

We also added a conditioning ablation that keeps the stochastic model fixed but replaces the realised conditional means observed by the planner with their unconditional expectations. Removing conditioning increases NN costs by approximately 6–65% in 13 of 15 tested settings (Appendix J, Table 4). The increase also occurs at h=0h=0, showing that the value of conditioning on the population law is distinct from the value of labels.

Finally, label-aware PPO (LA-PPO) outperforms label-blind PPO on Laser, Traffic, and DFP2 for h≥0.25h\ge 0.25. Neural policies are a natural extension of our finite tables. Our regularity results (Theorems 6, 7 and 15) make the restricted Q-function Hölder on a compact space, hence approximable by neural networks. They perform better on several benchmarks, and their convergence guarantees are future work.

2. Corollary 11: assumptions and finite-population lift.

We have clarified the hypotheses and quantities entering the bound. We assume the regularity of the limiting reward and dynamics (Assumption 2), the corresponding uniform regularity of the finite-agent model, convergence of (fN,FN)(f_N,F_N) to (f,F)(f,F), and the initial-information measurability condition.

Here, MNM_N is the worst-case expected empirical Wasserstein error for NN i.i.d. samples on I×X×AI\times\mathcal X\times A. The terms εNf\varepsilon_N^f and εNF\varepsilon_N^F bound the discrepancies between finite-agent and limiting rewards and dynamics, respectively. The exponent αMP∈(0,1]\alpha_{\mathrm{MP}}\in(0,1] is the Hölder exponent from Mekkaoui and Pham (2026, Thm. 3.3) and CC is the constant from their finite-population comparison theorem, depending on the model parameters but not on NN or the initial state.

For deterministic labels ui=i/Nu_i=i/N and uniform λ\lambda, the block coupling is ΓN(du,{i})=1[(i−1)/N,i/N)(u)du.\Gamma_N(\mathrm du,\lbrace i\rbrace)=\mathbf{1}_{[(i-1)/N,i/N)}(u)\mathrm du. The resulting lift μNλ[uN,x]\mu_N^\lambda[\mathbf u^N,x] has exactly the required label marginal λ\lambda, with label approximation error ℓN=1/(2N)\ell_N=1/(2N). Thus the deterministic labels are compatible with the lift. The regularity, model convergence, and initial-information hypotheses remain separate assumptions.

For interval-valued actions, the quantile representation of the learned LL-Lipschitz kernel satisfies the randomised-feedback regularity condition. Corollary 10 supplies its near-optimality residual, and Corollary 11 yields VN(x)−VNπR,N(x)≤C(1+L)[ηR+gL+MNαMP+εNf+(εNF)αMP].V_N(x)-V_N^{\pi_{\mathcal R},N}(x)\le C(1+L)\left[\eta_{\mathcal R}+g_L+M_N^{\alpha_{\mathrm{MP}}}+\varepsilon_N^f+(\varepsilon_N^F)^{\alpha_{\mathrm{MP}}}\right]. Appendix G additionally provides a self-contained deployment estimate (Proposition 23), including the label-lift, sampling, and model errors, without assuming continuity of the learned greedy policy in the population law.

Reviewer_HqWPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies cooperative control of heterogeneous agent populations with common noise using a mean-field state that retains the association between persistent labels and individual states. It establishes regularity of the lifted control problem, uniform consistency of restricting relaxed actions to Lipschitz kernels, and approximation bounds for finite state and kernel nets. A classical asynchronous Q-learning scheme converges to the projected Bellman fixed point under a generative mean-field oracle. The paper then connects a greedy-policy residual to a conditional finite-population performance bound and examines learning, projection, and transfer errors in synthetic experiments.

Critical strengths
  1. The policy-restriction analysis goes beyond pointwise approximation. Appendix D combines the density of Lipschitz kernels with a compactness and Dini argument to obtain uniform consistency over the fixed-marginal population-law space. This is a useful contribution because it connects a compact kernel class suitable for finite approximation to the unrestricted relaxed control problem.

  2. The analysis distinguishes convergence to the projected fixed point from approximation of the restricted and unrestricted optimal control problems. Theorem 9 and Corollary 10 make this distinction explicit, and the numerical diagnostics similarly separate incomplete learning from representation error. The manuscript also acknowledges unfavorable comparisons and the limitations of its empirical nets.

Critical weaknesses
  1. The connection to the original control problem and the finite-population guarantee is not specified completely. Section 2.2 requires a uniform random variable independent of the initial state, but this does not ensure randomization conditional on both the label and the initial state. Corollary 11 also invokes population-approximation conditions without fully defining the error quantities, the label-coupling requirements, or the dependence of the bound’s constant. These omissions prevent verification of when the claimed deployment guarantee applies and when its right-hand side becomes small. Appendix G should identify the exact external theorem and provide the specialized verification argument, particularly because the greedy kernel can change discontinuously across population-law projection cells.

  2. The experiments only partly instantiate the method covered by the main guarantee. The main benchmarks use label quadrature, structured ramp-kernel subclasses, reachable-set nets with measured radii, and extensive HAT comparisons, whereas the theorem concerns nearest-neighbor learning on global internal nets of the full Lipschitz kernel class. The additional restrictions are acknowledged but not quantified, and the cybersecurity benchmark’s mathematical model is not specified in the supplied manuscript. Consequently, the results provide useful representation diagnostics but limited evidence for the complete continuous-label control guarantee.

Questions for authors
  1. What assumption guarantees sufficient randomization conditional on the label–state pair? Can the formulation be strengthened, or can the relaxed-kernel realization and feedback-existence claims be justified under the current assumption?

  2. Can Corollary 11 state the complete population-approximation assumptions, define all error quantities, and identify the precise comparison theorem being applied? Please explain how the proof uses the same kernel in the one-step Bellman comparison without assuming the selected kernel is continuous across population-law projection boundaries.

  3. Can the policy guarantee be stated for the finite-iteration table actually produced by Algorithm 1?

  4. How does the contribution differ from existing finite approximations of measure-valued control problems and quantized Q-learning under common noise? Please cite and compare Bayraktar et al. (2025), identifying the additional result provided by the fixed label marginal and Lipschitz-kernel consistency analysis. • Bayraktar, E., Bäuerle, N., & Kara, A. D. (2025). Finite approximations for mean-field type multi-agent control and their near optimality. Applied Mathematics & Optimization, 92(1), Article 7. https://doi.org/10.1007/s00245-025-10279-x

  5. Can the experimental specification identify the model, kernel class, net coverage, oracle, and applicable guarantee for each comparison? In particular, please provide the cybersecurity dynamics and parameters, the additional label-quadrature and kernel-subclass errors where relevant, and the actual training configurations and computational budgets. A matched comparison using the proposed NN method would also help establish whether label information or common-noise conditioning changes control performance.

1 comment
AuthorsOpenReview ↗

response to Reviewer HqWP

Thank you for the constructive review. New proofs are in the revised PDF, in blue. The code is in our anonymous repository.

1. Randomisation given the label and state.

We agree that independence from the initial state ξ\xi alone is not enough. Section 2.2 now requires a uniform random variable ZZ independent of the pair (U,ξ)(U,\xi) and the driving noises. This allows us to realize any relaxed kernel using a Borel randomisation map (Kallenberg 2002). Proposition 3 establishes the existence of an optimal randomised feedback for the lifted MDP, and the strengthened assumption ensures that this feedback can be implemented by an admissible contrl. Appendix G (proof of Corollary 11) further verifies the measurability and regularity of the feedback control used in the finite-agent system.

2. Corollary 11.

We have clarified the assumptions and error terms in Corollary 11. These include uniform regularity of the finite-agent model, convergence of fN,FNf_N,F_N to f,Ff,F, and initial-information measurability. We now define MNM_N as the worst-case expected empirical Wasserstein error, ϵNf,ϵNF\epsilon_N^f,\epsilon_N^F as the model mismatches, αMP\alpha_{\mathrm{MP}} as the Hölder exponent, and CC as a constant independent of NN and the initial state. For labels ui=i/Nu_i=i/N, the block coupling gives the required lift (19),with label error ℓN=1/(2N)\ell_N=1/(2N). The other assumptions must still hold separately.

Appendix G verifies the application of Mekkaoui and Pham (2026, Theorem 3.6). Our learned feedback is Borel (nearly optimal by Corollary 10,) and LL-Lipschitz in the label–state variables through the quantile representation (20).

3. Why continuity in the population law is not needed.

The greedy policy can jump between projection cells. Lemma 22 and Proposition 23 (Appendix G.1) now give a direct proof that does not require continuity in the population law. at each step, the finite-agent system selects a kernel using its current lifted law, and the comparison uses exactly the same kernel at exactly the same law. Proposition 23 gives V⋆(μNλ[uN,x])−VNπR,N(x)≤ϵB+ΔNf+γΩc0,K0(ΔNF)1−γ,V^\star(\mu_N^\lambda[\mathbf u^N,x])-V_N^{\pi_{\mathcal R},N}(x)\le\frac{\epsilon_B+\Delta_N^f+\gamma\Omega_{c_0,K_0}(\Delta_N^F)}{1-\gamma}, where the errors account for label approximation, sampling and model mismatch. When fN=ff_N=f and FN=FF_N=F, the finite-population errors vanish as NN grows for fixed L.

4. Guarantee for the actual learned table.

Corollary 10 now applies to any table Q^\widehat Q, including the table QnQ_n produced after nn updates. The policy error gains an additional term 2δ(Q^)2\delta(\widehat Q), where δ(Q^)=∥Q^−QR∥∞≤∥Q^−BRQ^∥∞1−γ.\delta(\widehat Q)=\Vert\widehat Q-Q_{\mathcal R}\Vert_\infty\le\frac{\Vert\widehat Q-\mathcal B_{\mathcal R}\widehat Q\Vert_\infty}{1-\gamma}. we obtain a finite-iteration guarantee in terms of the Bellman residual,and δ(Qn)→0\delta(Q_n)\to0 almost surely by Theorem 9.

5. Comparison with Bayraktar et al. (2025).

We now cite Bayraktar et al (2025) and compare with it in Section 1. They study exchangeable mean-field control with common noise on compact spaces and establish near-optimality through finite approximations of states and actions.

Our main difference is not simply the presence of labels. In our setting, actions are kernels on I×XI\times\mathcal X, and the unrestricted kernel class is not totally bounded, even when individual states and actions are finite. Thus, existing finite-net methods cannot be applied directly. Restricting to Lipschitz kernels restores compactness, while Theorem 7 proves that the resulting optimality loss vanishes uniformly over population laws. We then establish finite-net Q-learning convergence and explicit approximation bounds (Theorem 9), including guarantees for the learned greedy policy (Corollary 10). Finally, Proposition 23 gives a finite-population deployment bound without requiring continuity of the learned policy in the population law.

6. Experiments.

Appendix M now gives the model, kernel class, net coverage, oracle and applicable guarantee for each experiment(Table 9). It also provides the cybersecurity dynamics and parameters (Table 8), training budgets (Table 10), and discusses label-quadrature and kernel-subclass errors. We clarify that the finite-table convergence result applies to NN on the implemented finite surrogate. We also added matched NN comparisons. In the original congestion model, label and law information show no material advantage. In the new DFP2 benchmark (see appendix J), label-aware NN reduces cost by 20.9% and 16.0% at high heterogenety. Removing conditioning on the realized population law increases cost by 6–65% in 13 of 15 settings. Finally, NN is 11.6% above the Riccati reference (Appendix K) and 10.6% above the value-iteration reference (Appendix L) while the observed finite-population gaps decrease at roughly N−1/2N^{-1/2}

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised Manuscript and Code Availability

Dear Reviewers,

The code and additional experiments are available in our anonymous repository. All modifications to the revised paper are highlighted in blue.

We remain happy to address any further questions.

Thank you for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.