- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work studies cooperative heterogeneous MARL with agents subject to common noise. It proposes a finite-net Q-learning scheme and shows that restricting relaxed policies to Lipschitz kernels is uniformly consistent. Finally, it presents numerical simulations to validate the theory in settings of congestion and cybersecurity.
Critical strengths
None
Critical weaknesses
-
It is quite hard to read the paper due to the notation. For instance, almost all works in RL use for the discount parameter in the objective, but this work uses . Sections 2.1 and 2.2 also read a bit strangely. Ideally, they should be combined and written in a more cohesive way, as Section 2.2 is the infinite limit of 2.1.
-
A bigger concern I have is with the novelty of the work: for instance, as the authors point out, Mekkaoui & Pham (2026) do almost the same thing in the control-theoretic framework (so its only distinction is that it doesn't have the finite-net Q-learning scheme), and the works of Carmona, Laurrier, Tan (2023) presents a Q-learning scheme (but for finite spaces), and the works of Gu, Guo, Wei, Xu (2021) explicitly studies epsilon-nets of lifted state/action spaces. So my concern here is that this work seems to be spanned by the works above: in particular, I don't think any of the results here are particularly surprising, but maybe I'm missing some critical detail here and the authors could comment further on this (I'm certainly open to raising my score).
Questions for authors
- As mentioned above, mean-field approximation with finite-net Q-learning and finite-agent performance bound has been done a lot in the literature. I don't think prior work has shown that restricting relaxed policies to Lipschitz kernels is uniformly consistent, but I'm not sure how interesting the uniform convergence of restricted optimal values is, compared to (for instance) a uniform convergence of, say, every measurable kernel. Can the authors comment on this? Specifically, why is this the right restriction to study?
- I think a lot of the motivation for the mean-field's necessity comes from References 1,2,3, which was among the first works to establish that it suffices to look at your neighbors (or a local neighborhood) rather than the full population. Mean-field then said that it suffices to look at a representative agent, building on this. Can the authors provide a discussion of this?
- I think the empirical sampling ideas in (line 1081 of the manuscript) has been studied a lot in the mean-field sampling works. For this, references 4,5,6 should really be cited, and perhaps some of the analyses can be abstracted by referencing results from 4-6 directly?
References:
-
Scalable reinforcement learning for multiagent networked systems. Qu et. al. (2022)
-
Multi-agent reinforcement learning in stochastic networked systems. Qu et. al. (2020)
-
Scalable Multi-Agent Reinforcement Learning for Networked Systems with Average Reward. Lin et. al. (2021)
-
Mean-field sampling for cooperative multi-agent reinforcement learning. Anand et. al. (2025)
-
Graphon Mean-Field Subsampling for Cooperative Heterogeneous Multi-Agent Reinforcement Learning. Anand et. al. (2026)
-
Learning Approximate Nash Equilibria in Cooperative Multi-Agent Reinforcement Learning via Mean-Field Subsampling. Anand et. al. (2026)
response to Reviewer eDaC
We thank the reviewer for the careful review. Changes are in the revised PDF in blue
1. Notation and presentation.
Section 2.2 now explicitly introduces the N→∞ limit of Section 2.1. We now use γ for the discount factor and have renamed the exponent in Corollary 11 to αMP. Appendix N summarises the notation.
2. Novelty.
We agree that mean-field control, finite-net Q-learning, and finite-population approximation have each been studied previously. However, our contribution is not simply their combination: applying existing Q-learning results to the non-exchangeable mean-field model does not directly yield a convergent, near-optimal policy that can be deployed in finite heterogeneous populations. The difficulty is making the policy approximation, learning, and finite-population transfer guarantees compatible.
In the continuously labelled setting, relaxed actions are kernels on I×X, even when X and A are finite. Unlike the finite-dimensional lifted action spaces of Carmona et al. (2023) and Gu et al.(2021), the unrestricted Borel-kernel class is not totally bounded under d∞, so it cannot be directly approximated by finite nets. Restricting to L-Lipschitz kernels restores compactness, but introduces a potential loss of optimality. Theorem 7 resolves this obstruction by proving that the restriction loss gL→0 uniformly over all population laws. This is essential because the learned policy must be evaluated at population laws that are not known in advance.
Theorem 9 and Corollary 10 then quantify how state and kernel discretization, together with finite-table learning error, affect the greedy policy's Bellman residual. Crucially, this policy is can be discontinuous in the population law because of nearest neighbour projection. Its deployment therefore requires a comparison argument that does not assume such continuity. Proposition 23 establishes this directly, using regularity in the label–state variables and controlling label approximation, empirical sampling, and model mismatch.
Thus, whereas Mekkaoui and Pham (2026, Thm 3.6) establish finite-population transfer given an ε-optimal label–state-Lipschitz feedback, our results establish how such a feedback can be approximated learned and transferred with a quantified end-to-end loss. The uniform Lipschitz consistency, finite-net approximation analysis, and deployment bound are linked: each addresses an obstruction that prevents a direct application of the preceding literature. Section 1 compares these works
3. Why Lipschitz kernels and uniform convergence?
We use Lipschitz kernels because they make the problem manageable for learning and allow us to transfer the learned policy to finite populations. For a fixed L, the class of Lipschitz kernels is compact (Lemma 16), so finite nets exist, unlike for arbitrary measurable kernels, which can change abruptly. Lipschitz regularity also ensures that nearby population laws produce nearby state–action distributions (Lemma 14), helping us control disretisation errors. It is also important for finite-population transfer: two kernels can behave identically in the mean-field model but differently at the exact labels of finite agents. For example, a kernel that chooses action 1 at rational labels and 0 elsewhere is equivalent to always choosing 0 under a uniform label distribution, but not when finite-agent labels are i/N. Lipschtz regularity prevents this issue, and Proposition 23 quantifies the resulting label approximation error. Finally, our goal is not to approximate every measurable kernel, which is generally impossible uniformly, but to achieve nearly optimal performance. Theorem 7 shows that the loss from restricting to Lipschitz kernels goes to zero uniformly over population laws.
We now discuss Qu et al. (2020, 2022) and Lin et al. (2021) in Section 1 (related literature). These works exploit the decay of influence with graph distance, allowing agents to use local neighbourhoods. Our mean-field approach addresses the complementary dense interaction setting, where the population law summarize global interactions while continuous labels preserve heterogeneity.
We now cite Anand et al. (2025, 2026) and Anand and Karmarkar (2026) in Section 1 and Appendix G.1. Following the suggestion, we relate our Wasserstein sampling estimate to finite-space total-variation bounds, including the approach of Anand et al. (2025, Thm. D.2). Our setting differs because the samples are independent but not identically distributed, with action laws depending on agent labels and states. A short variance argument gives MN(Y)≤2r+2diam(Y)Nm, where Y is covered by m sets of diameter at most r. This is an empirical approximation estimate used in Proposition 23, not a new subsampling algorithm.