- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes MF-Diffuser for diffusion-based trajectory planning in large-scale offline multi-agent reinforcement learning. It treats each agent’s complete trajectory as a particle, models population interactions through mean-field information, and progressively increases the number of trajectories during denoising to reduce generation cost. It introduces a value-weighted training objective and provides theoretical analyses of mean-field approximation, planning error, and equilibrium properties. Experiments are conducted on Ising, Battle, and Gaussian Squeeze, with up to 10,000 agents.
Critical strengths
- The paper provides extensive theoretical analysis and proofs, attempting to explain the method from several perspectives, including mean-field approximation, performance error, and game-theoretic equilibria.
Critical weaknesses
- Experiments are limited to Ising, Battle, and Gaussian Squeeze. It remains unclear whether the method is effective on other common multi-agent tasks, heterogeneous agents, or scenarios requiring specialized coordination. Additional experiments ( for example: Multi-Agent MuJoCo or SMAC/SMACv2 etc) would help clarify its applicability.
- No code is provided.
- The overall novelty of the paper is relatively limited.
Questions for authors
-
In Appendix H.2.3 [2950–2954], the fitted slopes across the three environments are reported to lie in [−0.51, −0.47]. However, Figure 8 on page 56 shows −0.27, −0.21, and −0.25, respectively. Could the authors clarify which values are correct and whether the current results support the stated convergence rate?
-
Could the authors provide additional experiments on standard tasks such as SMAC, or scenarios with heterogeneous roles? Also, is the comparison with MF-CDMs-RL conducted under the same training budget? A cost comparison would help assess the benefit of the added value-weighted training objective.
-
Section 3.2 [190–192] states: “In practice we replace with during training…”
The paper claims that this replacement introduces only a constant factor, which can be absorbed into the score error and is independent of the number of agents and the planning horizon. However, I do not understand why a bounded Hessian of the reference log-density is sufficient to control the Sobolev error, which includes derivative terms, using the Euclidean error alone. Could the authors clarify the required assumptions and explain why the theoretical guarantees still apply to the practical training objective?
-
I am curious about the wording “SCALING OFFLINE MARL TO THOUSANDS OF AGENTS.” This is mainly a question about the title. Existing offline MARL methods, such as MADiff [1], DoF [2], and Oryx [3], have already considered scalability. What specifically does “thousands” emphasize here: the ability to handle this number of agents, the planning quality, or the computational efficiency at this scale?
-
Appendix G states that the mean-field interaction kernel is implemented using attention. Could the authors clarify its specific architectural differences from standard cross-agent attention or local graph attention?
If the authors address my concerns, I will increase my score.
References [1] MADiff: Offline Multi-agent Learning with Diffusion Models. NeurIPS 2024. [2] DoF: A Diffusion Factorization Framework for Offline Multi-Agent Reinforcement Learning. ICLR 2025. [3] Oryx: a Scalable Sequence Model for Many-Agent Coordination in Offline MARL. NeurIPS 2025.
6 comments
Response to Reviewer f1K8 (2/2)
Q3: Why a bounded Hessian of the reference log-density controls the Sobolev error. The reviewer is correct that it does not. The error is for , and a bound on the Hessian of controls neither term of this error. The constant in Remark B.3 () was derived under an implicit smoothness assumption on that we did not state, and stating it is now part of the remark.
The corrected statement adds it explicitly. If the score network is Lipschitz in its trajectory argument with constant — which is what spectral normalization, enabled by default in our implementation, enforces — then for the Gaussian reference measure at noise level , Young's inequality gives , so the Sobolev-to-Euclidean conversion costs a factor that is independent of and . That factor is absorbed into and changes no rate. The measured values across the sweep (0.18-0.42, App. H.3.1) already include it. We have restated Remark B.3 with this assumption and dropped the claim that a bounded reference Hessian suffices on its own.
Q4: What "thousands" emphasizes in the title. Scale of the setting, not a claim of efficiency. Two specific claims: (i) at the method still produces non-degenerate plans where joint-space models collapse (normalized return 4.2-12.3 for Joint Diffuser and 6.8-18.4 for MADiff on Ising, against 85.2-95.3 for ours; Table 8), and (ii) quality improves with rather than degrading (Cor. 4.1; PoC KL 1/N fit, ). Inference remains per denoising step and we say so in Sec. 3.4 and in the Limitations; the efficiency claim is limited to the 2.58x constant factor of the hierarchical schedule at fixed (App. H.4.6). If the title reads as an efficiency claim, the abstract's first sentence now makes the distinction explicit.
Q5: Architectural difference from standard cross-agent attention. The kernel is conditioned on the empirical mean field: the interaction term is $B_\theta
(\tau^i) = \frac{1}{N-1}\sum_{j\neq i} K_\theta(\tau^i,\tau^j) B_\theta(\tau^i,\tau^j)$, where is a dynamic k-nearest-neighbour graph convolution over the current noised states and is an attention kernel whose keys carry as additional context (App. G). Standard cross-agent attention keys contain neighbour features only; MADiff's inter-agent attention is of that form. The practical consequence is that the interaction term changes when the population distribution changes even if the local neighbourhood does not, which is what the Gaussian Squeeze visualisation shows (Fig. 5): the population-level mode structure is tracked, whereas factorized models lose a mode. The revised Sec. 3.1 states this contrast next to the architecture description.
W1: Benchmarks limited to three mean-field environments. Addressed by the two-type Ising experiment above, and by the scope statement now in Sec. 1. We agree that the current set does not establish generality; App. I says so and names the testbeds we have not run.
W2: No code. We will release the full training and inference code, the dataset-construction scripts, and the per-seed raw CSVs with the camera-ready.
W3: Limited novelty. We would locate the contribution in the guarantee rather than the architecture. To our knowledge this is the first trajectory-level planner with (i) a propagation-of-chaos bound that is dimension-free in , (ii) an end-to-end suboptimality decomposition in which the mean-field term decreases with while the offline-shift term does not, and (iii) an exploitability bound applying to team-competitive settings without monotonicity. The individual ingredients — mean-field diffusion, value tilting, coarse-to-fine sampling — exist separately; the composition and the guarantees are what we claim. The revised introduction states this in one paragraph and cites MADiff, DoF, and Oryx explicitly as the scalability precedents the reviewer names.
Thanks for the response and the additional experiments. I have two follow-up questions.
-
How does a uniform Lipschitz bound on the score network allow the Euclidean score error to control the derivative error?
-
The two-type Ising experiment varies coupling strength. Does the mean-field representation preserve the information needed for coordination when agents have complementary roles? Can you test this on a task that requires such coordination?
Response to Reviewer f1K8 (1/2)
We thank Reviewer f1K8 for the concrete list; we can act on all five points, and one of them exposed an error in the paper, which we report first.
Q1: The slope values in App. H.2.3 and Fig. 8 disagree. The reviewer is right that they cannot both be correct, and the figure is the correct one. The range quoted in H.2.3, [-0.51, -0.47], is the exploitability slope range reported in H.2.2 for a different quantity (normalized exploitability vs. N, per-environment fits -0.49/-0.51/-0.46); it was carried into the H.2.3 text when that paragraph was revised, and the values annotated in the Fig. 8 panels (-0.27, -0.21, -0.25) are what the Wasserstein-to-MFE divergence actually fits. We have corrected H.2.3 to report the figure's own numbers and removed the sentence claiming agreement with -1/2.
What this does and does not affect. The -1/2 rate is a statement about exploitability and about the PoC bound, not about W2 to the MFE flow, and the corrected text now says so explicitly. The exploitability fits (-0.49±0.04, -0.51±0.05, -0.46±0.06 for Ising, GS, Battle; App. H.2.2) are within bootstrap CIs of -0.5, and the PoC tightness measurement is the clean rate check: W22 falls from 3.00e-2 at N=102 to 3.0e-4 at N=104, a fitted slope of -0.98±0.02, i.e. W2∝N−0.49 with pooled R2=0.99 (Table 14). The W2 divergence to the MFE is a different and noisier quantity: it has a larger constant and an N-independent floor set by score error, so its observed slope is shallower once the floor dominates. Reading it as a rate measurement was the mistake; the divergence is qualitative evidence that the gap shrinks, and the rate evidence is the exploitability exponent and the PoC tightness fit.
Q2: Experiments on standard tasks (SMAC / heterogeneous roles); budget parity with MF-CDMs-RL. We have not run SMAC, MA-MuJoCo, or the Heterogeneous Multi-Agent Challenge within the rebuttal window, and we do not claim to have. The revised Limitations names them as the right testbeds and states why adapting our per-team protocol to their unit-type-conditional observation structure is nontrivial rather than a matter of compute. What we did run is the weakest heterogeneity test that stays inside our data pipeline, a two-type Ising model: two agent classes with coupling strengths λ1, λ2 (λ2/λ1=0.5), 50/50 mix, everything else as App. F.1, Medium and Mixed data, N=1000, 4 seeds. The class label is supplied as an input feature to every method, so the comparison is not a handicap. See C1 in the common response for the full table; in brief, MF-Diffuser reaches 84.2 ±1.6 (Medium) and 75.4 ±2.3 (Mixed) against 80.7 ±2.1 and 71.2 ±2.7 for MF-CDMs-RL. The heterogeneity costs us 3.7 points but costs the strongest baseline 3.9, so the ranking and the margin are preserved; removing the class label instead of supplying it costs a further 6.3 points, which locates the cost in the heterogeneous projection rather than in the generator.
On budget parity with MF-CDMs-RL: the two share the backbone, optimizer, batch size, 200 diffusion steps, the level schedule, and the same 4xA100 allocation; measured cost is 12.5 GPU-h (ours) against 14.2 (MF-CDMs-RL) at N=1000. We did not fix the gradient-step count in that run and we now state the budget as measured GPU-hours rather than imply step parity. We have since run the step-matched comparison: 200K gradient steps, Battle, Medium, N=1000, 3 seeds, giving 75.2 ±2.7 for MF-Diffuser against 68.1 ±2.9 for MF-CDMs-RL, a gap of 7.1 points against a converged gap of 7.4 (MF-CDMs-RL reaches 68.4 ±2.9 with 1.7x more steps). The gap therefore does not close with training. The cleaner control is still the within-architecture ablation: same backbone and budget, removing value weighting (λ=0) gives 66.5 on Battle against 75.8 with it (Table 2), which isolates the objective without any implementation confound.