acceptodds
Under review as a conference paper at ICLR 2027

Coordinated Denoising Stabilizes Cooperation in Diffusion Multi-Agent Reinforcement Learning

Abstract

Diffusion policies are promising for cooperative multi-agent reinforcement learning (MARL) because they can represent multimodal joint action distributions. However, existing methods supervise only the final executed action, leaving intermediate denoising steps unguided. This causes poor coordination when agents execute their denoising processes independently. We propose DiMAC, a novel diffusion-based offline MARL framework that injects centralized coordination supervision directly into the denoising process. The core component is a Denoising Process Critic (DPC) that evaluates intermediate denoising states according to their expected contribution to the eventual coordinated joint action. DiMAC further employs coalition value decomposition to translate centralized process values into decentralized agent-wise and pairwise guidance signals, along with reliability weighting to emphasize more stable denoising steps. This design preserves the Centralized Training with Decentralized Execution (CTDE) paradigm while optimizing coordination throughout the entire action generation process. Experiments on MPE and SMAC benchmarks show that DiMAC substantially outperforms prior offline MARL and diffusion-based methods.

Primary area:
reinforcement learning
Keywords:
MARL

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ojNbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DiMAC, a diffusion-based MARL framework that improves multi-agent coordination through process-level supervision. The motivation is interesting and well justified, and the proposed method offers a meaningful perspective beyond simply improving the diffusion backbone. The overall framework is reasonably designed, and the experimental results and ablation studies generally support the effectiveness of the main components. However, I still have some concerns about the reliability of the method, scalability, experimental consistency, and overall presentation. Overall, I find the work promising, but I think some aspects could be better justified and improved.

Critical strengths
  1. The paper tackles an interesting problem. Most existing diffusion-based MARL methods mainly focus on the final joint action, while DiMAC explicitly supervises the intermediate denoising process. I think this is a meaningful direction beyond simply improving the diffusion backbone.
  2. The overall method is well designed, and the ablation results generally support the main idea. In particular, removing DPC or using final-Q guidance leads to clear performance drops, suggesting that process-level supervision is indeed important.
Critical weaknesses
  1. I'm concerned about the OOD issue in coalition counterfactuals. The mixed denoising prefixes may not follow the actual policy distribution, so it's unclear how reliable the DPC predictions are on these inputs. The lower reliability weights reported for mixed prefixes also suggest this could be an issue.

  2. Pairwise coalition decomposition has roughly quadratic complexity in the number of agents, and DPC requires additional evaluations. It's unclear how well the method would scale to larger multi-agent settings.

  3. While the results on MPE are strong, DoF still performs better on several SMAC settings, including all 8m splits. This makes the general claim about improved coordination somewhat less convincing.

  4. Figure 2 uses different fonts and font sizes, which makes it look a bit inconsistent. Also, the method introduces quite a lot of notation throughout the derivation, making it harder to follow than necessary. A clearer and more consistent presentation would improve readability.

Questions for authors
  1. How accurate is DPC on coalition-mixed prefixes compared to normal policy-generated prefixes? It would be helpful to report prediction errors or calibration results for both cases.

  2. How does DiMAC scale with more agents? Could the authors provide runtime, memory usage, and DPC evaluation costs, ideally compared with DoF and MADiff?

  3. Do all baselines use the same checkpoint selection protocol and environment rollout budget? Results using the final checkpoint or offline validation for selection would help clarify this concern.

  4. Why is the current mask-variance-based reliability weighting necessary? Could the authors compare it with simpler timestep-based or fixed weighting schemes? Also, why does B=5 perform noticeably worse than B=3?

Reviewer_afJMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies diffusion-based policies for offline cooperative multi-agent reinforcement learning under centralized training with decentralized execution (CTDE). The authors argue that optimizing only the final joint action provides insufficient coordination supervision for intermediate denoising decisions. The proposed framework, DiMAC, learns a Denoising Process Critic (DPC) to estimate the expected terminal joint-action value conditioned on a denoising prefix. It then constructs process-level policy objectives using singleton and pairwise coalition effects and weights these objectives according to the sensitivity of DPC predictions to masked-prefix perturbations. Experiments show substantial improvements on several MPE settings and competitive, although mixed, results on SMAC. The paper also includes component ablations, gradient analyses, coordination statistics, and correlations between DPC predictions and rollout returns.

Critical strengths
  1. Equation (5) defines the value of a denoising prefix through its expected eventual joint-action value. The recursion in Equation (7) follows naturally from conditional expectation. Extending centralized coordination feedback into the internal generation process is a worthwhile direction.

  2. In Table 1, the reported score increases from 125.6 to 216.3 on Tag–Expert and from 136.4 to 203.8 on Spread–Expert relative to the strongest prior method. These gains warrant attention.

  3. The paper evaluates variants without DPC, with direct final-action Q guidance, with first-order decomposition, without decomposition, and without bootstrap regularization or reliability weighting. The additional return-correlation and target-selection analyses provide useful evidence about the proposed mechanisms.

  4. According to Equation (4) and the architecture description, execution uses local diffusion policies without querying the centralized DPC or requiring additional inter-agent communication.

Critical weaknesses

The proposed process supervision is appealing, and the ablations and gradient analyses support its empirical effectiveness. However, the introduction suggests that endpoint-only supervision leaves intermediate denoising steps without guidance. Since terminal Q-value gradients can already propagate through the denoising chain, the distinction is more accurately between implicit and explicit process supervision. The current evidence supports improved optimization, while the stronger explanation that the gains specifically arise from better coordination during denoising is somewhat less directly established.

The coalition-based objective is understandable at a high level, but the source of the reference prefixes is not sufficiently explicit. Because these references define the comparisons underlying the coalition effects, their construction matters for interpreting and reproducing the method.

The Direct final-Q comparison in Figure 3 is a relevant control for evaluating process supervision. However, the manuscript could state more explicitly which other components remain unchanged in this comparison. Appendix D also reports selecting the best periodically evaluated EMA checkpoint, but it is not fully clear whether the same selection procedure applies to the compared methods. These details are particularly useful for interpreting the smaller and less consistent improvements on SMAC.

Questions for authors

see weakness

Reviewer_njbnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The focus of the paper is DiMAC, a diffusion-based offline multi-agent reinforcement learning (MARL) method. The main idea is to inject centralized coordination supervision directly into the denoising process rather than supervising only the final joint action. It tries to address a critical problem: in CTDE-based diffusion MARL, decentralized denoising chains can drift and lose coordination even if the final action is evaluated centrally.

DiMAC was evaluated in the paper on MPE (Tag, Spread, World) and SMAC (3m, 8m, 5m_vs_6m, 2s3z) against several existing methods such as BC, MA-CQL, MA-TD3+BC, MAICQ, MADT, OMAR, MADiff, and DoF. DiMAC was shown to be consistently competitive and it also shows gains on coordination-sensitive tasks, though it does not dominate every SMAC split ablations indicate process-level DPC supervision is the main source of gains. Ablation stidies provide evidence that process-level DPC supervision can be the main source of gains

Critical strengths
  1. The paper identifies a main gap in diffusion-based MARL: coordination is usually supervised only at the final joint action, while the decentralized denoising process itself is left unsupervised and can drift. The proposed DiMAC shifts supervision from the endpoint to the denoising process.

  2. The Denoising Process Critic (DPC) scores intermediate denoising prefixes by their expected contribution to the eventual coordinated joint action. This turns terminal team-level feedback into step-wise supervision, which is the paper’s central technical contribution. Similar ideas are widely used elsewhere, but it's nice to see they also worked in the current context

  3. Carefully designed ablation studies

Critical weaknesses
  1. The DPC target depends on a centralized critic evaluated on policy-generated final actions, which may be out-of-distribution in offline MARL. It seems that the conservative QQ ensemble is only a heuristic mitigation, not a guarantee. Errors in QtotQ_{\rm tot} can propagate into process supervision and be amplified across denoising steps.

  2. The method uses random reference prefixes and only singleton and pairwise coalition terms. It does not formalize a proper Shapley-style decomposition or guarantee efficiency.

  3. It seems that the computational and scalability costs are high,

Questions for authors

N/A

Comments on the paper

1 comment

Dear Reviewers,

As the discussion period is nearing its end, we would appreciate your feedback on whether our responses and the new evidence address your main concerns.

Thank you again for your time and constructive feedback!

Best,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.