acceptodds
Under review as a conference paper at ICLR 2027

CGPO: Counterfactual Credit for Multi-Agent Generative Policy Optimization

Abstract

Diffusion policies offer expressive action representations for cooperative multi-agent control, but learning from shared team rewards requires distinguishing each agent's contribution to joint performance. We introduce Counterfactual Generative Policy Optimization (CGPO), an on-policy actor-critic framework that trains decentralized diffusion policies through counterfactual credit assignment. For each agent, a centralized critic estimates a counterfactual advantage by comparing the value of its executed action with the average value of alternatives independently sampled from its behavior policy, while holding teammates' actions fixed. This advantage weights PPO-style updates at each denoising step along the diffusion trajectory that generated the executed action. Our theoretical analysis shows that estimating the counterfactual baseline with independent samples introduces an additional gradient covariance term that scales as , and identifies conditions under which counterfactual credit reduces policy-gradient variance. Experiments on VMAS, MaMuJoCo, and MPE show that CGPO achieves the highest mean returns among the compared methods, with gains of 6.7-46.6% over the strongest compared baselines on MaMuJoCo.

Primary area:
reinforcement learning
Keywords:
Multi-Agent Reinforcement Learning, Credit Assignment, Counterfactual, Multi-Agent Diffusion

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BDocOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a method that combines agent-specific counterfactual advantages with PPO updates along each agent’s diffusion generation path. The method - CGPO compares each agent’s chosen action with its alternatives while holding teammates’ actions fixed, then uses the resulting credit to train every denoising step of its diffusion policy.

Critical strengths
  1. The credit ablations in table 4 are a useful evidence for the benefit of the agent-specific counterfactual advantage. The Global-advantage comparison is also informative because it tests whether retaining teammates factual actions helps beyond subtracting a state-dependent baseline.

  2. It is useful in my option to test the predicted sampling-noise contribution using independently estimated variance scales. Separately, the environment-branch evaluations assess candidate actions through simulator returns, avoiding reliance on the training critic to establish action quality.

Critical weaknesses
  1. Corollary 3 assumes an additive value function and exact values, while the practical method uses a learned critic, batch standardization, and clipped updates. Does CGPO have a lower total gradient variance than global advantage on the evaluated tasks?

  2. It seems as if the algorithmic contribution is largely a combination of existing techniques. Also, the paper does not compare its baseline with the score-weighted alternative in Remark 4, which accounts for how strongly different actions affect learning updates. This leaves unclear whether CGPO’s chosen baseline is competitive with an established approach to reducing noise in those updates.

Questions for authors
  1. (Weakness 1) For the same actor and critic, does CGPO have lower total gradient variance than Global advantage after batch standardization and PPO clipping?

  2. (Weakness 2) Can the authors clarify what algorithmic contribution distinguishes CGPO from combining COMA-style counterfactual credit with DPPO-style diffusion updates?

  3. (Weakness 2) With the same diffusion architecture and sampling budget, how does CGPO compare with Remark 4’s score-weighted baseline in final return and gradient variance?

Reviewer_E1JrOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies credit assignment for decentralized diffusion policies in cooperative multi-agent reinforcement learning. It proposes CGPO, an on-policy actor-critic method in which a centralized critic computes a counterfactual advantage for each agent. Following COMA, the advantage compares the value of the executed action with the mean value of M alternative actions sampled from the agent's own policy while teammates' actions are held fixed. Following DPPO, the advantage then weights PPO-style updates at every denoising step of the diffusion path. The theoretical analysis shows that finite sampling adds a covariance term proportional to 1/M. Under an additive team-value model, it derives a sampling threshold above which counterfactual credit has lower gradient variance than a global advantage. Experiments on MPE, MaMuJoCo and VMAS report the highest mean returns among the compared methods.

Critical strengths

S1. The paper expresses counterfactual credit through the transition-level path score of diffusion policies (Proposition 1), which avoids evaluating the marginal action likelihood. The evaluation spans three benchmark suites with five or four seeds and standard deviations, and the paired credit ablations hold the diffusion actor fixed. These are the main points in the paper's favour.

Critical weaknesses

W1. The paper presents itself as a credit-assignment method without engaging the credit-assignment problem in its experiments or its theory. The method combines two existing components, the Monte Carlo counterfactual baseline of COMA for continuous actions (Foerster et al., AAAI 2018) and the per-step PPO objective of DPPO (Ren et al., ICLR 2025). None of the compared methods is designed for credit assignment. They are general MARL algorithms (MAPPO, MASAC, MADDPG, HATD3, HASAC) and diffusion-policy methods (MADPMD, MASDAC, OMAD). The theory does not close this gap. Theorem 2 quantifies the variance of an M-sample Monte Carlo mean, and Corollary 3 assumes an additive team value that removes the interaction between agents. Appendix A.3 further predicts that the benefit of counterfactual credit grows with the number of agents. No experiment varies the number of agents, and the stated team sizes do not exceed six. MaMuJoCo offers scenarios that would test this prediction, such as ManySegmentSwimmer with a configurable number of agents and the strongly coupled CoupledHalfCheetah. The only direct check of the theory, Figure 7, confirms the 1/M law of Monte Carlo averaging. The predicted variance reduction relative to a global advantage is never measured.

W2. The paper's own ablations show that its gains over the baselines cannot be attributed to counterfactual credit. With the same counterfactual credit, a Gaussian actor in place of the diffusion actor reaches 4487.8 instead of 16746.8 on HalfCheetah 2x3 and 2179.0 instead of 9447.1 on Walker2d 2x3 (Tables 4 and 5). The Gaussian counterfactual variant falls below every external baseline in Table 1 on Walker2d 2x3, Swimmer 2x1 and Ant 4x2. On both Cooperative Navigation configurations, it is even worse than the Gaussian variant without a counterfactual baseline (-36.15 versus -32.79 and -53.8 versus -46.22). Conversely, the Global advantage variant, which keeps the diffusion actor and drops teammate-conditioned credit, already exceeds every external baseline on Ant 2x4 and Ant 2x4d (Table 4). The comparison in Table 1 therefore conflates the credit signal with the actor parameterisation and the training pipeline. The paper does not explain why a five-step diffusion actor, evaluated with zero noise, returns 3.7 and 4.3 times as much as a Gaussian actor on these two locomotion tasks. The Gaussian variants also reuse the hyperparameters of the diffusion actor (Appendix B.4), and the paper itself calls this comparison incomplete. The behavioural analyses in Section 5.3 and Appendix C.4 compare CGPO only with Factual-Q and MAPPO, not with the Global advantage variant.

W3. The comparison with the baselines is not made on equal terms. The configuration of CGPO was selected on the evaluation tasks themselves. On VMAS, the reported runs use the values of M and K with the highest final return in the sweeps of Figures 5 and 6. These are M = 8 for balance and M = 16 with K = 2 for sampling. These sweeps use the same tasks and the same seeds {0, 1, 2, 3} as Table 2. Further task-specific overrides change the clip ratio, the sample budget, the discount and the critic warm-up, and the KL gate is enabled only for MaMuJoCo (Table 3). The paper does not describe how any of these values were chosen. No comparable tuning is reported for the baselines. On VMAS they are BenchMARL implementations. On MPE and MaMuJoCo, MAPPO, HATD3, HASAC, MADPMD and MASDAC are reported at 10M steps following OMAD, against 3M steps for CGPO and OMAD (Appendix B.3). The paper does not state whether these results were rerun or taken from the OMAD paper, or whether they use the same zero-noise evaluation as CGPO. Without equal tuning effort and a common evaluation protocol, the margins in Tables 1 and 2 do not show that CGPO is the stronger method.

Questions for authors

Q1 (clarification). Were the baseline results in Table 1 rerun under the protocol of CGPO or taken from Li et al. (2026c), and how were the hyperparameters of CGPO and of each baseline chosen? Results with the CGPO configuration fixed in advance and with the baselines tuned under the same budget would address W3.

Q2 (requested experiments). What explains the gap between the diffusion and Gaussian actors under the same counterfactual credit (Tables 4 and 5)? A comparison in which each actor is tuned separately, with and without teammate-conditioned credit, would show how much of the gain comes from the credit signal.

Q3 (requested experiments). Can the authors evaluate CGPO on MaMuJoCo scenarios with a varied number of agents, such as ManySegmentSwimmer, and on strongly coupled tasks such as CoupledHalfCheetah? Reporting the gradient variance of CGPO and of the Global advantage variant as the number of agents grows would test Corollary 3 and Appendix A.3 directly. The team sizes used in balance and sampling should also be stated.

1 comment

No response to this review has been posted to date. The reviewer's assessment therefore stands, and the reviewer would gladly consider a response from the authors before the discussion period closes.

Reviewer_9FD1OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Counterfactual Generative Policy Optimization (CGPO), an on-policy actor–critic framework for cooperative multi-agent reinforcement learning with decentralized diffusion policies. For each agent, the method independently samples alternative actions from its behavior policy while holding the centralized context and teammates’ executed actions fixed. A centralized critic evaluates the executed joint action and these alternatives, and their value difference defines an agent-specific advantage. This advantage then weights PPO-style updates along the recorded denoising trajectory that generated the executed action.

Experiments cover 13 task configurations across MPE, MaMuJoCo, and VMAS, supplemented by credit-signal ablations, sampling-budget studies, and behavioral diagnostics.

Critical strengths
  1. Constructing informative agent-specific learning signals from shared team rewards is an important problem. CGPO provides a clearly specified way to combine a centralized action-value critic with decentralized diffusion actors.

  2. The Factual-Q and Global advantage ablations help isolate the effect of the credit signal while retaining the diffusion-policy architecture and other training components. The supplementary Gaussian-policy comparisons provide additional context.

Critical weaknesses

My primary concern is that the core algorithm appears to combine established components without sufficiently identifying a new technical contribution. Equations (5)–(6) use the familiar COMA-style baseline that marginalizes the focal agent’s action while fixing teammates’ actions. Equation (8) estimates this baseline through independent Monte Carlo sampling, an extension to continuous actions already discussed in the original COMA paper. Equations (10)–(11) then apply PPO likelihood ratios and clipping to diffusion denoising transitions, following the optimization structure of DPPO. The method constructs a centered critic-based advantage and uses it in an otherwise familiar actor–critic update. A combination of existing ideas can be valuable, but the paper needs to explain which aspects of this integration require a nontrivial technical solution and what new insight follows from them.

Questions for authors
  1. What precisely distinguishes CGPO from a direct combination of a Monte Carlo COMA-style baseline and DPPO? Please identify the mathematical or algorithmic component that requires a new technical solution. If the core contribution is the integration itself, what makes that integration nontrivial, and how should its contribution be distinguished from those of the underlying methods?

  2. How does CGPO compare with a matched diffusion-policy PPO baseline using a learned state-value function and GAE?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.