- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents a method that combines agent-specific counterfactual advantages with PPO updates along each agent’s diffusion generation path. The method - CGPO compares each agent’s chosen action with its alternatives while holding teammates’ actions fixed, then uses the resulting credit to train every denoising step of its diffusion policy.
Critical strengths
-
The credit ablations in table 4 are a useful evidence for the benefit of the agent-specific counterfactual advantage. The Global-advantage comparison is also informative because it tests whether retaining teammates factual actions helps beyond subtracting a state-dependent baseline.
-
It is useful in my option to test the predicted sampling-noise contribution using independently estimated variance scales. Separately, the environment-branch evaluations assess candidate actions through simulator returns, avoiding reliance on the training critic to establish action quality.
Critical weaknesses
-
Corollary 3 assumes an additive value function and exact values, while the practical method uses a learned critic, batch standardization, and clipped updates. Does CGPO have a lower total gradient variance than global advantage on the evaluated tasks?
-
It seems as if the algorithmic contribution is largely a combination of existing techniques. Also, the paper does not compare its baseline with the score-weighted alternative in Remark 4, which accounts for how strongly different actions affect learning updates. This leaves unclear whether CGPO’s chosen baseline is competitive with an established approach to reducing noise in those updates.
Questions for authors
-
(Weakness 1) For the same actor and critic, does CGPO have lower total gradient variance than Global advantage after batch standardization and PPO clipping?
-
(Weakness 2) Can the authors clarify what algorithmic contribution distinguishes CGPO from combining COMA-style counterfactual credit with DPPO-style diffusion updates?
-
(Weakness 2) With the same diffusion architecture and sampling budget, how does CGPO compare with Remark 4’s score-weighted baseline in final return and gradient variance?
No response to this review has been posted to date. The reviewer's assessment therefore stands, and the reviewer would gladly consider a response from the authors before the discussion period closes.