- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work proposes Advantage Clipped Policy Optimization (ACPO). This author observes that very hard and very easy prompts produce long positive and negative tails of the product of the importance sampling and advantage. Therefore, this work thinks that instead of clipping the importance ratio as PPO does, this work argues that the existing PPO-like RL algorithm should clip the product of the importance ratio and the advantage to [-,], and drops the gradient of the tokens whose product falls outside this range. In terms of the experiments of Qwen2.5-Math-7B and Qwen3 models, ACPO improves the weighted accuracy on math benchmarks by 1.6–4.0 points over PPO and 0.6–1.5 points over GRPO, and reaches the best accuracy of each baseline in fewer training steps. The paper also provides a second-moment comparison with PPO and connects ACPO to gradient clipping in policy mirror descent with convergence guarantees.
Critical strengths
I think the idea of connecting ACPO to gradient clipping in policy mirror descent is interesting, as it offers an optimization perspective on clipping in policy optimization. The method is easy to integrate into existing RL training frameworks such as verl without complex modifications. The evaluation setting is clearly reported, and the improvements are consistent across all eight matched comparisons across four model sizes and two base algorithms (PPO and GRPO). Overall, it is a complete submission.
Critical weaknesses
(1) I really doubt the core claim in this work. This work states that the policy update is mainly determined by the product of the importance ratio and the advantage, and therefore clipping this product, rather than the ratio alone. However, clipping to is equivalent to allowing the ratio to go up to . Under the GRPO setting used in this work, I suspect that the group-normalized advantages are rarely large enough to trigger clipping near , so that for most tokens ACPO effectively allows a much wider ratio range than GRPO. If this is the case, the gains of GRPO-AC may simply come from loosening the clipping, which is already known to improve performance and preserve entropy (e.g., Clip-Higher in DAPO), rather than from clipping the product itself.
(2) Table 1 reports the best accuracy over all training checkpoints, and all results appear to come from a single run. I do not know whether there is any model checkpoint selection for each task. The GRPO gains (0.6–1.5 points) are actually within typical run-to-run variance noise, so I cannot be convinced by this gain. The reported speedup over PPO is also measured against the baseline’s best checkpoint and is inflated by a weak PPO baseline as shown in Appendix D.3, so I am not convinced by this efficiency gain as well.
Questions for authors
I strongly encourage the authors report the distribution of and the fraction of tokens clipped by ACPO versus GRPO during training. Also, comparisons with GRPO and DAPO using a larger or no ratio clipping and a sensitivity analysis of are needed to support the core claim.