acceptodds
Under review as a conference paper at ICLR 2027

Towards Better Training Signal: Advantage Clipped Policy Optimization

Abstract

Reinforcement learning (RL) has become a cornerstone for improving the reasoning capabilities of large language models (LLMs), but the need for on-policy data substantially limits training efficiency. Reusing off-policy data through importance sampling (IS) can improve efficiency but introduce considerable instability. Hence, algorithms such as PPO and GRPO widely adopt IS-ratio clipping to stabilize training. However, training stability and gradient estimate are mainly determined by the product of IS ratio and advantage. To further stabilize training, we propose ACPO, which clips the product of the IS ratio and the advantage, leading to more stable gradient estimates. We also establish a connection between ACPO and gradient clipping in policy mirror descent (PMD), which is a standard technique to stabilize optimization process, and prove the convergence of clipped-PMD under the standard RL setting. Experiments on widely used mathematical reasoning benchmarks show that ACPO consistently outperforms PPO and GRPO in both accuracy and training efficiency, delivering 4-6% points gains on standard math benchmarks, with Qwen3-8B+PPO. Hence, ACPO is a practical and effective alternative to conventional IS-ratio clipping for RL post-training of LLMs.

Primary area:
reinforcement learning
Keywords:
Policy Optimization, Mirror Descent, Post-Training

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wkveOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes Advantage Clipped Policy Optimization (ACPO). This author observes that very hard and very easy prompts produce long positive and negative tails of the product of the importance sampling and advantage. Therefore, this work thinks that instead of clipping the importance ratio as PPO does, this work argues that the existing PPO-like RL algorithm should clip the product of the importance ratio and the advantage to [-α\alpha,α\alpha], and drops the gradient of the tokens whose product falls outside this range. In terms of the experiments of Qwen2.5-Math-7B and Qwen3 models, ACPO improves the weighted accuracy on math benchmarks by 1.6–4.0 points over PPO and 0.6–1.5 points over GRPO, and reaches the best accuracy of each baseline in fewer training steps. The paper also provides a second-moment comparison with PPO and connects ACPO to gradient clipping in policy mirror descent with convergence guarantees.

Critical strengths

I think the idea of connecting ACPO to gradient clipping in policy mirror descent is interesting, as it offers an optimization perspective on clipping in policy optimization. The method is easy to integrate into existing RL training frameworks such as verl without complex modifications. The evaluation setting is clearly reported, and the improvements are consistent across all eight matched comparisons across four model sizes and two base algorithms (PPO and GRPO). Overall, it is a complete submission.

Critical weaknesses

(1) I really doubt the core claim in this work. This work states that the policy update is mainly determined by the product of the importance ratio and the advantage, and therefore clipping this product, rather than the ratio alone. However, clipping rA^r\hat{A} to [−α,α][-\alpha, \alpha] is equivalent to allowing the ratio rr to go up to α\/∣A^∣\alpha\/|\hat{A}|. Under the GRPO setting used in this work, I suspect that the group-normalized advantages are rarely large enough to trigger clipping near r≈1r\approx1, so that for most tokens ACPO effectively allows a much wider ratio range than GRPO. If this is the case, the gains of GRPO-AC may simply come from loosening the clipping, which is already known to improve performance and preserve entropy (e.g., Clip-Higher in DAPO), rather than from clipping the product itself.

(2) Table 1 reports the best accuracy over all training checkpoints, and all results appear to come from a single run. I do not know whether there is any model checkpoint selection for each task. The GRPO gains (0.6–1.5 points) are actually within typical run-to-run variance noise, so I cannot be convinced by this gain. The reported speedup over PPO is also measured against the baseline’s best checkpoint and is inflated by a weak PPO baseline as shown in Appendix D.3, so I am not convinced by this efficiency gain as well.

Questions for authors

I strongly encourage the authors report the distribution of ∣A^∣|\hat{A}| and the fraction of tokens clipped by ACPO versus GRPO during training. Also, comparisons with GRPO and DAPO using a larger ϵhigh\epsilon_{high} or no ratio clipping and a sensitivity analysis of are needed to support the core claim.

Reviewer_PFxhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes Advantage Clipped Policy Optimization (ACPO), which replaces conventional importance-ratio clipping with clipping of the product of the importance ratio and estimated advantage. The method is applied to PPO and GRPO for LLM reinforcement-learning post-training. The authors provide a second-moment comparison under assumptions, relate the method to gradient-clipped policy mirror descent (PMD), and evaluate it on four mathematical-reasoning benchmarks using Qwen2.5-Math-7B and Qwen3 models. The reported best-checkpoint weighted accuracies improve over the corresponding PPO and GRPO baselines.

Critical strengths
  1. The proposed modification to the PPO/GRPO surrogate is simple and implementable without introducing another model. This makes the idea practically relevant to LLM post-training.

  2. The experiments cover multiple model sizes and report consistent improvements in best-checkpoint weighted accuracy across the matched PPO and GRPO comparisons, suggesting that the idea merits further investigation.

Critical weaknesses
  1. The claimed connection between gradient-clipped PMD and the ACPO objective is not established by valid equivalence transformations (Section 4 and Appendix B, particularly pp. 6 and 14). PMD rescales the entire statewise advantage vector by alpha/max(alpha, ||A(s,.)||), whereas Appendix B replaces this with elementwise clipping clip(A(s,a), -alpha, alpha). The two operations are generally different. Moreover, importance sampling does not justify replacing r * clip(A, -alpha, alpha) with clip(r*A, -alpha, alpha), since clipping is nonlinear. The convergence theorems concern an idealized PMD update with a KL proximity term, not the implemented ACPO surrogate after these substitutions and removal of that term. Thus, the stated PMD theory does not directly establish convergence or stability of ACPO.

  2. Several reported experimental settings and evaluation procedures require clarification (Section 5 and Appendix D.1). The authors state that they ultimately selected a learning rate of 5e-7 for Qwen3-1.7B PPO-type runs, but the next sentence attributes the reported Qwen3-1.7B results to a learning rate of 1e-6. Table 2 also specifies a maximum response length of 8196 tokens for Qwen2.5-Math-7B while its own footnote gives the model's maximum context length as 4096, without explaining the extension or truncation policy. Finally, Table 1 selects the best weighted accuracy across evaluation benchmarks during training; it is unclear whether a separate validation set was used for checkpoint selection. These issues affect interpretation and reproducibility of the empirical claims.

Questions for authors
  1. Could you provide a rigorous derivation connecting the statewise norm-rescaled PMD update to the elementwise ACPO surrogate? In particular, how do you justify the elementwise clipping step and the placement of the importance ratio inside the nonlinear clipping operator? Which convergence statement, if any, applies to the actual ACPO updates?

  2. Which learning rate was used to produce the reported Qwen3-1.7B PPO/PPO-AC results: 5e-7 or 1e-6? Please reconcile the conflicting statements in Appendix D.1 and provide the final run configurations.

  3. How was a maximum response length of 8196 implemented for Qwen2.5-Math-7B given the stated 4096-token context limit? Was context extension, truncation, or another configuration used? Please clarify whether 8196 is intentional.

  4. How were checkpoints selected for Table 1? Was there a held-out validation set, or were the four reported test benchmarks used directly to select the best training step?

Reviewer_cXyoOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes approaches to stability for RL in LLMs. It proposes a heuristic for clipping policy gradient signals that is usually a source of instability in gradient steps with off-policy data. Concretely, they suggest clipping the product of the IS weights and advantage, instead of only the IS weights as usual. The work provinces theoretical results claiming variance reduction (under certain assumptions) and establishing a connection between the method and Mirror Policy Descent. Empirically, it presents results on math reasoning benchmarks, claiming improvements in final performance and sample efficiency against PPO and GRPO baselines.

Critical strengths
  • Stability of RL in LLMs is a very relevant problem to unlocking further post-training scaling.

  • The paper is easy to follow. The proposed method is simple and easy to understand.

Critical weaknesses
  • The work presents several claims that are either vague, unclear or unsupported, either as motivation or as takeaways from the experimental evidence.

    • “Training stability and gradient estimates are mainly determined by the product of IS ratio and advantage.” This is used in the abstract as a motivation of the work, but it is also the hypothesis investigated in the work.
    • “Reusing off-policy data through importance sampling (IS) can improve efficiency but introduce considerable instability.” This claim is in several places in the work. It is not factually wrong, but it is important to qualify how IS leads to instability (i.e., IS leads to increased variance in the gradient estimation)
    • “Explicitly computing per-token gradients ∇θlogπθ(at | st) is computationally expensive and impossible”. It also needs a better justification. Note that per-token gradients are computed for gradient descent, but usually as an average and not isolatedly materialized.
  • The work proposes theoretical results to support the method but does not discuss or analyze if the assumptions are realistic. Is the retention assumption valid? Should we expect the gradient norms after clipping to me roughly the same for different methods?

    • Also, the whole connection with the Mirror Policy Descent is presented as a main contribution of the paper but I had a hard time to see why that is really relevant to the presented method and which insight/understanding it should bring, especially given that MPD is not really used or compared against in this work, or any particular property of MPD is highlighted as a motivation to support the proposed method. It looks tangential to the core point of the paper.
    • It would be more interesting to bring theoretical results on the bias introduced by clipping, or under the convergence properties of the proposed objective.
  • At some point the paper states “Vanilla PPO and GRPO can exhibit rapid entropy decay” - which is indeed what prior literature supports - but the experiments show a different trend for GRPO: the entropy is actually increasing over the course of learning. This raises concerns about the experimental setup and the claims about entropy.

  • In the experimental design there are also a few important concerns:

    • The paper does not report confidence intervals, so it is unclear what is the statistical significance of the results. It is unclear if the gains are not just observation noise.
    • The paper’s central point of study is about clipping mechanisms, but the only presented baselines are PPO and GRPO, although, as they paper presents, there is a lot of literature on different clipping strategies for RL stability. The paper should at least bring methods like DAPO/CISPO/GAPO.
    • Furthermore, the paper introduces a heuristic that relies on a hyperparameter, but there was no analysis on the sensitivity of this hyperparameter choice or how to select it based on the problem.
Questions for authors

Please see my points above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.