acceptodds
Under review as a conference paper at ICLR 2027

When Clipping Becomes a Learning Signal: A Permutation Baseline for Policy Optimization

Abstract

Reinforcement learning with verifiable rewards (RLVR) has improved mathematical reasoning in large language models. Methods such as Group Relative Policy Optimization (GRPO) use clipping to stabilize training by discouraging large policy changes. However, we identify a component of the clipped objective whose gradient acts as an additional learning signal outside the clipping interval and can counteract reward-guided updates. This component equals the average clipped objective over all within-group reward permutations, which we call the permutation baseline. In this work, we propose Permutation-Corrected Policy Optimization (PerCPO), which removes this source of interference with reward-guided updates by subtracting this baseline from the original clipped objective in closed form. For binary rewards, the resulting updates follow the direction indicated by rewards and are naturally attenuated outside the clipping interval, restoring gradients suppressed by clipping. We further show that the PerCPO objective increases along a path that improves accuracy with minimal Kullback–Leibler divergence from the sampling policy, even where the standard clipped objective plateaus. Extensive experiments on mathematical reasoning benchmarks demonstrate the effectiveness of PerCPO. For instance, on Qwen2.5-Math-7B, PerCPO outperforms GRPO by 3.47% in average accuracy across seven benchmarks.

Primary area:
reinforcement learning
Keywords:
RLVR, policy optimization, LLM, clipping mechanism, mathematical reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mu6UOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors study how the clipping mechanism can provide information to the LLM RL training and propose PerCPO, which subtracts the permutation baseline to restore and rescale gradients outside the clipping interval. Experiments on 3 Qwen models across 7 mathematical reasoning benchmarks show improvements over GRPO and the evaluated clipping variants.

Critical strengths
  1. The paper tackles a important problem for clipping in LLM RL.
  2. Experiments over multiple benchmarks is provided.
  3. The method is explained theoretically to some extent.
Critical weaknesses

Experiments

  • The experiment setup raises concerns. All the Qwen-Math model can be improved under random reward (arxiv.org/pdf/2506.10947) due to contamination. Over the uncontaminated Qwen3 model, the improvement becomes more marginal. Also, evaluating pass@1 is very limiting to examine RLVR performace. Could you report the avg@16 or avg@8 along with pass@16?
  • Clipping information would accumulate over models with very long rollout. However, current experiment only uses 4K-capped Qwen-Math model and non-thinking mode from Qwen3. Whether the method and result can be extended to real reasoning model remains a gap. Current performance improvement might just due to training heuristics, as only single-run result is report.
  • It remains uncertain whether the results can be extended to further model families.
  • The method is quite close to SAPO (arxiv.org/pdf/2511.20347v2), but the author didn't provide any discussion or comparison to it. Also, the method is close to BAPO, and the author discussed it without any comparison.

Method novelty & soundness

  • why the permutation baseline should be treated as a harmful signal that needs to be removed? I don't fully understand. I mean a gradient that survives reward permutation doesn't necessarily mean it's bad, where clipping is exactly designed to constrain policy changes.
  • the ablation doesn't justify the benefit of the specific permutation-based correction. A simple baseline is to retain a fixed fraction of the unclipped gradient outside the clipping interval, and also, I think a comparison against SAPO can see this question better.
Questions for authors

See weaknesses section. Please answer the questions and address the concerns.

1 comment

We thank the reviewer for his/her constructive comments and provide our point-wise replies as follows!

W1/Q1. Given that random rewards can improve Qwen-Math and the gains on Qwen3 appear smaller, why subtract the permutation baseline if it could provide useful regularization?

A1. Both Shao et al.’s random-reward experiment and our permutation analysis break the link between response correctness and reward assignment. However, our permutations preserve the observed counts of positive and negative rewards within each prompt group, and hence its reward mean and standard deviation. Shao et al. instead give each response the same fixed probability of a positive reward, independently of the other responses and of the group’s observed correctness rate.

We agree that a gradient surviving reward permutation could still provide useful regularization. To test whether retaining part of BB improves accuracy, we added an ablation on Qwen2.5-Math-7B in the revised manuscript (Table 3). We vary the fraction λ\lambda of BB subtracted from the clipped objective.

λ\lambda0 (GRPO)0.250.500.751 (PerCPO)
Seven-benchmark mean Avg@4 (%)38.5938.5239.3041.2742.85

Accuracy rises across the three partial corrections, and full subtraction gives the highest score.

In Section 4.2 of the revision, we uniformly permute rewards within each prompt group before every GRPO update on Qwen3-4B, so BB is the expected token objective. Policy entropy drops sharply without a sustained Avg@4 gain (41.18% initially, 38.35% at step 200, Figure 1(b)). Together with the scaling ablation, this supports subtracting BB.

W2/Q2. Can you report Avg@8 or Avg@16 alongside Pass@16, and do the gains hold across independent training runs?

A2. Compute constraints limited us to four responses per problem. In the revision, Table 1 reports Avg@4 and Appendix B reports Pass@4 at the same checkpoints. For space, we show seven-benchmark Avg@4 (%) as mean ± std over three independent training runs per model–method pair.

MethodQwen2.5-Math-1.5BQwen2.5-Math-7BQwen3-4B
GRPO33.43±0.9538.59±0.9346.91±2.10
Clip-Higher34.03±0.2741.48±0.5853.15±0.94
CISPO33.80±0.3541.70±0.3651.36±1.19
CE-GPPO29.84±1.3038.77±0.9853.24±1.35
PerCPO35.42±0.4742.85±0.5753.82±1.13

PerCPO leads on both metrics for all three models.

W3/Q3. Do PerCPO’s gains extend beyond the Qwen family, and how does it compare with SAPO and BAPO?

A3. In the revised manuscript, we evaluated all five main methods on two Llama models and added comparisons with SAPO and BAPO on both Llama and both Qwen2.5-Math models. The table reports Avg@4 (%) averaged across seven benchmarks, with mean ± std across three independent training seeds.

ModelSAPOBAPOPerCPO
Qwen2.5-Math-1.5B32.07±1.5131.05±0.6135.42±0.47
Qwen2.5-Math-7B40.58±0.2039.49±0.3342.85±0.57
Llama-3.2-1B-Instruct7.89±0.129.14±0.3310.61±0.14
Llama-3.2-3B-Instruct18.90±1.1519.86±0.7520.10±0.65

PerCPO outperforms SAPO and BAPO on all four models and achieves the highest mean Avg@4 among all seven methods on both Llama models. Appendix B of the revised manuscript provides further analysis of SAPO and BAPO.

W4/Q4. Do PerCPO’s gains extend to longer reasoning rollouts, such as those generated in thinking mode?

A4. Due to compute constraints, we did not train Qwen3-4B in thinking mode or any dedicated thinking models, so the cumulative effect of clipping on longer rollouts remains untested. However, Appendix C of the revised manuscript compares Qwen3-4B’s responses to the same math problem after training with each of the five methods in our main experiments. Even in non-thinking mode, the responses from all five models show the kind of step-by-step reasoning often associated with thinking models. In this setting, PerCPO exceeds GRPO by 6.91 percentage points in mean Avg@4 across the seven benchmarks (53.82% vs. 46.91%, Table 1).

W5/Q5. Does PerCPO’s permutation-based correction outperform retaining a fixed fraction of the unclipped gradient outside the clipping interval?

A5. In the revised manuscript, we added an ablation on Qwen2.5-Math-7B that replaces the observed correctness rate pp of each prompt group with a fixed value cc. Each fixed setting retains a fraction 1−c1-c of the unclipped gradient below the lower bound and cc above the upper bound.

Ratec=0c=0c=0.25c=0.25c=0.50c=0.50c=0.75c=0.75c=1c=1Group rate pp (PerCPO)
Seven-benchmark mean Avg@4 (%)38.1038.9739.3441.4941.0442.85

PerCPO exceeds the best fixed setting tested (c=0.75c=0.75) by 1.36 percentage points, favoring the use of each group’s observed correctness rate to set the gradient fractions.

Reviewer_J34sOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper analyzes GRPO's clipped objective through within-group reward permutations, derives the policy-dependent component that remains after permutation averaging, and proposes PerCPO to subtract this closed-form baseline. For binary rewards, the resulting objective restores or attenuates gradients outside the clipping interval according to the group's correctness rate. In an idealized model that treats each complete response as a single action, the authors also show that the corrected objective strictly increases along a minimum-KL path toward higher accuracy. Experiments cover Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, and Qwen3-4B in non-thinking mode, reporting higher average Pass@1 than the compared clipping strategies across seven mathematics benchmarks, together with component ablations and an analysis of update-count sensitivity.

Critical strengths

S1: The analytical perspective is interesting and directly motivates the method.

Section 4.1 explicitly derives the component that remains after reward-permutation averaging in Equations (7)–(9). Section 4.2 then subtracts it to obtain a closed-form objective without enumerating permutations during training. For binary rewards, Equation (13) and Figure 2 clearly explain how the correction changes gradients outside the clipping interval. This establishes a clear connection between the diagnosis and the method, contributing more than an empirical adjustment of clipping thresholds.

S2: The ablations are reasonably thorough, and the main results show consistent average gains.

Table 2 compares restoration alone, attenuation alone, bounded restoration, and the full method. Full PerCPO achieves 42.55% average accuracy, compared with 39.93% for Restore-Only, 40.12% for Attenuation-Only, and 41.51% for the Bounded variant, supporting the combination of restoration and attenuation. Figure 4 also reports sensitivity to the number of updates. In Table 1, PerCPO achieves the highest average accuracy on all three models, exceeding each model's strongest competing baseline by approximately 1.35–1.53 percentage points. These results make the method promising enough to warrant further validation.

Critical weaknesses

W1: The experimental coverage is limited.

All evaluated models belong to Qwen, and the paper does not test thinking mode or dedicated thinking models. Although the ablations are useful, the current results do not establish whether the gains extend beyond these settings. Improvements on another model family would make me willing to reconsider my score.

W2: Checkpoint selection is unclear.

The paper does not explain how the reported checkpoints are selected or whether all methods follow the same rule. Some table results also differ from the endpoints of the training curves. This needs clarification to assess the comparisons fairly.

Questions for authors

Q1: Are there existing results or a limited additional evaluation on a non-Qwen model family showing whether PerCPO's gains over GRPO and relevant clipping baselines persist? Are there also any evaluations in thinking mode or on dedicated thinking models? Please identify the models and operating modes explicitly. Evidence of gains across model families is the most likely to change my current score. Thinking-setting results would also help establish the method's scope, without requiring a comprehensive new benchmark study during rebuttal.

Q2: Please specify the checkpoint-selection rules used for all methods in Tables 1–2 and explain how the table results correspond to Figures 1 and 4. Is selection based on an independent validation set? Which checkpoints are reported for the CE-GPPO runs that collapse early? Comparisons under a consistent selection rule would directly affect my assessment of the headline gains.

2 comments

We thank the reviewer for the encouraging feedback and thoughtful suggestions!

W1/Q1. Do PerCPO’s gains over GRPO and other clipping baselines extend beyond Qwen? Have you evaluated thinking mode or dedicated thinking models?

A1. In the revised manuscript, we added experiments on Llama-3.2-1B-Instruct and Llama-3.2-3B-Instruct, using three independent training seeds for every model–method combination. Due to space constraints, we report only Avg@4 (%) averaged across seven benchmarks, with mean ± std across training seeds. Appendix B of the revised manuscript provides the full results for each benchmark.

MethodLlama-3.2-1B-InstructLlama-3.2-3B-Instruct
GRPO8.73±0.0917.86±0.32
Clip-Higher9.80±0.4818.62±0.91
CISPO9.14±0.2919.30±0.35
CE-GPPO8.14±0.1517.41±0.79
SAPO7.89±0.1218.90±1.15
BAPO9.14±0.3319.86±0.75
PerCPO10.61±0.1420.10±0.65

PerCPO improves over GRPO by 1.88 percentage points on 1B and 2.24 points on 3B. Its mean also exceeds the strongest baseline on each model, by 0.81 points over Clip-Higher on 1B and 0.24 points over BAPO on 3B.

We used Qwen3-4B in non-thinking mode throughout. Given the compute and time available for the rebuttal, we did not evaluate Qwen3-4B in thinking mode or any dedicated thinking models. Appendix C of the revised manuscript compares Qwen3-4B’s responses to the same math problem after training with each of the five methods in our main experiments. Even in non-thinking mode, the responses from all five models show the kind of step-by-step reasoning often associated with thinking models.

W2/Q2. How were checkpoints selected for Tables 1–2, including CE-GPPO runs that collapsed early, and how do the reported scores relate to Figures 1 and 4?

A2. In the original submission, we trained each method once per setting. For every method in Tables 1–2, we selected the evaluated checkpoint with the highest mean Avg@4 (reported in the original submission as the mean of four Pass@1 evaluations) across the seven benchmarks and reported all seven scores from that checkpoint. As in many prior studies, we selected checkpoints based on benchmark performance. We used the evaluation benchmarks themselves for selection, with no separate validation set. The same rule applied to CE-GPPO runs that collapsed early: we selected the highest-scoring checkpoint among those evaluated in that run.

Figures 1 and 4 show how performance evolves during training. Figure 1 tracks GRPO and PerCPO on Qwen2.5-Math-7B, and Figure 4 compares different numbers of policy updates per rollout. The curve endpoints correspond to the last evaluation, whereas the checkpoint selected for a table can occur earlier.

In the revised manuscript, we report results from three independent training runs for each main experiment. We re-evaluated the saved checkpoints from the original training run and conducted two additional training runs with independent seeds. We applied the same checkpoint-selection rule separately to each of the three runs. Table 1 reports the mean and standard deviation of the resulting benchmark scores across runs.

The authors have addressed most of my concerns; accordingly, I lean toward raising my previous score to 3.

Reviewer_kBBcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Permutation-Corrected Policy Optimization (PerCPO), which modifies the clipped GRPO objective by subtracting a permutation baseline. The authors show that averaging the clipped objective over within-group reward permutations leaves a nonzero component whose gradient can interfere with reward-guided updates. PerCPO removes this component in closed form, allowing gradients beyond the clipping bounds while reducing their magnitude. The paper also provides a theoretical analysis showing that the corrected objective increases along a minimum-KL path toward higher accuracy. Experiments on three language models and seven mathematical reasoning benchmarks show that PerCPO consistently improves average accuracy over GRPO and several alternative clipping methods.

Critical strengths
  1. An interesting analysis of clipping. The paper identifies a component of the clipped GRPO objective that remains after averaging over reward permutations. I find this perspective useful, as it reveals a property of clipping that is not immediately apparent from the standard PPO formulation.

  2. A clean mathematical formulation. The permutation baseline has a closed-form expression and can be subtracted directly from the original objective. For binary rewards, the resulting gradient behavior is particularly simple: updates preserve the advantage direction and are attenuated outside the clipping interval.

  3. Good performance across different models. PerCPO achieves the highest average accuracy among the compared methods on all three models. The improvement is especially noticeable on Qwen3-4B, where the average score increases from 47.64 to 55.24. The comparisons with Clip-Higher, CISPO, and CE-GPPO are relevant.

  4. Useful analysis of the objective components. The restore-only and attenuation-only ablations help explain the effect of the proposed correction. The experiments varying the number of policy updates per rollout also provide useful information about when the method may become unstable.

Critical weaknesses
  1. The interpretation of the permutation baseline is somewhat overstated. The derivation shows that the clipped objective contains a component that remains under reward permutation. However, this does not necessarily mean that the component is harmful. Clipping is intended to constrain policy updates, and the permutation baseline may partly reflect this regularization effect. The paper would benefit from stronger evidence that removing this component is preferable to retaining it.

  2. The theoretical analysis is limited to an idealized setting. Proposition 1 considers complete responses as actions and studies a specific minimum-KL path along which accuracy increases. In practice, PerCPO uses token-level importance ratios and updates a neural policy over a much larger space. It is unclear how much the theoretical result tells us about the actual training dynamics, particularly whether the correction can lead to excessively large policy changes.

  3. The relationship between PerCPO and existing gradient-restoration methods could be clearer. Methods such as CISPO and CE-GPPO also preserve gradients beyond clipping boundaries. Although PerCPO is derived differently, its practical effect is closely related to modifying the strength of these gradients. A more direct comparison of their gradient coefficients and optimization behavior would help clarify what the permutation-based formulation offers beyond existing approaches.

  4. The experimental results lack sufficient information about variability. The authors repeat Pass@1 evaluation four times, but this does not measure variation across independent training runs. Some of the improvements over the strongest competing methods are relatively small, around 1.35–1.53 percentage points on average. Results with multiple training seeds would make these comparisons more convincing.

  5. Training stability deserves more investigation. Figure 4 shows that increasing the number of policy updates per rollout substantially degrades PerCPO on Qwen3-4B. Since the method restores gradients that standard clipping would suppress, its behavior under larger policy deviations is important. I would like to see more analysis of KL divergence, clipping ratios, and gradient magnitudes to understand when the correction remains stable.

Questions for authors
  1. The permutation baseline is interpreted as an unwanted learning signal. Could the authors provide more evidence that this component is harmful rather than serving as part of the intended regularization effect of clipping?

  2. Proposition 1 considers a minimum-KL path with complete responses as actions. How closely does this analysis reflect token-level optimization in practice? Does PerCPO offer any guarantees on policy divergence or update stability?

  3. What is the main practical difference between PerCPO and existing gradient-restoration methods such as CISPO and CE-GPPO? Could the authors compare their effective gradient weights outside the clipping interval?

  4. Are the improvements consistent across independent training seeds? Reporting standard deviations for the main experiments would help assess the reliability of the gains over the strongest baselines.

  5. Figure 4 shows considerable sensitivity to the number of policy updates per rollout on Qwen3-4B. Could the authors provide KL divergence and clipping-ratio statistics to explain this behavior?

1 comment

We thank the reviewer for the thoughtful questions and constructive feedback!

W1/Q1. Could the permutation baseline serve as a useful regularizer?

A1. We agree that clipping can provide useful regularization. However, Section 4.1 shows that the gradient of the permutation baseline BB opposes the reward-supported update when Ai>0A_i > 0 and ρ>U\rho > U, or when Ai<0A_i < 0 and ρ<L\rho < L. Even after subtracting BB, PerCPO retains only a fraction of the unclipped slope outside the clipping interval.

In Section 4.2 of the revised manuscript, we uniformly permute verifier rewards within each prompt group before every GRPO update, making BB the expected token objective. Figure 1(b) shows a sharp drop in policy entropy without a sustained gain in test Avg@4, which ends at 38.35% after 200 steps, below its initial 41.18%.

Table 3 of the revised manuscript shows how varying the fraction λ\lambda of BB subtracted from the clipped objective affects accuracy when Qwen2.5-Math-7B is trained with the original verifier rewards.

λ\lambda0 (GRPO)0.250.500.751 (PerCPO)
Seven-benchmark mean Avg@4 (%)38.5938.5239.3041.2742.85

Accuracy rises across the partial corrections. Full subtraction performs best, supporting removal of BB in this setting.

W2/Q2. How does Proposition 1 relate to token-level training, and does PerCPO guarantee bounded policy divergence or stable updates?

A2. We agree that the minimum-KL path in Proposition 1 describes an idealized setting in which each action is a complete response. We included it as a separate mathematical observation and have removed it from the revised manuscript. PerCPO is derived directly from the token-level clipped objective under within-group reward permutations, independently of this proposition. We do not claim a theoretical bound on policy divergence or a guarantee of update stability. In A5, we examine update stability empirically in a specific training setting.

W3/Q3. How do PerCPO’s gradient weights differ from those of CISPO and CE-GPPO outside the clipping interval?

A3. Appendix B of the revised manuscript compares the per-token gradient coefficients of all five methods in the main experiments, as well as SAPO and BAPO. Each entry is the coefficient multiplying Ai∇θlog⁡πθA_i\nabla_\theta\log\pi_\theta.

Methodρ<L\rho < L, Ai<0A_i < 0ρ<L\rho < L, Ai>0A_i > 0L<ρ<UL < \rho < Uρ>U\rho > U, Ai>0A_i > 0ρ>U\rho > U, Ai<0A_i < 0
GRPO00ρ\rhoρ\rho00ρ\rho
Clip-Higher00ρ\rhoρ\rho00ρ\rho
CISPOρ\rhoρ\rhoρ\rhomin⁡(ρ,5)\min(\rho,5)min⁡(ρ,5)\min(\rho,5)
CE-GPPOβ1L\beta_1 Lρ\rhoρ\rhoβ2U\beta_2 Uρ\rho
SAPOsis_isis_isis_isis_isis_i
BAPO00ρ\rhoρ\rho00ρ\rho
PerCPO(1−p)ρ(1-p)\rho(1−p)ρ(1-p)\rhoρ\rhopρp\rhopρp\rho

Clip-Higher uses a larger UU, and BAPO selects L,UL,U per batch. For SAPO, si=ρ/cosh⁡2(τi(ρ−1)/2)s_i=\rho/\cosh^2(\tau_i(\rho-1)/2), where τi\tau_i depends on the advantage sign. CISPO caps the importance weight, and CE-GPPO restores suppressed gradients with fixed coefficients. PerCPO uses the group’s correctness rate pp to both restore gradients that GRPO suppresses and attenuate those it retains outside the interval.

Appendix B of the revised manuscript also reports training dynamics to provide further insight into optimization behavior.

W4/Q4. Are the gains consistent across independent training seeds?

A4. We trained every model–method combination in the main experiments with three independent seeds. Table 1 of the revised manuscript reports mean ± std across these runs. Due to space constraints, the table below shows only the mean Avg@4 (%) across the seven benchmarks.

MethodQwen2.5-Math-1.5BQwen2.5-Math-7BQwen3-4B
GRPO33.43±0.9538.59±0.9346.91±2.10
Clip-Higher34.03±0.2741.48±0.5853.15±0.94
CISPO33.80±0.3541.70±0.3651.36±1.19
CE-GPPO29.84±1.3038.77±0.9853.24±1.35
PerCPO35.42±0.4742.85±0.5753.82±1.13

PerCPO’s mean exceeds the strongest baseline on each model by 1.39, 1.15, and 0.58 percentage points, respectively. Appendix B also reports Pass@4 at the same checkpoints and results for Llama-3.2-1B-Instruct and Llama-3.2-3B-Instruct under the same three-seed protocol.

W5/Q5. What do the training dynamics reveal about the accuracy drops in Figure 4(b)?

A5. In Appendix B of the revised manuscript, we compare Qwen3-4B runs with 8 and 16 policy updates per rollout. In the 16-update run, the accuracy drops in Figure 4(b) coincide with spikes in approximate KL, clip fraction, and gradient norm. Policy entropy also changes sharply during this period, briefly surging before falling close to zero. The 8-update run shows much smaller fluctuations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.