- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors study how the clipping mechanism can provide information to the LLM RL training and propose PerCPO, which subtracts the permutation baseline to restore and rescale gradients outside the clipping interval. Experiments on 3 Qwen models across 7 mathematical reasoning benchmarks show improvements over GRPO and the evaluated clipping variants.
Critical strengths
- The paper tackles a important problem for clipping in LLM RL.
- Experiments over multiple benchmarks is provided.
- The method is explained theoretically to some extent.
Critical weaknesses
Experiments
- The experiment setup raises concerns. All the Qwen-Math model can be improved under random reward (arxiv.org/pdf/2506.10947) due to contamination. Over the uncontaminated Qwen3 model, the improvement becomes more marginal. Also, evaluating pass@1 is very limiting to examine RLVR performace. Could you report the avg@16 or avg@8 along with pass@16?
- Clipping information would accumulate over models with very long rollout. However, current experiment only uses 4K-capped Qwen-Math model and non-thinking mode from Qwen3. Whether the method and result can be extended to real reasoning model remains a gap. Current performance improvement might just due to training heuristics, as only single-run result is report.
- It remains uncertain whether the results can be extended to further model families.
- The method is quite close to SAPO (arxiv.org/pdf/2511.20347v2), but the author didn't provide any discussion or comparison to it. Also, the method is close to BAPO, and the author discussed it without any comparison.
Method novelty & soundness
- why the permutation baseline should be treated as a harmful signal that needs to be removed? I don't fully understand. I mean a gradient that survives reward permutation doesn't necessarily mean it's bad, where clipping is exactly designed to constrain policy changes.
- the ablation doesn't justify the benefit of the specific permutation-based correction. A simple baseline is to retain a fixed fraction of the unclipped gradient outside the clipping interval, and also, I think a comparison against SAPO can see this question better.
Questions for authors
See weaknesses section. Please answer the questions and address the concerns.
We thank the reviewer for his/her constructive comments and provide our point-wise replies as follows!
A1. Both Shao et al.’s random-reward experiment and our permutation analysis break the link between response correctness and reward assignment. However, our permutations preserve the observed counts of positive and negative rewards within each prompt group, and hence its reward mean and standard deviation. Shao et al. instead give each response the same fixed probability of a positive reward, independently of the other responses and of the group’s observed correctness rate.
We agree that a gradient surviving reward permutation could still provide useful regularization. To test whether retaining part of B improves accuracy, we added an ablation on Qwen2.5-Math-7B in the revised manuscript (Table 3). We vary the fraction λ of B subtracted from the clipped objective.
Accuracy rises across the three partial corrections, and full subtraction gives the highest score.
In Section 4.2 of the revision, we uniformly permute rewards within each prompt group before every GRPO update on Qwen3-4B, so B is the expected token objective. Policy entropy drops sharply without a sustained Avg@4 gain (41.18% initially, 38.35% at step 200, Figure 1(b)). Together with the scaling ablation, this supports subtracting B.
A2. Compute constraints limited us to four responses per problem. In the revision, Table 1 reports Avg@4 and Appendix B reports Pass@4 at the same checkpoints. For space, we show seven-benchmark Avg@4 (%) as mean ± std over three independent training runs per model–method pair.
PerCPO leads on both metrics for all three models.
A3. In the revised manuscript, we evaluated all five main methods on two Llama models and added comparisons with SAPO and BAPO on both Llama and both Qwen2.5-Math models. The table reports Avg@4 (%) averaged across seven benchmarks, with mean ± std across three independent training seeds.
PerCPO outperforms SAPO and BAPO on all four models and achieves the highest mean Avg@4 among all seven methods on both Llama models. Appendix B of the revised manuscript provides further analysis of SAPO and BAPO.
A4. Due to compute constraints, we did not train Qwen3-4B in thinking mode or any dedicated thinking models, so the cumulative effect of clipping on longer rollouts remains untested. However, Appendix C of the revised manuscript compares Qwen3-4B’s responses to the same math problem after training with each of the five methods in our main experiments. Even in non-thinking mode, the responses from all five models show the kind of step-by-step reasoning often associated with thinking models. In this setting, PerCPO exceeds GRPO by 6.91 percentage points in mean Avg@4 across the seven benchmarks (53.82% vs. 46.91%, Table 1).
A5. In the revised manuscript, we added an ablation on Qwen2.5-Math-7B that replaces the observed correctness rate p of each prompt group with a fixed value c. Each fixed setting retains a fraction 1−c of the unclipped gradient below the lower bound and c above the upper bound.
PerCPO exceeds the best fixed setting tested (c=0.75) by 1.36 percentage points, favoring the use of each group’s observed correctness rate to set the gradient fractions.