- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper identifies Spurious Policy Shift (SPS), a phenomenon where large token-level probability changes may result from probability redistribution among contextually interchangeable tokens rather than meaningful policy changes. To address the overly conservative clipping in existing policy optimization methods, the authors propose SPS-aware Clipping for Policy Optimization (SCPO), which detects SPS using probability changes between sampled tokens and their contextual candidates and adaptively relaxes the upper clipping bound for positive-advantage updates. Experiments on mathematical reasoning benchmarks with Qwen3-4B and Qwen3-8B demonstrate improvements over existing policy optimization methods while maintaining stable training dynamics.
Critical strengths
-
The author highlights an overlooked limitation of token-level clipping: large probability changes do not necessarily indicate meaningful policy shifts. This provides a potentially valuable perspective for improving policy optimization in LLM reinforcement learning.
-
SCPO achieves competitive or superior performance across multiple mathematical reasoning benchmarks and model scales, while maintaining stable training dynamics. The method is also relatively straightforward to integrate into GRPO frameworks.
-
The paper provides detailed empirical analyses of SPS prevalence and its contribution to suppressed positive-advantage updates during training. These findings help connect the identified phenomenon to the limitations of conventional clipping and provide empirical motivation for the proposed method.
Critical weaknesses
-
The proposed method assumes that high-probability tokens under the old policy are contextually interchangeable, which may not always hold. Moreover, relying on the old-policy Top-K candidates may miss genuine probability redistribution involving previously low-probability tokens. The paper does not independently validate whether the detected SPS cases correspond to genuinely interchangeable tokens or evaluate the precision and recall of the proposed detection criteria. This weakens the evidence supporting the paper's central hypothesis.
-
The SPS detection mechanism relies on several manually specified thresholds and hyperparameters, but the paper provides no systematic sensitivity analysis to justify their values.
Questions for authors
-
How reliable is the assumption that the old policy's Top-K candidates are contextually interchangeable? Could the candidate construction introduce selection bias if probability redistribution has already occurred under the old policy? Have the authors evaluated the detection accuracy of SPS against independently identified SPS cases, for example, through precision and recall? Such evidence would help establish whether the proposed detector captures genuine SPS rather than merely satisfying the probability-based definition.
-
How were the three SPS detection thresholds (,,)) and the candidate set size (K) determined? Can the authors provide sensitivity analyses demonstrating that SCPO remains effective under different hyperparameter settings?
-
Why are the results primarily reported using avg@16 and pass@16? Although avg@16 estimates single-sample accuracy, it would be helpful to clarify whether SCPO's improvements persist under greedy decoding and different sampling budgets (e.g., pass@1, pass@4, and pass@8), particularly given the reported differences in policy entropy.
5 comments
Q2. How were the thresholds selected, and how sensitive are the results?
We selected a well-performing configuration in preliminary experiments and then fixed it. The SCPO experiments in the main paper used , , and . Appendix D.1 now specifies this configuration.
controls the minimum probability change, controls how much that change is offset by other candidates, controls local probability mass conservation, and sets the candidate-set size. We varied one parameter at a time and trained each configuration independently for 50 steps, including the default. We used greedy decoding on the same evaluation set and gave equal weight to all six benchmarks.
| Parameter change (default → tested) | Greedy accuracy (%) | Change vs. default (pp) |
|---|---|---|
| Default configuration | 27.79 | 0.00 |
| from 0.0005 to 0.001 | 26.95 | −0.84 |
| from 0.995 to 0.990 | 26.56 | −1.23 |
| from 0.995 to 0.990 | 26.33 | −1.46 |
| from 4 to 8 | 28.58 | +0.79 |
Among the tested configurations, increasing or lowering or reduced greedy accuracy by 0.84–1.46 percentage points relative to the default. Increasing from 4 to 8 raised accuracy by 0.79 percentage points. If the candidates span the entire vocabulary apart from the sampled token, probability normalization makes the local conservation condition automatic. The minimum probability change threshold still applies, and tracking more candidates increases the work per position. We compare relaxation strategies in Appendix C.1 of the revised paper. Over the tested settings, the changes in accuracy were modest, and the default configuration performed well.
Q3. avg@16, greedy decoding, and sampling budgets
We report avg@16 and pass@16 in the main paper to capture both per-answer accuracy and problem coverage at a common sampling budget. In the supplementary evaluations, SCPO outperformed GRPO under greedy decoding and at every reported sampling budget. The tables report Qwen3-4B scores (%) averaged equally across the six benchmarks.
| Method | Greedy pass@1 | pass@1 | pass@4 | pass@8 | pass@16 |
|---|---|---|---|---|---|
| GRPO | 40.82 | 43.24 | 51.42 | 54.49 | 57.38 |
| SCPO | 49.68 | 52.44 | 60.68 | 63.89 | 67.10 |
| Method | Greedy avg@1 | avg@1 | avg@4 | avg@8 | avg@16 |
|---|---|---|---|---|---|
| GRPO | 40.82 | 39.50 | 41.62 | 41.44 | 41.01 |
| SCPO | 49.68 | 47.96 | 50.46 | 49.68 | 49.70 |
SCPO led by 9.20, 9.26, and 9.40 percentage points in sampled pass@1, pass@4, and pass@8, respectively. The gaps in sampled avg@k ranged from 8.24 to 8.84 percentage points. SCPO's advantage was already present under greedy decoding and single-sample decoding, and it persisted at 16 samples per problem.
We appreciate the reviewer’s questions about how SPS candidates are selected and how the results depend on the training and evaluation settings. In Appendix D.1 of the revised paper, we explain how and why we construct the candidate set and specify the training thresholds. We report the single-parameter sensitivity analysis in Appendix C.4 and the results across sampling budgets in Appendix C.5. We highlight the additions in blue.
Q1. Candidate validity, old-policy selection bias, and detector accuracy
We select the old policy's Top-K alternatives to the sampled token as contextual candidates. This lets us reuse rollout probabilities in the current decoding context and track probability mass within the selected set without a separate semantic scoring step. The choice balances candidate coverage and implementation cost. High probability alone cannot guarantee semantic interchangeability. We also considered token embedding similarity as a proxy for semantic similarity. In the candidate-set ablation in Section 5.3, we replaced Top-K with nearest neighbors in the static embedding space and held the other components fixed. The scores below give equal weight to the six mathematics benchmarks.
In this ablation, Top-K improved avg@16 and pass@16 by 6.54 and 8.40 percentage points, respectively. With embedding-based candidates, we also observed late-stage declines in entropy and reward similar to those in GRPO. These observations led us to use Top-K to select contextual candidates.
We agree with the reviewer that using the old policy can bias candidate coverage. For example, a plausible alternative that had already lost probability mass under the old policy may fall outside the Top-K set. Independent token-level labels are unavailable in our study, so detector precision and recall remain unmeasured. Many English tokens are fragments of words, which makes their semantics dependent on context. If future work establishes a better criterion for token-level semantic equivalence, we will test it within SCPO.