acceptodds
Under review as a conference paper at ICLR 2027

Spurious Policy Shift: Do Not Be Too Conservative for Your Policy Optimization

Abstract

Reinforcement learning (RL) has become a key component of post-training for large language models (LLMs), where token-level clipping stabilizes off-policy optimization by controlling token-wise importance sampling (IS) ratios between the current policy and the rollout policy. However, we empirically identify a phenomenon termed spurious policy shift (SPS): a large shift in the sampled token's probability may simply reflect probability mass shifting among contextually interchangeable alternatives at the same decoding state, rather than a meaningful change in the model's generation policy. Furthermore, we observe that SPS-affected tokens occupy a substantial portion of upper-clipped positive advantage updates and suppress policy gain, suggesting that the standard token-level clipping is too conservative and may unnecessarily suppress positive policy updates. Building on this observation, we propose SPS-aware clipping for policy optimization (SCPO) that identifies SPS through probability mass shifting between the sampled token and its contextually interchangeable candidate tokens under the old and new policies, and selectively relaxes clipping for such tokens to avoid suppressing positive policy updates. Experimental results show that our method outperforms state-of-the-art methods on multiple math reasoning benchmarks without compromising training stability. Our code is available at: https://anonymous.4open.science/r/scpo

Primary area:
reinforcement learning
Keywords:
Reinforcement learning, Large Language Model, policy optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_EErbOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper identifies Spurious Policy Shift (SPS), a phenomenon where large token-level probability changes may result from probability redistribution among contextually interchangeable tokens rather than meaningful policy changes. To address the overly conservative clipping in existing policy optimization methods, the authors propose SPS-aware Clipping for Policy Optimization (SCPO), which detects SPS using probability changes between sampled tokens and their contextual candidates and adaptively relaxes the upper clipping bound for positive-advantage updates. Experiments on mathematical reasoning benchmarks with Qwen3-4B and Qwen3-8B demonstrate improvements over existing policy optimization methods while maintaining stable training dynamics.

Critical strengths
  • The author highlights an overlooked limitation of token-level clipping: large probability changes do not necessarily indicate meaningful policy shifts. This provides a potentially valuable perspective for improving policy optimization in LLM reinforcement learning.

  • SCPO achieves competitive or superior performance across multiple mathematical reasoning benchmarks and model scales, while maintaining stable training dynamics. The method is also relatively straightforward to integrate into GRPO frameworks.

  • The paper provides detailed empirical analyses of SPS prevalence and its contribution to suppressed positive-advantage updates during training. These findings help connect the identified phenomenon to the limitations of conventional clipping and provide empirical motivation for the proposed method.

Critical weaknesses
  • The proposed method assumes that high-probability tokens under the old policy are contextually interchangeable, which may not always hold. Moreover, relying on the old-policy Top-K candidates may miss genuine probability redistribution involving previously low-probability tokens. The paper does not independently validate whether the detected SPS cases correspond to genuinely interchangeable tokens or evaluate the precision and recall of the proposed detection criteria. This weakens the evidence supporting the paper's central hypothesis.

  • The SPS detection mechanism relies on several manually specified thresholds and hyperparameters, but the paper provides no systematic sensitivity analysis to justify their values.

Questions for authors
  • How reliable is the assumption that the old policy's Top-K candidates are contextually interchangeable? Could the candidate construction introduce selection bias if probability redistribution has already occurred under the old policy? Have the authors evaluated the detection accuracy of SPS against independently identified SPS cases, for example, through precision and recall? Such evidence would help establish whether the proposed detector captures genuine SPS rather than merely satisfying the probability-based definition.

  • How were the three SPS detection thresholds (τ\tau,γ\gamma,η\eta)) and the candidate set size (K) determined? Can the authors provide sensitivity analyses demonstrating that SCPO remains effective under different hyperparameter settings?

  • Why are the results primarily reported using avg@16 and pass@16? Although avg@16 estimates single-sample accuracy, it would be helpful to clarify whether SCPO's improvements persist under greedy decoding and different sampling budgets (e.g., pass@1, pass@4, and pass@8), particularly given the reported differences in policy entropy.

5 comments

We appreciate the reviewer’s questions about how SPS candidates are selected and how the results depend on the training and evaluation settings. In Appendix D.1 of the revised paper, we explain how and why we construct the candidate set and specify the training thresholds. We report the single-parameter sensitivity analysis in Appendix C.4 and the results across sampling budgets in Appendix C.5. We highlight the additions in blue.

Q1. Candidate validity, old-policy selection bias, and detector accuracy

We select the old policy's Top-KK alternatives to the sampled token as contextual candidates. This lets us reuse rollout probabilities in the current decoding context and track probability mass within the selected set without a separate semantic scoring step. The choice balances candidate coverage and implementation cost. High probability alone cannot guarantee semantic interchangeability. We also considered token embedding similarity as a proxy for semantic similarity. In the candidate-set ablation in Section 5.3, we replaced Top-KK with nearest neighbors in the static embedding space and held the other components fixed. The scores below give equal weight to the six mathematics benchmarks.

Candidate constructionavg@16pass@16
Contextual Top-KK from the rollout policy49.7067.10
Static embedding neighbors43.1658.70

In this ablation, Top-KK improved avg@16 and pass@16 by 6.54 and 8.40 percentage points, respectively. With embedding-based candidates, we also observed late-stage declines in entropy and reward similar to those in GRPO. These observations led us to use Top-KK to select contextual candidates.

We agree with the reviewer that using the old policy can bias candidate coverage. For example, a plausible alternative that had already lost probability mass under the old policy may fall outside the Top-KK set. Independent token-level labels are unavailable in our study, so detector precision and recall remain unmeasured. Many English tokens are fragments of words, which makes their semantics dependent on context. If future work establishes a better criterion for token-level semantic equivalence, we will test it within SCPO.

Q2. How were the thresholds selected, and how sensitive are the results?

We selected a well-performing configuration in preliminary experiments and then fixed it. The SCPO experiments in the main paper used τ=0.0005\tau=0.0005, γ=η=0.995\gamma=\eta=0.995, and K=4K=4. Appendix D.1 now specifies this configuration.

τ\tau controls the minimum probability change, γ\gamma controls how much that change is offset by other candidates, η\eta controls local probability mass conservation, and KK sets the candidate-set size. We varied one parameter at a time and trained each configuration independently for 50 steps, including the default. We used greedy decoding on the same evaluation set and gave equal weight to all six benchmarks.

Parameter change (default → tested)Greedy accuracy (%)Change vs. default (pp)
Default configuration27.790.00
τ\tau from 0.0005 to 0.00126.95−0.84
γ\gamma from 0.995 to 0.99026.56−1.23
η\eta from 0.995 to 0.99026.33−1.46
KK from 4 to 828.58+0.79

Among the tested configurations, increasing τ\tau or lowering γ\gamma or η\eta reduced greedy accuracy by 0.84–1.46 percentage points relative to the default. Increasing KK from 4 to 8 raised accuracy by 0.79 percentage points. If the candidates span the entire vocabulary apart from the sampled token, probability normalization makes the local conservation condition automatic. The minimum probability change threshold still applies, and tracking more candidates increases the work per position. We compare relaxation strategies in Appendix C.1 of the revised paper. Over the tested settings, the changes in accuracy were modest, and the default configuration performed well.

Q3. avg@16, greedy decoding, and sampling budgets

We report avg@16 and pass@16 in the main paper to capture both per-answer accuracy and problem coverage at a common sampling budget. In the supplementary evaluations, SCPO outperformed GRPO under greedy decoding and at every reported sampling budget. The tables report Qwen3-4B scores (%) averaged equally across the six benchmarks.

MethodGreedy pass@1pass@1pass@4pass@8pass@16
GRPO40.8243.2451.4254.4957.38
SCPO49.6852.4460.6863.8967.10
MethodGreedy avg@1avg@1avg@4avg@8avg@16
GRPO40.8239.5041.6241.4441.01
SCPO49.6847.9650.4649.6849.70

SCPO led by 9.20, 9.26, and 9.40 percentage points in sampled pass@1, pass@4, and pass@8, respectively. The gaps in sampled avg@k ranged from 8.24 to 8.84 percentage points. SCPO's advantage was already present under greedy decoding and single-sample decoding, and it persisted at 16 samples per problem.

Reviewer_ZoNhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper identifies spurious policy shift as a phenomena where reinforcement learning algorithms aggressively clip advantages for tokens that have only undergone redistribution of probability mass among contextually-interchangeable tokens. In such cases, the paper argues, the higher advantage value of the specific token does not reflect a meaningful change in model behavior, and thus should not necessarily be clipped. To compute spurious policy shift, the authors of the paper present three metrics. They present SCPO, an algorithm which modifies GRPO to only clip advantages when there is a meaningful policy shift, as computed using these metrics. They present Qwen3-4B model results in the main paper and Qwen3-8B results in the appendix that demonstrates SCPO outperforms baselines like GRPO and DCPO across a variety of math benchmarks.

Critical strengths

The main strength of the paper is its characterization of the spurious policy shift phenomenon - it is well-defined and the authors present a convincing set of metrics to calculate when the difference in old and new token probabilities is meaningful. Their GRPO analysis supports their conclusions that the majority of tokens with large differences in probabilities are due to spurious policy shifts. This is an interesting finding, and warrants further investigation to identify the impact of these spurious shifts on training.

Critical weaknesses

I think the paper has the following critical weaknesses, due to which I am not fully convinced that performance improvements can be attributed purely to using the metrics in SCPO to identify aggressive clipping.

  1. I do not feel the main experiment paper results support the benefits of SCPO, while only some appendix results show an unexplained improvement in performance, which contributes to my negative assessment of the paper despite its interesting findings. The Qwen3-8B results, which show larger performance increases than the Qwen3-4B results, are not reported in the main paper. In the main paper Qwen3-4B results, the pass@16 performance difference between SCPO and the second-best performing algorithm is less than 3% for Minerva, AMC’23, and MATH500, while the avg@16 performance difference between SCPO and DCPO is under 3% for all 6 evaluations. The benefits of SCPO are not clearly demonstrated by the main paper evaluations, and the curves from those training runs reported in Figure 3 are not very informative due to slight performance improvements that may be attributable to evaluation noise. In my opinion, the paper would benefit significantly if the results section is rewritten to better explain the sizeable performance improvements reported in Table 3 in Appendix C (between 2.9-3.75% avg@16 and 3-11% pass@16 performance gap for 4 out of 6 evaluations).
  2. Additionally, I am not sure SCPO is generalizable across model families or task types, given that results are only presented for 2 models of the same model family on math question-answering, with limited performance improvements for one model. There are no evaluations on code generation or non-math question-answering tasks which are commonly used to evaluate the performance of reinforcement learning algorithms for finetuning LLMs. Thus, it seems that while their approach increases overall model entropy, there isn't clear evidence that the increase leads to performance improvements on tasks where response diversity may be much more valuable.
Questions for authors
  1. Why have evaluations been conducted on only one model family and math question-answering tasks? Can the authors provide any evidence for SCPO generalizing across models or tasks?
  2. Why does the paper currently not examine or discuss the Qwen3-8B results in the main paper?
  3. Is it possible that overall model entropy increases due to clipping being relaxed to reinforce tokens in high-entropy positions while continuing to clip low-entropy positions? Can the authors describe the percentage of tokens classified as spurious policy shifts that are present in high-entropy token positions?
5 comments

We thank the reviewer for raising questions about the scope of our evaluation and the relationship between SPS and entropy. In the revised paper, Appendix C.2 presents a mathematical reasoning comparison on Llama-3.2-3B-Instruct, and Appendix C.3 reports an entropy analysis of SPS positions at four training steps. The full Qwen3-8B comparison, including the new DAPO and GSPO results, now appears in the Scaling subsection. The additions are highlighted in blue.

Q1. Why did we choose Qwen and mathematical reasoning, and does SCPO generalize?

We chose Qwen and mathematical reasoning to align with common setups for RL baselines such as DAPO. SCPO's clipping rule operates on policy probabilities and advantages. To test it on a different model family, we compared SCPO with GRPO using Llama-3.2-3B-Instruct as the base model. SCPO outperformed GRPO on both metrics with this backbone as well.

Base modelGRPO avg@16 / pass@16SCPO avg@16 / pass@16
Qwen3-4B41.01 / 57.3849.70 / 67.10
Llama-3.2-3B-Instruct24.95 / 40.6529.37 / 46.21

To assess cross-task generalization, we trained Qwen3-4B with SCPO on instruction-following data and used IFEval to measure how well the model satisfied the constraints in each prompt.

Qwen3-4B methodIFEval score (%)
Base model80.78
GRPO84.33
SCPO88.37

On IFEval, SCPO scored 4.04 percentage points higher than GRPO and 7.59 points higher than the base model. This result provides preliminary evidence that SCPO's gains extend beyond mathematical reasoning to instruction following.

Q2. Why were the 8B results placed in the appendix?

Our compute budget before submission did not allow us to complete the 8B comparisons with DAPO and GSPO, so we placed the available results in the appendix. We have now completed both comparisons and included the full table in the main text of the revised paper. We will retain it there in the final version.

On Qwen3-8B, SCPO achieved the highest equally weighted mean pass@16 across the six benchmarks (70.62%), maintaining its lead from 4B. It also led the token-level methods on both avg@16 and pass@16. GSPO, which uses sequence-level optimization, also performed strongly and achieved the highest mean avg@16 (56.68%). SCPO reached 56.39% on avg@16, only 0.29 percentage points behind GSPO, and led GSPO by 1.56 points on pass@16.

This comparison follows the pattern we observed at 4B, with SCPO leading the token-level methods and GSPO remaining a strong sequence-level baseline. DCPO's performance was less consistent across the two model sizes. Its coarser clipping relaxation may have contributed to this difference.

Qwen3-8B methodAIME’24 avg / passAIME’25 avg / passAMC’23 avg / passMATH500 avg / passMinerva avg / passOlympiadBench avg / passEqually weighted mean avg / pass
GRPO33.96 / 56.7527.08 / 40.7381.41 / 93.7788.40 / 93.7341.11 / 50.6551.03 / 61.3353.83 / 66.16
DAPO36.32 / 58.8929.17 / 43.3383.54 / 95.8390.49 / 95.8043.22 / 52.8253.08 / 63.5655.97 / 68.37
GSPO36.94 / 60.0030.00 / 43.3384.17 / 96.6791.14 / 96.6043.95 / 53.5553.89 / 64.2056.68 / 69.06
DCPO31.67 / 60.0025.63 / 40.0080.47 / 95.0084.51 / 93.8041.27 / 55.1548.50 / 62.2652.01 / 67.70
SCPO37.08 / 66.7230.00 / 51.7385.16 / 95.0090.05 / 94.4141.54 / 50.4454.50 / 65.3956.39 / 70.62

Compared with GRPO, SCPO gained 4.46 percentage points on pass@16 and 2.56 points on avg@16. Its pass@16 lead came primarily from AIME’24 and AIME’25, where it outperformed the strongest baseline by 6.72 and 8.40 points, respectively. It also led by 1.19 points on OlympiadBench. GSPO achieved the highest pass@16 on AMC’23 and MATH500, and DCPO led on Minerva. With 16 responses sampled per problem, SCPO solved a larger share of AIME problems at least once.

Q3. Are detected SPS positions concentrated in the highest-entropy quintile?

Let HH be the top 20% of valid response positions ranked by old-policy entropy, SS the positions identified as SPS, and EE the positions with positive advantage and a policy probability ratio above the original upper clipping bound. We report P(H∣S)P(H\mid S) as requested by the reviewer. We also report P(H∣S∩E)P(H\mid S\cap E) to examine positions directly relevant to upper clipping.

Prespecified training stepSPS positionsPositions in S∩ES\cap EP(H∣S)P(H\mid S)P(H∣all valid positions)P(H\mid\text{all valid positions})P(H∣S∩E)P(H\mid S\cap E)P(H∣E)P(H\mid E)
1145,37011435.1166%20.0001%29.8246%40.8163%
10153,3179934.4234%20.0001%32.3232%37.1901%
100140,83439829.3707%20.0000%23.8693%35.3965%
200120,21633127.7392%20.0000%19.0332%28.4360%

At the four prespecified SCPO training steps, 27.74%–35.12% of detected SPS positions fell in the highest-entropy quintile, above the 20% background rate. Within EE, the high-entropy share of SPS positions was 4.87–11.53 percentage points lower than that of EE as a whole at every step. At each step, the number of SPS positions in EE matched the logged count of positions where relaxation changed the clipped objective.

These results support SCPO's design. SCPO uses compensating probability shifts and local probability mass conservation among contextual candidates to relax the upper bound at selected positions across entropy levels. We leave the causal effect of this selection on subsequent policy entropy to future work.

Reviewer_1Tk8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes "spurious policy shift" (SPS): a large change in a sampled token's probability that is offset by the opposite change in the total mass of its top-K (K = 4) rollout-policy alternatives. Two scores are defined over these alternatives:

  1. Counter-balancing b_i: how much of the token's probability change is offset by the opposite change in candidate mass.
  2. Local conservation q_i: how close the combined mass of the token and its candidates stays to constant.

On Qwen3-4B (DAPO data, six math benchmarks, avg@16 / pass@16), SCPO reaches 49.70 / 67.10, versus 48.53 / 63.76 for DCPO and 46.94 / 63.04 for GSPO. On Qwen3-8B it beats GRPO by +2.56 / +4.35.

Critical strengths

The diagnostic analysis (ρ_change, ρ_SPS, ρ_clip, ρ_gain over training) gives a concrete, measurable view of which positive updates upper clipping removes in later training.

The method is simple, needs only quantities the rollout already provides, plugs into standard GRPO, and is evaluated on two model scales against several recent clipping baselines.

Critical weaknesses

SPS is a real policy change. If the sampled token's probability rises by Δ and the top-4 alternatives lose the same mass, the total-variation distance at that state is |Δ|. The model now emits a different token with probability Δ, which is a genuine change in behavior.

Removing the upper clip ties on the primary metric. In Table 2, "w/o upper clipping" gives the same mean avg@16 as SCPO (49.71 vs 49.70 by my computation). The only gap is pass@16 (64.35 vs 67.10), and it comes mainly from AIME'24 (43.33 vs 56.67), where the difference is 4 of 30 problems.

No variance, a likely weak baseline. All results are single runs with no seeds or variance, so these gaps cannot be judged. GRPO's reward declines after the early stage, and DAPO (which already uses Clip-Higher) is barely above GRPO on avg@16, which suggests the baselines are not well tuned.

Missing comparisons. There is no comparison with a uniformly raised upper bound at the same effective width (λ·ε_high), and no comparison with BAPO, Clip-Cov / KL-Cov, or CISPO.

Questions for authors

What values of λ, ρ, ε_high, and ε_low were used? How many training steps, and how many seeds per method?

How does SCPO compare with Clip-Higher at a fixed width ε_high·λ, and with a uniform upper relaxation applied to a random subset of tokens of the same size as the SPS set?

5 comments

We thank the reviewer for the careful assessment. We provide the hyperparameters and details of the experimental setup below and have added them to Appendix D.1 of the revised paper. SCPO continued to outperform the fixed Clip-Higher and equal-size random controls, supporting the choice of SPS positions for relaxation. We have added these comparisons to Appendix C.1. The additions are highlighted in blue.

Q1. Hyperparameters, training steps, and random seeds

The SCPO runs reported in the paper used the same clipping parameters at both model sizes.

Modelλ\lambdaρ\rhoϵhigh\epsilon_{\mathrm{high}}ϵlow\epsilon_{\mathrm{low}}
Qwen3-4B410.20.2
Qwen3-8B410.20.2

We used training seed 42 for all runs, trained for 2 full epochs, and evaluated the final checkpoint. The training set contains 13,044 examples. We batched 32 prompts at a time and dropped the final incomplete batch, for 407 rollout iterations per epoch and 814 in total. As described in Section 5.1 and Appendix D.1, we sampled 16 responses per problem from the same checkpoint at a temperature of 1.0 and top-pp of 1.0.

Q2. Fixed Clip-Higher and equal-size random relaxation

To address the reviewer’s questions about how much to relax clipping and which tokens should receive relaxation, we compared fixed Clip-Higher and equal-size random relaxation and added a fixed Clip-Higher control restricted to SPS positions. SCPO achieved the best results on both metrics.

Global fixed Clip-Higher yielded a modest gain over GRPO, consistent with the DAPO results in the paper. DAPO uses a fixed increase in ϵhigh\epsilon_{\mathrm{high}} to 0.28. Our control performed slightly below DAPO overall, which we attribute to the additional optimizations in DAPO, including dynamic sampling.

For the equal-size random control, we randomly selected as many tokens as were affected by SPS-based relaxation and applied the SCPO update rule to those tokens. This control performed slightly better than GRPO and still lagged behind SCPO by a substantial margin. For the SPS-only fixed Clip-Higher control, we used the same ϵhigh=0.28\epsilon_{\mathrm{high}}=0.28 at SPS positions. This control performed slightly below SCPO and better than the other controls. These results suggest that focusing clipping relaxation on SPS tokens can improve performance.

We used the final checkpoints for evaluation and sampled 16 responses per problem. Each column reports an equally weighted average across the six benchmarks.

Methodavg@16 (%)pass@16 (%)
GRPO41.0157.38
Fixed Clip-Higher41.3259.37
Equal-size random control42.8360.90
SPS-only fixed Clip-Higher48.9964.41
SCPO49.7067.10

Responses to other weaknesses

Meaning of “spurious” policy shift. SPS still changes the policy. When the probabilities of two candidates shift from (py,pz)(p_y, p_z) to (py+δ,pz−δ)(p_y + \delta, p_z - \delta), the sampled token’s probability ratio is 1+δ/py1 + \delta/p_y. Even a small transfer of probability mass can yield a large ratio when pyp_y is sufficiently small. A ratio computed for one token can obscure compensation among contextual candidates. This local phenomenon motivates selective relaxation. As discussed in the main text, we considered using embeddings and other approaches to assess semantic equivalence. We ultimately chose Top-KK on practical grounds, including ease of implementation and computational complexity. If future work validates a better way to identify semantically equivalent candidates, we will explore it in SCPO.

Removing the upper clipping bound. In the ablation reported in Section 5.3, removing the upper bound left mean avg@16 nearly unchanged and lowered mean pass@16 by 2.75 percentage points. The variant without an upper bound still performed well on the mathematical reasoning tasks. Given the original role of clipping in GRPO, removing the upper bound may affect training stability. Assessing that effect requires larger-scale experiments beyond the scope of this paper. We plan to examine it in future work.

Variance and baselines. We used a single training seed for the experiments in the submitted paper, consistent with most current studies on policy optimization. We omitted Clip-Cov, KL-Cov, and CISPO because their design motivations differ from ours. BAPO shares some of our motivation. In several reproduction attempts before submission, however, we observed unstable training and did not recover the results reported in the BAPO paper. We therefore omitted BAPO from the main text of the paper.

Comments on the paper

2 comments

We thank the reviewers for their careful assessment and constructive feedback. We have consolidated our responses to the main questions raised in the reviews to clarify the paper’s contributions.

1. How does SCPO represent contextual alternatives?

Semantic equivalence is difficult to assess at the token level. Many tokens are fragments of words with little meaning in isolation, and whether two tokens can be substituted without changing meaning depends on the surrounding context. SCPO therefore uses the rollout policy’s Top-KK alternatives to the sampled token as a practical heuristic. These candidates compete under the same prefix, and we can reuse their rollout probabilities to track compensating probability shifts without a separate semantic scoring step.

In the ablation in Section 5.3, Top-KK achieved 49.70% avg@16 and 67.10% pass@16, compared with 43.16% and 58.70% when we substituted static embedding neighbors and held the rest of SCPO fixed. These downstream gains motivated our choice of Top-KK. The candidate set can miss plausible alternatives that already had low probability under the rollout policy, and token-level detector precision and recall remain unmeasured.

2. How did SCPO perform at 8B, and why was the table in the appendix?

Our compute budget before submission limited the 8B comparison, so we placed the available results in the appendix. We have since completed the DAPO and GSPO comparisons and moved the full table to the Scaling subsection of the revised paper. On Qwen3-8B, SCPO achieved the highest mean pass@16 (70.62%) and led the token-level methods on both metrics. GSPO was a strong sequence-level baseline. SCPO’s 56.39% avg@16 was within 0.29 percentage points of GSPO’s 56.68%, and SCPO led GSPO by 1.56 points on mean pass@16.

3. What do the Clip-Higher and random controls show?

With the same upper clipping width of 0.280.28, applying Clip-Higher only at SPS positions scored 48.99% avg@16 and 64.41% pass@16, versus 41.32% and 59.37% when applied globally. Applying SCPO-style relaxation to an equal number of randomly selected positions scored 42.83% and 60.90%. SCPO achieved the highest scores at 49.70% and 67.10%, exceeding SPS-only fixed Clip-Higher by 0.71 and 2.69 percentage points. These comparisons support targeting SPS positions and show a further gain from adaptive relaxation in this setting (Appendix C.1).

Dear all, The discussion phase will end soon. The authors have provided their responses to your comments. Some of you have started involving discussion, but all of us can take action now. If the provided responses do not address (even part of) your concerns, this is your chance to raise it. Please engage with the authors as soon as possible. We really appreciate your efforts! Thanks, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.