acceptodds
Under review as a conference paper at ICLR 2027

Conformal Prediction-Enabled Sequence-level Policy Optimization for Reasoning Language Models

Abstract

Reinforcement learning improves the reasoning capability of large language models through verifiable rewards for mathematical and coding tasks. In token-level methods such as GRPO, clipping acts on individual token ratios while the reward and advantage apply to a complete response. Recent sequence-level methods align these units but still use fixed clipping bands, whose tail probabilities change as the score distribution evolves. We propose *Conformal Prediction-Enabled Sequence-level Policy Optimization* (CSPO), which adaptively calibrates sequence-level clipping bounds using recent training statistics. The resulting band targets a specified marginal tail budget and adapts to policy drift; calibration conditional on response length requires additional assumptions. We bound the probability that a fresh score falls outside the band by the target miscoverage level plus explicit CDF discrepancy and drift terms. Under additional moment conditions, this bound also controls the objective and gradient error relative to the unclipped GSPO power surrogate. Experiments with Qwen3-1.7B series show improvements over GSPO on difficult mathematical reasoning benchmarks and ALFWorld, with clipping endpoints that adapt during training.

Primary area:
reinforcement learning
Keywords:
reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_W396OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Fixed clipping intervals can become mismatched with the changing distribution of policy ratios during reinforcement learning. Motivated by this limitation, the paper proposes Conformal Prediction-Enabled Sequence-level Policy Optimization (CSPO), which replaces GSPO’s fixed clipping bounds with empirical quantiles of sequence-level power scores collected across recent training steps. A rolling buffer determines the bounds according to a prescribed marginal tail budget. The analysis bounds the probability that a newly sampled score falls outside the clipping interval using the target miscoverage level, empirical CDF discrepancies, and policy drift. Under additional assumptions, it also bounds the objective and gradient error relative to the unclipped GSPO surrogate. Experiments with Qwen3-1.7B variants report an average mathematical Pass@1 of 34.11%, compared with GSPO’s 33.36%, and an ALFWorld overall success rate of 64.06%, compared with 53.12% for the strongest reported baseline.

Critical strengths

CSPO presents a conceptually clear and simple extension of GSPO. By using recent training scores to adapt clipping bounds, it addresses changes in the score distribution during RL training. The experiments cover both interactive tasks in ALFWorld and mathematical reasoning, with encouraging overall results. The analysis also explicitly accounts for data reuse and policy drift, and carefully limits its optimization conclusions to comparisons with the GSPO surrogate.

Critical weaknesses
  1. The abstract and opening introduction emphasize the mismatch between token-level clipping and sequence-level rewards. However, GSPO already moves clipping to the sequence level, while CSPO’s main contribution is replacing fixed clipping bounds with empirical quantiles from recent training steps. The paper should motivate this specific design more directly and explain why the proposed tail budget is an appropriate quantity to control.

  2. The experiments exclude relevant adaptive-clipping methods such as BAPO, BandPO, and ACPO, despite discussing them in the appendix. Without these comparisons and details of how GSPO’s clipping bounds were tuned, it is difficult to determine whether the gains come from the proposed adaptation or simply better clipping bounds. Evaluation is also restricted to Qwen3 variants at the 1.7B scale.

  3. The improvements are uneven. On ALFWorld, Pick shows a substantial gain, while Look and Heat tie the strongest baselines and Clean and Pick2 remain below their respective best baselines. The mathematical average improves over GSPO by only 0.75 percentage points. Results across multiple independent training runs are not reported, leaving the reliability of these gains unclear.

  4. The theoretical results bound the objective and gradient error relative to the unclipped GSPO surrogate, but do not establish why the proposed clipping rule provides effective regularization. Removing clipping trivially eliminates this error, as the paper acknowledges. The analysis does not explain how the chosen tail budget balances training stability against the removal of useful gradients, especially when samples outside the interval have different advantages and gradient magnitudes.

  5. The training curves provide limited evidence for the proposed explanation. Figure 5 reports overall clip fractions, but does not separate clipping at the upper and lower bounds or compare these curves with GSPO using fixed bounds. These measurements would help connect the changing clipping bounds to the stated motivation.

Questions for authors
  1. Why were the adaptive-clipping methods discussed in the appendix excluded from the experiments? How does CSPO compare with methods such as BAPO, BandPO, or ACPO under the same training conditions? Please also specify how GSPO’s fixed clipping bounds were selected and tuned. This would help determine whether adaptation improves performance beyond choosing better clipping bounds.

  2. What supports applying empirical quantiles to sequence-level scores rather than token-level ratios? Have you evaluated the same method for selecting clipping bounds with token-level clipping? This comparison would help distinguish the benefit of adaptive bounds from that of the GSPO sequence-level formulation.

  3. How many independent training runs and evaluation samples were used for Tables 1 and 2, and how were the reported checkpoints selected? Can you provide the mean and standard deviation across runs for CSPO and the baselines? Please also report the ALFWorld task counts and explain how the All score is calculated, since it is not the simple average of the six task scores.

  4. Can you separately report the fractions of scores below the lower bound and above the upper bound across training steps? Please also distinguish these from the fractions actually clipped: positive-advantage samples above the upper bound and negative-advantage samples below the lower bound. Comparing these curves with GSPO using fixed bounds would help assess whether adaptation reduces excessive or insufficient clipping.

  5. How should the objective and gradient error bounds guide the choice of α\alpha? Since unclipped updates have zero clipping error, what theoretical or empirical evidence shows that the selected tail budget preserves useful gradients while preventing unstable updates? In particular, how does the fraction of scores outside the interval relate to the gradient magnitude removed by clipping and to training stability?

4 comments

[W1, W2, Q1, Q2] Motivation, adaptive clipping comparisons, and token ratios.

Thank you for pointing out these questions. We will make the clipping pattern the starting point of the abstract and introduction: fixed bounds can impose substantially different amounts of clipping at different training stages. CSPO uses recent score quantiles to maintain more consistent tail coverage as the policy evolves. The motivation is to avoid excessive early restriction while keeping clipping active later.

We have now implemented the core clipping rules of BAPO, BandPO, and ACPO from their public code in the same ALFWorld trainer, and completed the requested token comparison, which we call token-level CPO. The adaptive baseline experiments follow the CSPO reference configuration. Evaluation uses 128 episodes every five iterations.

MethodPickLookCleanHeatCoolPick2All
GSPO58.0666.6745.0061.5422.2220.8331.25
DAPO76.9270.0072.0075.0056.5233.3353.12
BAPO70.0057.1473.3381.8245.4530.0051.56
BandPO76.4761.5470.3772.7354.8470.0056.25
ACPO75.8657.1477.7856.2539.1342.1146.09
CSPO93.3370.0068.0075.0060.0044.0064.06
token-level CPO85.2972.7385.1981.8265.0075.0068.75

All entries are success rates in percent. As in Table 1, each column is its historical maximum over evaluation steps, including the evaluation before training. The maxima in one row can come from different checkpoints.

CSPO has a higher All score than BAPO, BandPO, and ACPO. The added token-level CPO achieves 68.75% All, compared with 64.06% for CSPO, and improves five of the six category scores over CSPO; CSPO remains stronger on Pick. These results show that the quantile rule also works with token ratios, with token-level CPO performing best overall in this ALFWorld comparison. The full results are in Appendix D.1.4, Table 9 of the revised paper.

Sequence scores give the tail budget a direct interpretation at the response level: it specifies a target fraction of response scores outside the clipping interval, aligning the calibration unit with the reward and advantage. With token ratios, the budget instead targets token tail mass, which can be concentrated in a few responses or spread across many. The new result supports applying quantile calibration at either granularity. Our CSPO versus GSPO comparison keeps the sequence score W=exp⁡(S/L)W=\exp(S/L) and surrogate fixed to examine the effect of adapting the bounds.

The GSPO bounds and their additional comparison are addressed in our response to Reviewer hWK6, W1. The completed 4B results appear in our response to Reviewer r34E, W1/Q1/Q2.

[W3, Q3] Uneven gains, independent runs, checkpoint selection, and ALFWorld task counts.

The ALFWorld gains differ by task. In the original table, CSPO's largest advantage is on Pick, while DAPO is stronger on Clean and GRPO on Pick2. In the added comparison, BAPO and ACPO are stronger on Clean, while BandPO is stronger on Pick2. We will describe the results at this level rather than suggesting that CSPO improves every task.

The math results for seeds 42, 314, and 271 are listed in our response to Reviewer r34E and in Appendix D.1.2, Table 7.

At an ALFWorld evaluation step tt, let nc(t)n_c(t) be the number of episodes of category cc, and let sc(t)s_c(t) be its number of successes. Then

=100\,\frac{\sum_c s_c(t)}{\sum_c n_c(t)}, \qquad \sum_c n_c(t)=128.$$ Thus All is weighted by episode counts at that evaluation. The reported All is $\max_t\mathrm{All}(t)$; it is not the average of the six category maxima. Category counts can change when the environments reset. | Method | Successful episodes / total | |---|---:| | GSPO | 40 / 128 | | DAPO | 68 / 128 | | BAPO | 66 / 128 | | BandPO | 72 / 128 | | ACPO | 59 / 128 | | CSPO | 82 / 128 | > *[W4, Q5] Clipping and training stability.* Our choice $\alpha=0.1$, $K=2$ has been discussed in **Appendix C.3**. There is a trade-off in choosing $\alpha$: a smaller $\alpha$ gives a wider clipping interval and allows more updates to remain unclipped, but as $\alpha \to 0$ the interval becomes too loose and CSPO approaches an effectively unclipped update, risking unstable or collapsed training. Conversely, a larger $\alpha$ narrows the interval and makes clipping more aggressive, which can over-constrain policy updates and limit learning. **Figure 12 (Appendix D.5)** shows smaller early gradient fluctuations for CSPO than GSPO: the standard deviation over steps 1–150 is **0.135 versus 0.150**. Both methods settle into smooth trajectories during the middle stage. Unclipped training develops large late spikes, reaching **1.44 and 5.91**, while GSPO and CSPO remain near 0.1. Together with the reward curves in **Figure 11 (Appendix D.4)** and clipping patterns in **Figure 8 (Appendix D.2)**, these observations support our interpretation: fixed GSPO clipping may restrict early learning, while removing clipping entirely permits occasional large gradients later. CSPO maintains moderate clipping throughout training, balancing early learning with later stability. > *[W5, Q4] Separating both tails from actual clipping.* **Figure 8 (Appendix D.2)** separately plots $\widehat P(W<a)$, $\widehat P(W>b)$, $\widehat P(A<0,W<a)$, and $\widehat P(A>0,W>b)$ for GSPO and CSPO at both model sizes. All four use the same denominator of valid response evaluations; the last two are joint fractions whose sum gives actual clipping. CSPO maintains balanced tails and a small, persistent actual clipping fraction, while GSPO's fractions change more across training.

Thank you for the additional experiments and clarifications. The adaptive clipping comparisons, token level results, and training diagnostics address several of my concerns. I will therefore raise my score.

One final question concerns the scope of the study. Since the quantile calibration rule also applies to token level clipping, why was the method originally formulated specifically at the sequence level? Did it originate from observations of GSPO’s optimization behavior? The stronger reported token level CPO result makes me interested in understanding how this scope was determined.

Thank you for your thoughtful follow-up and for recognizing the value of our additional experiments. We appreciate the opportunity to further clarify the motivation behind our initial sequence-level formulation. The original sequence-level formulation was motivated by observations of GSPO's clipping behavior. With fixed bounds, the fraction of response scores outside the interval was initially high and declined during training, motivating calibration against recent score distributions to target more consistent marginal tail coverage.

The sequence-level setting provided a controlled comparison: retaining GSPO's score, surrogate, and clipping granularity isolated the effect of adaptive versus fixed bounds. It also has a natural modeling rationale in single-turn mathematical RLVR. Unlike multi-step MDPs involving environment interactions, such tasks can be formulated as contextual bandit problems: given a prompt (context), the policy generates a complete response (action) and receives a verifiable reward upon completion. Thus, the response is a natural unit for quantile calibration, giving the tail budget a direct interpretation as a target marginal fraction of response scores outside the interval. This does not imply an inherent advantage over token-level calibration.

The additional experiment demonstrates the broader applicability of the quantile rule. Under the historical-maximum evaluation protocol in Table 9, token-level CPO achieves 68.75% All success on ALFWorld, exceeding CSPO's 64.06%. The rule therefore supports both granularities, although their tail budgets have different units.

The revision will distinguish the empirical origin in GSPO, the initial choice of sequence-level optimization for controlled evaluation, and the broader applicability demonstrated by token-level CPO.

Reviewer_r34EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CSPO adapts GSPO’s clipping bounds using empirical quantiles of recent sequence-level scores. The paper bounds the probability that a fresh score falls outside the interval, accounting for empirical-to-population CDF discrepancies and distribution drift. Under additional assumptions, it also bounds how much clipping changes the unclipped GSPO objective and gradient. Experiments cover mathematical reasoning and ALFWorld, both with Qwen3 models at the 1.7B scale.

Critical strengths
  1. The method is simple and practical. Rolling quantiles let the clipping interval change with the score distribution instead of keeping the same numerical bounds throughout training. This is straightforward to implement, and Figure 6 reports training-step and actor-update times of a similar order to GSPO.

  2. The reported ALFWorld improvement is substantial. CSPO achieves 64.06% overall success, compared with 53.12% for DAPO. This is the strongest empirical result in the paper and makes the method worth investigating further.

Critical weaknesses
  1. The evidence for a reliable advantage over GSPO is limited. CSPO improves all six math benchmarks, but the average gain is only 0.75 percentage points, with a 0.24-point gain on AIME25. Without uncertainty estimates across training runs, it is hard to judge how reliable these improvements are. The ALFWorld result is stronger, but both settings use only 1.7B models, leaving the benefit at larger scales untested. Since the main change is the choice of clipping bounds, a comparison with well-tuned fixed GSPO bounds is central to assessing the contribution. The paper does not report those baseline values or how they were selected. Comparisons with related adaptive clipping methods would also help establish the value of the quantile rule.

  2. The calibration theory has limited empirical support. The tail bound depends on empirical-to-population CDF discrepancies and distribution drift, whose magnitudes are not measured in the experiments. Figure 5 provides a useful training-score diagnostic, but does not establish tail control on independent fresh scores. It therefore remains unclear whether the bound is informative during training. The optimization analysis characterizes distortion relative to the unclipped GSPO surrogate; it does not establish why the quantile rule should outperform fixed clipping. A direct evaluation of fresh-score tail exceedance would make the connection between the theory and the experiments more convincing.

Questions for authors
  1. How consistent are the math gains across independent training runs? Results with uncertainty estimates would help judge whether the small improvements over GSPO are reliable.

  2. How were GSPO’s clipping bounds selected? A comparison with tuned fixed bounds would help determine whether rolling quantiles provide an advantage beyond choosing a better clipping interval. Results on a 4B or 8B model would also help assess whether that advantage extends beyond the 1.7B setting.

  3. Beyond the training-score diagnostic in Figure 5, is there evidence that the clipping interval controls tail exceedance on independent fresh scores? This would help assess whether the theoretical bounds are informative during training.

2 comments

[W1, Q1, Q2] Independent training runs, fixed GSPO bounds, and a larger model.

Thank you for these questions. The GSPO and CSPO results in Table 2 use seed 42. We added runs with seeds 314 and 271 for both methods, using the same budget of 500 training steps.

SeedMethodAIME24AIME25AMCMATH500MinervaOly.Average
42GSPO15.088.3342.8673.3622.6837.8333.36
42CSPO16.878.5743.3674.2223.1938.4434.11
314GSPO13.777.7240.7972.4621.1036.4932.05
314CSPO14.259.0342.1673.4022.1637.1533.03
271GSPO13.278.3240.7772.0721.0836.1031.93
271CSPO15.359.6041.9973.6022.6337.1033.38

CSPO improves all six benchmarks for each of the three seeds. Across the three seeds, Average Pass@1 is 33.50 ± 0.55% for CSPO and 32.45 ± 0.79% for GSPO (mean ± sample standard deviation). The complete comparison appears in Appendix D.1.2, Table 7 of the revised paper.

Scores are Pass@1 percentages based on 200 responses per problem. Each benchmark reports its highest observed score, and Average is computed from the six unrounded benchmark scores.

The fixed GSPO comparison is described in our response to Reviewer hWK6, W1. We have also completed the requested larger model experiment with Qwen3-4B-Base. Both methods use training seed 314 and a budget of 200 training steps. CSPO uses β=1\beta=1, α=0.1\alpha=0.1, and K=2K=2.

The following results use the final checkpoint at step 200 for both methods. We sample 200 responses per problem with temperature 0.8, top-p 1.0, and a maximum response length of 8192 tokens.

MethodAIME24AIME25AMCMATH500MinervaOly.Average
GSPO18.4012.4251.7079.3230.9345.9139.78
CSPO18.4315.3052.5480.1333.3845.7240.92

CSPO improves the average by 1.14 percentage points, with gains on five of the six benchmarks. The largest gains are on AIME25 and Minerva; GSPO is slightly stronger on OlympiadBench. The results appear in Appendix D.1.3, Table 8 of the revised paper; the corresponding training reward curves are in Appendix D.4, Figure 11.

[W2, Q3] Connecting calibration and optimization theory to the experiments.

The added experiments connect the theory to the clipping pattern experienced during optimization. Appendix D.2, Figures 8 and 9 show that fixed GSPO clipping restricts a larger share of weighted contributions early in training and becomes less restrictive later. CSPO maintains more consistent empirical tail coverage and a modest weighted clipping fraction across stages. This pattern offers a plausible explanation for its advantage over the tested fixed GSPO setting: it avoids much of the early restriction while retaining stable clipping as training proceeds.

Appendix D.3, Figure 10 examines how the rules treat the same recorded scores and advantages. The reward and gradient comparisons in Appendix D.4, Figure 11 and Appendix D.5, Figure 12 are discussed in our response to Reviewer hWK6, W4, alongside the unclipped benchmark results.

We have now evaluated independent fresh score tail exceedance for CSPO at three stages of training, with a target of α=0.1\alpha=0.1:

Training stageFresh tail exceedance95% confidence interval
Early12.89%11.55%–14.26%
Middle12.48%11.21%–13.82%
Late13.75%12.44%–15.10%

Fresh tail exceedance stays between 12.48% and 13.75% across these stages. This supports a consistent tail pattern on independent responses, with an upward deviation of 2.48–3.75 percentage points from the nominal 10% target. The theoretical bound retains CDF discrepancy and drift terms to account for departures from that target.

We argue that the training clipping diagnostics also carry direct information about optimization. The actual clipping fractions measure the proportion of training responses satisfying A>0,W>bA>0,W>b or A<0,W<aA<0,W<a, whose surrogate contributions enter a clipped branch. The separately reported tail fractions count scores outside the interval regardless of advantage. Together, these quantities show the clipping applied during training and how its pattern changes as learning proceeds.

Thank you for the additional experiments. The three-seed results and fixed-bound comparisons help address my empirical concerns. However, the 4B improvement is modest and mainly concentrated on AIME25 and Minerva. The clipping curves support a plausible explanation, but it doesn't necessarily mean that more stable clipping causes the performance gains. Could the gains instead mainly come from less restrictive clipping early in training?

Reviewer_hWK6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This works studied improving the GSPO algorithm by adaptively control its clipping to achieve better performance. The canonical GSPO performs a fixed clipping hyperparameter on the length-normalized tempered sequence-level importance sampling ratio. However, as the training process continues, such clipping coefficient may not be effective, too aggressive or too loose. The proposed CSPO algorithm uses quantile to set the clipping coefficient. The quantile is estimated by the empirical distribution from the tempered ratios in the recent rollouts, and hyperparamter is the tail budget. With tune of the budget parameter, the estimated quantile can adaptively control the clipping coefficient. In experiments, CSPO gives better performance compared to GSPO and other clipping-based counterparts, like CISPO. Under controllable CDF drift assumption, the probability of the ratio falls in the estimated region can be controlled by the budget hyperparameter, and the gap between the CSPO objective and CSPO-without-clipping objective and their gradients can both be bounded by the budget hyperparameter.

Critical strengths

The idea is intuitive and easy to follow. The theorem as far as I known is correct. The performance improvement in Alfworld is decent.

Critical weaknesses

I didn't find the sweep of hyperparameter selection for GSPO counterpart. Essentially the CSPO is a dynamic clipping version of GSPO, would be good to show such dynamic is better than the best of static. I didn't find the hyperparameter optimization of GSPO baseline in the work, apologies if I miss it.

The illustration of the clipping fraction dynamics during the training for GSPO would be a good metric comparing with CSPO.

It is interesting to see that in the initial phase of the training for DAPO-Math data in Figure 2, GSPO is actually left behind but coming up over time. Does it indicate the quantile with fixed budget may not be the fundamental conservative quantity that leads to better performance?

Another good reason of using clipping besides efficiency is stability. Any stress test of extreme off-policynss to show the adaptivity in CSPO also work?

This may sound like a boilerplate, but I don't see the value of the theorems claimed in the work. It is basic estimation + gradual drift assumption.

Questions for authors

Please find first 4 comments in Critical Weaknesses.

2 comments

[W1] Hyperparameter selection for GSPO.

Thank you for raising this question. In Tables 1 and 2, GSPO uses ϵlow=3×10−4\epsilon_{\mathrm{low}}=3\times10^{-4} and ϵhigh=4×10−4\epsilon_{\mathrm{high}}=4\times10^{-4}, following the setting in Section 5.1 of the GSPO paper. These values give the interval [0.9997,1.0004][0.9997,1.0004].

We have completed training and evaluation for two additional GSPO configurations with half and twice these widths on DAPO-Math-17k. The table below compares the three fixed settings and CSPO. The default GSPO and CSPO rows reproduce Table 2.

MethodAIME24AIME25AMCMATH500MinervaOly.Average
GSPO, (1.5×10−4,2×10−4)(1.5\times10^{-4},2\times10^{-4})14.578.8042.6073.5822.1037.0633.12
GSPO, (3×10−4,4×10−4)(3\times10^{-4},4\times10^{-4})15.088.3342.8673.3622.6837.8333.36
GSPO, (6×10−4,8×10−4)(6\times10^{-4},8\times10^{-4})14.938.5242.3472.9722.7537.2133.12
CSPO, α=0.1\alpha=0.1, K=2K=216.878.5743.3674.2223.1938.4434.11

Values are Pass@1 percentages, reporting the highest observed score for each benchmark. Average is the arithmetic mean of the six benchmark scores.

Both GSPO, (1.5×10−4,2×10−4)(1.5\times10^{-4},2\times10^{-4}) and GSPO, (6×10−4,8×10−4)(6\times10^{-4},8\times10^{-4}) achieve an average of 33.12, compared with 33.36 for the original GSPO setting and 34.11 for CSPO. The original setting is therefore the strongest of the three fixed settings tested. We have added this comparison in Appendix D.1.1, Table 6 of the revised paper.

[W2, W3] Clipping dynamics and the reward curves.

Our interpretation is that CSPO's more moderate and stable clipping pattern helps explain its advantage over the tested fixed GSPO setting. The relevant pattern includes which responses reach the clipped branches and how the weighted clipping fraction changes over training, with each response weighted by Wi∣Ai∣W_i|A_i|. We have added these trajectories for the completed 1.7B and 4B runs in Appendix D.2, Figures 8 and 9.

GSPO has a large weighted clipping fraction early in training, which decreases as training proceeds. In the 1.7B run, this fraction falls from about 28% in the early window to 17% in the late window. CSPO stays near 5% across the early, middle, and late windows. This suggests that fixed GSPO clipping may be overly conservative early on, suppressing many weighted contributions while the policy is learning the task. As the score distribution contracts, its fixed bounds impose less restriction, providing a plausible explanation for GSPO's later improvement.

CSPO maintains similar empirical coverage of both tails while continuing to clip a stable share of contributions throughout training. This gives it a more consistent clipping pattern across stages: learning can proceed with less early restriction, while clipping remains active as the policy evolves. The four fractions in Figure 8 distinguish this persistent clipping from merely falling outside the interval.

Appendix D.4, Figure 11 now shows two reward comparisons. The left panel combines the original 1.7B CSPO and default GSPO curves with the additional unclipped run: removing clipping gives faster early learning, while CSPO achieves higher reward through much of the middle and later stages. The right panel compares CSPO and GSPO at 4B. These 4B runs also supply the clipping diagnostics: CSPO maintains higher reward through most of training, together with more consistent weighted clipping fraction and a higher final benchmark average.

[W4] The role of clipping and stability under greater policy mismatch.

We have completed a 500-step unclipped GSPO comparison on Qwen3-1.7B-Base. We set α=0\alpha=0 with β=1\beta=1 to disable policy ratio clipping; the remaining model, data, and training hyperparameters follow the main math experiments in Table 2.

Figure 11 (Appendix D.4) shows that unclipped training improves reward faster early on, but CSPO achieves higher reward through much of the middle and later stages. Figure 12 (Appendix D.5) also shows late gradient norm spikes in the unclipped run, while GSPO and CSPO remain smooth. Together, these observations support retaining moderate clipping as training proceeds.

The benchmark comparison is:

MethodAIME24AIME25AMCMATH500MinervaOly.Average
GSPO15.088.3342.8673.3622.6837.8333.36
CSPO16.878.5743.3674.2223.1938.4434.11
Unclipped, α=0\alpha=013.928.6341.8373.4121.9937.2332.84

[W5] What do the theoretical results contribute?

The theory explains how a quantile estimated from past training scores relates to the scores encountered during the next step. The central decomposition is

ptband≤α+2εt+2Dt,p_t^{\mathrm{band}}\leq \alpha+2\varepsilon_t+2D_t,

where εt\varepsilon_t measures the discrepancy between the retained empirical CDFs and their population counterparts, and DtD_t measures the change from the retained score distribution to the current one. This makes explicit the two reasons why the observed tail mass can differ from α\alpha: estimation from training data and policy drift.

The tail budget gives the clipping rule a target that remains comparable as the score distribution changes. The objective and gradient bounds then describe the optimization cost of imposing that clipping, under their stated moment assumptions. Appendix D.2 measures the corresponding pattern through tail exceedance, actual clipping, weighted clipping fraction, and surrogate objective gap. The empirical question is whether this pattern supports learning across stages; the reward and benchmark comparisons address that question. The calibration statement retains the CDF discrepancy and drift terms because the scores are recorded along the training path.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.