- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Fixed clipping intervals can become mismatched with the changing distribution of policy ratios during reinforcement learning. Motivated by this limitation, the paper proposes Conformal Prediction-Enabled Sequence-level Policy Optimization (CSPO), which replaces GSPO’s fixed clipping bounds with empirical quantiles of sequence-level power scores collected across recent training steps. A rolling buffer determines the bounds according to a prescribed marginal tail budget. The analysis bounds the probability that a newly sampled score falls outside the clipping interval using the target miscoverage level, empirical CDF discrepancies, and policy drift. Under additional assumptions, it also bounds the objective and gradient error relative to the unclipped GSPO surrogate. Experiments with Qwen3-1.7B variants report an average mathematical Pass@1 of 34.11%, compared with GSPO’s 33.36%, and an ALFWorld overall success rate of 64.06%, compared with 53.12% for the strongest reported baseline.
Critical strengths
CSPO presents a conceptually clear and simple extension of GSPO. By using recent training scores to adapt clipping bounds, it addresses changes in the score distribution during RL training. The experiments cover both interactive tasks in ALFWorld and mathematical reasoning, with encouraging overall results. The analysis also explicitly accounts for data reuse and policy drift, and carefully limits its optimization conclusions to comparisons with the GSPO surrogate.
Critical weaknesses
-
The abstract and opening introduction emphasize the mismatch between token-level clipping and sequence-level rewards. However, GSPO already moves clipping to the sequence level, while CSPO’s main contribution is replacing fixed clipping bounds with empirical quantiles from recent training steps. The paper should motivate this specific design more directly and explain why the proposed tail budget is an appropriate quantity to control.
-
The experiments exclude relevant adaptive-clipping methods such as BAPO, BandPO, and ACPO, despite discussing them in the appendix. Without these comparisons and details of how GSPO’s clipping bounds were tuned, it is difficult to determine whether the gains come from the proposed adaptation or simply better clipping bounds. Evaluation is also restricted to Qwen3 variants at the 1.7B scale.
-
The improvements are uneven. On ALFWorld, Pick shows a substantial gain, while Look and Heat tie the strongest baselines and Clean and Pick2 remain below their respective best baselines. The mathematical average improves over GSPO by only 0.75 percentage points. Results across multiple independent training runs are not reported, leaving the reliability of these gains unclear.
-
The theoretical results bound the objective and gradient error relative to the unclipped GSPO surrogate, but do not establish why the proposed clipping rule provides effective regularization. Removing clipping trivially eliminates this error, as the paper acknowledges. The analysis does not explain how the chosen tail budget balances training stability against the removal of useful gradients, especially when samples outside the interval have different advantages and gradient magnitudes.
-
The training curves provide limited evidence for the proposed explanation. Figure 5 reports overall clip fractions, but does not separate clipping at the upper and lower bounds or compare these curves with GSPO using fixed bounds. These measurements would help connect the changing clipping bounds to the stated motivation.
Questions for authors
-
Why were the adaptive-clipping methods discussed in the appendix excluded from the experiments? How does CSPO compare with methods such as BAPO, BandPO, or ACPO under the same training conditions? Please also specify how GSPO’s fixed clipping bounds were selected and tuned. This would help determine whether adaptation improves performance beyond choosing better clipping bounds.
-
What supports applying empirical quantiles to sequence-level scores rather than token-level ratios? Have you evaluated the same method for selecting clipping bounds with token-level clipping? This comparison would help distinguish the benefit of adaptive bounds from that of the GSPO sequence-level formulation.
-
How many independent training runs and evaluation samples were used for Tables 1 and 2, and how were the reported checkpoints selected? Can you provide the mean and standard deviation across runs for CSPO and the baselines? Please also report the ALFWorld task counts and explain how the All score is calculated, since it is not the simple average of the six task scores.
-
Can you separately report the fractions of scores below the lower bound and above the upper bound across training steps? Please also distinguish these from the fractions actually clipped: positive-advantage samples above the upper bound and negative-advantage samples below the lower bound. Comparing these curves with GSPO using fixed bounds would help assess whether adaptation reduces excessive or insufficient clipping.
-
How should the objective and gradient error bounds guide the choice of ? Since unclipped updates have zero clipping error, what theoretical or empirical evidence shows that the selected tail budget preserves useful gradients while preventing unstable updates? In particular, how does the fraction of scores outside the interval relate to the gradient magnitude removed by clipping and to training stability?
4 comments
[W3, Q3] Uneven gains, independent runs, checkpoint selection, and ALFWorld task counts.
The ALFWorld gains differ by task. In the original table, CSPO's largest advantage is on Pick, while DAPO is stronger on Clean and GRPO on Pick2. In the added comparison, BAPO and ACPO are stronger on Clean, while BandPO is stronger on Pick2. We will describe the results at this level rather than suggesting that CSPO improves every task.
The math results for seeds 42, 314, and 271 are listed in our response to Reviewer r34E and in Appendix D.1.2, Table 7.
At an ALFWorld evaluation step , let be the number of episodes of category , and let be its number of successes. Then
=100\,\frac{\sum_c s_c(t)}{\sum_c n_c(t)}, \qquad \sum_c n_c(t)=128.$$ Thus All is weighted by episode counts at that evaluation. The reported All is $\max_t\mathrm{All}(t)$; it is not the average of the six category maxima. Category counts can change when the environments reset. | Method | Successful episodes / total | |---|---:| | GSPO | 40 / 128 | | DAPO | 68 / 128 | | BAPO | 66 / 128 | | BandPO | 72 / 128 | | ACPO | 59 / 128 | | CSPO | 82 / 128 | > *[W4, Q5] Clipping and training stability.* Our choice $\alpha=0.1$, $K=2$ has been discussed in **Appendix C.3**. There is a trade-off in choosing $\alpha$: a smaller $\alpha$ gives a wider clipping interval and allows more updates to remain unclipped, but as $\alpha \to 0$ the interval becomes too loose and CSPO approaches an effectively unclipped update, risking unstable or collapsed training. Conversely, a larger $\alpha$ narrows the interval and makes clipping more aggressive, which can over-constrain policy updates and limit learning. **Figure 12 (Appendix D.5)** shows smaller early gradient fluctuations for CSPO than GSPO: the standard deviation over steps 1–150 is **0.135 versus 0.150**. Both methods settle into smooth trajectories during the middle stage. Unclipped training develops large late spikes, reaching **1.44 and 5.91**, while GSPO and CSPO remain near 0.1. Together with the reward curves in **Figure 11 (Appendix D.4)** and clipping patterns in **Figure 8 (Appendix D.2)**, these observations support our interpretation: fixed GSPO clipping may restrict early learning, while removing clipping entirely permits occasional large gradients later. CSPO maintains moderate clipping throughout training, balancing early learning with later stability. > *[W5, Q4] Separating both tails from actual clipping.* **Figure 8 (Appendix D.2)** separately plots $\widehat P(W<a)$, $\widehat P(W>b)$, $\widehat P(A<0,W<a)$, and $\widehat P(A>0,W>b)$ for GSPO and CSPO at both model sizes. All four use the same denominator of valid response evaluations; the last two are joint fractions whose sum gives actual clipping. CSPO maintains balanced tails and a small, persistent actual clipping fraction, while GSPO's fractions change more across training.Thank you for the additional experiments and clarifications. The adaptive clipping comparisons, token level results, and training diagnostics address several of my concerns. I will therefore raise my score.
One final question concerns the scope of the study. Since the quantile calibration rule also applies to token level clipping, why was the method originally formulated specifically at the sequence level? Did it originate from observations of GSPO’s optimization behavior? The stronger reported token level CPO result makes me interested in understanding how this scope was determined.
Thank you for your thoughtful follow-up and for recognizing the value of our additional experiments. We appreciate the opportunity to further clarify the motivation behind our initial sequence-level formulation. The original sequence-level formulation was motivated by observations of GSPO's clipping behavior. With fixed bounds, the fraction of response scores outside the interval was initially high and declined during training, motivating calibration against recent score distributions to target more consistent marginal tail coverage.
The sequence-level setting provided a controlled comparison: retaining GSPO's score, surrogate, and clipping granularity isolated the effect of adaptive versus fixed bounds. It also has a natural modeling rationale in single-turn mathematical RLVR. Unlike multi-step MDPs involving environment interactions, such tasks can be formulated as contextual bandit problems: given a prompt (context), the policy generates a complete response (action) and receives a verifiable reward upon completion. Thus, the response is a natural unit for quantile calibration, giving the tail budget a direct interpretation as a target marginal fraction of response scores outside the interval. This does not imply an inherent advantage over token-level calibration.
The additional experiment demonstrates the broader applicability of the quantile rule. Under the historical-maximum evaluation protocol in Table 9, token-level CPO achieves 68.75% All success on ALFWorld, exceeding CSPO's 64.06%. The rule therefore supports both granularities, although their tail budgets have different units.
The revision will distinguish the empirical origin in GSPO, the initial choice of sequence-level optimization for controlled evaluation, and the broader applicability demonstrated by token-level CPO.
Thank you for pointing out these questions. We will make the clipping pattern the starting point of the abstract and introduction: fixed bounds can impose substantially different amounts of clipping at different training stages. CSPO uses recent score quantiles to maintain more consistent tail coverage as the policy evolves. The motivation is to avoid excessive early restriction while keeping clipping active later.
We have now implemented the core clipping rules of BAPO, BandPO, and ACPO from their public code in the same ALFWorld trainer, and completed the requested token comparison, which we call token-level CPO. The adaptive baseline experiments follow the CSPO reference configuration. Evaluation uses 128 episodes every five iterations.
All entries are success rates in percent. As in Table 1, each column is its historical maximum over evaluation steps, including the evaluation before training. The maxima in one row can come from different checkpoints.
CSPO has a higher All score than BAPO, BandPO, and ACPO. The added token-level CPO achieves 68.75% All, compared with 64.06% for CSPO, and improves five of the six category scores over CSPO; CSPO remains stronger on Pick. These results show that the quantile rule also works with token ratios, with token-level CPO performing best overall in this ALFWorld comparison. The full results are in Appendix D.1.4, Table 9 of the revised paper.
Sequence scores give the tail budget a direct interpretation at the response level: it specifies a target fraction of response scores outside the clipping interval, aligning the calibration unit with the reward and advantage. With token ratios, the budget instead targets token tail mass, which can be concentrated in a few responses or spread across many. The new result supports applying quantile calibration at either granularity. Our CSPO versus GSPO comparison keeps the sequence score W=exp(S/L) and surrogate fixed to examine the effect of adapting the bounds.
The GSPO bounds and their additional comparison are addressed in our response to Reviewer hWK6, W1. The completed 4B results appear in our response to Reviewer r34E, W1/Q1/Q2.