acceptodds
Under review as a conference paper at ICLR 2027

Beyond Uniform Token-Level Trust Region in LLM Reinforcement Learning

Abstract

Reinforcement learning is widely used to improve reasoning in large language models. A central design choice is how much policy change to allow at each token of a response. Uniform token-level thresholds ignore both token position and preceding policy divergence. Yet early changes can affect longer continuations, and divergence can accumulate along the prefix. To address this, we introduce CPPO (Cumulative Prefix Divergence Policy Optimization), a token-level masking rule that combines position-weighted divergence with a cumulative prefix budget. Divergence at earlier positions receives greater weight, and the prefix budget limits the cumulative weighted divergence. We derive a finite-horizon policy improvement bound under exact token divergence constraints. With a sufficiently small prefix budget, it is tighter than the bound for uniform token-level limits at the same nominal threshold. Experiments on mathematical reasoning show that CPPO outperforms the evaluated baselines across model sizes. Ablation studies support the effectiveness of both components.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, LLM, RLVR, Trust Region, PPO

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_7nNyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CPPO, which adjusts token-level gradient masking using position-weighted divergence and a cumulative prefix budget. The authors derive a policy improvement bound under exact TV constraints and report higher AIME best validation scores across four Qwen3 configurations.

Critical strengths
  1. The method accounts for token position and preceding divergence, specifically addressing the issue where a uniform threshold overlooks prefix information.
  2. Theoretical analysis explicitly identifies the conditions under which prefix constraints tighten the surrogate error bound.
  3. Ablation studies cover two core components, position-weight ordering and hard/soft masking, facilitating an analysis of their respective contributions to the design.
Critical weaknesses
  1. The same set of AIME benchmarks is used for both checkpoint selection and result reporting, lacking an independent test evaluation to verify the performance gains.
  2. Results across multiple training seeds or uncertainty estimates are not reported, making it difficult to determine the stability of the minimum 0.91 percentage-point advantage.
  3. The theory relies on exact TV constraints, yet the implementation employs reduced-TV masking, which does not guarantee adherence to these constraints, without quantifying actual constraint violations.
  4. Comparisons with DPPO match the nominal threshold rather than the actual divergence allowance; a more comprehensive threshold sweep is needed to rule out the influence of overall constraint stringency.
Questions for authors

NaN

4 comments
AuthorsOpenReview ↗

Response to Reviewer 7nNy (1/2): W1, W2, W4

We thank the reviewer for the detailed and constructive comments. We apologize for the late reply: the additional experiments required substantial compute and time. We agree with the points raised and ran the corresponding experiments; we summarize them below and have added the corresponding tables and figures to the revised paper.

W1: independent test evaluation. We evaluated the selected checkpoints on seven benchmarks that are used neither for training nor for checkpoint selection: mathematics (MATH-500, AMC23, OlympiadBench, Minerva), science (GPQA-Diamond), and code (LiveCodeBench, HumanEval+). Checkpoints are still selected by the AIME average, and each benchmark uses its own prompt and official grader (Avg@16, T = 0.7, top-p = 0.95). The seed-1 checkpoints were not retained, so we use seeds 2 and 3 on Qwen3-1.7B-Base (mean ± std) and seed 2 on Qwen3-30B-A3B-Base:

Model, methodMATH-500AMC23OlympiadBenchMinervaGPQA-DiamondLiveCodeBenchHumanEval+
1.7B-Base, Uniform (DPPO)72.21 ± 0.2747.35 ± 1.5533.75 ± 0.5427.32 ± 0.9129.06 ± 0.747.36 ± 0.0862.92 ± 1.07
1.7B-Base, CPPO (ours)73.15 ± 0.5249.46 ± 0.7734.39 ± 0.6928.54 ± 0.1630.08 ± 0.818.70 ± 1.1563.24 ± 1.10
Δ+0.94+2.11+0.64+1.22+1.02+1.34+0.32
30B-A3B-Base, Uniform (DPPO)94.3092.0363.6546.9758.9025.5883.77
30B-A3B-Base, CPPO (ours)95.1495.1670.0745.2060.7326.2984.72
Δ+0.84+3.13+6.42−1.77+1.83+0.71+0.95

Although training uses only mathematical problems, CPPO is better on all seven benchmarks on Qwen3-1.7B-Base and on six of the seven on Qwen3-30B-A3B-Base, including science and code, so the gains do not come from selecting checkpoints on the reporting benchmarks. In addition, all baselines in the paper were re-run by us and their checkpoints were selected with the same protocol, so AIME-based selection applies equally to every method.

W2: seeds and uncertainty. We ran two additional seeds on Qwen3-1.7B-Base and one on Qwen3-30B-A3B-Base. AIME24/25/26 Avg@16 (seed 1 is the run in the paper; mean ± std over seeds):

Model, methodSeed 1Seed 2Seed 3AIME avgAIME24AIME25AIME26
1.7B-Base, Uniform (DPPO)10.9011.1810.4910.86 ± 0.3514.86 ± 0.798.75 ± 1.308.95 ± 1.27
1.7B-Base, CPPO (ours)12.7812.4312.7812.66 ± 0.2016.25 ± 1.0411.67 ± 0.9110.07 ± 0.32
30B-A3B-Base, Uniform (DPPO)49.2351.11–50.17 ± 1.3356.56 ± 1.0340.31 ± 1.0353.65 ± 3.98
30B-A3B-Base, CPPO (ours)54.7953.34–54.07 ± 1.0362.09 ± 3.8342.19 ± 1.3357.92 ± 2.06

On Qwen3-1.7B-Base, the 95% confidence intervals of the AIME average (t-distribution, three seeds), [12.16, 13.17] for CPPO and [9.99, 11.72] for the baseline, do not overlap. On both models, every CPPO seed scores above every baseline seed.

W4: actual divergence allowance and threshold sweep. We agree that matching the nominal threshold does not match the actual allowance. When the prefix budget is inactive, CPPO's position weights relax the effective threshold on the score to δ/wt∈[0.15,0.1875]\delta/w_t\in[0.15, 0.1875] (δ=0.15\delta=0.15, wmin⁡=0.8w_{\min}=0.8); the prefix budget lowers it when divergence accumulates. To test whether the gain comes from a looser overall threshold, we swept the uniform threshold over this range on Qwen3-1.7B-Base (AIME24/25/26 Avg@16):

MethodThreshold on the scoreAIME avgAIME24AIME25AIME26
Uniform, δ=0.15\delta=0.150.15010.9013.968.3310.41
Uniform, δ=0.175\delta=0.1750.17510.7613.129.589.58
Uniform, δ=0.20\delta=0.200.20011.1114.5810.837.92
CPPO, δ=0.15\delta=0.15, wmin⁡=0.8w_{\min}=0.8 (3 seeds)bt/wtb_t/w_t (adaptive)12.66 ± 0.2016.25 ± 1.0411.67 ± 0.9110.07 ± 0.32

Loosening the uniform threshold to CPPO's allowance barely moves the score and does not reproduce CPPO's gain. This complements the ablation in Figure 4 (middle): keeping the same set of position weights but randomly reassigning them to positions leaves the score close to uniform masking. Together, these indicate that the improvement comes from which positions receive the tighter allowance, not from the overall stringency.

AuthorsOpenReview ↗

Response to Reviewer 7nNy (2/2): W3

W3: full-vocabulary TV in the theory versus Top-K TV masking, and actual constraint violations. We agree that the theory is stated for the full-vocabulary (exact) TV and that masking with the Top-K TV does not guarantee the constraints; Section 3.3 notes that masking is not a projection, so masked positions can still change through shared parameters. We first quantify how far the Top-K TV is from the full-vocabulary TV, then check directly how often the updated policy violates the constraints; we have added the corresponding figures to the revised paper.

1. How far is the Top-K TV from the full-vocabulary TV? The TV summed over μ\mu's top-KK tokens is a lower bound Lt≤DtL_t\le D_t; adding the probability masses outside these tokens gives an upper bound Utup≥DtU^{\mathrm{up}}_t\ge D_t; and the mass-normalized Top-KK TV UtU_t rescales LtL_t by the captured mass. Since yt∼μy_t\sim\mu, batch means of DtD_t can be estimated without bias from the sampled tokens, so each approximation can be compared with DtD_t directly.

  • Accuracy. On post-trained Qwen3-1.7B, the top 20 tokens capture more than 99.9% of the probability mass, and LtL_t, UtU_t, and UtupU^{\mathrm{up}}_t match DtD_t from the first step (ratios 1.000, 1.000, and 1.014). On Qwen3-1.7B-Base, the high initial entropy leaves more tail mass: early in training Lt/Dt=0.877L_t/D_t=0.877 and Ut/Dt=1.064U_t/D_t=1.064. After step 90, both ratios stay within 0.998–1.003.
  • Cost. Computing the full-vocabulary TV during training would require returning full-vocabulary log probabilities (151,936 tokens) from the rollout engine, which must serialize and transfer each of them across processes. Already at 4 prompts × 8 responses of length 512, this raises step time 39× (13.9 s → 550.9 s) and peak host memory from 153 GB to 846 GB, which is infeasible at the training scale (64 prompts × 8, length 8192). With K=20K=20 or K=100K=100, the step time is 1.0–1.1× that of Binary TV, with negligible memory overhead.
  • Effect on the mask. We ran Algorithm 1 on the same batches with the full-vocabulary TV and with each approximation. On Qwen3-1.7B-Base, the masks differ more early in training (steps 1–10); once training stabilizes (steps 101–110), every rule masks fewer than 0.004% of tokens. On post-trained Qwen3-1.7B, the rules agree from the start: in steps 1–10 and 21–30, the full-vocabulary TV, LtL_t, and UtU_t each mask 0.010% and 0.011% of tokens, and Binary TV 0.0082% and 0.0088%.

2. How often does the updated policy actually violate the full-vocabulary TV constraints? We audited CPPO runs on Qwen3-1.7B-Base and on post-trained Qwen3-1.7B (the latter with the fixed budget rate δb=0.015\delta_b=0.015, over 30 rollout steps). At every rollout step, we scored the same batch twice without gradients, before the first and after the last optimizer update. We then checked the two conditions of Theorem 1 with the full-vocabulary TV: the token limit wtDt≤δw_tD_t\le\delta and the prefix budget ∑j≤twjDj≤δ+δbWt−1\sum_{j\le t}w_jD_j\le\delta+\delta_bW_{t-1}. The TV is computed over essentially the whole vocabulary (more than 99.9% of μ\mu's probability mass).

Model, rollout stepsTokens outside the full-vocabulary TV trust region, before updateAfter update
Qwen3-1.7B-Base, 1–500.015%0.027%
Qwen3-1.7B-Base, 51–1100.0011%0.0015%
Qwen3-1.7B (post-trained), 1–300.0093%0.011%
  • One rollout step of updates adds only 0.006 (base) and 0.0017 (post-trained) percentage points of tokens outside the region on average; the rest already exists before the update, because the training and rollout engines differ slightly.
  • Over training, the share does not grow: on the base model it drops after the first 20 steps and then stays between 0.0004% and 0.0024% per 10-step window through step 110; on the post-trained model it stays between 0.010% and 0.012%.

Connecting the theory to the gains. Although masking does not guarantee the full-vocabulary TV constraints, the policy produced by CPPO's updates violates them on a vanishing fraction of tokens, so Eq. (15) describes the training that produced our results. The theorem's role is to identify where divergence is costly: earlier positions have larger coefficients, and divergence accumulates along the prefix. It does not prescribe absolute thresholds; driving them to zero would remove the residual term but also the surrogate improvement Lμ′(π)L'_\mu(\pi). Consistently, CPPO improves over uniform masking with both Binary and Top-KK TV (Figure 5, right). Its gain therefore comes mainly from the position- and prefix-aware allocation of the budget that the bound motivates, rather than from small differences in the divergence estimate.

We hope these additions address your concerns, and we would appreciate it if you would reconsider your assessment in light of them.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer 7nNy,

Thank you again for your detailed review. As the discussion period ends on November 18 (AoE), we would like to kindly check whether our responses have addressed your concerns. In brief, we (1) evaluated the selected checkpoints on seven held-out benchmarks in mathematics, science, and code, (2) reported results over multiple seeds with confidence intervals, (3) quantified how closely the policies produced by CPPO's updates satisfy the theoretical constraints, and (4) compared against uniform thresholds loosened to CPPO's effective range. All corresponding tables and figures have been added to the revised paper.

If any concerns remain, we would be glad to clarify them before the discussion closes. If our responses have resolved them, we would be grateful if you would consider updating your assessment.

Best regards, The Authors

I update my score to 3

Reviewer_Q8P5OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose Cumulative Prefix Divergence Policy Optimization (CPPO), a method that masks tokens based on a cumulative prefix budget that is weighted based on position. They also derive a finite-horizon performance-difference decomposition when assuming exact token divergence constraints, showing that cumulative prefix constraints can tighten policy improvement bounds. They train four Qwen3 models on DAPO-Math-17k using CPPO, evaluating on AIME 2024, 2025, 2026, showing consistent improvements in mathematical reasoning performance over several baselines such as GRPO, CISPO, etc.

Critical strengths
  1. The paper addresses a well-motivated limitation of existing uniform token-level trust regions, proposing a relatively simple modification to token-level masking. This is consequential to the overall assessment as the paper identifies a meaningful limitation in existing approaches and provides a practical solution that could be useful even beyond the current experimental setting.

  2. The empirical results are positive, with CPPO achieving the highest validation performance across all models (including dense and MoE models), while comparing with several other baselines. This is consequential to the overall assessment as this shows that the proposed method offers practical improvements instead of just proposing a theoretically motivated method.

Critical weaknesses
  1. The experiments seem to be missing some important checks on robustness. In particular, it is unclear how many seeds were used in the experiments, and variability across seeds/confidence intervals for the validation performance appear to be omitted. Providing these would help strengthen the robustness and reliability of the observed improvements. Additionally, the evaluation is only done on AIME benchmarks; providing evaluation results on additional mathematical reasoning benchmarks (and even other general reasoning benchmarks) would strengthen the improvement claims of the proposed method.
Questions for authors
  1. Could the authors provide the multi-seed results as well as confidence intervals for the validation performance results?

  2. It seems as though the validation performance on the 30B-A3B-Base MoE appears to vary substantially across the different methods. Are there any possible reasons for this? Could more careful tuning help make the results more comparable across methods? The proposed method also seems to see the most gains on 30B-A3B-Base; could the authors discuss whether it is because CPPO provides additional benefits for MoEs, or any other reasons?

  3. How closely do the theoretical divergence constraints correspond to the behavior of the actual CPPO updates? In particular, can the authors quantify the discrepancy (eg. empirically) between the masking rule used in CPPO and the assumptions in the theoretical results? This would help connect the theory to the empirical improvements.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Q8P5 (1/2): W1 & Q1, Q2

We thank the reviewer for the careful review and for recognizing both the motivation and the practical value of CPPO. We apologize for the late reply: the additional experiments required substantial compute and time. We summarize the new results below and have added the corresponding tables and figures to the revised paper.

W1 & Q1: multiple seeds and confidence intervals. We ran two additional seeds of CPPO and the uniform token-level baseline (DPPO) on Qwen3-1.7B-Base and one on Qwen3-30B-A3B-Base. AIME24/25/26 Avg@16 (seed 1 is the run in the paper; mean ± std over seeds):

Model, methodSeed 1Seed 2Seed 3AIME avgAIME24AIME25AIME26
1.7B-Base, Uniform (DPPO)10.9011.1810.4910.86 ± 0.3514.86 ± 0.798.75 ± 1.308.95 ± 1.27
1.7B-Base, CPPO (ours)12.7812.4312.7812.66 ± 0.2016.25 ± 1.0411.67 ± 0.9110.07 ± 0.32
30B-A3B-Base, Uniform (DPPO)49.2351.11–50.17 ± 1.3356.56 ± 1.0340.31 ± 1.0353.65 ± 3.98
30B-A3B-Base, CPPO (ours)54.7953.34–54.07 ± 1.0362.09 ± 3.8342.19 ± 1.3357.92 ± 2.06

On Qwen3-1.7B-Base, the 95% confidence intervals of the AIME average (t-distribution, three seeds), [12.16, 13.17] for CPPO and [9.99, 11.72] for the baseline, do not overlap. On both models, every CPPO seed scores above every baseline seed.

W1 & Q1: evaluation beyond AIME. We evaluated the selected checkpoints on seven benchmarks that are used neither for training nor for checkpoint selection, covering mathematics, science, and code. Checkpoints are still selected by the AIME average, and each benchmark uses its own prompt and official grader (Avg@16, T = 0.7, top-p = 0.95). The seed-1 checkpoints were not retained, so we use seeds 2 and 3 on 1.7B (mean ± std) and seed 2 on 30B:

Model, methodMATH-500AMC23OlympiadBenchMinervaGPQA-DiamondLiveCodeBenchHumanEval+
1.7B-Base, Uniform (DPPO)72.21 ± 0.2747.35 ± 1.5533.75 ± 0.5427.32 ± 0.9129.06 ± 0.747.36 ± 0.0862.92 ± 1.07
1.7B-Base, CPPO (ours)73.15 ± 0.5249.46 ± 0.7734.39 ± 0.6928.54 ± 0.1630.08 ± 0.818.70 ± 1.1563.24 ± 1.10
30B-A3B-Base, Uniform (DPPO)94.3092.0363.6546.9758.9025.5883.77
30B-A3B-Base, CPPO (ours)95.1495.1670.0745.2060.7326.2984.72

Although training uses only mathematical problems, CPPO is better on all seven benchmarks on Qwen3-1.7B-Base and on six of the seven on Qwen3-30B-A3B-Base, including science and code.

Q2: why do the baselines differ so much on Qwen3-30B-A3B-Base? Three properties of this setting enlarge the discrepancy between the training policy and the rollout policy:

  1. Without routing replay, MoE models show a larger end-to-end policy shift than dense models.
  2. Each rollout step takes 8 minibatch updates (2–4 for the dense models), so the data are staler.
  3. Responses are 16k tokens instead of 8k, and later tokens drift further from the rollout policy (Figure 6 in Appendix C).

Together, these make it more likely that tokens which are truly outside the trust region, and which have large likelihood ratios, enter the gradient. GRPO and CISPO decide clipping from the token's likelihood ratio, which is a single-sample estimate of that token's TV. They are therefore more exposed to such gradient spikes, consistent with CISPO's collapse and GRPO's lower score. Without routing replay, the GRPO and CISPO scores reported by DPPO (Qi et al., 2026) are consistent with ours.

Could more careful tuning narrow the gap? Tuning baselines at 30B scale is expensive and outside the scope of this work. We therefore used each baseline's recommended hyperparameters for the corresponding model size, and its default configuration when no such setting is reported. As sanity checks, our TRM-Avg/Max scores on Qwen3-8B-Base match those reported by TRM (Li et al., 2025), and our 30B DPPO, CISPO, and GRPO scores are in line with those reported by DPPO (Qi et al., 2026).

Why does CPPO gain most on the MoE model? We attribute this to the combination of architecture and training setting rather than an MoE-specific mechanism. Without routing replay, MoE updates are less stable, and 8 updates per rollout step increase staleness. Both raise the share of tokens whose divergence matters. In this regime, allocating the trust region by position and prefix, rather than uniformly, stabilizes training the most. The 30B gain is also stable across seeds: CPPO reaches 54.07 ± 1.03 versus 50.17 ± 1.33 for DPPO over two seeds (+3.90).

AuthorsOpenReview ↗

Response to Reviewer Q8P5 (2/2): Q3

Q3: how closely do the theoretical constraints correspond to the actual CPPO updates? We quantify the discrepancy directly and have added the corresponding figures to the revised paper.

Theory versus implementation. Theorem 1 and Eq. (15) apply to an updated policy π\pi that satisfies, with the full-vocabulary TV DtD_t, (i) the token limit wtDt≤δw_tD_t\le\delta and (ii) the prefix budget ∑j≤twjDj≤δ+δbWt−1\sum_{j\le t}w_jD_j\le\delta+\delta_bW_{t-1}. CPPO differs in two ways: it measures divergence with the K=20K=20 Top-KK TV instead of DtD_t, and it does not constrain π\pi directly but masks gradient contributions, so masked positions can still change through shared parameters (Section 3.3). We first quantify the approximation in the score, then check directly whether the updated policy satisfies (i) and (ii), which accounts for both differences at once.

1. Top-K TV versus full-vocabulary TV. The TV summed over μ\mu's top-KK tokens is a lower bound Lt≤DtL_t\le D_t; adding the probability masses outside these tokens gives an upper bound Utup≥DtU^{\mathrm{up}}_t\ge D_t; and the mass-normalized Top-KK TV UtU_t rescales LtL_t by the captured mass. Since yt∼μy_t\sim\mu, batch means of DtD_t can be estimated without bias from the sampled tokens, so each approximation can be compared with DtD_t directly.

  • Accuracy. On post-trained Qwen3-1.7B, the top 20 tokens capture more than 99.9% of the probability mass, and LtL_t, UtU_t, and UtupU^{\mathrm{up}}_t match DtD_t from the first step (ratios 1.000, 1.000, and 1.014). On Qwen3-1.7B-Base, the high initial entropy leaves more tail mass: early in training Lt/Dt=0.877L_t/D_t=0.877 and Ut/Dt=1.064U_t/D_t=1.064. After step 90, both ratios stay within 0.998–1.003.
  • Cost. With K=20K=20 or K=100K=100, the step time is 1.0–1.1× that of Binary TV, with negligible memory overhead. With the full vocabulary (151,936 tokens), the rollout engine must serialize and transfer every log probability across processes: already at 4 prompts × 8 responses of length 512, step time grows 39× (13.9 s → 550.9 s) and peak host memory from 153 GB to 846 GB, infeasible at the training scale (64 prompts × 8, length 8192).
  • Effect on the mask. We ran Algorithm 1 on the same batches with the full-vocabulary TV and with each approximation. On Qwen3-1.7B-Base, the masks differ more early in training (steps 1–10); once training stabilizes (steps 101–110), every rule masks fewer than 0.004% of tokens. On post-trained Qwen3-1.7B, the rules agree from the start: in steps 1–10 and 21–30, the full-vocabulary TV, LtL_t, and UtU_t each mask 0.010% and 0.011% of tokens, and Binary TV 0.0082% and 0.0088%.

2. Does the policy produced by the actual updates satisfy (i) and (ii)? We audited CPPO runs on Qwen3-1.7B-Base and on post-trained Qwen3-1.7B (the latter with the fixed budget rate δb=0.015\delta_b=0.015, over 30 rollout steps). At every rollout step, we scored the same batch twice without gradients, before the first and after the last optimizer update, and checked (i) and (ii) with the full-vocabulary TV. The TV is computed over essentially the whole vocabulary (more than 99.9% of μ\mu's probability mass).

Model, rollout stepsTokens outside the full-vocabulary TV trust region, before updateAfter update
Qwen3-1.7B-Base, 1–500.015%0.027%
Qwen3-1.7B-Base, 51–1100.0011%0.0015%
Qwen3-1.7B (post-trained), 1–300.0093%0.011%
  • One rollout step of updates adds only 0.006 (base) and 0.0017 (post-trained) percentage points of tokens outside the region on average; the rest already exists before the update, because the training and rollout engines differ slightly.
  • Over training, the share does not grow: on the base model it drops after the first 20 steps and then stays between 0.0004% and 0.0024% per 10-step window through step 110; on the post-trained model it stays between 0.010% and 0.012%.

Connecting the theory to the gains. The policy produced by CPPO's updates satisfies the theorem's conditions on all but a vanishing fraction of tokens, so Eq. (15) describes the training that produced our results. The theorem's role is to identify where divergence is costly: earlier positions have larger coefficients, and divergence accumulates along the prefix. It does not prescribe absolute thresholds; driving them to zero would remove the residual term but also the surrogate improvement Lμ′(π)L'_\mu(\pi). Consistently, CPPO improves over uniform masking with both Binary and Top-KK TV (Figure 5, right). Its gain therefore comes mainly from the position- and prefix-aware allocation of the budget that the bound motivates, rather than from small differences in the divergence estimate. Loosening the uniform threshold to CPPO's effective range (0.15 to 0.20) gives an AIME average of at most 11.11 versus 12.66 for CPPO, and randomly reassigning the position weights leaves the score close to uniform masking (Figure 4, middle).

We hope these results address your questions.

Thanks for the additional results and discussions; they have addressed most of my concerns. Just regarding the MoE training, given that routing replay substantially stabilizes and improves training, is there a reason why this setting was not used in the baselines? (methods such as CISPO even collapse without it, but perform significantly better with it) This would make the comparisons for the MoE training substantially more reliable. Although there is not enough time to run this for the rebuttal, I would suggest that the authors also test this setting prior to the camera ready submission, if accepted.

Given the other additional results and discussion, I have raised my score to 4.

AuthorsOpenReview ↗

Thank you for raising your score

Thank you very much for reading our responses carefully and for raising your score. We are glad that the additional results addressed most of your concerns, and we appreciate this constructive suggestion.

We agree that routing replay substantially stabilizes MoE training and that comparisons with it enabled would make the MoE results more complete. Following your suggestion, we will add results on Qwen3-30B-A3B-Base with routing replay enabled in the camera-ready version, if accepted.

Thank you again for your time and for the thoughtful feedback throughout the review process.

Reviewer_VDGYOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a token-level trust region regularization, more finegrained to DPPO (2026, Qi et al.). It dynamically adjusts trust region clipping bound by:

  1. linear decay by assigning greater weight to earlier tokens, enforcing stricter limits on policy changes at the beginning of a response and relaxing them toward the end, and

  2. limiting the total divergence (i.e., if earlier tokens diverges too much, the divergence for the current token is strictly reduced or completely nullified).

The authors mathematically prove the effectiveness of the proposed method that, under the same nominal threshold, this prefix-based control provides a tighter finite-horizon policy improvement bound than uniform limits. The authors evaluate their method on AIME benchmarks, showing that CPPO consistently outperforms major baselines on Qwen variants ranging from 1.7B to 30B parameters.

Critical strengths
  1. Very intuitive and elegant. This is not a fancy method but it simply works.

  2. Both theoretically sound and empirically outperforming.

Critical weaknesses
  1. Empirical evaluations are only done on AIME datasets. It's not diverse enough. Need data points from other coding/reasoning benchmarks.

  2. Also, since RL experiments tend to have variations. Candlestick charts with clear stddevs, or a table with plus and minus stddev work better to show statistical significance of the results.

Questions for authors

I'm willing to bump up my rating once the above weaknesses are resolved.

3 comments
AuthorsOpenReview ↗

Response to Reviewer VDGY

We thank the reviewer for finding CPPO "very intuitive and elegant" and both theoretically sound and empirically strong. We apologize for the late reply: the additional experiments required substantial compute and time. We have addressed both weaknesses below and have added the corresponding tables to the revised paper.

W1: evaluation beyond AIME. We evaluated the selected checkpoints on seven benchmarks that are used neither for training nor for checkpoint selection: mathematics (MATH-500, AMC23, OlympiadBench, Minerva), science (GPQA-Diamond), and code (LiveCodeBench, HumanEval+). Checkpoints are still selected by the AIME average, and each benchmark uses its own prompt and official grader (Avg@16, T = 0.7, top-p = 0.95). The seed-1 checkpoints were not retained, so we evaluate seeds 2 and 3 on Qwen3-1.7B-Base (mean ± std) and seed 2 on Qwen3-30B-A3B-Base:

Model, methodMATH-500AMC23OlympiadBenchMinervaGPQA-DiamondLiveCodeBenchHumanEval+
1.7B-Base, Uniform (DPPO)72.21 ± 0.2747.35 ± 1.5533.75 ± 0.5427.32 ± 0.9129.06 ± 0.747.36 ± 0.0862.92 ± 1.07
1.7B-Base, CPPO (ours)73.15 ± 0.5249.46 ± 0.7734.39 ± 0.6928.54 ± 0.1630.08 ± 0.818.70 ± 1.1563.24 ± 1.10
Δ+0.94+2.11+0.64+1.22+1.02+1.34+0.32
30B-A3B-Base, Uniform (DPPO)94.3092.0363.6546.9758.9025.5883.77
30B-A3B-Base, CPPO (ours)95.1495.1670.0745.2060.7326.2984.72
Δ+0.84+3.13+6.42−1.77+1.83+0.71+0.95

Although training uses only mathematical problems, CPPO is better on all seven benchmarks on Qwen3-1.7B-Base and on six of the seven on Qwen3-30B-A3B-Base, including science and code.

W2: variation across runs. Following your suggestion, we report mean ± standard deviation over three seeds on Qwen3-1.7B-Base and two seeds on Qwen3-30B-A3B-Base (AIME24/25/26 Avg@16; seed 1 is the run in the paper):

Model, methodSeed 1Seed 2Seed 3AIME avgAIME24AIME25AIME26
1.7B-Base, Uniform (DPPO)10.9011.1810.4910.86 ± 0.3514.86 ± 0.798.75 ± 1.308.95 ± 1.27
1.7B-Base, CPPO (ours)12.7812.4312.7812.66 ± 0.2016.25 ± 1.0411.67 ± 0.9110.07 ± 0.32
30B-A3B-Base, Uniform (DPPO)49.2351.11–50.17 ± 1.3356.56 ± 1.0340.31 ± 1.0353.65 ± 3.98
30B-A3B-Base, CPPO (ours)54.7953.34–54.07 ± 1.0362.09 ± 3.8342.19 ± 1.3357.92 ± 2.06

On both models, the gain in the AIME average (+1.80 and +3.90) is well beyond the seed-to-seed standard deviation, and every CPPO seed scores above every baseline seed.

Across seeds, model sizes, and subjects, CPPO generally outperforms the uniform baseline. We hope these additions resolve both concerns, and we would be grateful if you would take them into account in your final assessment.

Thanks authors for the response. I am satisfied with the results. Raised my rating to 4.

AuthorsOpenReview ↗

Thank you for raising your score

Thank you very much for your positive feedback and for raising your score. Your suggestions on evaluation beyond AIME and on variation across runs have clearly strengthened the paper, and the corresponding results are now part of the revised version. We sincerely appreciate your time and support throughout the review process.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank all reviewers for their thoughtful and constructive feedback, and we apologize for the delayed response: the additional experiments, especially on Qwen3-30B-A3B-Base, required substantial compute and time. We summarize the discussion below; each reviewer's thread contains the full results for that reviewer's questions, and all new tables and figures have been added to the revised paper (as highlighted in blue color) .

What the reviewers found valuable.

  • Motivation and simplicity. CPPO addresses "a well-motivated limitation of existing uniform token-level trust regions" with "a practical solution that could be useful even beyond the current experimental setting" (Q8P5); it is "very intuitive and elegant ... not a fancy method but it simply works" (VDGY), and accounts for the prefix information that a uniform threshold overlooks (7nNy).
  • Theory. The analysis "explicitly identifies the conditions under which prefix constraints tighten the surrogate error bound" (7nNy) and is "theoretically sound" (VDGY).
  • Empirical gains. CPPO achieves the highest validation performance across all four models, dense and MoE (Q8P5, VDGY).

How the new results address the concerns.

ConcernRaised byNew evidence
Robustness across seedsVDGY, Q8P5, 7nNyCPPO improves the AIME average by +1.80 on Qwen3-1.7B-Base (three seeds; non-overlapping 95% intervals) and by +3.90 on Qwen3-30B-A3B-Base (two seeds). On both models, every CPPO seed scores above every baseline seed.
Evaluation beyond AIMEVDGY, Q8P5, 7nNyOn seven held-out mathematics, science, and code benchmarks, used neither for training nor for checkpoint selection, CPPO is better on all seven on 1.7B and on six of seven on 30B.
Full-vocabulary TV in the theory vs. Top-K TV in the implementationQ8P5, 7nNyThe Top-K TV closely matches the full-vocabulary TV at negligible cost. The policy produced by CPPO's actual updates lies outside the full-vocabulary TV trust region on only about 0.0015% (1.7B-Base, after step 50) to 0.01% (post-trained 1.7B) of tokens, so the theorem's conditions hold for the training that produced our results.
Nominal vs. actual threshold of the uniform baseline7nNyLoosening the uniform threshold across CPPO's effective range (0.15 to 0.20) gives at most 11.11, versus 12.66 for CPPO.
Baseline spread on the 30B MoE modelQ8P5Explained by MoE updates without routing replay, more updates per rollout step, and longer responses; our baseline scores are consistent with those reported in prior work.

Summary. The new experiments strengthen the paper's claims: CPPO's gains are robust across seeds and model sizes, carry over to held-out mathematics, science, and code benchmarks, are not explained by a looser threshold, and come from updates that satisfy the theory's constraints on all but a vanishing fraction of tokens. We believe these results address the reviewers' concerns, and we thank the reviewers and the AC for their time.

Dear Reviewers,

Thank you for the review. The authors have responded to the points you raised, and the response period closes on 18 November (AoE). I would be grateful if you could post a brief reply on whether your concerns are resolved, partly resolved, or still open, so that the authors have a fair chance to address them. If anything still needs clarifying, please ask now while they can still respond.

Regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.