- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CPPO, which adjusts token-level gradient masking using position-weighted divergence and a cumulative prefix budget. The authors derive a policy improvement bound under exact TV constraints and report higher AIME best validation scores across four Qwen3 configurations.
Critical strengths
- The method accounts for token position and preceding divergence, specifically addressing the issue where a uniform threshold overlooks prefix information.
- Theoretical analysis explicitly identifies the conditions under which prefix constraints tighten the surrogate error bound.
- Ablation studies cover two core components, position-weight ordering and hard/soft masking, facilitating an analysis of their respective contributions to the design.
Critical weaknesses
- The same set of AIME benchmarks is used for both checkpoint selection and result reporting, lacking an independent test evaluation to verify the performance gains.
- Results across multiple training seeds or uncertainty estimates are not reported, making it difficult to determine the stability of the minimum 0.91 percentage-point advantage.
- The theory relies on exact TV constraints, yet the implementation employs reduced-TV masking, which does not guarantee adherence to these constraints, without quantifying actual constraint violations.
- Comparisons with DPPO match the nominal threshold rather than the actual divergence allowance; a more comprehensive threshold sweep is needed to rule out the influence of overall constraint stringency.
Questions for authors
NaN
4 comments
Response to Reviewer 7nNy (2/2): W3
W3: full-vocabulary TV in the theory versus Top-K TV masking, and actual constraint violations. We agree that the theory is stated for the full-vocabulary (exact) TV and that masking with the Top-K TV does not guarantee the constraints; Section 3.3 notes that masking is not a projection, so masked positions can still change through shared parameters. We first quantify how far the Top-K TV is from the full-vocabulary TV, then check directly how often the updated policy violates the constraints; we have added the corresponding figures to the revised paper.
1. How far is the Top-K TV from the full-vocabulary TV? The TV summed over 's top- tokens is a lower bound ; adding the probability masses outside these tokens gives an upper bound ; and the mass-normalized Top- TV rescales by the captured mass. Since , batch means of can be estimated without bias from the sampled tokens, so each approximation can be compared with directly.
- Accuracy. On post-trained Qwen3-1.7B, the top 20 tokens capture more than 99.9% of the probability mass, and , , and match from the first step (ratios 1.000, 1.000, and 1.014). On Qwen3-1.7B-Base, the high initial entropy leaves more tail mass: early in training and . After step 90, both ratios stay within 0.998–1.003.
- Cost. Computing the full-vocabulary TV during training would require returning full-vocabulary log probabilities (151,936 tokens) from the rollout engine, which must serialize and transfer each of them across processes. Already at 4 prompts × 8 responses of length 512, this raises step time 39× (13.9 s → 550.9 s) and peak host memory from 153 GB to 846 GB, which is infeasible at the training scale (64 prompts × 8, length 8192). With or , the step time is 1.0–1.1× that of Binary TV, with negligible memory overhead.
- Effect on the mask. We ran Algorithm 1 on the same batches with the full-vocabulary TV and with each approximation. On Qwen3-1.7B-Base, the masks differ more early in training (steps 1–10); once training stabilizes (steps 101–110), every rule masks fewer than 0.004% of tokens. On post-trained Qwen3-1.7B, the rules agree from the start: in steps 1–10 and 21–30, the full-vocabulary TV, , and each mask 0.010% and 0.011% of tokens, and Binary TV 0.0082% and 0.0088%.
2. How often does the updated policy actually violate the full-vocabulary TV constraints? We audited CPPO runs on Qwen3-1.7B-Base and on post-trained Qwen3-1.7B (the latter with the fixed budget rate , over 30 rollout steps). At every rollout step, we scored the same batch twice without gradients, before the first and after the last optimizer update. We then checked the two conditions of Theorem 1 with the full-vocabulary TV: the token limit and the prefix budget . The TV is computed over essentially the whole vocabulary (more than 99.9% of 's probability mass).
| Model, rollout steps | Tokens outside the full-vocabulary TV trust region, before update | After update |
|---|---|---|
| Qwen3-1.7B-Base, 1–50 | 0.015% | 0.027% |
| Qwen3-1.7B-Base, 51–110 | 0.0011% | 0.0015% |
| Qwen3-1.7B (post-trained), 1–30 | 0.0093% | 0.011% |
- One rollout step of updates adds only 0.006 (base) and 0.0017 (post-trained) percentage points of tokens outside the region on average; the rest already exists before the update, because the training and rollout engines differ slightly.
- Over training, the share does not grow: on the base model it drops after the first 20 steps and then stays between 0.0004% and 0.0024% per 10-step window through step 110; on the post-trained model it stays between 0.010% and 0.012%.
Connecting the theory to the gains. Although masking does not guarantee the full-vocabulary TV constraints, the policy produced by CPPO's updates violates them on a vanishing fraction of tokens, so Eq. (15) describes the training that produced our results. The theorem's role is to identify where divergence is costly: earlier positions have larger coefficients, and divergence accumulates along the prefix. It does not prescribe absolute thresholds; driving them to zero would remove the residual term but also the surrogate improvement . Consistently, CPPO improves over uniform masking with both Binary and Top- TV (Figure 5, right). Its gain therefore comes mainly from the position- and prefix-aware allocation of the budget that the bound motivates, rather than from small differences in the divergence estimate.
We hope these additions address your concerns, and we would appreciate it if you would reconsider your assessment in light of them.
Follow-up on our response
Dear Reviewer 7nNy,
Thank you again for your detailed review. As the discussion period ends on November 18 (AoE), we would like to kindly check whether our responses have addressed your concerns. In brief, we (1) evaluated the selected checkpoints on seven held-out benchmarks in mathematics, science, and code, (2) reported results over multiple seeds with confidence intervals, (3) quantified how closely the policies produced by CPPO's updates satisfy the theoretical constraints, and (4) compared against uniform thresholds loosened to CPPO's effective range. All corresponding tables and figures have been added to the revised paper.
If any concerns remain, we would be glad to clarify them before the discussion closes. If our responses have resolved them, we would be grateful if you would consider updating your assessment.
Best regards, The Authors
I update my score to 3
Response to Reviewer 7nNy (1/2): W1, W2, W4
We thank the reviewer for the detailed and constructive comments. We apologize for the late reply: the additional experiments required substantial compute and time. We agree with the points raised and ran the corresponding experiments; we summarize them below and have added the corresponding tables and figures to the revised paper.
W1: independent test evaluation. We evaluated the selected checkpoints on seven benchmarks that are used neither for training nor for checkpoint selection: mathematics (MATH-500, AMC23, OlympiadBench, Minerva), science (GPQA-Diamond), and code (LiveCodeBench, HumanEval+). Checkpoints are still selected by the AIME average, and each benchmark uses its own prompt and official grader (Avg@16, T = 0.7, top-p = 0.95). The seed-1 checkpoints were not retained, so we use seeds 2 and 3 on Qwen3-1.7B-Base (mean ± std) and seed 2 on Qwen3-30B-A3B-Base:
Although training uses only mathematical problems, CPPO is better on all seven benchmarks on Qwen3-1.7B-Base and on six of the seven on Qwen3-30B-A3B-Base, including science and code, so the gains do not come from selecting checkpoints on the reporting benchmarks. In addition, all baselines in the paper were re-run by us and their checkpoints were selected with the same protocol, so AIME-based selection applies equally to every method.
W2: seeds and uncertainty. We ran two additional seeds on Qwen3-1.7B-Base and one on Qwen3-30B-A3B-Base. AIME24/25/26 Avg@16 (seed 1 is the run in the paper; mean ± std over seeds):
On Qwen3-1.7B-Base, the 95% confidence intervals of the AIME average (t-distribution, three seeds), [12.16, 13.17] for CPPO and [9.99, 11.72] for the baseline, do not overlap. On both models, every CPPO seed scores above every baseline seed.
W4: actual divergence allowance and threshold sweep. We agree that matching the nominal threshold does not match the actual allowance. When the prefix budget is inactive, CPPO's position weights relax the effective threshold on the score to δ/wt∈[0.15,0.1875] (δ=0.15, wmin=0.8); the prefix budget lowers it when divergence accumulates. To test whether the gain comes from a looser overall threshold, we swept the uniform threshold over this range on Qwen3-1.7B-Base (AIME24/25/26 Avg@16):
Loosening the uniform threshold to CPPO's allowance barely moves the score and does not reproduce CPPO's gain. This complements the ablation in Figure 4 (middle): keeping the same set of position weights but randomly reassigning them to positions leaves the score close to uniform masking. Together, these indicate that the improvement comes from which positions receive the tighter allowance, not from the overall stringency.