- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper examines pointwise clipping of the forward-KL objective in on-policy self-distillation (OPSD). In OPSD a student is trained on its own rollouts against the same model conditioned on a reference solution, and each vocabulary-wise term is capped at before summing. The authors report that this widely adopted clipping causes text degeneration and give two theoretical results:
- Proposition 1: relative to exact forward KL, the clipped objective reverses the logit-gradient sign on every clipped token, and on active tokens with .
- Proposition 2: within a fixed clipping region, the clipped objective is minimized by putting zero mass on clipped tokens and inflating each active token to , so it does not restore the student toward the teacher.
Empirically, four matched clipped/unclipped pairs of Qwen3-4B runs (200 updates on OpenThoughts) vary which side thinks. They are implemented in verl with a frozen SGLang teacher restricted to its top-128 tokens. Clipped runs produce 25–450× more terminal loops than their unclipped twins (Fig. 1, Table 3). Inside repetitions, the clipped student's exit mass falls to 0.41–0.70 of the teacher's while 71–84% of the teacher's exit mass is clipped (Fig. 2a), and the copy-token gradient under clipping points toward continuing the repetition where exact KL would correct it (Fig. 2b–c). On AIME 2024/2025, clipped runs loop far more often and generally score below their twins (Fig. 3, Tables 5–6), although no run, clipped or unclipped, exceeds the base model.
Critical strengths
1. A focused, well-controlled diagnosis of a widely copied design choice. Pointwise clipping is inherited by many follow-up OPSD methods (Section 2), yet its effect had not been isolated. Matched runs that differ only in the clip produce a large and consistent effect on terminal loops in all four families (Fig. 1; Table 3, at most 2 terminal loops per phase in any twin vs. 26–381 in the clipped runs). This is directly useful to practitioners, and the recommendation in Section 6 to report degeneration across checkpoints rather than only the best checkpoint is well motivated.
2. The mechanism is linked to measurable training quantities, not only stated. Propositions 1–2 are correct and easy to verify. The authors then test their prediction at the positions that matter: exit mass at the first copy, the fraction of copy positions inside the reversal band , and both objectives' copy-token gradients evaluated on the same positions (Fig. 2c). This makes the causal story considerably more convincing than a correlation between clipping and loops alone.
Critical weaknesses
1. The empirical evidence is too thin and partly confounded to support general conclusions.
- Run coverage: each configuration is a single trajectory (one seed), with one model (Qwen3-4B), one dataset, one threshold () and small updates (30 problems × 1 rollout).
- Effect sizes: the headline 25–450× ratios divide by twin counts of 1–15, so they are unstable.
- Family M is uninformative: its unclipped twin already scores 0.047 on AIME 2025 at update 25 and 0 thereafter (Table 5), because it stops closing the thinking tag.
- Faithfulness of the reproduction: no run in any configuration improves over the base model, and the unclipped F and L twins also end clearly below it (e.g., AIME 2024 F-no-clip 0.661 and L-no-clip 0.614 vs. base 0.727). The setup therefore does not reproduce the OPSD gains it claims to reproduce, and it departs from OPSD in ways that affect the clipped set. The loss is computed on the teacher's top-128 tokens renormalized at , rather than the full vocabulary (Section 3), and rollouts use top-p 0.8 / top-k 20.
Taken together, it is unclear whether the failure characterizes clipped OPSD in general or this particular non-improving regime. A sweep, multiple seeds for at least one family, and one full-vocabulary run would substantially strengthen the claim.
2. The theory is modest, the causal link remains inferential, and no remedy is tested.
- Theory: Proposition 1 follows almost immediately from the fact that clipped terms are constant (already noted by Chen et al., 2026b) while softmax normalization couples all logits through . Proposition 2 is a standard Lagrangian calculation on a fixed region. Neither handles the fact that the clipped set changes during training, which the authors acknowledge (Section 4.3).
- Causality: the paper itself states that the trajectory evidence "cannot establish" the path from gradient reversal to loops.
- Missing intervention: the most direct test is a variant that keeps the magnitude cap but removes the sign reversal, e.g., reweighting each term with a stop-gradient factor instead of replacing it with a constant. If loops disappear under such a variant while the stylistic-token protection is retained, the mechanism would be established and the paper would also offer a usable fix.
As written, the paper diagnoses a problem but leaves practitioners without a recommended objective. The unclipped runs also degrade, and the paper never shows whether the instability that originally motivated clipping reappears without it.
Minor. The text has many grammatical errors (e.g., "No study reports that pointwise clipped objective contribute…", "Sections4.3", "forward kl"), and Fig. 2 is very dense. The phase-portrait discussion in Appendix A uses informal language ("shatters", "false attractor") under a hold-other-logits-fixed assumption that does not hold in training; it would be better shortened or clearly labeled as intuition.
Questions for authors
- (Clarification; affects my assessment of faithfulness.) Why do none of the runs, including the unclipped twins, improve over the base model, unlike the original OPSD results? Have you verified that your setup reproduces OPSD's reported gains under its own configuration, in particular with the full-vocabulary loss rather than top-128 renormalization?
- (Clarification.) How sensitive is the loop onset to ? Do larger thresholds, which make the clipped set smaller and closer to 1, delay or remove the failure, as Proposition 1 would predict?
- (Could change my assessment.) Have you tried an objective that caps the magnitude of large terms without zeroing their gradient, such as the stop-gradient reweighting above? Does it prevent the loops while retaining the stability benefit that motivated clipping?
- (Clarification.) For family M, both arms collapse and the unclipped twin scores zero from update 25. Should M be treated as evidence about clipping at all, or would the claims hold with only F, K and L?
- (Clarification.) Did the unclipped twins show the stylistic-token domination or instability that OPSD originally reported? If not, what do you recommend practitioners use instead of pointwise clipping?
Dear Reviewer,
Thank you so much for your thoughtful initial reviews!
With only two days left before the discussion window closes, I wanted to send a friendly reminder to check out the authors' rebuttal. They have put a lot of effort into answering the specific questions you raised.
Whenever you have a spare moment today, please drop a brief reply in the thread to let the authors know if their responses have addressed your concerns, or if any questions remain. Your timely feedback makes a huge difference in ensuring a fair and constructive review process!
Warm regards