acceptodds
Under review as a conference paper at ICLR 2027

When Clipping Reverses Correction: Failure Dynamics of Pointwise Forward-KL On-Policy Self-Distillation

Abstract

On-policy self-distillation (OPSD) trains a student on its own generated responses using feedback from the same model conditioned on privileged information. On mathematical reasoning, the original OPSD study finds that stylistic tokens can dominate the training signal over math-related tokens, and that pointwise clipping of the forward KL objective stabilizes training. Pointwise clipping caps each vocabulary-wise forward KL term at a fixed threshold before summing over the vocabulary. Follow-up studies have adopted this clipping, but its effect on training has not been directly examined. In matched training runs differing only in whether clipping is applied, we observe that clipped runs produce substantially more repetitions that persist to the end of the response than their unclipped counterparts. We trace this failure to the clipped objective. We prove that the clipped objective can fail to correct the student toward the teacher and can instead push clipped and unclipped token probabilities away from its teacher. Our training runs agree with this analysis: inside repetitions, the clipped student places less probability than its teacher on leaving the repetition, and more on continuing it, whereas the unclipped runs stay close to their teachers.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models (LLMs), on-policy distillation (OPD), knowledge distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BYf8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines pointwise clipping of the forward-KL objective in on-policy self-distillation (OPSD). In OPSD a student is trained on its own rollouts against the same model conditioned on a reference solution, and each vocabulary-wise term di=pilog⁡(pi/qi)d_i = p_i\log(p_i/q_i) is capped at τ\tau before summing. The authors report that this widely adopted clipping causes text degeneration and give two theoretical results:

  • Proposition 1: relative to exact forward KL, the clipped objective reverses the logit-gradient sign on every clipped token, and on active tokens with pi<qi<pi/PAp_i < q_i < p_i/P_A.
  • Proposition 2: within a fixed clipping region, the clipped objective is minimized by putting zero mass on clipped tokens and inflating each active token to pi/PAp_i/P_A, so it does not restore the student toward the teacher.

Empirically, four matched clipped/unclipped pairs of Qwen3-4B runs (200 updates on OpenThoughts) vary which side thinks. They are implemented in verl with a frozen SGLang teacher restricted to its top-128 tokens. Clipped runs produce 25–450× more terminal loops than their unclipped twins (Fig. 1, Table 3). Inside repetitions, the clipped student's exit mass falls to 0.41–0.70 of the teacher's while 71–84% of the teacher's exit mass is clipped (Fig. 2a), and the copy-token gradient under clipping points toward continuing the repetition where exact KL would correct it (Fig. 2b–c). On AIME 2024/2025, clipped runs loop far more often and generally score below their twins (Fig. 3, Tables 5–6), although no run, clipped or unclipped, exceeds the base model.

Critical strengths

1. A focused, well-controlled diagnosis of a widely copied design choice. Pointwise clipping is inherited by many follow-up OPSD methods (Section 2), yet its effect had not been isolated. Matched runs that differ only in the clip produce a large and consistent effect on terminal loops in all four families (Fig. 1; Table 3, at most 2 terminal loops per phase in any twin vs. 26–381 in the clipped runs). This is directly useful to practitioners, and the recommendation in Section 6 to report degeneration across checkpoints rather than only the best checkpoint is well motivated.

2. The mechanism is linked to measurable training quantities, not only stated. Propositions 1–2 are correct and easy to verify. The authors then test their prediction at the positions that matter: exit mass at the first copy, the fraction of copy positions inside the reversal band pcopy<qcopy<pcopy/PAp_{\text{copy}}<q_{\text{copy}}<p_{\text{copy}}/P_A, and both objectives' copy-token gradients evaluated on the same positions (Fig. 2c). This makes the causal story considerably more convincing than a correlation between clipping and loops alone.

Critical weaknesses

1. The empirical evidence is too thin and partly confounded to support general conclusions.

  • Run coverage: each configuration is a single trajectory (one seed), with one model (Qwen3-4B), one dataset, one threshold (τ=0.05\tau=0.05) and small updates (30 problems × 1 rollout).
  • Effect sizes: the headline 25–450× ratios divide by twin counts of 1–15, so they are unstable.
  • Family M is uninformative: its unclipped twin already scores 0.047 on AIME 2025 at update 25 and 0 thereafter (Table 5), because it stops closing the thinking tag.
  • Faithfulness of the reproduction: no run in any configuration improves over the base model, and the unclipped F and L twins also end clearly below it (e.g., AIME 2024 F-no-clip 0.661 and L-no-clip 0.614 vs. base 0.727). The setup therefore does not reproduce the OPSD gains it claims to reproduce, and it departs from OPSD in ways that affect the clipped set. The loss is computed on the teacher's top-128 tokens renormalized at T=1.1T=1.1, rather than the full vocabulary (Section 3), and rollouts use top-p 0.8 / top-k 20.

Taken together, it is unclear whether the failure characterizes clipped OPSD in general or this particular non-improving regime. A τ\tau sweep, multiple seeds for at least one family, and one full-vocabulary run would substantially strengthen the claim.

2. The theory is modest, the causal link remains inferential, and no remedy is tested.

  • Theory: Proposition 1 follows almost immediately from the fact that clipped terms are constant (already noted by Chen et al., 2026b) while softmax normalization couples all logits through log⁡Z\log Z. Proposition 2 is a standard Lagrangian calculation on a fixed region. Neither handles the fact that the clipped set changes during training, which the authors acknowledge (Section 4.3).
  • Causality: the paper itself states that the trajectory evidence "cannot establish" the path from gradient reversal to loops.
  • Missing intervention: the most direct test is a variant that keeps the magnitude cap but removes the sign reversal, e.g., reweighting each term with a stop-gradient factor sg(min⁡(1,τ/di))\mathrm{sg}(\min(1,\tau/d_i)) instead of replacing it with a constant. If loops disappear under such a variant while the stylistic-token protection is retained, the mechanism would be established and the paper would also offer a usable fix.

As written, the paper diagnoses a problem but leaves practitioners without a recommended objective. The unclipped runs also degrade, and the paper never shows whether the instability that originally motivated clipping reappears without it.

Minor. The text has many grammatical errors (e.g., "No study reports that pointwise clipped objective contribute…", "Sections4.3", "forward kl"), and Fig. 2 is very dense. The phase-portrait discussion in Appendix A uses informal language ("shatters", "false attractor") under a hold-other-logits-fixed assumption that does not hold in training; it would be better shortened or clearly labeled as intuition.

Questions for authors
  1. (Clarification; affects my assessment of faithfulness.) Why do none of the runs, including the unclipped twins, improve over the base model, unlike the original OPSD results? Have you verified that your setup reproduces OPSD's reported gains under its own configuration, in particular with the full-vocabulary loss rather than top-128 renormalization?
  2. (Clarification.) How sensitive is the loop onset to τ\tau? Do larger thresholds, which make the clipped set smaller and PAP_A closer to 1, delay or remove the failure, as Proposition 1 would predict?
  3. (Could change my assessment.) Have you tried an objective that caps the magnitude of large terms without zeroing their gradient, such as the stop-gradient reweighting above? Does it prevent the loops while retaining the stability benefit that motivated clipping?
  4. (Clarification.) For family M, both arms collapse and the unclipped twin scores zero from update 25. Should M be treated as evidence about clipping at all, or would the claims hold with only F, K and L?
  5. (Clarification.) Did the unclipped twins show the stylistic-token domination or instability that OPSD originally reported? If not, what do you recommend practitioners use instead of pointwise clipping?
1 comment

Dear Reviewer,

Thank you so much for your thoughtful initial reviews!

With only two days left before the discussion window closes, I wanted to send a friendly reminder to check out the authors' rebuttal. They have put a lot of effort into answering the specific questions you raised.

Whenever you have a spare moment today, please drop a brief reply in the thread to let the authors know if their responses have addressed your concerns, or if any questions remain. Your timely feedback makes a huge difference in ensuring a fair and constructive review process!

Warm regards

Reviewer_sqZpOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines pointwise clipping of the forward-KL objective used in on-policy self-distillation. It shows that clipping can reverse the logit gradient on underrepresented teacher tokens and, within a fixed clipping region, favors assigning zero probability to the clipped tokens. Four matched training pairs on Qwen3-4B connect clipping to substantially more terminal repetitions. The paper then uses exit-mass and copy-token diagnostics to investigate how this happens. I see the main contribution as a useful diagnosis of a particular training objective, with a relatively compact mathematical explanation.

Critical strengths
  • I like the distinction between removing a token's direct loss gradient and removing all pressure on its logit. The softmax coupling makes the latter intuition wrong, and the paper explains the consequence clearly. This is an easy detail to miss when treating clipping as a harmless stabilization trick.
  • Clipping increases terminal loops across all four thinking configurations, with particularly large separation in family F: 2,250 versus 5 loops over training. Holding the teacher and privileged context fixed within each pair is a sensible way to isolate the effect of clipping in this implementation.
  • The first-copy exit mass and the comparison of both objectives on the same sampled positions help connect the output behavior to the proposed explanation. I also appreciate that the authors explicitly acknowledge that these observations support, rather than establish, the full mechanism.
Critical weaknesses
  1. Earlier work already notes the loss of direct term gradients [1] and shows that the clipped objective need not be nonnegative [2]. The two propositions sharpen this picture, but share a simple explanation. Fix A,CA,C, let PA=∑i∈ApiP_A=\sum_{i\in A}p_i, and define ri=pi/PAr_i=p_i/P_A on AA and zero on CC. Then

    Dclip(q)=PADKL(r∥q)+PAlog⁡PA+∣C∣τ.D_{\mathrm{clip}}(q)=P_A D_{\mathrm{KL}}(r\|q)+P_A\log P_A+|C|\tau.

    This gives the logit gradient PA(q−r)P_A(q-r) and, after checking feasibility, the boundary optimum q=rq=r, which finite softmax logits can only approach. I like the interpretation: clipping effectively replaces the teacher with a distribution that drops the clipped tokens. Still, these are fairly direct consequences of standard KL geometry. The main novelty therefore rests on connecting this observation to repetition in actual training; the fixed-region results alone do not establish those dynamics.

  2. Original OPSD already ablates clipping and finds it helpful in that setting [3], making the abstract's claim of no direct examination too broad. Here, top-128 renormalization, full-parameter updates, and longer rollouts change the setting substantially. The paired comparison is valid, but I am unsure how broadly the result transfers. One training seed per configuration also leaves its variability unclear.

  3. If it uses Equation (1), 1−qcopy1-q_{\mathrm{copy}} measures exit conditional on the top-128 support, whose total mass can differ between student and teacher. If it uses raw full-vocabulary probabilities, a clarification would resolve this concern. This matters because the diagnostic supports the proposed mechanism.

[1] Yutong Chen et al. DualOPSD: Adaptive Privileged Teachers for On-Policy Self-Distillation.

[2] Yangyang Feng et al. PAST: Privileged Adaptation from Complete Student Trajectories for On-Policy Self-Distillation.

[3] Siyan Zhao et al. Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models.

Questions for authors
  1. Does the loop-rate separation persist in a representative full-vocabulary paired run?
  2. Does Figure 2 use top-128-normalized or full-vocabulary probabilities?
  3. Has any matched pair been repeated with another training seed?
1 comment

Dear Reviewer,

Thank you so much for your thoughtful initial reviews!

With only two days left before the discussion window closes, I wanted to send a friendly reminder to check out the authors' rebuttal. They have put a lot of effort into answering the specific questions you raised.

Whenever you have a spare moment today, please drop a brief reply in the thread to let the authors know if their responses have addressed your concerns, or if any questions remain. Your timely feedback makes a huge difference in ensuring a fair and constructive review process!

Warm regards

Reviewer_WhjhOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies whether pointwise forward-KL clipping can make on-policy self-distillation go wrong. It shows that clipping can reverse corrective logit gradients and favor distributions that drop clipped tokens entirely. Four paired training comparisons link clipping to substantially more repetitive outputs.

Critical strengths
  • The paper asks a clear, practical question about a training trick that several follow-up works have adopted.

  • The fixed-region optimum analysis goes beyond noting that clipping removes gradients: it shows what the objective actually favors.

  • The paired runs show a large, consistent increase in repetition across four thinking-mode configurations.

Critical weaknesses
  • The gradient-reversal result also appears in the nearly simultaneous paper, “Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoning”, which this draft is older than this paper but does not cite or discuss. It's true that the mentioned paper is likely to be after conference submission, but clarifying the overlap would help establish what is new here.

  • All runs use one model and one training seed. The top-128 renormalized loss also differs from the original full-vocabulary OPSD setup.

  • The diagnostics fit the proposed explanation, but do not establish that gradient reversal drives the loops. A direct intervention would make the case stronger.

Questions for authors

see weakness

1 comment

Dear Reviewer,

Thank you so much for your thoughtful initial reviews!

With only two days left before the discussion window closes, I wanted to send a friendly reminder to check out the authors' rebuttal. They have put a lot of effort into answering the specific questions you raised.

Whenever you have a spare moment today, please drop a brief reply in the thread to let the authors know if their responses have addressed your concerns, or if any questions remain. Your timely feedback makes a huge difference in ensuring a fair and constructive review process!

Warm regards

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.