acceptodds
Under review as a conference paper at ICLR 2027

Mismatch Matters: On-Policy Distillation Beyond Token Agreement

Abstract

On-policy distillation (OPD) has emerged as a core component of modern LLM post-training pipelines, yet we reveal a failure mode: degenerate agreement, where students exploit repetitive loops to achieve near-perfect token agreement with the teacher despite globally flawed responses. We therefore shift our focus from agreement to teacher–student mismatch and find that the mismatch tokens can be mainly categorized to two types: student-excess tokens and student-deficit tokens. Specifically, student‑excess tokens are those generated by the student but assigned near‑zero probability by the teacher; their log‑ratio corrections grow unbounded and destabilize the update. Student‑deficit tokens, in contrast, are preferred by the teacher but rarely sampled by the student; their absence blocks the transfer of the teacher’s reasoning patterns. To tackle these mismatch directions, we propose TIDE (Token-level Independent Deficit–Excess correction), which applies bounded Hellinger shaping to suppress the most severe sampled excesses and an analytic teacher top- injection to restore deficient probability mass without requiring deficit tokens to be sampled. Across mathematical reasoning benchmarks with multiple Qwen3 teacher–student pairs, TIDE consistently outperforms standard OPD and recent token-selection and reward-shaping baselines. Besides, the gains of TIDE are more pronounced under strong teacher–student mismatch, where it improves Avg@8 from to , reduces average response length by a factor of , and substantially reduces formatting failures. Code is available at https://anonymous.4open.science/r/TIDE-B1B2.

Primary area:
generative models
Keywords:
Large language models, On Policy Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NecPOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

TIDE addresses two forms of teacher-student mismatch in on-policy distillation: unstable negative updates for student-excess tokens, and the rarity of student-deficit tokens in student rollouts. It combines gated Hellinger suppression with analytic recovery over the teacher's top-K tokens. The experiments cover nine math benchmarks and two teacher-student pairs.

Critical strengths

S1. The controlled-prefix and teacher-continuation experiments help diagnose a failure of OPD: local agreement can coexist with repetitive responses that still fail.

S2. Under strong mismatch, TIDE raises Avg@8 from OPD's 6.9% to 20.3%, a substantial improvement. Responses are also shorter and less likely to omit an answer.

S3. The component ablations and sensitivity studies help assess the method's design choices. Using the final checkpoint for every method avoids selecting checkpoints on the evaluation benchmarks.

Critical weaknesses

W1. It is unclear how much mismatch gating contributes beyond ordinary analytic teacher supervision. The deficit-only variant already reaches 18.32% Avg@8, close to AOPD, yet the experiments include neither ungated top-K cross-entropy nor a coverage-matched random gate. These controls would help establish whether the proposed selection rule accounts for the gains.

W2. The deficit score responds to teacher truncation even when the student matches the teacher. When q=p, Eq. 3 gives dt=−logPt, where Pt is the teacher's probability mass on its top-K set. The score is positive whenever Pt<1, so the deficit loss still pushes probability into that set despite perfect agreement. Positions can thus be selected because of the teacher's probability spread, with no actual student deficit. This also raises a question about the claim that matched positions receive no supervision.

W3. Appendix E.3 renormalizes teacher probabilities and removes the 1K factor from AOPD. The original AOPD objective uses unnormalized probabilities and includes that factor. This modification changes the strength of distribution matching relative to positive policy-gradient updates, so the reported 2.0-point Avg@8 gap needs to be checked using the published loss or a guidance-weight sweep.

W4. The two teacher-student pairs differ in student initialization, teacher size, and teacher strength, making it hard to attribute their different behavior to mismatch. The paper also reports neither variation across training seeds nor accuracy confidence intervals. These omissions matter for the smaller gains: 0.1 Avg@8 points over FiRe-OPD in the weak setting and 0.2 Pass@8 points over AOPD in the strong setting.

Questions for authors

Q1. How does TIDE compare with ungated top-K cross-entropy and a random gate with the same coverage?

Q2. How much of the deficit score comes from teacher truncation? Does TIDE prevent entry into loops, or increase the probability of exiting an existing loop?

Q3. Does the gain over AOPD remain with its published loss and a tuned guidance weight?

Q4. Please report variation across training seeds and the initial overlap values for both teacher-student pairs.

Reviewer_Eu1BOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper focuses on training LLMs with on-policy distillation (OPD). It first establishes that when there is a significant gap between the teacher and the student distribution, OPD can converge to a degenerate student which generates looping responses with lots of repetition.

This behavior is attributed to how OPD treats token positions with a huge mismatch between the teacher and student distributions. For positions where the student puts high mass on its generated token and the teacher puts low mass (called student-excess tokens), their contribution to update can be huge, which can destabilize the update. For the opposite case where the teacher puts relatively high mass on a token compared to the student (called student-deficit tokens), student rarely samples them and isn't incentivized to learn to produce them.

The proposed method TIDE aims to fix both theses issues. For the student-excess tokens, it applies a transformation to the log probability-ratio that ensures that this ratio stays bounded. For the student-deficit tokens, it adds a cross-entropy term to the loss that directly incentivizes the student to produce these tokens. It is shown that TIDE substantially outperforms OPD in cases where there is a big gap between the initial student and teacher distributions.

Critical strengths

Studying OPD is a timely topic. The issues discussed with it and the remedies seem natural and easy to incorporate into existing pipelines.

Critical weaknesses

Implications for more realistic setups unclear: It is unclear to what extent the degeneration issue discussed in the paper is a product of the particular setup being considered vs an issue with OPD in more realistic setups. In many of the frontier model training setups where OPD is used these days [5, 6] , it is done after offline supervised finetuning (SFT) on teacher generated traces. However, the paper shows the degeneration issue to appear in the case with a strong student-teacher mismatch (the student is a base model while the teacher is a post-trained Qwen3-8B) without the SFT phase. It is unclear whether the training dynamics would exhibit the same degeneration in this more commonplace setup. Although there is another weak mismatch experiment in the paper, I could not find a corresponding degeneration analysis. Moreover, it does not include the initial SFT phase on its OPD-teacher traces, so its implications for such a setup is unclear.

As a consequence, it is also unclear to what extent the remedies being proposed would be needed in the more realistic setup with initial SFT. For example, the additional cross-entropy term introduced in the loss is there so that the student is incentivized to produce tokens which receive high probability mass under the teacher. One can imagine that the initial SFT phase already serves this motive. Therefore, the effectiveness and need of proposed method needs to be evaluated in this more realistic setup.

Unclear contribution over existing literature: The issues discussed in this paper have already been pointed out in some of the initial works on the topic. These works also propose mitigations for these issues. It is unclear in what sense the mitigations proposed in this paper improve over existing proposals. For example, MiniLLM [1] already discusses the issue of student producing degenerate sentences. DistiLLM [2] discusses the gradient stabilization issue (similar to the student-excess issue discussed in this paper), and proposes skewed version of KL divergence to fix it. EOPD [3] explicitly says that the reverse KL objective used in on-policy distillation can "ignores teacher modes that the student does not visit", and adds a forward KL term to fix it (GKD [4] does something similar as well). Thus, the issues with OPD pointed by this paper are not new. While the mitigations are similar in spirit (e.g. adding forward KL loss is very similar to the additional cross-entropy loss term introduced), their exact form differs. However, the value-addition of these differences is unclear.

On the presentation side, while the paper cites many of these prior works, many times it doesn't do it in a way that makes it clear that the prior works already discussed these issues. For example, on the degeneration issue, the paper says "We identify student-induced teacher hacking as a previously overlooked failure mode of OPD", but this degeneration issue has already been pointed out in prior work by MiniLLM [1]. It would be helpful to clearly discuss in the main paper body what the prior work has done with respect to the issues being discussed and what the current paper adds on top.

Statistical significance of the results: I couldn't find error bars for the reported performance numbers. In the somewhat more realistic setup (with weak student-teacher gap), the gains of the proposed method are small and in the absence of error bars, it is unclear whether these are statistically significant.

Overall, I think the paper would benefit from clearly discussing what its conceptual contributions are over prior work, using more realistic setups to evaluate the proposed method and using clear error bars to distinguish signal from noise.

[1] https://proceedings.iclr.cc/paper_files/paper/2024/hash/8ac015d409635f196f9e3e9dcfb9a94e-Abstract-Conference.html

[2] https://proceedings.mlr.press/v235/ko24c.html

[3] https://openreview.net/pdf?id=J5i09faOOf

[4] https://openreview.net/forum?id=3zKtaqxLhW

[5] https://arxiv.org/pdf/2505.09388

[6] https://thinkingmachines.ai/blog/on-policy-distillation/

Questions for authors

Response to concerns raised in the weakness section would be helpful.

Reviewer_mETTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies failure modes of sampled-token on-policy distillation (OPD), particularly repetitive student trajectories on which the conditioned teacher also favors repetition, producing low local divergence despite poor response quality. The authors distinguish student-excess tokens, which can receive large negative log-ratio corrections, from student-deficit tokens, which are preferred by the teacher but rarely sampled by the student. They propose TIDE, combining independently gated Hellinger-shaped suppression of sampled excess tokens with analytic cross-entropy supervision over the teacher’s top-K candidates. Experiments on nine mathematical reasoning benchmarks show substantial improvements over vanilla OPD for a Qwen3-8B → Qwen3-1.7B-Base pair, with smaller gains in a second, more closely aligned teacher–student pair.

Critical strengths
  1. The paper provides concrete diagnostics of a practically relevant OPD failure mode. The controlled prefix-repetition experiment and teacher-continuation study support the observation that the teacher can become locally confident in continuing a degenerate student trajectory. Reporting accuracy alongside response length, missing-answer rates, and repetition statistics provides a more informative picture than accuracy alone.

  2. TIDE shows meaningful empirical improvements under the challenging base-student initialization. In the strong-mismatch setting, it improves Avg@8 from 6.9% for vanilla OPD to 20.3%, while reducing average response length from 22,395 to 7,294 tokens. The component ablations and sensitivity studies provide useful evidence that both branches contribute. The shared training setup and use of final checkpoints also make the comparisons easier to interpret.

Critical weaknesses
  1. The experiments do not sufficiently isolate the contribution beyond existing distributional supervision and bounded-update methods. Analytic supervision of unsampled teacher-preferred tokens is already central to AOPD, while on-policy forward-KL distillation is an established alternative to sampled-token reverse-KL updates. The important question is therefore whether TIDE’s independent mismatch gates and Hellinger shaping improve upon these simpler alternatives. The deficit-only ablation remains gated and does not replace an all-position top-K forward-KL baseline. Likewise, the matched-only control uses a different selection rate and does not control for random selection at the same rate or effective update scale. Useful comparisons would include ungated top-K distillation, random gates with matched supervision budgets, and clipped or tanh-shaped excess updates under identical gates.

  2. This matters because the advantage over the strongest reported baseline is considerably smaller than the headline comparison: under strong mismatch, TIDE exceeds AOPD by approximately 2.0 Avg@8 points and 0.2 Pass@8 points, while AOPD has shorter outputs and a lower missing-answer rate. Under weak mismatch, TIDE’s Avg@8 is only 0.1 point above FiRe-OPD. No variability across independent training runs is reported. Moreover, the two mismatch settings also differ in model family, teacher size, and student post-training history, so they do not isolate mismatch as the cause of the difference in gains.
    Several mechanistic and theoretical claims are stronger than the analysis supports. First, the deficit score measures discrepancy from a renormalized, truncated teacher target, not solely mismatch with the original teacher. Let (P_t(K)=\sum_{v\in K_t}p(v\mid s_t)). Even when (q=p), Eq. (3) gives [ d_t=-\log P_t(K)>0 ] whenever the teacher has probability mass outside the top-K set. At a selected position, the deficit loss still has logit gradient (q-\bar p_t), with (\bar p_t) extended by zero outside (K_t). Thus, this branch also concentrates probability onto the truncated support even in the absence of teacher–student mismatch. Proposition 2 is algebraically correct, but this truncation effect complicates the interpretation as pure deficit recovery and calls for reporting retained teacher mass and sensitivity to K.
    Second, Proposition 1 establishes a Hellinger-gradient identity for an ungated expectation over all vocabulary tokens. It does not establish the same identity for the implemented negative-only, quantile-gated update. The claim in Section 3.1 that hard clipping or tanh cannot correspond to a well-defined divergence is also incorrect in general. For example, the convex generator [ f(r)=\int_1^r \operatorname{clip}(\log u,-c,c),du,\qquad c>0, ] defines an f-divergence whose fixed-state gradient corresponds to a clipped log-ratio coefficient. Consequently, divergence consistency alone does not establish the proposed transformation’s advantage.
    Finally, low-KL agreement on corrupted or repetitive prefixes was already studied by Xin et al. (2026), which this paper cites. The controlled repetition analysis is useful, but the claim that this is a “previously overlooked” failure mode should be narrowed.

Questions for authors
  1. Can you compare TIDE with all-position top-K forward-KL distillation, random selection at matched keep-rates, and clipped/tanh excess corrections under the same gates? These comparisons would help establish the contribution of mismatch-based selection and the particular Hellinger transformation.

  2. What fraction of teacher probability mass does K = 16 retain, and how does this vary across selected and unselected positions? Can you separate the effect of correcting teacher–student mismatch from concentrating probability onto a truncated teacher support, including a sensitivity study over K?

  3. Can you report independent training runs and uncertainty for the comparison with AOPD? A controlled initialization or warm-up sweep within the same teacher–student family would also help substantiate the mismatch-dependent interpretation.

  4. Does TIDE primarily prevent entry into repetitive states or help the student escape after entering them? The teacher-continuation experiment suggests that the teacher itself often prefers continuing the loop. Evidence about gate activation and teacher-preferred exit-token probabilities around loop onset would clarify the claimed mechanism.


Additional Comments

  1. The abstract reports a 3.6× reduction in response length, whereas Table 2 gives 22,395/7,294 ≈ 3.07×. Please reconcile these numbers.

  2. Section 2 introduces a trajectory-level reverse-KL objective, but the subsequent update uses only the immediate token-level log ratio. Please explicitly distinguish this local surrogate from the full trajectory-objective gradient, which generally includes effects on future visited states.

  3. Appendix D.3 measures accessibility under four independent draws from the same state. This is a useful diagnostic, but four complete rollouts per prompt generally visit different subsequent states and should not be equated with four draws at every fixed prefix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.