acceptodds
Under review as a conference paper at ICLR 2027

Calibrating Teacher–Student Discrepancy for On-Policy Distillation

Abstract

On-policy distillation (OPD) improves reasoning models by learning the token-level discrepancy between a stronger teacher and an on-policy student. However, this discrepancy does not purely reflect the capability gap between the teacher and the student: it also contains deviations arising from the teacher itself, which are consequently mixed into the observed teacher–student discrepancy and indiscriminately learned by standard OPD during training. This issue is further exacerbated by privileged OPD, where privileged information induces larger teacher-side likelihood shifts, thereby encouraging the student to learn more of the teacher's own deviation. We introduce Calibrated On-Policy Distillation (Cal-OPD), which estimates the teacher's self-deviation region through positive and negative privileged interventions and calibrates the original teacher–student discrepancy by retaining only the component that lies beyond this region. Experiments on mathematical reasoning benchmarks show that, while retaining only about 52–65% of the original teacher–student discrepancy as the optimization signal, Cal-OPD consistently outperforms standard OPD and its variants across model scales.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Reasoning Models, Post-Training, Teacher–Student Discrepancy

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jteCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether teacher supervision in OPD includes fluctuations caused by changes in context. It calls changes in the teacher's likelihoods on fixed student tokens TSD. The authors examine this sensitivity through contextual interventions involving instructions, evaluations, answers, and solutions, and use these observations to propose Cal-OPD. Cal-OPD scores the same student trajectory using the original teacher prompt and prompts with positive and negative evaluations to estimate a teacher likelihood interval for each token. If the student's likelihood falls within this interval, the distillation advantage is set to zero. Otherwise, only the signed residual beyond the interval boundary is retained. Experiments compare Cal-OPD with standard OPD and its variants across two Qwen3 teacher and student configurations and six mathematics benchmarks. The paper also examines intervention types, the interval relaxation factor, and signal filtering strategies.

Critical strengths
  1. The calibration rule is clear and easy to understand. In Table 3, Cal-OPD achieves the highest average mathematics accuracy in both teacher and student configurations, supporting the practical value of this selective supervision approach.
  2. The main text provides a useful analysis of intervention types and calibration strength. Comparisons across contextual interventions and ablations of the relaxation factor show that excessive calibration can filter out useful supervision, helping clarify when the method is appropriate.
Critical weaknesses
  1. My main concern is that the link between TSD under evaluative interventions and unhelpful supervision remains insufficiently supported. The authors acknowledge that calibration using solution-level interventions can remove task-relevant information. For the positive and negative evaluative interventions used in the main method, however, similar likelihood changes and their concentration on connective tokens do not establish that the attenuated signals are unhelpful for planning or error correction. The results support the effectiveness of the calibration rule, but attributing the gains to removing unhelpful teacher fluctuations requires more direct evidence.
  2. Evaluation covers only mathematics tasks and the Qwen3 family. It remains unclear whether the evaluative interventions and calibration gains transfer to other tasks or model families, so the scope of the conclusions needs to be clearer.
Questions for authors
  1. For the evaluative interventions used in the main method, could you provide more direct evidence that the attenuated signals are unhelpful for learning? For example, you could analyze how these signals relate to subsequent reasoning correctness in existing trajectories, or test whether semantically similar paraphrases of the evaluation prompts produce stable intervals and training gains. This would also help clarify the limits of the proposed interpretation of TSD.
  2. Could you discuss when evaluative interventions would be appropriate for non-mathematical tasks or other model families, whether the prompts would need adjustment, and how these considerations limit the conclusions? If preliminary results are already available, one representative setting would be helpful.
Reviewer_4oX7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper is motivated by finding that the teacher’s logprob on some stylistic tokens (wait, maybe, however etc) moves around a lot just by changing the teacher prompt, even when the student rollout is fixed. so they argue part of the teacher-student gap in opd is just teacher noise, not real knowledge. They use a contrastive prompting trick (a pos/neg prompt pair, like “verifier says this is correct” vs “verifier says this is wrong”) to measure how much the teacher fluctuates on each token, which gives an interval. then when computing the opd advantage they take this interval into account: if the student is inside it, don’t learn that token, otherwise only learn the part outside. Experiments on qwen3 models on math benchmarks show it helps a bit over standard opd, and responses get shorter.

Critical strengths
  1. the analysis part is interesting and pretty clean. They keep the student rollout fixed and only change the teacher prompt, so the shift really comes from the teacher. The finding that giving the teacher a correct vs wrong answer moves its logprob in mostly the same direction is useful, especially for the privileged opd and self-distillation work, since it suggests a lot of the “privileged signal” might just be the prompt perturbing the teacher.
  2. The method is simple w/ just 2 extra teacher forwards and changing the advantage, so it’s easy to plug into existing opd code. showed performance improvement on qwen3
Critical weaknesses
  1. the 20k evaluation budget might not be enough to reflect the true scores. what is the truncation rate? in my experience these qwen3 models usually need 32k for competition math with the 2507 thinking models. the teacher numbers suggest heavy truncation: 4B-Thinking-2507 gets 53.1 on AIME25 here vs. 81.3 on the model card, and 30B-A3B-Thinking-2507 gets 65.4 vs. 85.0. This matters more for this paper than usual, because Cal-OPD produces much shorter responses than OPD. under a tight cap, shorter outputs get truncated less, so part of the gain could just be fewer truncations rather than better reasoning.

  2. in training, the same pos/neg verifier templates are applied to every rollout regardless of correctness, so one of the two is always false. for a wrong rollout, the false “correct” label can raise teacher likelihood on the wrong tokens and push U toward the student, which weakens exactly the correction we would want (and lambda can amplifies it). same for correct rollouts with the false “wrong” label on L

Questions for authors

what is the truncation rate per method and benchmark at 20k? can you re-evaluate at ~32k? this could change my assessment.

the paper’s premise is that high-TSD tokens (such as wait, maybe) are mostly noise and not good for learning. but in my experience some of these stylistic tokens are very important for OPD, and training only on them can already improve performance. what would happen if you only train only on high-TSD tokens? if that also improves over the student, the “noise” interpretation might not hold.

is eval using thinking mode?

Reviewer_AAXrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper empirically characterizes Teacher Self-Deviation (TSD), defined as changes in a teacher’s token-level log-probabilities under contextual interventions while holding the student-generated trajectory fixed. Across the examined interventions, these changes arise without task-specific knowledge, often remain aligned under contrasting feedback or reference correctness, and occur more frequently on discourse expressions than on numerical and mathematical tokens. Motivated by these observations, the authors propose Calibrated On-Policy Distillation (Cal-OPD), which uses contrasting teacher-side interventions to estimate a token-specific interval and retains only the teacher–student discrepancy beyond its boundary. Experiments on six mathematical reasoning benchmarks with two Qwen3 teacher–student configurations report average accuracy gains of 2.3 and 3.1 percentage points over standard OPD, and the highest average performance among the evaluated baselines. Additional analyses examine intervention choice, calibration strength, training dynamics, and comparisons with alternative ways of reducing the supervision signal.

Critical strengths
  1. Promising empirical performance. Cal-OPD achieves the highest reported average accuracy across six mathematical reasoning benchmarks in both evaluated teacher–student configurations, improving over standard OPD by 2.3 and 3.1 percentage points. These results suggest practical benefit across the two tested scales, although statistical uncertainty estimates are needed to establish confidence in the margins.

  2. Systematic empirical analysis of teacher contextual sensitivity. The paper extends prior observations through controlled comparisons across intervention types, thresholds, and teacher models. This provides a more detailed characterization of how teacher supervision changes with context, although the connection between these findings and the downstream performance gains remains unresolved.

Critical weaknesses
  1. The interpretation of TSD is stronger than the supporting evidence. The analysis establishes contextual sensitivity in teacher probabilities, but does not establish that the corresponding portion of the original teacher–student discrepancy represents harmful supervision. Downstream improvements support the usefulness of the calibration procedure without demonstrating that the proposed explanation accounts for those gains. The claims should be narrowed to distinguish the measured phenomenon from its hypothesized role in distillation.

  2. The empirical advantage needs stronger statistical and compute-controlled validation. The paper does not report confidence intervals for performance differences or clearly document variability across independent training runs. This is particularly important for the 0.4-point average improvement over the strongest TSD-filtering control. Furthermore, equal training steps do not imply equal computational budgets: in the 30B→4B configuration, Cal-OPD takes approximately 56% longer than standard OPD. A comparison under matched training compute or wall-clock budgets would help establish whether its gains justify the additional cost. Statistically reliable improvements under these controls would substantially strengthen the empirical case.

Questions for authors
  1. Statistical reliability. How many independent training runs underlie the reported results? Could you report variability across training seeds and confidence intervals for the performance differences against standard OPD, the strongest competing baseline, and TSD-Filter? The 0.4-point average improvement over TSD-Filter particularly needs uncertainty estimates.

  2. Matched training budgets. Does Cal-OPD retain its advantage over standard OPD under matched computational or wall-clock budgets? For the 30B→4B configuration, Table 12 reports approximately 56% longer training time. Either performance-versus-training-time curves on identical hardware or an OPD run extended to the same budget would help assess this trade-off.

  3. Testing the role of TSD. Could you test whether the correspondence between measured TSD and token-specific attenuation is important? One suggested control is to shuffle Cal-OPD’s multiplicative attenuation weights among tokens with similar original advantage sign, magnitude, and student entropy. This would preserve the zero-advantage fraction and approximately preserve overall signal attenuation while disrupting the TSD-specific assignment. Comparing this control with Cal-OPD would help establish whether contextual sensitivity contributes information beyond these simpler token properties.

  4. Comparison with REOPOLD. How does Cal-OPD relate to REOPOLD, which also selectively regulates teacher–student log-ratio supervision, but attributes the difficulty to extreme negative signals and entropy collapse? A comparison in one representative setting, with matched training budgets, would help establish the additional benefit of intervention-based calibration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.