acceptodds
Under review as a conference paper at ICLR 2027

Distilling Strong Teachers into Small Models: Teacher-Preferred Distribution Alignment Matters

Abstract

On-policy distillation (OPD) has emerged as an effective approach for transferring post-trained capabilities from strong teacher models to students. However, its effectiveness can deteriorate substantially in cross-origin settings, where the teacher and student are initialized from different base models. We show that, unlike in same-origin distillation, the teacher and student can exhibit substantial distributional mismatch in this setting. As a result, applying distillation supervision primarily to student-preferred tokens may yield weak or misaligned optimization signals, limiting capability transfer even from stronger teachers. Motivated by this observation, we systematically compare three strategies for leveraging teacher-preferred tokens: Online SFT, Teacher Top- OPD, and Hybrid Teacher Top- OPD. Teacher Top- OPD consistently outperforms the alternatives in our experiments. We further analyze three factors that determine its effectiveness: the implementation of the reverse KL (RKL) objective, renormalization of the truncated teacher distribution, and the choice of . Our analysis leads to a simple recipe, Teacher Top- OPD with renormalization, which achieves state-of-the-art performance across multiple experiments and outperforms prior baselines by up to 10.4 points. Scaling experiments show that increasing training compute consistently improves student performance, while stronger teachers yield a higher performance ceiling. Notably, this recipe enables a distilled Qwen3-14B student to outperform the official Qwen3-14B model by 14.7 points on AIME24/25 and AMC23. These findings demonstrate that teacher-preferred token supervision is key to effective cross-origin distillation from strong teachers to small students.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language model, post training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LeKjOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the distribution mismatch problem between teacher and student models in on-policy distillation, particularly the distributional differences encountered in cross-source distillation scenarios. The authors systematically compare three supervision strategies:Online SFT, Teacher Top-K, and Hybrid Teacher Top-K. Experimental results demonstrate that, under the settings examined, Teacher Top-K consistently outperforms the other two strategies.

Critical strengths
  1. The paper is well-written, clearly structured, and properly formatted, offering good readability.

  2. The authors conducted comprehensive experiments comparing various distillation strategies and further analyzed why Teacher Top-K performed better under the examined settings, providing empirical evidence regarding the impact of different supervision strategies on distillation performance.

Critical weaknesses

1 The authors present Online SFT and Hybrid OPD as two methods proposed in this paper, but provide insufficient explanation of their relationship to prior work and their incremental contributions. Specifically, the paper does not discuss the relationship between Online SFT and ImitKD, nor does it clearly explain the substantive differences between Hybrid OPD and existing mixed-distribution distillation objectives.

2 The novelty of the paper is relatively limited. Both Online SFT and Hybrid OPD, which the authors claim to propose, have fairly direct precedents in prior work, but their substantive differences from and incremental contributions over prior work have not been sufficiently clarified. Furthermore, in the experiments reported in this paper, neither method outperforms the existing Teacher Top-K OPD approach, failing to demonstrate a performance advantage over the existing approach.

Questions for authors

1 What specific hypotheses did the authors aim to test with Online SFT and Hybrid Teacher Top-K? Since comparisons with Teacher Top-K involve variations in supervision forms, loss functions, and target distributions, could controlled experiments be conducted to isolate the impact of these individual factors on performance?

2 Although the paper covers Qwen3 student models of various sizes, the experiments are limited to a single model family. Have the authors validated their main conclusions using student models other than Qwen3?

3 Section 4 defines the support set for Hybrid Top-K as the Top-K set selected by the student; however, Section 4.1 states that a teacher weight of 100% corresponds to Teacher Top-K. Based on the aforementioned definition, this scenario should yield a teacher distribution match over the student's Top-K support set. The authors are requested to clarify which support set was actually used in the experiments and whether the method of constructing the support set remained consistent across different mixing ratios.

4 Section 5.3 finds that K=32 yields the best performance under the settings examined. Does this choice remain optimal across different teacher and student sizes, as well as different tasks?

Reviewer_dc8POpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies distribution mismatch when distilling strong teachers into smaller, cross-origin students through on-policy distillation. Conventional OPD evaluates teacher signals mainly on student-preferred or sampled tokens. When teacher and student modes differ, these tokens may have low teacher probability, producing gradients that suppress the student's current choices without identifying which teacher-preferred tokens should gain probability. The paper compares Online SFT, Teacher Top-K OPD, and Hybrid Teacher Top-K OPD, establishes the need to renormalize the truncated teacher distribution to avoid a mass-shrinking shortcut, and analyzes support size, teacher capability, student capacity, and training budget.

Critical strengths
  1. The paper provides a clear and convincing explanation of cross-origin OPD failure: supervision on student-sampled tokens supplies mostly negative suppression, whereas teacher Top-K tokens explicitly identify useful probability-increase directions.

  2. The theoretical and empirical treatment of renormalization is strong. The paper explains how unnormalized truncated RKL admits a mass-shrinking shortcut and empirically verifies the resulting loss of probability mass on the teacher support.

  3. Experiments cover mathematical reasoning, code generation, several teacher-student scales, multiple benchmarks, and longer training schedules. Teacher Top-K produces large and consistent gains over Sample OPD and Student Top-K OPD and shows encouraging scaling behavior up to a 14B student.

Critical weaknesses
  1. Most experiments remain within the Qwen3 ecosystem. The claimed cross-origin setting appears to involve different post-training origins or teacher configurations rather than genuinely different tokenizers, base-model families, or architectures, limiting the breadth of the conclusion.

  2. Teacher Top-K requires querying the teacher's Top-K distribution at every student-generated prefix. Teacher inference, communication, and storage costs may be substantial, but complete GPU-time, throughput, and query-cost comparisons are not reported.

Questions for authors
  1. Please specify how every pair labeled cross-origin differs in its base model, tokenizer, and post-training data. If these pairs share the same Qwen3 base and vocabulary, do the authors agree that the evidence primarily supports post-training distribution mismatch rather than general cross-family distillation?

  2. Based on existing logs, what are the per-step teacher computation, communication volume, throughput, and total GPU time of Teacher Top-K relative to Sample OPD and Student Top-K OPD? Substantial Top-K access costs would affect the practical value of the reported gains.

Reviewer_gj54OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies on-policy distillation when the teacher and student come from different base models. It argues that student-preferred supervision, such as sampled-token or student Top-K OPD, gives weak signals in this setting because the teacher ranks the student’s preferred tokens very low. It compares three teacher-preferred alternatives, and Teacher Top-K OPD works best. It then analyzes the implementation choices behind Teacher Top-K, mainly renormalization and the support size, and settles on renormalized Teacher Top-K. Experiments and validations involve math and code with three Qwen3 teacher-student pairs.

Critical strengths
  • The implementation analysis is useful in practice. The paper shows that the REINFORCE-style surrogate matches direct RKL only when renormalization is applied consistently. Without renormalization, the student shrinks its mass on the teacher support and the math average drops from 19.55 to 2.26. The training diagnostics make this failure easy to see.

  • The gains over student-preferred OPD are consistent across all three pairs and both domains. They are large with the stronger teacher This is the main evidence that teacher-preferred supervision is the better default for these pairs.

Critical weaknesses
  • The final recipe, renormalized Teacher Top-K with K=32, seems to be the same objective as the local support matching loss of Fu et al. (2026). And the mass-shrinking result is expected: without renormalization, the truncated KL is not a proper divergence on the support. Renormalization is also compared only with this improper variant. It is not compared with a mass-preserving objective, such as the tail-bucket objective in Huang (2026), which the paper cites but does not test

  • All pairs are Qwen3 models sharing a tokenizer. The same- vs. cross-origin contrast in Figure 2 also changes the student from 4B to 1.7B. The win over Online SFT and Hybrid comes mostly from the 1.7B student (at 8B it is 45.13 vs. 45.00 on math). The +14.7 headline compares a math-trained student (distilled from the much stronger Qwen3-30B-A3B-Instruct-2507) with the general-purpose official Qwen3-14B. No other OPD method is run at 14B, so the number says more about the teacher than about the recipe.

Questions for authors

Beyond the analysis, what differs between your final recipe and the objective of Fu et al. (2026)?

Hybrid Top-K is defined on the student-selected support, so \alpha=1 should give renormalized Student Top-K. But Section 4.1 treats the 100% setting as Teacher Top-32 OPD. Which support was used in Figure 3, and which \alpha is reported in Table 1?

Comments on the paper

1 comment

We thank all reviewers for their careful reading and constructive feedback. We are glad that the reviewers appreciated our investigation of how to effectively distill large, strong teachers into smaller students. In particular, the distribution-mismatch analysis was found convincing (dc8P), and the analysis of renormalization and objective implementation was recognized as informative and practically useful (gj54, dc8P). The reviewers also appreciated the empirical comparisons of supervision strategies (LeKj), the consistent gains over student-preferred OPD across the evaluated settings (gj54, dc8P), and the investigation of teacher strength, student capacity, and training budget (dc8P).

The main concerns were the positioning of our contribution relative to prior work, the breadth of the teacher–student settings, the absence of alternative distillation baselines at 14B, comparisons with other objectives, the definition of Hybrid OPD, the robustness of the choice of K, and the computational cost of teacher Top-K supervision. We have added experiments and provided clarifications in our individual responses, as summarized below.

ConcernReviewerOur actions
1. Central contribution and relationship to prior workgj54, LeKjClarified the central question: why do small students not always benefit from stronger teachers, and how can supervision design improve capability transfer? Positioned the contribution around the analysis of distribution mismatch, supervision signals, objective implementation, and student capacity, and explicitly attributed Online SFT to ImitKD.
2. Generality beyond the original model pairsgj54, dc8P, LeKjAdded Skywork-7B → DeepSeek-R1-Distill-Qwen-1.5B experiments. Teacher Top-K achieves 44.30% average math accuracy, compared with 39.97% for Sample-token OPD, 42.29% for Student Top-K, 33.62% for Online SFT, and 43.42% for Hybrid OPD.
3. Distillation baselines with a 14B studentgj54, dc8PAdded five-method comparisons using Qwen3-30B-A3B-Instruct-2507 → Qwen3-14B-Base. With the same teacher and student, Teacher Top-K achieves 55.38%, compared with 38.25%, 36.38%, 53.93%, and 54.31% for Sample-token OPD, Student Top-K, Online SFT, and Hybrid OPD, respectively.
4. Comparison with a mass-preserving objectivegj54Reproduced the tail-bucket objective of Huang (2026). It achieves 16.47% average math accuracy versus 17.97% for Teacher Top-K, with substantially longer responses and transient truncation rates approaching 100%. Explicitly preserving tail mass therefore does not by itself ensure stability under the tested settings.
5. Hypotheses behind the supervision comparisonsLeKjClarified the questions tested by hard-label supervision and mixed targets. Added Teacher Top-1 and JS-divergence comparisons, which achieve 13.64% and 14.86%, respectively, versus 17.97% for Teacher Top-32. Separately, the controlled mixture-weight ablation fixes the support and loss: final average accuracy increases from 14.9% to 19.6% as the teacher weight increases from 25% to 100%.
6. Hybrid OPD support and mixing coefficientgj54, LeKjAcknowledged the error in the method description and clarified that all mixing ratios use the teacher-selected Top-32 support, with both distributions separately renormalized on that support. Thus, α = 1 recovers Teacher Top-K; the Hybrid results in Table 1 use α = 0.5.
7. Choice of K at a larger student scaleLeKjAdded a 14B-student sweep over K = {4, 16, 32, 64}, yielding average math accuracies of 55.07%, 54.86%, 55.38%, and 51.49%. These results support K = 32 as an effective choice in this additional setting, while showing that a larger support does not necessarily improve performance.
8. Computation, communication, and storage costsdc8PReported component timings around training step 200 and estimated communication and storage requirements. Teacher Top-32 exchanges 256 bytes per supervised position for indices and probabilities. Clarified that observed timing differences include differences in generated trajectory lengths and do not isolate the overhead of Top-K selection.

Our individual responses provide the point-by-point explanations and full experimental tables behind this summary. The additional experiments broaden the evidence for our findings, while our conclusions remain scoped to the configurations and objectives evaluated.

We are sincerely grateful to the reviewers for helping us make the paper more precise and better supported, and to the Area Chair for their time and consideration. We look forward to continuing the discussion.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.