acceptodds
Under review as a conference paper at ICLR 2027

Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recall

11 views

Abstract

Logit-based knowledge distillation (KD) is a popular technique used to train smaller, more capable language models (LMs) via supervision from stronger teachers. It remains unclear, however, whether its benefits are consistent across training stages. Through controlled experiments, we find that forward Kullback-Leibler (KL) distillation–the standard KD formulation–with post-trained teachers behaves fundamentally differently during mid-training, an intermediate phase of self-supervised learning on curated corpora. Surprisingly, while forward KD simultaneously improves reasoning and factual recall during pre-training relative to standard next-token prediction (NTP), it instead slows factual recall acquisition during mid-training despite continued reasoning gains. We trace this stage dependence to an asymmetry in teacher confidence across data domains and the student’s evolving knowledge state: teachers are more confident on procedural than knowledge-intensive data, while students acquire low-entropy factual knowledge earlier in training. To mitigate this imbalance, we propose Switch Distillation, a simple mid-training objective that distills on tokens where the teacher is confident, using teacher predictive entropy as a lightweight routing signal, and otherwise falls back to cross-entropy. Switch Distillation consistently outperforms existing distillation objectives across teacher sizes. Relative to standard NTP, it achieves 1.61–1.71x the reasoning performance and 1.13–1.19x the knowledge and commonsense performance while preserving 96.7-96.8% of factual recall. Crucially, these benefits persist after post-training: Switch Distillation closes the factual recall gap while maintaining 1.25–1.32x and 1.13–1.20x gains in reasoning and knowledge and commonsense, respectively. Our results demonstrate that stage-aware knowledge distillation extracts substantially more useful supervision per token under a fixed data budget.

Primary area:
foundation or frontier models, including LLMs
Keywords:
knowledge distillation, mid-training, data-efficient learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_TrkcOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates logit-based knowledge distillation during the LLM mid-training phase. It uncovers the reasoning-recall tradeoff, where standard KD improves reasoning but slows factual recall acquisition. To resolve this, it proposes Switch Distillation, which uses teacher predictive entropy to route tokens KD and NTP, successfully preserving factual recall while boosting reasoning.

Critical strengths
  • The paper identifies a previously uncharacterized, stage-dependent reasoning-recall tradeoff when applying KD during mid-training. This phenomenon is unveiled through sufficient experiments in the paper.
  • The paper links the tradeoff to an insightful interaction between asymmetric teacher confidence across data domains, student knowledge states, and optimization gradients.
  • The proposed Switch Distillation is a lightweight method, requiring no extra parameters or additional model passes beyond standard KD.
Critical weaknesses
  • The paper adopts an experiment-driven narrative approach, but there is not much introduction to the proposed method and comparative experiments, which limits the readers’ understanding.
  • The paper lacks a direct comparison with continuous entropy-aware distillation methods on the same mid-training data mixture, both theoretically and empirically.
  • The experiments evaluate performance on general mixtures. It remains unclear if the method is robust when applied to highly specialized, knowledge-heavy vertical domains.
Questions for authors
  • How does the proposed method fundamentally differ from existing selective distillation or token-level routing methods (like TRKD or EOPD)?
  • Is there any theoretical elaboration on the reason-recall tradeoff that is not dependent on the benchmark performance? Or what is the motivation for this observation and why are these two abilities?
  • With the recent shift toward long-form CoT and synthetic reasoning traces in mid-training, does the reasoning-recall tradeoff and entropy asymmetry still hold, or do structural reasoning tokens eliminate the need for hard token-level routing?
  • What is the training throughput or memory footprints compared to standard NTP or KD?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Trkc (1/2)

Thank you for your thoughtful review and for recognizing the significance of the stage-dependent reasoning–recall tradeoff, our explanatory analysis, and the simplicity of Switch Distillation. Your suggestions have helped us improve the paper’s presentation and motivated new experiments. Responding point-by-point:

1. Presentation and experimental narrative

We appreciate your suggestion to introduce the method and comparative experiments earlier. Our motivation for Switch Distillation comes from our observation of the reasoning–recall tradeoff, a novel empirical phenomenon that persists across teacher sizes, distillation mixing coefficients, and model families, and is a central contribution of this work. Our experiment-driven narrative first establishes the phenomenon (§3), investigates its underlying causes (§ 4), and uses these findings to motivate Switch Distillation (§5).

Our analysis shows that teacher entropy differs systematically across data domains (§4.1), unresolved factual knowledge is concentrated at high-entropy positions (§4.2), and KD can attenuate ground-truth supervision at these positions (§4.3). Together, these observations motivate distilling selectively where the teacher is confident. We agree that this progression would benefit from a clearer roadmap. We revised the introduction to preview Switch Distillation and its comparisons with existing methods, while retaining the empirical motivation for our routing criterion.

2. Comparison with TRKD and EOPD

Thank you for highlighting these related approaches; they provide useful points of clarification for our method design.

TRKD. We already include a direct empirical comparison against TRKD under the same mid-training setup and data mixture (§6.1, Tables 1–2), with methodological differences discussed in App. A.1. TRKD was originally proposed to mitigate in-context learning degradation during pretraining KD. It excludes the lowest-entropy tokens from distillation and applies a mixture of forward KL and CE to remaining tokens. Switch Distillation takes the opposite approach during mid-training: it applies reverse KL to low-entropy tokens and retains ground-truth CE elsewhere. This distinction is motivated by our finding that teacher uncertainty impedes factual knowledge acquisition during mid-training. Empirically, Switch Distillation outperforms TRKD across teacher sizes, with improvements persisting after post-training.

EOPD. EOPD was originally formulated for on-policy distillation, a post-training paradigm that relies on student-generated rollouts, instead of general pre-/mid-training. Consequently, a direct implementation changes the training setting as well as the distillation objective.

Nevertheless, we ran a new EOPD-inspired method that is entirely off-policy, but switches between RKL for teacher-confident tokens, and FKL otherwise. Under controlled comparisons, the EOPD-inspired method matches Switch Distillation on reasoning but trails on factual recall (−3.4) and knowledge (−1.5).

MethodReasoningFactual RecallKnowledge & Commonsense
Switch Dist44.729.349.3
EOPD-inspired44.525.947.8

3. Explanation of the reasoning–recall tradeoff

We appreciate the opportunity to clarify that our explanation extends beyond downstream outcomes. §4 identifies three interacting factors:

  • Teacher uncertainty (§4.1, Fig. 3): Teachers exhibit lower predictive entropy on procedural domains (e.g., math and instruction-following) than on knowledge-intensive text (e.g., Wikipedia), with lower entropy corresponding to more reliable supervision.
  • Student knowledge acquisition (§4.2, Fig. 4): Students acquire low-entropy factual knowledge earlier during pre-training, leaving unresolved facts concentrated in high-entropy regions by mid-training.
  • Optimization dynamics (§4.3, Fig. 5; App. C.3): Our analytical derivations show that KD increasingly attenuates ground-truth supervision as teacher entropy rises, explaining why factual acquisition can lag behind NTP during mid-training.

We also observe similar entropy asymmetries across recent model families (OLMo 3 7B Instruct, Qwen3 8B, Gemma 3 12B Instruct, and Granite 3.3 8B Instruct; App. D.3), suggesting that these observations to other modern instruct models.

Why reasoning and factual recall?

We distinguish these capabilities as they capture complementary aspects of learning: acquiring generalizable reasoning procedures vs. retrieving specific factual knowledge. We study them separately because KD affects them differently during mid-training. We do not claim a universal tradeoff: its direction depends on teacher reliability across domains and the student’s existing knowledge. A teacher more reliable on factual than procedural content could, in principle, yield a different tradeoff, although our analysis finds the opposite confidence pattern across the instruction-tuned teachers examined.

AuthorsOpenReview ↗

Response to Reviewer Trkc (2/2)

4. Switch Distillation on specialized, knowledge-heavy vertical domains.

Following established mid-training practice [1-6], our primary experiments use Dolmino Mix 2 to study mid-training under a broad data distribution. However, this mixture also includes specialized, knowledge-intensive sources such as scientific literature (peS2o), Wikipedia, and Stack Exchange. Moreover, our analysis in Fig. 3 shows that lower teacher entropy consistently corresponds to higher teacher prediction accuracy within every Dolmino domain. This supports the relevance of our entropy-based routing criterion to specialized, knowledge-intensive text. Corroborating this, Switch Distillation achieves statistically significant improvements over competing baselines on knowledge-intensive benchmarks, including MMLU, MMLU-Pro, ARC-C, and AGIEval (Table 1), showing it can directly benefit knowledge-intensive capabilities.

5. Long-form CoT and synthetic reasoning traces

Thank you for raising this timely question. Our existing analysis (App. D.2) already establishes that the procedural–factual entropy asymmetry persists across recent model families, including OLMo 3, Qwen3, Gemma 3, and Granite 3.3, some of which incorporate synthetic reasoning data during training.

But to directly address this concern, we conducted an additional entropy analysis of 200 OpenR1-Math long-form synthetic reasoning traces and 300 Wikipedia documents, using Qwen 3 8B and Olmo 2 13B Instruct as teachers. Even after excluding structural tokens (e.g., formatting markers and reasoning delimiters), teacher entropy remains substantially lower on CoT than on factual text (0.98 vs. 6.44 nats for Qwen3; 2.62 vs. 6.02 for OLMo). Switch Distillation also selects CoT tokens more frequently than Wikipedia tokens (84.5% vs. 16.1%; 29.6% vs. 4.9%), with most selected CoT tokens corresponding to substantive reasoning content (64% and 57%, respectively). Together, these findings show that the entropy asymmetry extends across recent model families and persists on long-form reasoning traces, rather than arising solely from predictable structural tokens. While this supports the continued relevance of entropy-guided routing, establishing the downstream reasoning–recall tradeoff under CoT-heavy mid-training requires additional controlled experiments, which we leave to future work.

6. Training throughput and memory

Switch Distillation requires no additional model forward passes or trainable parameters relative to standard online KD. Its only additional operations are computing teacher predictive entropy and selecting the lowest-entropy tokens (§5). We measured steady-state throughput and peak GPU memory under identical hardware and training configurations, and distillation with a 7B teacher.

MethodTokens/s/GPUThroughput vs. KDPeak GPU Memory
NTP141.0k5.32×66.4 GiB
Standard KD26.5k1.00×102.3 GiB
Switch Distillation25.7k0.97×96.8 GiB

Switch Distillation achieves 97% of standard KD throughput, with comparable peak memory usage (96.8 versus 102.3 GiB). Thus, the additional routing overhead is small relative to KD, although both distillation approaches are substantially more expensive than NTP.

Thank you again for your constructive suggestions; we have incorporated these additional analyses and clarifications in the revised manuscript.

References:

[1] Walsh et al. 2 OLMo 2 Furious: Advancing the Open Foundation Model Ecosystem. COLM, 2025.

[2] OLMo Team. OLMo 3 Technical Report. 2025.

[3] Gemma Team. Gemma 3 Technical Report. 2025.

[4] Meta Superintelligence Labs. Introducing Muse Glimmer: An Open Agentic Model That Runs on Your Device. 2026.

[5] Yang et al. Qwen3 Technical Report. 2025.

[6] Kimi Team. Kimi K2: Open Agentic Intelligence. 2025.

Thank you for your response and detailed experiments. The clarification regarding the theoretical analysis have addressed my main concerns. I have update my score accordingly.

Thank you for your thoughtful feedback and for reconsidering your score! We greatly appreciate your careful engagement with our work throughout the review process. Your comments were very helpful in guiding our additional analysis and strengthening the paper, and we’re glad that our response has addressed your concerns.

Reviewer_vCEYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses a problem in logit distillation during mid-training, which is lower factual recall compared to next token prediction loss. They claim in pre-training the student already learned facts that are low entropy, so most of the remaining gain should come from the high entropy tokens. However, the teacher is uncertain in high entropy tokens and provides less supervision compared to next token prediction. They suggest an approach that for tokens with high teacher entropy (top 80-percentile tokens) switches to CE loss (next token prediction) and for lower entropy tokens uses the reverse-KL loss. (bottom 20%)

They tested their method with large-scale experiments of 60B tokens and achieved a better reasoning/factual recall tradeoff across different teacher sizes of 1B, 7B, 13B. The student size remains 1B during their experiments.

Critical strengths
  1. Large Scale experiments (this makes the results solid)
  2. Simple Idea that make it useful for practice (the idea is easy to use and hence can become consequantial for the distilation feild.
  3. Reasoning, recall tradeoff that they identified during mid training is very interesting.
Critical weaknesses
  1. Lowering the distillation temperature can also sharpen the teacher distribution. A comparison of your method to the curve of different temperatures can better show the benefit of entropy routing vs other alternatives for sharpening.
  2. There is no abliation on bigger student models above 1B.
Questions for authors

Your explanation connects the usefulness of distillation to both teacher confidence and the student’s existing knowledge, but the routing rule uses only teacher entropy. For example, two positions can have equally low teacher entropy while differing substantially in teacher–student disagreement, and therefore in the supervision distillation provides. An abliation on teacher-student kl routing would be very interesting.

3 comments
AuthorsOpenReview ↗

Response to Reviewer vCEY

Thank you for your time and positive reception of the paper! We found your feedback insightful and encouraging. With your input, we have added several new experiments that have strengthened our paper. Addressing your comments point-by-point below:

1. Comparison with temperature-tuned KD

Thank you for this suggestion. We agree that temperature tuning is an important baseline for assessing whether the benefits of Switch Distillation can be recovered by simply sharpening the teacher distribution. We swept conventional forward KD over T∈{0.5,1,2,4}T \in \{0.5, 1, 2, 4\}, including our original choice of �0��0�. The results are summarized below:

MethodReasoningRecallKnowledge/Commonsense
KD (T=0.5)30.928.843.4
KD (T=1)32.928.445.6
KD (T=2)36.128.947.0
KD (T=4)29.826.843.1
NTP26.130.341.3
Switch Dist.44.729.349.3

We find that T=2T=2 is the best among the tested KD temperatures for all capability categories. Importantly, Switch Distillation substantially outperforms even the best temperature-tuned KD baseline (+8.6 points on reasoning and +2.3 points on knowledge/commonsense), while achieving comparable recall (+0.4 points). These results suggest that the benefits of Switch Distillation cannot be recovered simply by sharpening the teacher distribution through temperature scaling.

2. Evaluation on larger students

We agree that evaluating students larger than 1B would provide valuable evidence about scaling. We selected OLMo 2 1B because its publicly available Stage-1 pretraining checkpoint enables controlled experiments of mid-training objectives. OLMo 2 does not provide an intermediate-sized checkpoint between 1B and 7B, and extending our experiments to 7B students would substantially increase computational cost. Furthermore, 1B-scale models are widely used in controlled pretraining studies, where conducting multiple training runs makes computational efficiency and reproducibility important considerations [1–5]. In addition, we have independently replicated our main findings in a different model family, SmolLM, using a 360M student and 1.7B teacher (Appendix D.1). This provides additional evidence that our takeaways are not specific to OLMo 2 or the 1B student size. We acknowledge that larger-student evaluation is a necessary direction for future work.

3. Teacher-student KL-based routing

Thank you for the great suggestion. We agree that teacher-student disagreement is a natural alternative to teacher entropy for determining which tokens should receive distillation. To investigate this, we implemented an additional routing baseline that replaces teacher entropy with token-level teacher–student KL divergence while keeping the remaining components of Switch Distillation unchanged. Specifically, we run a new ablation where we apply reverse-KL distillation to the 20% of tokens (matching Switch Distillation’s selection rate) with the highest teacher–student KL divergence, and cross-entropy (CE) to the remaining tokens:

MethodReasoningRecallKnowledge/Commonsense
Forward KD36.128.947.0
KL routing (new)42.126.148.6
Switch Dist.44.729.349.3

KL-based routing improves reasoning over conventional KD (+6.0 points), but underperforms Switch Distillation across all three capability categories, particularly factual recall (-3.2 points).

We hypothesize that teacher-student disagreement is a weaker routing signal because high disagreement does not necessarily imply reliable teacher supervision: it may surface where the teacher is incorrect, or where the student has already learned the ground-truth continuation. In contrast, teacher entropy prioritizes positions where the teacher is most confident.

Importantly, Switch Distillation already implicitly accounts for teacher-student disagreement within the selected tokens through the reverse-KL term: definitionally, when the student matches the teacher, the divergence and its gradient vanish, whereas disagreement produces a learning signal. This means that teacher entropy determines where distillation is applied, while reverse KL determines how the student is updated.

References:

[1] Midtraining Bridges Pretraining and Posttraining Distributions, Liu et al., ICML 2026.

[2] DataDecide: How to Predict Best Pretraining Data with Small Experiments, Magnusson et al., ICML 2025.

[3] Distilled Pretraining: A Modern Lens of Data, In-Context Learning and Test-Time Scaling, Goyal et al., ICLR 2026.

[4] Data-efficient Pre-training by Scaling Synthetic Megadocs. Kim et al., COLM 2026.

[5] The Finetuner’s Fallacy: When to Pretrain with Your Finetuning Data. DatologyAI Team, Preprint.

Thanks for your detailed response. All my concerns except bigger model ablations are addressed. The reasoning behind why bigger models for ablation are not feasible to authors is also reasonable, and hence I maintain my score.

Thank you for your thoughtful feedback and for taking the time to consider our response! We appreciate your acknowledgment of our additional experiments and clarifications, as well as your understanding of the computational constraints on larger-model runs.

Reviewer_xssNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work focuses on logit-based distillation during mid-training. The authors first conduct experiments using a 1B OLMo 2 with the teacher OLMo 2 1B/7B/13B on Dolmino Mix 1124. They found when training from scratch, the forward KL performs better than next-token prediction (NTP), on both reasoning and factual recall. But they found the results shift a bit when they resume training from an existing checkpoint (4T-token Stage-1 on 60B tokens). They found the reasoning still improves, but the factual recall performance failed to beat the NTP. Authors conject the reason is that the on information-intense text, i.e. web and wikipedia, the teacher could get high entropy distribution which hurts the learning dynamic of the student on ground truth. Based on the observation, they propose Switch Distillation, which applies reverse KL to only to those tokens where the teacher gives low entropy, and applies cross-entropy to the rest tokens. The experimental results across reasoning and factual recall tasks confirms the effectiveness

Critical strengths
  1. Important research question. Mid-training is now standard practice. Exploring distillation at that stage is valuable for the community.
  2. The proposed method Switch Distillation is simple and free to add. Comprehensive evaluation demonstrates its effectiveness.
Critical weaknesses
  1. I am not fully convinced the gains come from the "switch". Always CE changes almost nothing (Table 3), uniform normalization costs 5.4 reasoning points (App. B.3), and random routing under the same normalization costs 6.5. When the q=20%, the routed token already carries about 4 times the weight of a CE token before the \tau^2 factor. So the method reads as full cross-entropy and a up-weighted RKL term. I’m also a bit confused by distilling only where the teacher is nearly one-hot, where the soft target carries least information. In table 3, the Teacher-Top-1 ablation says soft targets still matter there, but nothing in the paper explains it. I suggest the author to conduct experiments that can confirm the gain comes from KD rather then just token selection or reweighting.
  2. Everything is a single run, and only one student size is tested.
Questions for authors

Can you report two to three seeds, at least for NTP vs. Switch Distillation with the 7B teacher?

2 comments
AuthorsOpenReview ↗

Response to Reviewer xssN

Thank you for your careful engagement with our work and for recognizing the importance of mid-training distillation. We appreciate your positive assessment of Switch Distillation, and your suggestions have made our paper stronger. Addressing your comments below:

1. Do Switch Distillation’s gains come from KD rather than token selection or reweighting?

We agree that Switch Distillation combines several design choices: (1) teacher entropy-based routing selection, (2) separate partition normalization of the CE and KD losses, and (3) soft teacher supervision. Our ablations disentangle their contributions:

Entropy-based routing matters: Random routing under identical normalization reduces average reasoning by 6.5 points, while teacher-correct routing (e.g., routing only when the teacher’s top-1 choice matches the ground truth) reduces it by 4.4 points. Soft teacher supervision matters: Replacing RKL with cross-entropy on the teacher’s argmax prediction reduces reasoning by 6.4 points and knowledge/commonsense by 2.8 points.

Separate partition normalization matters: Using uniform token normalization reduces reasoning by 5.4 points (Appendix B.3), but reasoning remains substantially above NTP. Thus, gains cannot be attributed solely to upweighting distilled tokens.

Retaining CE is not the primary driver: Applying CE to all tokens indeed changes reasoning by only −0.3 points (Table 3). This indicates that retaining CE supervision on the distilled subset is not essential to the method’s effectiveness. Rather, our key finding is that gains primarily come from selectively applying distillation only where the teacher is confident, which our ablations isolate. We clarify this point in our revision.

New weighted-CE ablation:

To isolate the contribution of soft teacher supervision beyond entropy-based teacher weighting, we ran a new token-reweighted CE ablation that preserves SD's entropy-based routing mask and loss coefficients but replaces the RKL term with CE: L=T2∣S∣∑t∈SCE(yt)+1∣Sˉ∣∑t∈SˉCE(yt)\mathcal{L} = \frac{T^2}{|S|} \sum_{t \in S} \mathrm{CE}(y_t) + \frac{1}{|\bar{S}|} \sum_{t \in \bar{S}} \mathrm{CE}(y_t).

MethodReasoningRecallKnowledge/Commonsense
NTP26.130.341.3
Forward KD36.128.947.0
Weighted CE (new)36.329.247.4
Switch Distillation44.729.349.3

Our new weighted-CE control reveals that entropy-based token weighting contributes substantially to the gains, but cannot fully explain them. Replacing weighted CE with soft teacher supervision (e.g., the original Switch Distillation) on selected tokens yields an additional 8.4-point improvement in reasoning, demonstrating that both components are important.

2. Why distill at low-entropy positions, where the teacher is nearly one-hot?

Low teacher entropy serves to identify positions where the teacher provides relatively confident supervision; it does not imply that the teacher distribution contains no additional information. To quantify this, we sampled low-entropy positions from our training data and found that only 11.3% have a teacher top-1 probability exceeding 0.99.

Even when the teacher strongly favors one token, its non-top-1 probabilities still encode relative preferences over alternative continuations. For example, consider the sequence “The theory of relativity was developed by”, the teacher might strongly favor “Albert” for the next token while assigning some probability to “Einstein,” both of which lead to valid completions. Preserving these probabilities provides a richer optimization signal than supervising only the teacher’s argmax prediction.

Our Teacher-Top-1 ablation supports this interpretation: retaining the full teacher distribution (instead of CE everywhere) improves reasoning by 6.4 points and knowledge/commonsense by 2.8 points. Thus, even at low-entropy positions, soft teacher supervision provides useful information.

3. Robustness.

Additional random seeds.

Following your suggestion, we conducted three-seed replications of NTP and Switch Distillation with the 7B teacher. Switch Distillation consistently improves reasoning (44.47 ± 0.55 versus 26.38 ± 0.24 for NTP; mean ± standard deviation). Factual recall is also stable (29.38 ± 0.19 versus 30.38 ± 0.17). These results confirm that our main findings are robust across seeds.

Student model sizes.

We use an OLMo 2 1B student because its publicly available Stage-1 checkpoint enables controlled mid-training experiments. The next available size is 7B, for which extending our experiments would incur substantially greater computational cost (our existing distillation runs require up to ~5,300 H200 GPU-hours). Importantly, we also replicate our findings in a separate model family, SmolLM, with a 360M student and 1.7B teacher (App. D.1), demonstrating that our findings are not specific to OLMo 2 1B. We acknowledge larger-student evaluation as an important direction for future work.

AuthorsOpenReview ↗

Thanks for your thoughtful review! We look forward to further discussion with you

Thank you again for your thoughtful review! We hope our responses and additional experiments have helped address your concerns. As the ICLR discussion period draws to a close, we would be grateful to hear whether any concerns remain, and would be happy to provide any further clarification.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author General Response

We thank all reviewers again for your careful reading, constructive feedback, and recognition of the importance of studying distillation during mid-training. Your suggestions have motivated several new experiments that we believe have much improved our paper!

We would also like to emphasize that our paper makes three complementary contributions: (1) identifying a previously uncharacterized, stage-dependent reasoning–recall tradeoff in pre-training distillation; (2) investigating its underlying mechanisms through teacher confidence, student knowledge acquisition, and optimization dynamics; and (3) proposing Switch Distillation, a simple entropy-guided objective that improves the reasoning–recall tradeoff, with gains persisting through post-training. We view the first two findings as independently valuable insights into the behavior of pre-training distillation, and are excited about the broader research directions they open for rethinking how language models learn at different training stages.

Below, we summarize the key points addressed during rebuttal:

  1. Understanding Switch Distillation’s Gains (xSSN, vCEY): Reviewers questioned whether the downstream improvements arise from soft teacher supervision, token reweighting, or entropy-based routing. In response, we set up several new ablation experiments: (1) a weighted-CE baseline preserving the routing mask and loss normalization, which Switch Distillation outperforms by 8.4 reasoning points; (2) a temperature sweep over conventional KD, where Switch Distillation outperforms the best setting by 8.6 reasoning points; and (3) teacher-student KL-based routing, which improves over standard KD but underperforms Switch Distillation across all three capability categories. Together with our existing ablations, these results underscore the complementary importance of entropy-based selection and soft teacher supervision.

  2. Comparisons with existing methods and understanding the reasoning–recall tradeoff (TrKC): We clarified how Switch Distillation differs from TRKD, which we already evaluated under matched conditions, and introduced a new EOPD-inspired baseline, which matches Switch Distillation on reasoning but trails by 3.4 points on factual recall. We further clarified that the reasoning–recall tradeoff arises from asymmetric teacher reliability across procedural and factual domains, together with the student’s existing knowledge and the optimization dynamics of KD. To assess whether our findings extend to modern mid-training data, we analyzed long-form synthetic reasoning traces and found that teacher entropy remains substantially lower on reasoning than factual text, even after excluding structural tokens. We also clarified the evidence and limitations concerning specialized knowledge-intensive domains and revised the introduction to better communicate the motivation and contributions of our method.

  3. Robustness and computational efficiency (xSSN, vCEY, TrKC): We conducted three-seed replications confirming consistent downstream improvements (for reasoning: 44.47 ± 0.55 vs. 26.38 ± 0.24 for NTP), highlighted that our findings independently replicate with the SmolLM family, and clarified the computational constraints of larger-student experiments. Furthermore, we measured training efficiency, finding that Switch Distillation achieves 97% of standard KD throughput with comparable GPU memory usage, demonstrating that entropy-based routing introduces minimal additional overhead relative to standard KD.

We hope our additional experiments and clarifications have addressed the main concerns raised during review. We welcome any further questions or feedback and would be happy to address them during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.