- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates logit-based knowledge distillation during the LLM mid-training phase. It uncovers the reasoning-recall tradeoff, where standard KD improves reasoning but slows factual recall acquisition. To resolve this, it proposes Switch Distillation, which uses teacher predictive entropy to route tokens KD and NTP, successfully preserving factual recall while boosting reasoning.
Critical strengths
- The paper identifies a previously uncharacterized, stage-dependent reasoning-recall tradeoff when applying KD during mid-training. This phenomenon is unveiled through sufficient experiments in the paper.
- The paper links the tradeoff to an insightful interaction between asymmetric teacher confidence across data domains, student knowledge states, and optimization gradients.
- The proposed Switch Distillation is a lightweight method, requiring no extra parameters or additional model passes beyond standard KD.
Critical weaknesses
- The paper adopts an experiment-driven narrative approach, but there is not much introduction to the proposed method and comparative experiments, which limits the readers’ understanding.
- The paper lacks a direct comparison with continuous entropy-aware distillation methods on the same mid-training data mixture, both theoretically and empirically.
- The experiments evaluate performance on general mixtures. It remains unclear if the method is robust when applied to highly specialized, knowledge-heavy vertical domains.
Questions for authors
- How does the proposed method fundamentally differ from existing selective distillation or token-level routing methods (like TRKD or EOPD)?
- Is there any theoretical elaboration on the reason-recall tradeoff that is not dependent on the benchmark performance? Or what is the motivation for this observation and why are these two abilities?
- With the recent shift toward long-form CoT and synthetic reasoning traces in mid-training, does the reasoning-recall tradeoff and entropy asymmetry still hold, or do structural reasoning tokens eliminate the need for hard token-level routing?
- What is the training throughput or memory footprints compared to standard NTP or KD?
4 comments
Response to Reviewer Trkc (2/2)
4. Switch Distillation on specialized, knowledge-heavy vertical domains.
Following established mid-training practice [1-6], our primary experiments use Dolmino Mix 2 to study mid-training under a broad data distribution. However, this mixture also includes specialized, knowledge-intensive sources such as scientific literature (peS2o), Wikipedia, and Stack Exchange. Moreover, our analysis in Fig. 3 shows that lower teacher entropy consistently corresponds to higher teacher prediction accuracy within every Dolmino domain. This supports the relevance of our entropy-based routing criterion to specialized, knowledge-intensive text. Corroborating this, Switch Distillation achieves statistically significant improvements over competing baselines on knowledge-intensive benchmarks, including MMLU, MMLU-Pro, ARC-C, and AGIEval (Table 1), showing it can directly benefit knowledge-intensive capabilities.
5. Long-form CoT and synthetic reasoning traces
Thank you for raising this timely question. Our existing analysis (App. D.2) already establishes that the procedural–factual entropy asymmetry persists across recent model families, including OLMo 3, Qwen3, Gemma 3, and Granite 3.3, some of which incorporate synthetic reasoning data during training.
But to directly address this concern, we conducted an additional entropy analysis of 200 OpenR1-Math long-form synthetic reasoning traces and 300 Wikipedia documents, using Qwen 3 8B and Olmo 2 13B Instruct as teachers. Even after excluding structural tokens (e.g., formatting markers and reasoning delimiters), teacher entropy remains substantially lower on CoT than on factual text (0.98 vs. 6.44 nats for Qwen3; 2.62 vs. 6.02 for OLMo). Switch Distillation also selects CoT tokens more frequently than Wikipedia tokens (84.5% vs. 16.1%; 29.6% vs. 4.9%), with most selected CoT tokens corresponding to substantive reasoning content (64% and 57%, respectively). Together, these findings show that the entropy asymmetry extends across recent model families and persists on long-form reasoning traces, rather than arising solely from predictable structural tokens. While this supports the continued relevance of entropy-guided routing, establishing the downstream reasoning–recall tradeoff under CoT-heavy mid-training requires additional controlled experiments, which we leave to future work.
6. Training throughput and memory
Switch Distillation requires no additional model forward passes or trainable parameters relative to standard online KD. Its only additional operations are computing teacher predictive entropy and selecting the lowest-entropy tokens (§5). We measured steady-state throughput and peak GPU memory under identical hardware and training configurations, and distillation with a 7B teacher.
| Method | Tokens/s/GPU | Throughput vs. KD | Peak GPU Memory |
|---|---|---|---|
| NTP | 141.0k | 5.32× | 66.4 GiB |
| Standard KD | 26.5k | 1.00× | 102.3 GiB |
| Switch Distillation | 25.7k | 0.97× | 96.8 GiB |
Switch Distillation achieves 97% of standard KD throughput, with comparable peak memory usage (96.8 versus 102.3 GiB). Thus, the additional routing overhead is small relative to KD, although both distillation approaches are substantially more expensive than NTP.
Thank you again for your constructive suggestions; we have incorporated these additional analyses and clarifications in the revised manuscript.
References:
[1] Walsh et al. 2 OLMo 2 Furious: Advancing the Open Foundation Model Ecosystem. COLM, 2025.
[2] OLMo Team. OLMo 3 Technical Report. 2025.
[3] Gemma Team. Gemma 3 Technical Report. 2025.
[4] Meta Superintelligence Labs. Introducing Muse Glimmer: An Open Agentic Model That Runs on Your Device. 2026.
[5] Yang et al. Qwen3 Technical Report. 2025.
[6] Kimi Team. Kimi K2: Open Agentic Intelligence. 2025.
Thank you for your response and detailed experiments. The clarification regarding the theoretical analysis have addressed my main concerns. I have update my score accordingly.
Thank you for your thoughtful feedback and for reconsidering your score! We greatly appreciate your careful engagement with our work throughout the review process. Your comments were very helpful in guiding our additional analysis and strengthening the paper, and we’re glad that our response has addressed your concerns.
Response to Reviewer Trkc (1/2)
Thank you for your thoughtful review and for recognizing the significance of the stage-dependent reasoning–recall tradeoff, our explanatory analysis, and the simplicity of Switch Distillation. Your suggestions have helped us improve the paper’s presentation and motivated new experiments. Responding point-by-point:
1. Presentation and experimental narrative
We appreciate your suggestion to introduce the method and comparative experiments earlier. Our motivation for Switch Distillation comes from our observation of the reasoning–recall tradeoff, a novel empirical phenomenon that persists across teacher sizes, distillation mixing coefficients, and model families, and is a central contribution of this work. Our experiment-driven narrative first establishes the phenomenon (§3), investigates its underlying causes (§ 4), and uses these findings to motivate Switch Distillation (§5).
Our analysis shows that teacher entropy differs systematically across data domains (§4.1), unresolved factual knowledge is concentrated at high-entropy positions (§4.2), and KD can attenuate ground-truth supervision at these positions (§4.3). Together, these observations motivate distilling selectively where the teacher is confident. We agree that this progression would benefit from a clearer roadmap. We revised the introduction to preview Switch Distillation and its comparisons with existing methods, while retaining the empirical motivation for our routing criterion.
2. Comparison with TRKD and EOPD
Thank you for highlighting these related approaches; they provide useful points of clarification for our method design.
TRKD. We already include a direct empirical comparison against TRKD under the same mid-training setup and data mixture (§6.1, Tables 1–2), with methodological differences discussed in App. A.1. TRKD was originally proposed to mitigate in-context learning degradation during pretraining KD. It excludes the lowest-entropy tokens from distillation and applies a mixture of forward KL and CE to remaining tokens. Switch Distillation takes the opposite approach during mid-training: it applies reverse KL to low-entropy tokens and retains ground-truth CE elsewhere. This distinction is motivated by our finding that teacher uncertainty impedes factual knowledge acquisition during mid-training. Empirically, Switch Distillation outperforms TRKD across teacher sizes, with improvements persisting after post-training.
EOPD. EOPD was originally formulated for on-policy distillation, a post-training paradigm that relies on student-generated rollouts, instead of general pre-/mid-training. Consequently, a direct implementation changes the training setting as well as the distillation objective.
Nevertheless, we ran a new EOPD-inspired method that is entirely off-policy, but switches between RKL for teacher-confident tokens, and FKL otherwise. Under controlled comparisons, the EOPD-inspired method matches Switch Distillation on reasoning but trails on factual recall (−3.4) and knowledge (−1.5).
3. Explanation of the reasoning–recall tradeoff
We appreciate the opportunity to clarify that our explanation extends beyond downstream outcomes. §4 identifies three interacting factors:
We also observe similar entropy asymmetries across recent model families (OLMo 3 7B Instruct, Qwen3 8B, Gemma 3 12B Instruct, and Granite 3.3 8B Instruct; App. D.3), suggesting that these observations to other modern instruct models.
Why reasoning and factual recall?
We distinguish these capabilities as they capture complementary aspects of learning: acquiring generalizable reasoning procedures vs. retrieving specific factual knowledge. We study them separately because KD affects them differently during mid-training. We do not claim a universal tradeoff: its direction depends on teacher reliability across domains and the student’s existing knowledge. A teacher more reliable on factual than procedural content could, in principle, yield a different tradeoff, although our analysis finds the opposite confidence pattern across the instruction-tuned teachers examined.