- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies why is the distillation from larger teacher models to smaller models produce traces that are not fully compatible with smaller reasoning models, where the paper argues that this is because the stronger models traces might contain a few tokens that have low probability under the student model. To test this hypothesis, they propose Interleaved-Policy Distillation (IPD): During the generation of teacher tokens if any token has a probability below a specfic threshold, the student samples a replacement token, and the teacher continues reasoning. Then the resulting trajectory is used directly for SFT. A couple of experiments across multiple model families (Qwen3, EXAONE), sizes (0.6B to 1.7B), and datasets (s1K, LIMO) demonstrate that teacher-propose IPD consistently outperforms direct distillation, reinforcement learning (GRPO), on-policy distillation (OPD), and various baseline ablations.
Critical strengths
1- The paper presents a simple approach to improve reasoning distillation from larger teacher models to smaller student models. The approach is easy to understand and does not require changes to the standard SFT training procedure.
2- The paper provides extensive experiments that include four settings, two training datasets, three thresholds, and seven evaluation benchmarks, showing consistent improvements over several baselines. The analysis also provides interesting insights into reasoning distillation, showing that neither mean surprisal nor the number of correct traces alone explains knowledge transfer. The results also highlight the importance of proposal direction and preserving teacher guidance.
Critical weaknesses
1- The paper clearly highlights a strong link between extreme-tail surprisal and distillation performance. However, it remains not very clear whether lowering that surprisal is what actually drives the performance gains. As shown in Figure 4, methods with very similar extreme-tail surprisal values still end up with different performance levels, so surprisal reduction alone isn't the only cause of this improvement.
2- The evaluation is focused on math reasoning benchmarks and scientific Q&A. Extending this to multi-turn tasks or code-generation tasks would help verify the generalizability of the findings.
Questions for authors
1- The results in Figures 3 and 4 suggest a link between extreme-tail surprisal and distillation performance. Can the authors provide further evidence that reducing extreme-tail surprisal directly contributes to performance improvements, rather than the gains being driven by other changes in the generated reasoning trajectories?
2- Do the authors expect IPD to provide similar improvements for larger students or more open-ended reasoning tasks?
2 comments
Dear Reviewer z76V, Thank you again for your review. Your feedback helped us clarify our interpretation of extreme-tail surprisal. As the discussion period nears its end, could you let us know whether our initial response above addresses your concerns? We would be glad to clarify any remaining points.
Initial Response
Reviewer z76V
Thank you for taking the time to review our work. Your questions about the interpretation of extreme-tail surprisal helped us distinguish more clearly between an observed association and an explanation of the performance gains. We appreciate the opportunity to clarify this distinction.
Extreme-tail surprisal and the source of improvement (W1, Q1)
Our analysis shows that lower extreme-tail student surprisal is associated with better transfer and serves as a primary indicator of learning compatibility, but it does not fully explain downstream performance. T-P produces stronger students than S-P even when their training datasets have comparable extreme-tail mean surprisal (Figure 4). This suggests that the source of the reasoning tokens matters alongside their compatibility with the student.
We agree, however, that the current experiments do not isolate the causal contribution of surprisal reduction from other changes in the reasoning trajectories. IPD selectively revises teacher proposals during generation, changing both student support and subsequent reasoning. Our experiments support the practical benefit of adapting teacher-led reasoning to student support, while disentangling the contributions of improved compatibility and changes in reasoning content remains an important direction for future work.
Model scale and task scope (W2, Q2)
Our goal was to test whether adapting teacher supervision can improve reasoning transfer to students smaller than 3.8B. This focus is motivated by successful reasoning distillation with roughly 800–1,000 traces in Qwen2.5-32B-Instruct [1, 2], alongside reported degradation when the 3.8B Phi-4-Mini model is fine-tuned on s1K [3].
The current comparisons show that IPD benefits both 0.6B and 1.7B students under the same 8B teacher, but they do not establish how the gain changes for larger students. We expect the value of adaptation to depend on how well the student supports the teacher’s reasoning. IPD should therefore help most when a student struggles to learn directly from teacher traces, and less when it already learns well from them, as the 32B models above do.
Other verifiable tasks, such as code reasoning with executable tests, offer a natural extension, because our current data-construction pipeline uses a correctness verifier to select full training traces. Multi-turn tasks, open-ended generation, and interactive agents would require suitable alternatives for assessing and selecting supervision; we leave these extensions for future work. In the revised version, we have added both limitations to a new Limitations and Extensions section (Section 6).
References
[1] Muennighoff et al. s1: Simple test-time scaling. 2025.
[2] Ye et al. LIMO: Less is more for reasoning. 2025.
[3] Xu et al. Phi-4-Mini-Reasoning: Exploring the limits of small reasoning language models in math. 2025.