- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focuses on on policy distillation. The author proposes Mixed-Policy Distillation (MPD), which constructs a student-generated prefix with a teacher-generated continuation, to solve the problem that the teacher is not familiar to the student-generated states.
Critical strengths
1 The paper is logically structured, the method is easy to follow, and the diagrams are clear and intuitive.
2 Experimental results demonstrate that the proposed method outperforms offKD and the original OPD, while also showing strong performance on OOD data.
Critical weaknesses
1 In the experiments section, the main results table compares the method only against two basic baselines—offKD and OPD—while omitting comparisons with other methods that also employ mixed OPD strategies, such as Speculative KD and TRD , which the paper mentions.
2 As shown in Figure 2, while a mixed policy might use a single concatenation to steer the trajectory away from regions of overlap, it is possible that the trajectory could subsequently drift into areas requiring "teacher support." Relying on a single correction step seems somewhat crude.
Questions for authors
1 Why is the location of the student's highest entropy necessarily the optimal point for the teacher to take over? High entropy indicates student uncertainty, but it does not guarantee that the teacher can provide the most effective guidance at that specific point.
2 Does MPD retain its performance advantage over OPD when the computational budget is held constant? Since MPD requires generating the full student response to identify the point of highest entropy, it incurs additional computational costs.
3 comments
My question still concerns the link between student uncertainty and the benefits of teacher intervention; I hope the author can provide a clearer explanation of why teacher guidance is more effective in this context.
Explanation for Entropy-Guided Teacher Intervention at Reasoning Forks
Thank you for the follow-up. We use entropy to place teacher intervention before an uncertain reasoning choice determines the rest of the rollout. At a reasoning fork, the student may assign substantial probability to several continuations. If it chooses an unproductive branch, later tokens build on that choice, and teacher supervision must operate on the resulting states. Taking over before this choice allows the teacher to steer the continuation toward a more productive path.
Compared with intervening during a routine stretch of generation, intervention at such a fork has a greater opportunity to change the direction of the entire suffix. For MPD, the learning benefit we seek is a useful continuation precisely where the student struggles to continue independently. The student-generated prefix connects this continuation to its own partial reasoning; the teacher suffix demonstrates how to resolve the uncertain choice and develop the solution afterward. Distillation on this trajectory therefore teaches the student how to proceed through a difficulty encountered in its own generation.
Intervention studies support this motivation. In CURE, regenerating trajectories from high-entropy rather than random cut points improves the trained model's average accuracy from 51.4% to 53.7% under the same reported training budget and hyperparameters (Section 4.5). Shen et al. further show that entropy-triggered delegation to a stronger model improves completed responses through sparse token interventions (Appendix C.3). These results support using high-entropy positions to intervene on the continuation and construct more useful trajectories for learning.
Clarifications on MPD’s Teacher Intervention Strategy, Handoff Design, and Training Efficiency
Q1. Speculative KD and TRD
Our experimental budget prioritizes standard reference baselines and broad evaluation of MPD. The completed four-model, three-domain, two-teacher study has already consumed substantial compute and funding. Each additional method requires implementation and adaptation, hyperparameter tuning, validation, and full training and evaluation. The remaining budget cannot support these additional comparisons.
Speculative KD is an ICLR 2025 publication that replaces student token proposals using teacher feedback. TRD's cited arXiv version is listed as under review and refines student trajectories under teacher guidance. For recent preprints such as TRD, adding a benchmark entails implementing and validating another training pipeline in our setting, alongside the cost of the training itself.
MPD instead uses a student prefix followed by a continuous teacher continuation. We discuss these distinct intervention mechanisms in related work and use recognised methods (OffKD and OPD) as the experimentally evaluated reference points.
Q2. Does one handoff provide enough teacher support?
One handoff initiates a complete continuation in which the teacher keeps generating subsequent tokens. It is not a one-token correction followed by an immediate return to independent student generation. The student prefix selects the learner-visited state, and the teacher then demonstrates an extended path forward. This is why a single handoff can still provide sustained guidance.
The paper also varies the number of cut points M selected from the same original student rollout:
In this reported comparison, increasing M substantially increases teacher generation without a monotonic accuracy improvement. It motivates the simple default as a practical cost–performance choice. The multi-cut study also does not directly establish the benefit of repeatedly alternating policies within one finalized sequence.
Q3. Why should the teacher take over at high student entropy?
High entropy identifies student uncertainty, which makes it an interpretable place to offer a demonstration. Maximum KL instead identifies disagreement between student and teacher. The paper compares these two signals with random handoff in the stated science setting.
The learning intuition is that a teacher continuation can show how to resolve an uncertain branch while remaining anchored in the student's reasoning. This is consistent with findings about high-entropy reasoning forks in Beyond the 80/20 Rule, while TIP also demonstrates the relevance of teacher–student disagreement.
Neither result guarantees an optimal handoff. We use entropy as a motivated practical rule and PPL/overlap as diagnostics of the resulting trajectory, rather than a proof that teacher benefit is maximal at the selected token.
Q4. Time to accuracy and generation overhead
MPD reaches high accuracy earlier in the existing Qwen3.5-4B science runs. The first evaluated checkpoints meeting each threshold are:
Times are cumulative logged training-step durations at the first qualifying evaluation.
At step 20, MPD achieves 65.29%, close to its late-training mean of 65.46%. The recorded training-step time includes the complete student rollout, including the discarded suffix, and MPD's teacher-continuation stage. The continuous teacher demonstration is designed to help the student exploit difficult examples sooner, motivating the additional generation work per update.