- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies why on-policy self-distillation is effective, with a focus on how the reference provided to the teacher interacts with the student's current reasoning state. Through controlled experiments on mathematical reasoning, the authors show that references from other problems can also be useful, while references specific to the target problem provide larger gains. More importantly, exchanging reference-induced guidance between different reasoning trajectories reduces its training benefit, even when the trajectories are for the same problem. The paper further shows that most of the improvement is expressed through the thinking stage rather than final-answer generation. The results suggest that matching guidance to the student's reasoning state helps the model complete reasoning more efficiently under limited token budgets.
Critical strengths
- The paper studies an interesting and valuable question: why on-policy self-distillation works. Understanding the role of the reference and its interaction with the student's reasoning state is useful for this line of research.
- Some findings are insightful. In particular, the observation that using the trained policy only during the thinking stage retains most of the improvement provides a useful perspective on where the benefit of self-distillation comes from.
Critical weaknesses
- The model coverage is limited. The experiments only consider Qwen3 models at two relatively small scales. It is unclear whether the same conclusions hold for other model families.
- The evaluation mainly focuses on mathematical reasoning. Although several math datasets are included, the task coverage is still limited. It is unclear whether the observed effects also hold for other types of tasks, such as code generation or general reasoning.
Questions for authors
- How can the findings be used to improve the training method in practice? For example, can they guide the design of a more effective teacher reference or supervision strategy, beyond explaining the behavior of the current method?
- Do the authors expect different results on other model families or non-mathematical benchmarks? Additional experiments would help clarify how general these conclusions are.
- Could other types of teacher references also be useful? For example, instead of a worked solution to a specific problem, could the reference be a summary of the solution strategy for a class of problems?