- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates an interesting question in on-policy distillation (OPD): why can it improve students’ task performance while also causing excessively long and repetitive generation? The authors treat the teacher’s probability signals on student-generated content as training feedback and examine both successful and deteriorating training runs. Through repeated sampling and per-problem analysis, they find that successful training primarily makes students more likely to produce correct solutions they were already capable of generating. By tracking response preferences before and after training, the paper further shows how pathological responses can be progressively amplified by favorable training signals. The authors propose filtering excessively long responses and initializing students through supervised fine-tuning to mitigate this behavior.
Critical strengths
The paper provides a clear and coherent analysis of generation degradation. The authors find that certain repetitive responses already receive relatively favorable signals before training, after which the student increasingly favors them. They also show that excluding excessively long responses from the training loss can mitigate the degradation. This finding offers useful insight into understanding and diagnosing distillation training.
Critical weaknesses
-
The main insights are close to those of existing work, making the additional contribution primarily confirmatory. Prior studies have already discussed how repetitive outputs can receive favorable signals and become progressively amplified, and have used repeated sampling to analyze how distillation improves sampling efficiency. Although this paper tracks initial responses and subsequent preference changes in greater detail, its results mainly reinforce existing explanations rather than establishing an equally distinctive new mechanism. This limits the paper’s novelty as an analysis study.
-
The claims about capability changes are broader than the experimental coverage supports. The analysis relies on a single 1.5B student that already possesses reasoning ability, while both teachers are obtained by further reinforcement learning from that same model. Therefore, the results provide limited support for the general claim in the abstract that “distillation does not expand the student’s capabilities.
Questions for authors
Please see the weakness section.