- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies why on-policy self-distillation (OPSD), in which a model is distilled from a copy of itself that sees a privileged context (answer, worked solution, or its own rollout), sometimes causes “thinking collapse”: reasoning becomes much shorter and accuracy drops. The authors frame OPSD as pushing the model from controlled toward automatic processing, and use V-usable information (PVI) as a common measure. Experiments on Qwen3-8B and Gemma-4-E4B show that only long, self-generated contexts (RC, RU) cause collapse, which worsens with task difficulty (e.g., ZebraLogic XL, Qwen: 39% → ~5%) and carries over out of domain. A synthetic shortest-path task separates information density from context length, and PVI predicts both accuracy and reasoning shortening. The paper also finds that students trained with richer contexts follow malicious in-context content more readily.
Critical strengths
-
Controlled design. Data, recipe, number of steps and decoding are shared, and only c varies, across two model families and in-/out-of-domain tasks. ZebraLogic’s difficulty tiers provide a useful difficulty axis.
-
The paper moves from “collapse happens” to when and why. It shows that shortening alone is not harmful (MBPP+, ZebraLogic Small, and the length-matched AO+R/R+AO contexts that improve accuracy).
-
The synthetic shortest-path task varies information density (I_BFS) and context length (random-token padding) separately, with a verified unique shortest path. PVI then unifies the two axes as a predictor of accuracy and length reduction, and the </think>-probability analysis offers a plausible account of why padding shortens reasoning.
Critical weaknesses
-
The claim that only self-generated contexts cause collapse is confounded with length and style: RC/RU average ~5,500 tokens versus ~680 for FS, and there is no short self-generated context (e.g., truncated rollouts) or long human-written context. The AO+R/R+AO controls help but contain only the answer plus random tokens. RC and RU also behave almost identically, so correctness filtering appears not to matter, which should be stated. The paper does not separate premature convergence of the reasoning chain from imitation of the teacher’s style.
-
The ZebraLogic tiers show a monotone trend in both models, but no criterion predicts when collapse will be harmful (e.g., a regression on base accuracy or base reasoning length across tasks). The PVI relationship is established only for the teacher on the synthetic task (Qwen3-8B, 25 graphs) and is not shown to predict student collapse on natural tasks. The r=0.99 between teacher and student length rests on only a few context conditions.
-
Each configuration appears to be a single training run (seed 42, one epoch), and the CIs come from bootstrapping over problems or prompts, so they do not capture training variance; AIME has only 30 problems per year. In the seen/unseen comparison, the unseen set is selected from problems BASE failed on, whereas the seen set spans difficulty bins, so longer reasoning on unseen problems may reflect difficulty rather than adaptation.
Questions for authors
-
What is the final teacher–student JSD for each context type, and how does it evolve during training? Is the stronger imprint of RC/RU associated with a smaller gap because the context is on-policy?
-
The reasoning-PVI curves are aggregated over live rollouts only. Do the RC/RU curves differ between correct and incorrect rollouts, and are the late parts of the curves driven by a subset of long rollouts?
-
Would a short self-generated context, or a long non-self-generated natural-language context, still trigger (or avoid) collapse?
Response Part 1
We truly appreciate the reviewer's valuable feedback and the recognition of our work. We hope the following discussion can address the reviewer's remaining concerns.
We conducted the suggested experiments and found that Seg OPD retains higher reasoning accuracy on all eight datasets, as shown below. We also clarify that redraft SFT and segment-wise preference learning optimize different objectives and therefore need not produce matching gradient scales, as discussed in Appendix B.6. We attribute the superiority of our method to the following differences between the two objectives:
Our method explicitly trains the student to prefer teacher redrafts over their paired student segments. In contrast, redraft SFT increases redraft likelihood without explicitly discouraging the original segments, which may therefore remain likely choices at inference [1].
Our method emphasizes insufficiently learned revision preferences through the coefficient βσ(−βmπθ(p)), which decreases as the revision margin grows. Rescaling SFT loss is not equal to this margin-dependent weighting, as we analyzed in Appendix B.6.
For this rebuttal, we evaluate three variants of Seg-OPD\N. Specifically:
We kindly invite the reviewer to check Appendix B.6 for a more detailed analysis of these variants. For this rebuttal, we use Qwen3-4B as the student and train each compared method with different training seeds. We report the mean reasoning accuracy (%) in the following table:
From the table, we observe that:
Both gradient calibration and SFT loss-weight tuning can improve performance. This indicates that balancing redraft imitation with token-wise OPD still enables the student to benefit from teacher redrafts.
Our method outperforms the additional SFT variants. This indicates the effectiveness and practical value of our method, which explicitly teaches the student to prefer teacher redrafts over student segments, beyond adjusting the strength of redraft imitation.
[1] Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data. ICML 2024