- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper critically re-examines recent claims that self-distillation improves reasoning, focusing on SDPO and OPSD. Through reproductions and controlled experiments across Qwen3 models from 1.7B to 14B and Olmo-3-7B, the paper argues that several reported gains have alternative explanations. For SDPO on code, gains largely disappear under problem-disjoint evaluation, suggesting memorization of training problems rather than transferable reasoning. On science and tool-use tasks, supervised fine-tuning on final answers matches or exceeds SDPO, questioning whether privileged reasoning traces are necessary. For OPSD, the paper finds that performance is largely explained by transferring the teacher's thinking-mode behavior rather than by the particular privileged solution supplied to the teacher. Finally, it compares self-distillation updates with reward-optimization gradients and finds poor alignment on held-out prompts.
Critical strengths
- This is a valuable corrective study because it proposes concrete alternative explanations and changes the experimental setting to test them. The problem-disjoint coding split cleanly distinguishes generalization to new test cases of the same specification from generalization to unseen problems, showing that these are substantially different claims.
- The analysis of OPSD is particularly convincing. Varying the teacher's thinking mode independently of whether the reference solution is correct, absent, or shuffled directly separates thinking-mode transfer from privileged-information transfer. The output-format artifacts and response-length analysis provide additional evidence for the same explanation, and the results are reproduced across model sizes and families.
Critical weaknesses
- Two interpretations are stronger than the corresponding interventions support. Final-answer SFT matching SDPO shows that explicit rationales or privileged CoT supervision are unnecessary for these benchmark gains; it does not show that the task or the model's computation requires no reasoning. Similarly, the gradient experiment compares self-distillation updates on training prompts with a GRPO gradient on held-out prompts. Negative cosine similarity is evidence of poor alignment with generalizable reward optimization, but it does not directly contradict Proposition 3.1, which assumes the same prompt distribution, the ideal KL objective, and the true optimal teacher.
- The broad conclusion is based on two particular methods, SDPO and OPSD, and several related failure modes have also been reported by concurrent work discussed in the paper. The experiments provide strong evidence about these implementations, but they do not cover self-distillation teachers conditioned on the student's actual attempt and verifier outcome, or other methods that construct the teacher differently. The title and general conclusion should be scoped accordingly.
Questions for authors
- For the gradient analysis, what happens when both gradients are estimated on exactly the same prompts and checkpoint, before separately evaluating whether either direction improves held-out reward? This would distinguish objective misalignment from train-test distribution shift.
- Which conclusions are intended to extend to methods whose teacher observes the student's actual attempt and verifier outcome, rather than a privileged reference solution or thinking-mode prompt? This boundary materially affects whether the paper is about SDPO/OPSD or self-distillation more generally.
3 comments
Dear Reviewer Fvvk,
The authors have provided a detailed reply to your review. Could you please take a look, acknowledge the rebuttal, and give your response?
Thank you for serving as a reviewer for ICLR.
Best,
AC
Most of my concerns have been addressed and I remain my positive judgement to the work.
Thank you for your positive feedback on the value of the paper and its analysis! We revise our manuscript based on your feedback below. Please note that for the revision, all changes are in red and new material is Section 4.3, Section 4.5, and Appendices D.3.2, D.3.4, and D.6.1.
We hope our responses above have resolved your concerns about our analyses, and we would be glad to address any additional questions you may have.
References: