- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies on-policy self-distillation. The authors find that this approach reaches strong pass@1 but reduces output diversity, so pass@k barely improves as more samples are drawn. They argue that the teacher favors rollouts that resemble the demonstration, and demonstrations tend to be the most common solutions. Their theory supports this. Under a KL-regularized version of the objective, the optimal policy reweights rollouts by how well they match the demonstrations, rather than by correctness, so already likely solutions become even more likely.
Critical strengths
-
It points out a real and easy-to-miss cost of a popular method. Self-distillation methods are gaining attention mainly because of their strong pass@1. This paper shows that the gain can come with a loss in diversity that pass@1 does not reveal.
-
Clean theory:The theory gives an intuitive explanation: the teacher rewards rollouts that look like the demonstration, so common solutions keep getting reinforced.
Critical weaknesses
None
Questions for authors
Self-distillation often reaches higher pass@1, and a model with higher pass@1 has less room to improve as k grows. So part of the flatter pass@k curve may simply reflect a better starting point. Could the authors compare GRPO and self-distillation at checkpoints with similar pass@1, or report diversity as a function of pass@1 over training? This would make it clearer that the diversity gap comes from the method itself.
2 comments
Thank you for the detail rebuttal. I would keep my positive score.
Rebuttal
We thank the reviewer for appreciating the importance of our highlight of “easy-to-miss cost of a popular method” and for appreciating our analysis. We respond in detail below and revise the manuscript for clarity in incorporating the response.
pass@1 starting point and diversity
The reviewer raises a good question about the relation between the initial pass@1 performance and diversity. Indeed, they are related, and we argue that higher pass@1 in SDSD is obtained through a mechanism that reduces the diversity to a greater extent than methods like GRPO.
In terms of pass@k curves, in most experiments in Figure 3, Figure 4, SDSD has a higher pass@1 but pass@k has room to grow as more responses are sampled. GRPO overtakes SDSD when using a high number of k samples and achieves higher pass@K.
We agree with the reviewer that comparing the diversity of SDSD and GRPO when they have the same pass@1 is insightful. This way we disentangle the diversity from performance. We use the coding setup for this, where we use the semantic diversity described in the response “Direct measure of diversity” to reviewer 8vSi .
We compute the diversity metrics on the medium set of LCBv6 for SDSD and GRPO checkpoints with similar pass@1:
At closely matched pass@1, SDSD exhibits substantially lower semantic diversity than GRPO. At 79.9% pass@1, SDSD produces only 0.98 distinct correct algorithms and 1.22 distinct correct implementations per problem, with 92.7 pass@16, compared to 1.39-1.44 correct algorithms and 2.20-2.34 correct implementations, with 97.6-100.0 pass@16 for GRPO at 78.2-81.4% pass@1.
This shows that GRPO and SDSD have different mechanisms for increasing the pass@1 performance, and SDSD does so at a higher loss of diversity.