acceptodds
Under review as a conference paper at ICLR 2027

On-Policy Self-Distillation with Sampled Demonstrations Reduces Output Diversity

Abstract

On-policy self-distillation achieves strong pass@1 accuracy by using a single model as both teacher and student, with the teacher conditioned on a correct demonstration to provide dense token-level feedback. We show that this could come at a hidden cost: rollout diversity decreases and pass@k curves flatten (i.e., generating more rollouts fails to improve accuracy). We trace this to compounding biases in the design of self-distillation with sampled demonstrations. The teacher scores each student rollout while conditioned on a sampled correct rollout, channeling its feedback through the model's own biases. We theoretically analyze the optimal self-distillation policy and show that it tilts the base distribution by a pointwise conditional mutual information score between the student's rollout and the correct rollout used as context. Unlike the ideal optimal on-policy reinforcement learning (RL), which preserves probability ratios among equally correct rollouts, self-distillation can amplify existing probability gaps, concentrating mass on already-dominant modes. On a controlled graph path-finding task and science question-answering benchmarks, self-distillation confirms this pattern: competitive average performance but substantially lower functional and semantic diversity than RL models, and failure on out-of-distribution settings that require diverse strategies.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, reasoning, self-distillation, diversity, On-Policy Self-Distillation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_5njHOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies on-policy self-distillation. The authors find that this approach reaches strong pass@1 but reduces output diversity, so pass@k barely improves as more samples are drawn. They argue that the teacher favors rollouts that resemble the demonstration, and demonstrations tend to be the most common solutions. Their theory supports this. Under a KL-regularized version of the objective, the optimal policy reweights rollouts by how well they match the demonstrations, rather than by correctness, so already likely solutions become even more likely.

Critical strengths
  1. It points out a real and easy-to-miss cost of a popular method. Self-distillation methods are gaining attention mainly because of their strong pass@1. This paper shows that the gain can come with a loss in diversity that pass@1 does not reveal.

  2. Clean theory:The theory gives an intuitive explanation: the teacher rewards rollouts that look like the demonstration, so common solutions keep getting reinforced.

Critical weaknesses

None

Questions for authors

Self-distillation often reaches higher pass@1, and a model with higher pass@1 has less room to improve as k grows. So part of the flatter pass@k curve may simply reflect a better starting point. Could the authors compare GRPO and self-distillation at checkpoints with similar pass@1, or report diversity as a function of pass@1 over training? This would make it clearer that the diversity gap comes from the method itself.

2 comments

We thank the reviewer for appreciating the importance of our highlight of “easy-to-miss cost of a popular method” and for appreciating our analysis. We respond in detail below and revise the manuscript for clarity in incorporating the response.

pass@1 starting point and diversity

The reviewer raises a good question about the relation between the initial pass@1 performance and diversity. Indeed, they are related, and we argue that higher pass@1 in SDSD is obtained through a mechanism that reduces the diversity to a greater extent than methods like GRPO.

In terms of pass@k curves, in most experiments in Figure 3, Figure 4, SDSD has a higher pass@1 but pass@k has room to grow as more responses are sampled. GRPO overtakes SDSD when using a high number of k samples and achieves higher pass@K.

We agree with the reviewer that comparing the diversity of SDSD and GRPO when they have the same pass@1 is insightful. This way we disentangle the diversity from performance. We use the coding setup for this, where we use the semantic diversity described in the response “Direct measure of diversity” to reviewer 8vSi .

We compute the diversity metrics on the medium set of LCBv6 for SDSD and GRPO checkpoints with similar pass@1:

methodsteppass@1 (%)pass@16 (%)algorithmic divimplementation div
GRPO5078.297.61.392.34
GRPO6081.4100.01.442.20
SDSD1079.992.70.981.22

At closely matched pass@1, SDSD exhibits substantially lower semantic diversity than GRPO. At 79.9% pass@1, SDSD produces only 0.98 distinct correct algorithms and 1.22 distinct correct implementations per problem, with 92.7 pass@16, compared to 1.39-1.44 correct algorithms and 2.20-2.34 correct implementations, with 97.6-100.0 pass@16 for GRPO at 78.2-81.4% pass@1.

This shows that GRPO and SDSD have different mechanisms for increasing the pass@1 performance, and SDSD does so at a higher loss of diversity.

Thank you for the detail rebuttal. I would keep my positive score.

Reviewer_8vSiOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper investigates the loss of output diversity in on-policy self-distillation with sampled demonstrations. The authors theoretically show that SDSD can amplify existing probability imbalances among equally correct solutions, leading to reduced diversity. Through experiments on graph path-finding, science QA, and coding tasks, they demonstrate that SDSD achieves competitive pass@1 performance but exhibits lower diversity than RL, potentially harming out-of-distribution generalization.

Critical strengths

S1. Important Research Question and Theoretical Analysis

The paper investigates an important trade-off between accuracy and output diversity in on-policy self-distillation. The theoretical analysis provides insights into how demonstration-conditioned feedback can amplify existing biases and reduce output diversity.

S2. Comprehensive Experimental Evaluation

The paper validates its findings across controlled graph path-finding, science QA, and coding tasks. The controlled experiments are particularly useful for understanding how reduced diversity can affect out-of-distribution generalization.

Critical weaknesses

W1. Gap Between Theory and Practice

The theoretical analysis assumes a fixed teacher and a fixed demonstration distribution, whereas practical SDSD uses an evolving EMA teacher and on-policy demonstrations. It remains unclear to what extent the theoretical findings explain the dynamics of practical SDSD training.

W2. Limited Empirical Evidence of Diversity Reduction

While the controlled graph experiments demonstrate reduced solution diversity, the evaluation on real-world tasks relies primarily on pass@k, which does not directly measure output diversity. Moreover, the practical impact varies across tasks, as SDSD with feedback outperforms GRPO on pass@16 in coding. A more direct evaluation of semantic diversity and its impact on downstream performance would strengthen the findings.

Questions for authors
  1. DAIL (arXiv:2602.02405; ICML 2026) reports improvements in pass@k using self-distillation with expert demonstrations and a contrastive objective, which appears to contrast with this paper’s findings. Could the authors discuss the key differences between these approaches and explain which factors may account for the different outcomes?

  2. Could the authors evaluate output diversity using metrics beyond pass@k, such as the number of distinct solution strategies, semantic clustering, or pairwise similarity between generated responses? This would help distinguish actual diversity reduction from changes in solution accuracy.

  3. To what extent is the observed diversity reduction specific to demonstration-conditioned self-distillation, rather than a general consequence of reverse KL minimization? Could the authors further clarify the role of the proposed PCMI mechanism?

4 comments
AuthorsOpenReview ↗

Rebuttal response 1

We thank the reviewer for their insightful comments and for appreciating our important research question, theoretical analysis and for comprehensive experimental evaluation. We respond in detail below and revise the manuscript for clarity in incorporating the response.

Direct measure of diversity

We agree with the reviewers that a direct measure of diversity in real world cases is useful.

For this, we analyse the coding setup, that is easier to interpret, and where different measures of diversity arise more naturally. Here it is easier to distinguish between diversity in terms of algorithms, implementation or surface level.

For every problem in LCBv6 we sample 16 programs from GRPO and SDSD at a final checkpoint of step 80. We use LLM as a judge approach to compute two diversity scores. A Claude Opus 5.5 model reads all 16 programs of a set and counts how many different correct programs are in the set, according to two separate semantic diversity criterions. Each set is judged on its own, without knowing which method or checkpoint produced it.

  • Algorithm diversity: two programs are considered different if they have the different underlying algorithm.
  • Implementation diversity: two programs are considered different if they have different algorithms or implementation choices. Examples are recursion vs explicit stack, BFS vs DFS, memoization vs tabulation, or a different data structure with the same complexity.

We split problems by how hard they are for the base model: easy (the base model solves 16/16), medium (1–15/16) and hard (0/16). All three subsets showcase the same trends, but the medium split is the one where post-training methods produce the most effects (the easy subset is saturated, while the hard is almost impossible).

splitmodelalgorithmic divimplementation divpass@1 / pass@16
easyGRPO1.191.8199.2 / 100.0
SDSD1.001.06100.0 / 100.0
mediumGRPO1.462.3785.7 / 100.0
SDSD0.930.9389.2 / 92.7
hardGRPO0.050.051.0 / 5.4
SDSD0.050.054.3 / 5.4
allGRPO0.630.9939.5 / 46.6
SDSD0.440.4542.6 / 44.3

We observe that SDSD has lower diversity than GRPO both in terms of algorithmic diversity and implementation diversity.

This agrees with SDSD having higher pass@1 but lower pass@16 than GRPO. Thus SDSD is worse both in semantic diversity as well functional diversity in the coding setting.

AuthorsOpenReview ↗

Rebuttal response 2

Theory and Practice Relation

“The theoretical analysis assumes a fixed teacher and a fixed demonstration distribution, whereas practical SDSD uses an evolving EMA teacher and on-policy demonstrations.”

We agree with the reviewer about these differences, and it is important to highlight and explain them. Importantly, we argue that the theory conditions are sufficient and can explain the bias for diversity reduction, while the practical conditions can enhance this even more.

First, our derivation assumes demonstrations are sampled from a fixed policy, which more closely resembles approaches like OPSD or our external-demonstration graph experiments (Section 4.4, Section A.2) than sampling on-policy from the student. The demonstrations can be seen as anchoring points, encouraging the student to stay close to them. Sampling the demonstrations from the student thus removes this anchoring to external, fixed points and allows more collapse.

Second, our theoretical analysis assumes a teacher frozen at the base policy, whereas most practical SDSD implementations, including ours, employ a teacher as an EMA of the student itself. Moving the teacher from the fixed base towards the student itself brings more biases for collapse. At the limit, using the student as the teacher is the most biased, as it’s easy for the student to reinforce its own unconditional outputs.

Our experimental analysis in Section 4.4 (Table 2) supports these arguments. First, using fixed and diverse demonstrations slightly increases the diversity to 1.119 compared to standard SDSD with demonstrations sampled from the student of 1.034. Additionally, using an initial frozen base as a teacher brings another small increase in diversity to 1.138. Nevertheless, using both still has very low diversity compared to GRPO + diversity of 1.797. Thus, the practical differences contribute to the reduced diversity only to a minor degree.

Overall, the main mechanism described by the theory and argued in the paper doesn’t need the practical choices to explain the reduced diversity. The practical choices bring additional biases for collapse but only to a minor degree.

Reverse KL vs demonstration-conditioned self-distillation

We agree with the reviewer that reverse KL minimization is mode-seeking and reduces diversity (Bishop, 2006). This stems from the sampling from the student, while trying to match the teacher. We argue that SDSD has compounding biases on top of this.

First, optimizing standard RL with KL regularization to the base model can be seen as optimizing a reverse KL between the model and the optimal RL policy.

Eπ[R]−βDKL(π∥π0)=−βDKL(π∥πRL∗)+ConstE_{\pi}[R] - \beta D_{\mathrm{KL}}(\pi\Vert\pi_0) = -\beta D_{\mathrm{KL}}(\pi\Vert\pi^*_{RL}) + Const

Similarly, the SDSD objective (Eq. 3) equals

Eycorr[DKL(π∥π0(.∣x,ycorr))]+βDKL(π∥π0)=(1+β)DKL(π∥πSD∗)+ConstE_{y^{corr}}[D_{\mathrm{KL}}(\pi\Vert\pi_0(.|x,y^{corr}))] + \beta D_{\mathrm{KL}}(\pi\Vert\pi_0) = (1+\beta)D_{\mathrm{KL}}(\pi\Vert\pi^*_{SD}) + Const

Both methods therefore minimize a reverse KL to a target distribution. The difference lies in the targets. As all experiments in the paper show SDSD has lower diversity than regularised RL (GRPO) it means that the loss in diversity of SDSD cannot be attributed to the reverse KL aspect.

The optimal policy of SDSD shows the mechanism for compounding biases: unlike the RL optimum, which is tilted by task reward, the SDSD optimum is tilted by expected PCMI between the student rollout and the sampled correct demonstration. The problem is that the teacher is biased itself: it depends on its capability to relate the student rollout y and the correct demonstration ycorr and it depends on the sampling of y and ycorr. When the demonstrations are skewed, or when the teacher cannot understand the relation between y and ycorr this can lead to low PCMI and reduced diversity. We expand on this in the paragraph “On-policy RL is mode-seeking, SDSD is even more” of the manuscript.

Let’s consider a reverse KL baseline from Table 2 of the paper. We see that “SDSD + diverse demos, strong teacher” has good performance and good diversity. The objective of this model is also a reverse KL between student and teacher with fixed demonstrations and a strong underlying model.

min⁡πEycorr∼fixedDKL(π(.∣x)∥πstrong(.∣x,ycorr))+βDKL(π(.∣x)∥π0(.∣x))\min_{\pi} E_{y^{corr} \sim fixed} D_{\mathrm{KL}}(\pi(.|x)\Vert\pi_{strong}(.|x,y^{corr})) + \beta D_{\mathrm{KL}}(\pi(.|x)\Vert\pi_0(.|x))

This “SDSD + diverse demos, strong teacher” model achieves a diversity of 1.474 compared to standard SDSD of 1.034. So even though this model optimizes a reverse KL objective, it does not have the same extreme diversity loss as SDSD.

When using a weaker underlying teacher model, the diversity collapses to 1.138. Also when using the demonstrations coming from the student the diversity reduces to 1.031. Thus, the most important aspects for diversity are a teacher capable of understanding the alignment between student rollouts and demonstrations and having diverse demonstrations.

AuthorsOpenReview ↗

Rebuttal response 3

Relation to DAIL

We thank the reviewer for pointing out the DAIL paper, we will discuss it below and in the final manuscript.

DAIL method is an off-policy, offline training method that uses external, expert demonstrations to guide a student model. First, they generate a set of offline solutions using a teacher that has expert demonstrations in the context. Then they finetune the student on these offline solutions such that the student matches the teacher that has the initial expert demonstrations in context.

This approach is different from SDSD and approaches like SDPO and OPSD and some of these differences are beneficial in terms of diversity. First, DAIL is off-policy, it doesn’t sample from the student during student training, it only finetunes on offline dataset. Thus it avoids the mode seeking biases coming from using on-policy data, like in SDSD.

Then the teacher M(ri,t∣xi,si,ri,<t)M(r_{i, t} | x_i, s_i, r_{i, <t}) scores the solution rir_i that is generated from the expert solution sis_i. The solutions and demonstrations are aligned as one is generated from the other. This avoids another bias of SDSD where the teacher needs to understand the relation between an on-policy solution generated by a student, and a potentially different demonstration.

Additionally, DAIL also uses negative KL with a teacher with weaker demonstrations. We don’t analyse such a case since this is not standard for methods like SDSD, SDPO, OPSD. Nevertheless, table 9 in their paper, shows that DAIL without the negative term still improves in terms of pass@128.

Thus DAIL is consistent with the insights from our analysis: it improves the alignment in the teacher and uses good fixed demonstrations. These choices are possible because they use an offline, off-policy setting.

Thank you for the response. I am satisfied with the rebuttal results. I already have a positive score, so I've updated the soundness score to 3.

Reviewer_y7aXOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates diversity collapse in on-policy self-distillation with sampled demonstrations (SDSD). The authors theoretically show that demonstration-conditioned distillation can amplify probability imbalances among correct responses through PCMI-induced sharpening. Experiments on graph path-finding, science QA, and coding tasks demonstrate that SDSD achieves competitive Pass@1 but often suffers from reduced diversity and weaker Pass@k scaling.

Critical strengths
  1. The paper provides a meaningful theoretical explanation for diversity collapse in self-distillation.
  2. The controlled graph experiments effectively demonstrate the relationship between reasoning diversity and OOD generalization.
  3. Experiments across multiple models and tasks provide convincing evidence of the practical relevance of the problem.
Critical weaknesses

On science QA and coding tasks, diversity is mainly inferred from Pass@k curves, without directly measuring distinct reasoning strategies.

Questions for authors

Can the authors provide semantic diversity measurements on real reasoning benchmarks beyond the synthetic graph setting?

1 comment

We thank the reviewer for appreciating the paper theoretical insights, the controlled setting showcasing relation between diversity and OOD performance, and experimental validation in multiple settings.

Direct measure of diversity

We agree with the reviewers that a direct measure of diversity in real-world tasks is useful. We respond in detail below and revise the manuscript for clarity in incorporating the response.

For this, we analyse the coding setup, that is easier to interpret, and where different measures of diversity arise more naturally. Here it is easier to distinguish between diversity in terms of algorithms, implementation or surface level.

For every problem in LCBv6 we sample 16 programs from GRPO and SDSD at a final checkpoint of step 80. We use an LLM-as-a-judge approach to compute two diversity scores. A Claude Opus 5.5 model reads all 16 programs of a set and counts how many different programs are in the set, according to two separate semantic diversity criterions. Each set is judged on its own, without knowing which method or checkpoint produced it.

  • Algorithm diversity: two programs are considered different if they have the different underlying algorithm.
  • Implementation diversity: two programs are considered different if they have different algorithms or implementation choices. Examples are recursion vs explicit stack, BFS vs DFS, memoization vs tabulation, or a different data structure with the same complexity.

As in the graph setting, we count only correct solutions (programs that pass all tests): the algorithmic (implementation) diversity of a problem is the number of distinct algorithms (implementations) among its correct programs. A problem with no correct program counts as 0, so both methods are measured on the same set of questions.

We split problems by how hard they are for the base model: easy (the base model solves 16/16), medium (1–15/16) and hard (0/16). All three subsets showcase the same trends, but the medium split is the one where post-training methods produce the most effects (the easy subset is saturated, while the hard subset is almost impossible).

splitmodelalgorithmic divimplementation divpass@1 / pass@16
easyGRPO1.191.8199.2 / 100.0
SDSD1.001.06100.0 / 100.0
mediumGRPO1.462.3785.7 / 100.0
SDSD0.930.9389.2 / 92.7
hardGRPO0.050.051.0 / 5.4
SDSD0.050.054.3 / 5.4
allGRPO0.630.9939.5 / 46.6
SDSD0.440.4542.6 / 44.3

We also present the diversity of all rollouts, including the incorrect ones.

splitmodelalgorithmic div - allimplementation div - allpass@1 / pass@16
easyGRPO1.251.9499.2 / 100.0
SDSD1.001.06100.0 / 100.0
mediumGRPO2.023.2285.7 / 100.0
SDSD1.511.6689.2 / 92.7
hardGRPO5.507.971.0 / 5.4
SDSD5.387.534.3 / 5.4
allGRPO3.895.7539.5 / 46.6
SDSD3.634.9042.6 / 44.3

We observe that SDSD has lower diversity than GRPO both in terms of algorithmic diversity and implementation diversity both when considering correct and all programs.

This agrees with SDSD having higher pass@1 but lower pass@16 than GRPO. Thus SDSD is worse both in semantic diversity as well functional diversity in the coding setting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.