acceptodds
Under review as a conference paper at ICLR 2027

Disentangling Self-Distillation: Measuring and Modeling Acquisition and Retention

Abstract

Self-distillation with privileged context adapts a language model from demonstrations by letting the model, once conditioned on a reference response, teach its context-free copy token by token. Our taxonomy reveals existing methods differ along three entangled axes: (i) the *rollout source* (student or teacher), (ii) the *teacher coupling* (frozen, or an exponential moving average of the student at some coupling rate) and (iii) the *KL direction* (reverse or forward), yet these axes are usually studied in fixed combinations and have led to conflicting conclusions. We formalize a unifying framework to encompass all self-distillation methods vs classic supervised fine-tuning: we train every combination of the three axes, on Qwen2.5-7B and Ministral-3-3B across ordinary and contradictory tasks, totaling 1,200 adaptation runs, to systematically investigate the impact of the above axes. We propose a controlled model of the same objective to explain the resulting acquisition–retention trade-offs. We find that (i) the rollout source matters mostly where the task contradicts the pretrained behavior: there teacher rollouts raise acquisition well above what student rollouts achieve, with almost no change in retention; (ii) the teacher coupling changes acquisition most, on every task: acquisition rises with the coupling rate, then falls past a task-specific rate; (iii) switching the KL direction costs retention in one model but not the other so which axis to tune first depends on the model. The controlled model reproduces the three trends.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Self-Distillation, On-Policy Distillation, KL Divergence, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_AqZNOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper provides an empirical assessment of self-distillation with privileged context. The authors organize the existing methods along three axes: (1) on- or off-policy setup, (2) frozen or moving teacher, and (3) distillation loss (forward vs. reverse KL). All combinations of these three axes are trained on Qwen2.5-7B and Ministral-3-3B, tested on five tasks, two of which are in contradiction with the model's pre-trained behavior. The authors measure the improvements on new tasks (acquisition) and retention on general benchmarks.

There are three main findings: off-policy rollouts improve acquisition mostly on the contradictory tasks, and have small effects on retention; a moving teacher can help up to a certain point and then degrades learning beyond it, which is task and model dependent; the cost of switching the direction of the loss (forward vs. reverse) also depends on the model.

Critical strengths
  • The paper does a good job at reconciling conflicting findings in different papers that cover self-distillation with privileged information. By training every combination across three axes, the results can be compared within the same setup. The corresponding findings seem practically useful. For instance, off-policy rollouts seem to be most useful when the new task contradicts the model's base behavior, without substantially hurting retention.

  • Acquisition and retention are evaluated together and compared against SFT trained on the same demonstrations. The authors find that most self-distillation settings achieve a better trade-off (acquisition vs. retention) than SFT, but SFT still achieves the highest acquisition on most tasks.

Critical weaknesses
  • Conclusions about when each axis matters often rely on just a few cases (e.g., just two contradictory tasks, two models, and one type of privileged context). It remains unclear whether the findings would hold for setups with different privileged information (e.g., feedback or hints).

  • The explanatory role of the controlled model is overstated. The authors state that the model explains the trade-off between acquisition and retention, but the parameters are not measured on the LLMs and are instead selected to match the trends. This only shows that the trends can come from this mechanism, and not that it explains the results.

Questions for authors

How do setups with other types of privileged context behave (e.g., feedback, hints) and do the same findings hold?

4 comments
AuthorsOpenReview ↗

Response to Reviewer AqZN (1/2)

Thank you for raising the role of privileged context and the interpretation of the controlled model. We added runs with a hint as privileged context (new App. D.6) and a measurement relating the LLM tasks to the controlled model (new App. F.1.1), and we revised how the paper presents the controlled model.

W1-Q1. Conclusions about when each axis matters often rely on just a few cases (e.g., just two contradictory tasks, two models, and one type of privileged context). It remains unclear whether the findings would hold for setups with different privileged information (e.g., feedback or hints). How do setups with other types of privileged context behave (e.g., feedback, hints) and do the same findings hold?

We extended the evaluation on the privileged context and on the seeds.

Final-answer hint as privileged context. In the main grid, the privileged context is fixed to the full reference response, so that the rollout-source, coupling and KL comparisons all share the same supervision. To test whether the findings depend on this choice, we gave the teacher only the final answer as a hint: the answer letter on chemistry and the base-9 result on math-contradiction, one ordinary and one contradictory task. We repeated the coupling and KL-direction comparisons on Qwen with teacher rollouts (new App. D.6).

On chemistry, both comparisons give the same results as in the main text. The coupling curve keeps its shape: a frozen teacher is already close to the best rate, and the fastest rate falls below the untrained accuracy. With a frozen teacher and at the best coupling rate, switching to forward KL changes acquisition by at most about one point and retention by less than half a point. With less information in the context, acquisition peaks lower than with the full reference.

On math-contradiction, the coupling curve is now flat as well: coupling no longer helps. The hint contains no intermediate calculations, so the teacher has to write them itself. Tab. 11 (new, App. D.6) reports whether the teacher and the student write these calculations in base 9, as the task requires, or in base 10. With the full reference, the share written in base 9 rises with the coupling rate, from 7% with a frozen teacher to 89% at α=0.05\alpha = 0.05. With the hint, it stays below 4%: neither the teacher nor the student writes calculations in base 9, at any coupling rate. The gain from coupling that the main text reports on math-contradiction therefore comes with calculations written in base 9, which appear with the full reference and do not appear with the hint.

Feedback. Experiments with feedback as privileged context are running, and we will add them to App. D.6 before the end of the discussion period. Feedback raises a point that we now state as a limitation in the conclusion. The controlled model represents the effect of the privileged context as a fixed shift of the teacher toward the target behavior. Feedback depends on the current attempt and changes as the student's mistakes change, so the controlled model may not describe this case.

Seeds. We trained two new seeds for the five configurations of the math-contradiction panel of Fig. 3a, which reproduce its conclusions (App. D.5). Tab. 7 now reports the KL-direction effects seed by seed: no significant effect changes sign between seeds. A third seed for some of the main figures is running, and we will add it to the paper before the end of the discussion period.

Models. The experiments cover two models from different families and sizes, and the conclusion states this scope. Adding a model requires the full grid of 600 runs, which we could not run during the rebuttal.

AuthorsOpenReview ↗

Response to Reviewer AqZN (2/2)

W2. The explanatory role of the controlled model is overstated. The authors state that the model explains the trade-off between acquisition and retention, but the parameters are not measured on the LLMs and are instead selected to match the trends. This only shows that the trends can come from this mechanism, and not that it explains the results.

We agree. The controlled model reproduces the three trends by varying three quantities: initial-policy concentration, context strength, and feature overlap. We do not measure these quantities on the LLMs. The revised paper says so in the conclusion, and the abstract and Sec. 6 now describe the model as a way to interpret and reproduce the trends. Estimating the three quantities from a model and a task before training is listed as future work.

Until then, we can relate the rollout-source and coupling effects to differences we observe between tasks. These differences show how one task compares with another, without measuring the initial-policy concentration or the context strength themselves.

Rollout source. For this effect, we consider spatial and spatial-contradiction: they share the same inputs and the same form of privileged context, and only the expected output of spatial-contradiction contradicts the pretrained behavior. The table below gives, for the initial models and on both tasks, a. the validation set accuracy without context (student accuracy), b. with context (teacher accuracy), and c. a new measurement, the teacher utility on the student's path (defined below). We also report d. the advantage of teacher rollouts after training, i.e. their acquisition minus that of student rollouts (median over matched pairs, Tab. 5).

The teacher utility on the student's path is defined as follows (details in the new App. F.1.1). For the spatial tasks, a solution is a sequence of moves (such as (+1, 0); a reference solution is shown in App. C.5). We take the solution written in the ordinary convention, the convention the initial student follows. Before each move along this solution, we ask the teacher to predict the next move, and record the probability it gives to the correct one (which is the ordinary one for spatial, and the rotated one for spatial-contradiction). We then average those probabilities over the moves from 64 problems. A value of 1 would mean that the teacher always predicts the correct move from its context, regardless of the moves already written on the student's path.

initial modeltaskstudent accuracyteacher accuracyteacher utility on the student's pathadvantage of teacher rollouts
Qwenspatial26.2%58.0%0.48+2.4 pp
Ministralspatial19.6%65.4%0.40+2.5 pp
Qwenspatial-contradiction1.5%28.2%0.15+4.9 pp
Ministralspatial-contradiction0.8%20.0%0.16+4.4 pp

Two things happen on the contradictory task. First, the student's rollouts almost never contain the new behavior (accuracy near 1%), while the teacher's do (20.0-28.2%).

Second, the teacher utility on the student's path falls from 0.40-0.48 on the ordinary task to 0.15-0.16 on the contradictory one, even though its context has the same form in both tasks. We observe a larger advantage of teacher rollouts (e.g., +4.9 vs +2.4 pp for Qwen) where this utility is lower.

This is the behavior of the controlled model when the initial-policy concentration λ\lambda increases: the teacher utility on the student's path falls (new App. F.1.1), and the advantage of teacher rollouts grows (Fig. 1b). With respect to the ordinary task, the contradictory task thus behaves like a larger λ\lambda.

Coupling. The quantity is the context strength κ\kappa, how far the context alone moves the initial model toward the new task. Its closest observable is the acquisition reached with a frozen teacher. On Qwen (Fig. 2a), a frozen teacher reaches about 90% of the best SFT gain on tool-alpaca and chemistry, where it is nearly the best coupling, and 2% on the two spatial tasks (whose best rate is α=0.10\alpha = 0.10). In the model, the best rate α\alpha also increases when κ\kappa decreases (Fig. 2b). On Ministral this relation is less clear (Fig. 5).

The model was built to vary these two quantities separately, which the LLM tasks cannot do.

Thank you for the response and for running additional experiments. The new setup with two spatial tasks is really nice. Together with the revisions, it resolves my second concern about the role of the controlled model.

I have two minor reservations. First, the benefit of coupling seems to depend on the type of context, as it no longer helps on math-contradiction with the hint. The other reservation is about the hint experiment, where only the teacher rollouts are used on Qwen, so I'm not sure whether the finding about the rollout source would hold with a different conext.

Overall, I don't see these issues as critical and I think that the paper is a valuable contribution.

Dear Reviewer AqZN,

Thank you for your reply and for your positive assessment. We are glad that the comparison of the two spatial tasks resolves your concern about the controlled model.

Coupling and context. We agree that the benefit of coupling depends on the context. In the main grid, the best coupling rate already depends on the task and the model; the hint and the new feedback runs show that it also depends on the context (see Feedback below). Sec. 6.2 now states this dependence.

Rollout source and context. As you note, the paper compares the two rollout sources with the full reference only. Tab. 11 suggests what to expect with the hint on math-contradiction: the teacher's own responses then contain no calculations in base 9, so its rollouts would carry no more of the new behavior than the student's. Two follow-ups would test this directly, and we leave them to future work: measuring the teacher utility on the student's path (App. F.1.1) under other contexts, which requires extending it beyond the move-by-move solutions of the spatial tasks, and varying the context gradually within one task, from the final answer to the full reference.

Since our first answer, the revised paper also includes the feedback experiments and the third seed we announced, together with experiments on a third model family (Gemma).

Feedback. App. D.6.2 adds feedback as privileged context, on Qwen with student rollouts, since feedback describes the student's response. The feedback is written either by GPT-5 mini, on chemistry, or by the student itself (self-feedback), on chemistry and math-contradiction. As in the main text, acquisition still peaks at one coupling rate and falls beyond it, here at a slower rate (frozen on chemistry and α=0.05\alpha = 0.05 on math-contradiction, against 0.250.25 with the full reference).

Seeds. We trained a third seed for every run of Figs. 1a and 2a, and compare the three-seed averages with the two-seed averages of the main text in the new App. D.5.1. With three seeds, teacher rollouts still win every pair of the contradictory tasks by a wide margin, the retention differences barely change, and the best coupling rate of Fig. 2a is unchanged on all five tasks. The conclusions of Secs. 6.1 and 6.2 therefore hold.

Models. The full grid remained out of reach, but we repeated the comparison of Fig. 3a (chemistry and math-contradiction, teacher rollouts) on a third model family, Gemma-4-12B, with two seeds (new App. D.7). On math-contradiction, Gemma behaves like Qwen: switching to forward KL costs retention, so the coupling remains the axis to tune first. On chemistry, the differences stay within Gemma's much larger variation across seeds.

Thank you again for your constructive comments and your time reviewing our paper.

Best regards, The authors

Reviewer_fPHzOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies self-distillation with privileged context, where a model that sees a reference answer teaches a copy of itself that does not. The authors argue that existing methods differ along three axes: who generates the training responses (student or teacher), how the teacher is updated (frozen or an EMA of the student), and the KL direction (forward or reverse). They cross all three axes on Qwen2.5-7B and Ministral-3-3B over five tasks, two of which contradict the model's prior knowledge, and measure both acquisition of the new task and retention of general capabilities. They find that teacher rollouts help mainly on the contradictory tasks, that acquisition first rises and then falls as the teacher moves faster, and that the effect of the KL direction depends on the model. A small controlled model of the same objective is used to reproduce these trends. Finally, SFT reaches the highest acquisition on most tasks, but most self-distillation settings offer a better balance between acquisition and retention.

Critical strengths

1. A large and systematic study. The paper crosses all three axes (rollout source, teacher coupling, and KL direction) on two models and five tasks, for a total of 1,200 runs. The comparisons also rest on a sound statistical basis, using matched-pair Wilcoxon tests with Holm correction.

2. A useful map of the literature. Tab. 1 organizes a fast-growing set of self-distillation methods along the same three axes, and marks which axes each study actually compares and whether it measures retention. App. A backs each row with a short justification, for example noting where a study changes several choices at once so that an axis is not isolated. Even if the framework itself is not new, this table is a helpful reference for readers trying to compare scattered and sometimes conflicting results.

3. Transparent and reproducible. The authors release their code, configurations, and all run results. They also report the limits of their own controlled model openly.

Critical weaknesses

1. Results are expected and partly known. I could guess the main conclusions before reading them. For example, "The rollout source, by contrast, matters mostly on the contradictory tasks: there, teacher rollouts raise acquisition much more than on the ordinary tasks," "The teacher coupling matters on every task," and "SFT still reaches the highest acquisition on most tasks, but pays for it in retention." The last one is even stated as known in the paper's own introduction, which says SFT "is known to erode prior capabilities as it learns." The coupling result is also framed as new, although SDPO [1] (Sec. 4.3 and Tab. 4) already reports the same rise-then-fall pattern. The paper presents frozen versus moving teachers as an open conflict that it resolves, calling them "two sides of one curve" (Sec. 1) and claiming to "reconcile" the literature (Sec. 7). Yet the paper itself lists SDPO in Tab. 1. SDPO already shows this pattern in a single study. In that study, a moderate EMA or trust-region teacher beats a frozen one, while an unregularized teacher diverges. Beyond this pattern, the paper adds a denser sweep, the task- and model-dependence of the best rate, and the retention cost. These additions are useful. Still, they refine a known pattern rather than reveal a new one.

2. The paper makes overstated claims that its own content contradicts. Sec. 3 claims that "any self-distillation methods with privileged context (and even classic SFT) can be formalized through a single objective," with the loss limited to forward or reverse KL and the coupling limited to the EMA rule of Eq. 3. But Tab. 1 lists several methods that fall outside this. SDPO [1], Kim et al. [2], and Guo et al. [3] use the Jensen–Shannon divergence (JSD), and OPSD [4] also tests it; JSD is neither forward nor reverse KL. SDPO's trust-region teacher [1] also cannot be written as an EMA update. Similarly, Sec. 1 states that "the state of the art (Sec. 2) is still largely unaware of this taxonomy," while GKD [5] and Zhao et al. [6], both cited, already cross rollout source with KL direction, and [6] already presents a unified view that includes SFT. Please narrow these claims and position the framework relative to these works.

3. Significant presentation issues.

  • Key terms are not defined consistently. The abstract relies on acquisition and retention but never defines them. The paper then defines acquisition in different ways in different places. Sec. 1 says: "We measure acquisition, the accuracy gained on the new task." Sec. 4 says: "Acquisition is the accuracy gain: the mean task accuracy over the last three saved checkpoints of each run, minus the initial accuracy."

  • Long, complex sentences make the paper hard to follow. For example:

    "Self-distillation with privileged context has emerged as a middle ground: the same model, once conditioned on a demonstration, a verified answer, a document or environment feedback, becomes a dense token-level teacher for its context-free copy."

    "We argue that, chasing an individual champion, the literature fails to see the forest from the trees: we therefore set out to build a taxonomy of methods within the self-distillation family and provide a unifying framework (Sec.3), to then layout a systematic evaluation plan (Sec.4) that we carry out with complementary controlled modeling (Sec. 5) and experimental (Sec. 6) approaches."

  • Informal language does not suit a scientific paper. Examples include "This set of methods now appears under many names," "chasing an individual champion," "the literature fails to see the forest from the trees," and "providing a picture that is less sharp."

  • There are frequent grammar errors. For example: "each axis act on different object," "fix each of these axis," "Three choices in Eq. 2 are left open, that maps to the taxonomical axes," and "to then layout a systematic evaluation plan."

  • The contributions are not clearly stated. Contributions belong in the introduction, but here they are scattered across sections. Sec. 1 says "Our first contribution (Sec. 3) is to show that existing methods [...] can be written as a single objective," while Sec. 3 says "Our first contribution is to introduce a unifying framework." No second or third contribution is ever named.

4. Hyperparameters may be tuned on the evaluation set. Several key results report only the best configuration. Fig. 1 uses "each source at its best coupling α*," and Tab. 8 shows "the run with the highest acquisition over the coupling rates α and the three learning rates." The paper does not describe a separate validation split, so it seems these choices may have been made on the same evaluation set that is then reported. If so, the reported gains could be somewhat optimistic, especially for small sets such as tool-alpaca (97 examples). This may also slightly favor self-distillation in Tab. 8, where each family picks from 15 runs while SFT picks from 7. The Wilcoxon tests over all pairs would not be affected, so the main trends likely hold. If tuning was done on the evaluation set, results on a separate split would help; for example, the unused "2,000 examples in a separate test split" of the spatial tasks (App. B.3).


References

[1] Reinforcement Learning via Self-Distillation (Hübotter et al., ICML 2026)

[2] Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs? (Kim et al., arXiv 2026)

[3] When Should the Teacher Move? Temporal Coupling and Stability in Self On-Policy Distillation (Guo et al., arXiv 2026)

[4] Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models (Zhao et al., arXiv 2026)

[5] On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (Agarwal et al., ICLR 2024)

[6] Decoupling KL and Trajectories: A Unified Perspective for SFT, DAgger, Offline RL, and OPD in LLM Distillation (Zhao et al., arXiv 2026)

Questions for authors

Could the authors clarify how the best settings were selected?

3 comments
AuthorsOpenReview ↗

Response to Reviewer fPHz (1/3)

Thank you for the precise reading of our work. We have narrowed the two claims you pointed to, credited SDPO, GKD and Decoupling KL where they apply, revised the sentences you quote, and state how each setting was selected. All changes are marked in the revised PDF.

W1. (Results are expected). "The rollout source, by contrast, matters mostly on the contradictory tasks: there, teacher rollouts raise acquisition much more than on the ordinary tasks"

This result may seem intuitive once stated, but prior work suggested otherwise: student rollouts are the default choice in Tab. 1 and are credited with retention (Chen et al.), and Skill-SD reports that teacher rollouts collapse during training. Here, we find that teacher rollouts add a median +8.0 pp (Qwen) and +8.3 pp (Ministral) of acquisition on contradictory tasks, against +0.5 and +2.2 pp on the ordinary ones, and that retention changes by one point at most (Tab. 5). The one exception is Qwen chemistry, where student rollouts acquire 2.9 pp more.

W1. (Results are expected). "The teacher coupling matters on every task"

We now agree that this sentence alone is poor. The result is instead the next one: the rate at which acquisition peaks depends on the task and the model. On Qwen, at the same rate α=0.10\alpha = 0.10, a moving teacher loses 18.8 pp to a frozen one on tool-alpaca but gains 21.9 pp on spatial (Tab. 6).

W1. (Results are expected). "SFT still reaches the highest acquisition on most tasks, but pays for it in retention". The last one is even stated as known in the paper's own introduction, which says SFT "is known to erode prior capabilities as it learns

As you note, SFT's retention cost is known, and Sec. 1 cites it as such. What our comparison adds is the observation that "SFT still reaches the highest acquisition on most tasks". By contrast, SDFT reports that it "consistently outperforms SFT", with "higher new-task accuracy". In our grid, we observe that self-distillation generally outperforms SFT when considering the trade-off between acquisition and retention (17 of 20 settings on Qwen and 14 of 20 on Ministral are above the SFT front, Tab. 8), but not on acquisition alone, where SFT reaches the highest value except for Qwen on tool-alpaca and chemistry.

W1. Results are partly known. The coupling result is also framed as new, although SDPO [1] (Sec. 4.3 and Tab. 4) already reports the same rise-then-fall pattern. The paper presents frozen versus moving teachers as an open conflict that it resolves, calling them "two sides of one curve" (Sec. 1) and claiming to "reconcile" the literature (Sec. 7). Yet the paper itself lists SDPO in Tab. 1. SDPO already shows this pattern in a single study. In that study, a moderate EMA or trust-region teacher beats a frozen one, while an unregularized teacher diverges. Beyond this pattern, the paper adds a denser sweep, the task- and model-dependence of the best rate, and the retention cost. These additions are useful. Still, they refine a known pattern rather than reveal a new one.

SDPO indeed already shows the rise and fall: slow-moving teachers beat a frozen one, and an unregularized one diverges. We now credit it with this pattern in Sec. 2, and cite it in Secs. 1 and 6.2. What remains open is why a moving teacher helps in some studies and hurts in others. SDPO compares these teachers on one coding task, where the slow ones (rate 0.01) gain under two points over frozen. Kim et al. find a teacher moving at rate 0.05 below a frozen one on math reasoning. Our grid shows that the peak moves with the task and the model, so that one rate can help on one task and hurt on another. On Qwen at α=0.10\alpha = 0.10, a moving teacher loses 18.8 pp to a frozen one on tool-alpaca but gains 21.9 pp on spatial (medians over all rollout sources, KL directions and learning rates, Tab. 6). The same rate can thus fall on either side of the peak, which is the sense in which we help reconcile these reports. The controlled model suggests what moves the peak: the more the context alone helps the initial model, the slower the best rate (Sec. 6.2).

AuthorsOpenReview ↗

Response to Reviewer fPHz (2/3)

W2. (The paper makes overstated claims that its own content contradicts). Sec. 3 claims that "any self-distillation methods with privileged context (and even classic SFT) can be formalized through a single objective," with the loss limited to forward or reverse KL and the coupling limited to the EMA rule of Eq. 3. But Tab. 1 lists several methods that fall outside this. SDPO [1], Kim et al. [2], and Guo et al. [3] use the Jensen-Shannon divergence (JSD), and OPSD [4] also tests it; JSD is neither forward nor reverse KL. SDPO's trust-region teacher [1] also cannot be written as an EMA update.

You are right about the text of Sec. 3: it claimed to cover any method, but then wrote the divergence as a KL divergence and the teacher update as an EMA, which left out JSD and SDPO's trust-region teacher. The objective itself (Eq. 2) does not need these restrictions, and Sec. 3 now separates it from our experimental grid. Eq. 2 leaves three choices open: the rollout distribution μi\mu_i, the teacher's target qq, and the divergence dd. JSD is a choice of dd, a synchronized teacher is α=1\alpha = 1 in Eq. 3, and SDPO's trust-region teacher, an interpolation between the current and initial teachers, is a choice of qq. Eq. 2 thus covers the distillation term of every method in Tab. 1. Our grid tests a subset: forward and reverse KL, the two ends of the generalized JSD family, and EMA rates up to the point where acquisition collapses (Fig. 2a).

W2. (The paper makes overstated claims that its own content contradicts). Similarly, Sec. 1 states that "the state of the art (Sec. 2) is still largely unaware of this taxonomy," while GKD [5] and Zhao et al. [6], both cited, already cross rollout source with KL direction, and [6] already presents a unified view that includes SFT. Please narrow these claims and position the framework relative to these works.

We agree that "largely unaware" was too strong: GKD and Decoupling KL already separate the rollout source from the KL direction, and Decoupling KL places SFT in the same view. We now credit both in Sec. 1, and Decoupling KL's unified view in Sec. 3. Both use a fixed external teacher. In self-distillation, the teacher is a copy of the student, so how closely it follows the student becomes a third axis, and this axis moves both the teacher's targets and, with teacher rollouts, the trajectories themselves. As Tab. 1 shows, ours is the first self-distillation study to vary the three axes together. The sentence of Sec. 1 now reads: "In self-distillation, joint analysis of these axes remains limited: no study yet varies the three axes together, and most fix each of them, often implicitly (Sec. 2)."

W3. Significant presentation issues: A. Key terms are not defined consistently. B. Long, complex sentences make the paper hard to follow. C. Informal language does not suit a scientific paper. D. The contributions are not clearly stated.

We revised the text on each of these points. The abstract now defines acquisition and retention, the two sentences you quote are split, the informal expressions are replaced, the grammar errors are corrected, and Sec. 1 states the three contributions in one sentence. All changes are marked in the revised PDF.

AuthorsOpenReview ↗

Response to Reviewer fPHz (3/3)

W4. Hyperparameters may be tuned on the evaluation set. Several key results report only the best configuration. Fig. 1 uses "each source at its best coupling α∗\alpha^*," and Tab. 8 shows "the run with the highest acquisition over the coupling rates α\alpha and the three learning rates." The paper does not describe a separate validation split, so it seems these choices may have been made on the same evaluation set that is then reported. If so, the reported gains could be somewhat optimistic, especially for small sets such as tool-alpaca (97 examples). This may also slightly favor self-distillation in Tab. 8, where each family picks from 15 runs while SFT picks from 7. The Wilcoxon tests over all pairs would not be affected, so the main trends likely hold. If tuning was done on the evaluation set, results on a separate split would help; for example, the unused "2,000 examples in a separate test split" of the spatial tasks (App. B.3). Q1. Could the authors clarify how the best settings were selected?

You raised three concerns: (1) the configurations we highlight may have been selected on the same set on which we report them; (2) this could inflate the reported gains, especially on small sets such as tool-alpaca; and (3) in Tab. 8, self-distillation seems to pick from 15 runs and SFT from 7, which could favor self-distillation. You suggested a check on an unused test split. We answer each point.

(1) Where selection enters: First, the trends we report are established by the Wilcoxon tests (Tabs. 5-7), which compare all runs and use no selection, as you noted. Selection enters only where a figure or table displays a single configuration (Fig. 1, Fig. 3a, Tab. 8): it is shown at its best coupling rate among the five, and SFT at its best learning rate among seven. Epochs and learning rate follow fixed rules, stated in Sec. 4 and in the figure captions. The selection is made on the held-out set of each task, which is never used for training. The paper did not say this clearly. We now call this set the validation set and state the rules in Sec. 4, and we added a separate test split to check the selected results on unseen data (point 2).

(2) Our selection is fair and holds on unseen data: Our figures compare two settings (e.g. teacher vs student rollouts in Fig. 1). Each of the two is shown at its own best coupling rate, chosen among the same five rates, so both get the same number of tries. Choosing the best can make both values a little too high, but it gives no advantage to one of them. We also ran the check you suggested (new App. D.5). We kept the five configurations of the math-contradiction panel of Fig. 3a as selected on validation, trained them again with two new seeds, and evaluated these runs on an unused test split of 1,272 examples. Test acquisition is 0.1 to 0.9 points higher than validation acquisition for all five, and never lower. The new seeds reproduce the two large effects of Sec. 6.3 (+43.3 points from tuning the coupling, against +42.0 in the paper; +4.5 from switching to forward KL, against +5.8). The smallest effect (forward against reverse KL once the coupling is tuned) is not reproduced: +2.4 in the paper, -0.8 with the new seeds. Both values are within the difference between two seeds of one configuration (up to 2.5 points).

As for the small sets, we agree. On tool-alpaca (97 examples), differences of a few points between selected configurations are within sampling noise, and we do not draw conclusions from them. Tool-alpaca and chemistry have no unused split, so we could not run the same check there.

(3) Tab. 8: The caption was wrong, and we corrected it. For each self-distillation setting, the table shows the best of 5 runs: the five coupling rates, at one learning rate. SFT is shown as the best of 7 runs. SFT therefore has more tries than self-distillation. We also added Tab. 9, where self-distillation gets 15 tries (five coupling rates at each of three learning rates). This larger choice does not help self-distillation: the number of settings above the SFT front stays at 17 of 20 on Qwen, and goes from 14 to 11 of 20 on Ministral.

Reviewer_so12OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how self-distillation learns new skills while preserving existing capabilities. It separates three training choices, who generates the responses, how closely the teacher follows the student, and the direction of the KL loss, and tests their combinations across two models and five tasks. Teacher rollouts help most when the new task contradicts pretrained behavior; a moving teacher helps only up to a task-dependent rate; and the effect of KL direction varies by model. Compared with SFT, self-distillation usually retains more prior capability at the skill level it reaches, while SFT often achieves the highest new task performance.

Critical strengths
  • The paper offers a clear three-axis framework and tests the combinations systematically, making previously conflicting findings easier to compare.

  • The results are interesting and nuanced: teacher rollouts help most on contradictory tasks, the best teacher-coupling rate depends on the task, and KL direction behaves differently across models. The SFT comparison makes the acquisition–retention trade-off clear.

Critical weaknesses
  • Generality is limited: the experiments use two models, two seeds per configuration, and only the full reference response as privileged context. The constructed contradictory tasks isolate the effect well, but do not establish that it extends to other settings.

  • The controlled model reproduces the observed trends, but its proposed explanatory factors are not measured in the LLMs. It supports a plausible account of the findings rather than confirming their mechanism.

Questions for authors
  1. What evidence from the LLM runs supports the controlled model’s explanation of the rollout-source and coupling effects? Can the authors relate observable task differences to the factors varied in that model?

  2. Could the authors report acquisition and retention at the same checkpoint and confirm whether the reported trade-offs remain unchanged? Acquisition currently averages the last three checkpoints, while retention uses the final one.

  3. How stable are the smaller, model-dependent KL-direction effects across the two seeds? Showing the effect for each seed would help assess this conclusion.

4 comments
AuthorsOpenReview ↗

Response to Reviewer so12 (1/3)

Thank you for your questions; each of them led to a new analysis in the revised paper. We added a measurement relating the LLM tasks to the controlled model (new App. F.1.1), acquisition at the final checkpoint (new App. D.4), the KL-direction effects for each seed (Tab. 7), and a final-answer hint as privileged context (new App. D.6).

Q1. What evidence from the LLM runs supports the controlled model's explanation of the rollout-source and coupling effects? Can the authors relate observable task differences to the factors varied in that model?

We can relate an observable task difference to each of the two factors. These are observations on the runs: the factors themselves are not measured on the LLMs (see our answer to W2 below).

Rollout source. For this effect, we consider spatial and spatial-contradiction: they share the same inputs and the same form of privileged context, and only the expected output of spatial-contradiction contradicts the pretrained behavior. The table below gives, for the initial models and on both tasks, a. the validation set accuracy without context (student accuracy), b. with context (teacher accuracy), and c. a new measurement, the teacher utility on the student's path (defined below). We also report d. the advantage of teacher rollouts after training, i.e. their acquisition minus that of student rollouts (median over matched pairs, Tab. 5).

The teacher utility on the student's path is defined as follows (details in the new App. F.1.1). For the spatial tasks, a solution is a sequence of moves (such as (+1, 0); a reference solution is shown in App. C.5). We take the solution written in the ordinary convention, the convention the initial student follows. Before each move along this solution, we ask the teacher to predict the next move, and record the probability it gives to the correct one (which is the ordinary one for spatial, and the rotated one for spatial-contradiction). We then average those probabilities over the moves from 64 problems. A value of 1 would mean that the teacher always predicts the correct move from its context, regardless of the moves already written on the student's path.

initial modeltaskstudent accuracyteacher accuracyteacher utility on the student's pathadvantage of teacher rollouts
Qwenspatial26.2%58.0%0.48+2.4 pp
Ministralspatial19.6%65.4%0.40+2.5 pp
Qwenspatial-contradiction1.5%28.2%0.15+4.9 pp
Ministralspatial-contradiction0.8%20.0%0.16+4.4 pp

Two things happen on the contradictory task. First, the student's rollouts almost never contain the new behavior (accuracy near 1%), while the teacher's do (20.0-28.2%).

Second, the teacher utility on the student's path falls from 0.40-0.48 on the ordinary task to 0.15-0.16 on the contradictory one, even though its context has the same form in both tasks. We observe a larger advantage of teacher rollouts (e.g., +4.9 vs +2.4 pp for Qwen) where this utility is lower.

This is the behavior of the controlled model when the initial-policy concentration λ\lambda increases: the teacher utility on the student's path falls (new App. F.1.1), and the advantage of teacher rollouts grows (Fig. 1b). With respect to the ordinary task, the contradictory task thus behaves like a larger λ\lambda.

Coupling. The factor is the context strength κ\kappa, how far the context alone moves the initial model toward the new task. Its closest observable is the acquisition reached with a frozen teacher. On Qwen (Fig. 2a), a frozen teacher reaches about 90% of the best SFT gain on tool-alpaca and chemistry, where it is nearly the best coupling, and 2% on the two spatial tasks (whose best rate is α=0.10\alpha = 0.10). In the model, the best rate α\alpha also increases when κ\kappa decreases (Fig. 2b). On Ministral this relation is less clear (Fig. 5).

The model was built to vary these two factors separately, which the LLM tasks cannot do.

AuthorsOpenReview ↗

Response to Reviewer so12 (2/3)

Q2. Could the authors report acquisition and retention at the same checkpoint and confirm whether the reported trade-offs remain unchanged? Acquisition currently averages the last three checkpoints, while retention uses the final one.

We confirm that the trade-offs remain unchanged when acquisition is measured at the final checkpoint: teacher rollouts still raise acquisition on the contradictory tasks with almost no change in retention, acquisition still rises and then falls with the coupling rate, and the retention cost of the KL direction still depends on the model. The conclusions of Secs. 6.1 to 6.3 remain true. In the three test tables (Tabs. 5-7), no cell changes value by more than 2 pp and no significant effect changes sign. Six cells change significance level (two of the 16 acquisition cells of Tab. 5, one of the 64 of Tab. 6, three of the 22 of Tab. 7), in both directions; only Qwen chemistry on the KL axis (+1.4** to +0.8) and Qwen tool-alpaca at α=0.02\alpha = 0.02 (+2.5* to +2.1) lose significance, both at p = 0.06.

The initial reason for averaging the last three checkpoints was to reduce checkpoint-level variance: at the end of training, acquisition still varies from one saved checkpoint to the next, by 0.6 pp on median (computed over the 1,200 training runs of Tab. 3a). Acquisition was evaluated at every saved checkpoint (about 1 minute 30 seconds per checkpoint), while retention is a five-benchmark evaluation (about 15 minutes per checkpoint) and was only evaluated at the final checkpoint. We now state this motivation in Sec. 4.

We keep the three-checkpoint mean in the paper and added the same-checkpoint comparison as an ablation in App. D.4. The three full tables under the same-checkpoint measure are available on request.

Q3. How stable are the smaller, model-dependent KL-direction effects across the two seeds? Showing the effect for each seed would help assess this conclusion.

The KL-direction effects are stable across the two seeds. The table below repeats Tab. 7 (median difference forward - reverse in pp) and adds in parentheses the value for each of the two seeds taken alone.

Task (n pairs)Qwen acquisitionQwen retentionMinistral acquisitionMinistral retention
tool-alpaca (30)-0.7 (-1.4, -0.3)-0.9*** (-0.8***, -0.8***)-0.8 (-1.7, -1.5)-0.0 (-0.1, +0.1)
chemistry (30)+1.4** (+1.5*, +1.2**)-0.5*** (-0.6***, -0.4***)-0.2 (-0.0, +0.0)+0.3*** (+0.4***, +0.3***)
spatial (30)+0.3 (-0.4, +0.5)-0.3*** (-0.4***, -0.3**)+6.0* (+4.4**, +5.4)+0.1 (+0.2, +0.2)
spatial-contradiction (30)-0.4 (-0.5, -0.2)-0.4*** (-0.4***, -0.4***)+0.4 (+1.2, -0.0)+0.2 (+0.0, +0.2)
math-contradiction (30)+5.0*** (+5.5***, +5.1***)-0.9*** (-0.8***, -0.8***)+2.6 (+0.9, +2.6)+0.6** (+0.2, +0.6*)
ordinary (90)+0.7 (+0.2, +0.6)-0.5*** (-0.6***, -0.5***)+0.3 (+0.6, +0.0)+0.2*** (+0.2**, +0.3***)
contradictory (60)+1.1** (+1.0**, +0.7**)-0.5*** (-0.5***, -0.5***)+1.0 (+1.1*, +0.7)+0.3** (+0.1*, +0.3**)
all tasks (150)+0.8** (+0.5*, +0.6**)-0.5*** (-0.5***, -0.5***)+0.4 (+0.8*, +0.2)+0.2*** (+0.2***, +0.3***)
all tasks, lr 5e-6 (50)+0.8 (+0.7, +0.5*)-0.3*** (-0.3***, -0.3***)-0.2 (-0.4, -0.9)-0.0 (-0.0, +0.0)
all tasks, lr 1e-5 (50)+0.6 (+0.1, +0.5)-0.5*** (-0.6***, -0.5***)+1.1 (+1.3, +1.1)+0.1* (+0.1, +0.3**)
all tasks, lr 2e-5 (50)+1.3 (+1.5, +1.2)-1.0*** (-1.0***, -0.9***)+1.0* (+1.2*, +0.6)+1.1*** (+0.9***, +1.3***)

No significant effect of Tab. 7 changes sign between the two seeds. On retention, where the effect depends on the model, each seed gives the all-tasks value of Tab. 7: -0.5 pp on Qwen, and +0.2 and +0.3 pp on Ministral. The per-seed values are added to Tab. 7.

AuthorsOpenReview ↗

Response to Reviewer so12 (3/3)

W1. Generality is limited: the experiments use two models, two seeds per configuration, and only the full reference response as privileged context. The constructed contradictory tasks isolate the effect well, but do not establish that it extends to other settings.

We extended the evaluation on the privileged context and on the seeds.

Final-answer hint as privileged context. In the main grid, the privileged context is fixed to the full reference response, so that the rollout-source, coupling and KL comparisons all share the same supervision. To test whether the findings depend on this choice, we gave the teacher only the final answer as a hint: the answer letter on chemistry and the base-9 result on math-contradiction, one ordinary and one contradictory task. We repeated the coupling and KL-direction comparisons on Qwen with teacher rollouts (new App. D.6).

On chemistry, both comparisons give the same results as in the main text. The coupling curve keeps its shape: a frozen teacher is already close to the best rate, and the fastest rate falls below the untrained accuracy. With a frozen teacher and at the best coupling rate, switching to forward KL changes acquisition by at most about one point and retention by less than half a point. With less information in the context, acquisition peaks lower than with the full reference.

On math-contradiction, the coupling curve is now flat as well: coupling no longer helps. The hint contains no intermediate calculations, so the teacher has to write them itself. Tab. 11 (new, App. D.6) reports whether the teacher and the student write these calculations in base 9, as the task requires, or in base 10. With the full reference, the share written in base 9 rises with the coupling rate, from 7% with a frozen teacher to 89% at α=0.05\alpha = 0.05. With the hint, it stays below 4%: neither the teacher nor the student writes calculations in base 9, at any coupling rate. The gain from coupling that the main text reports on math-contradiction therefore comes with calculations written in base 9, which appear with the full reference and do not appear with the hint.

Feedback. Experiments with feedback as privileged context are running, and we will add them to App. D.6 before the end of the discussion period. Feedback raises a point that we now state as a limitation in the conclusion. The controlled model represents the effect of the privileged context as a fixed shift of the teacher toward the target behavior. Feedback depends on the current attempt and changes as the student's mistakes change, so the controlled model may not describe this case.

Seeds. We trained two new seeds for the five configurations of the math-contradiction panel of Fig. 3a, which reproduce its conclusions (App. D.5). We also report the KL-direction effects seed by seed (answer to Q3): no significant effect changes sign between seeds. A third seed for some of the main figures is running, and we will add it to the paper before the end of the discussion period.

Models. The experiments cover two models from different families and sizes, and the conclusion states this scope. Adding a model requires the full grid of 600 runs, which we could not run during the rebuttal.

W2. The controlled model reproduces the observed trends, but its proposed explanatory factors are not measured in the LLMs. It supports a plausible account of the findings rather than confirming their mechanism.

We agree. The controlled model reproduces the three trends by varying three quantities: initial-policy concentration, context strength, and feature overlap. We do not measure these quantities on the LLMs. The revised paper says so in the conclusion, and the abstract and Sec. 6 now describe the model as a way to interpret and reproduce the trends. Estimating the three quantities from a model and a task before training is listed as future work.

AuthorsOpenReview ↗

Follow-up experiments

Dear Reviewer so12,

The revised paper now includes the feedback experiments and the third seed announced in our answer, together with experiments on a third model family (Gemma).

Feedback. App. D.6.2 adds feedback as privileged context, on Qwen with student rollouts, since feedback describes the student's response. The feedback is written either by GPT-5 mini, on chemistry, or by the student itself (self-feedback), on chemistry and math-contradiction. As in the main text, acquisition still peaks at one coupling rate and falls beyond it, here at a slower rate (frozen on chemistry and α=0.05\alpha = 0.05 on math-contradiction, against 0.250.25 with the full reference).

Seeds. We trained a third seed for every run of Figs. 1a and 2a, and compare the three-seed averages with the two-seed averages of the main text in the new App. D.5.1. With three seeds, teacher rollouts still win every pair of the contradictory tasks by a wide margin, the retention differences barely change, and the best coupling rate of Fig. 2a is unchanged on all five tasks. The conclusions of Secs. 6.1 and 6.2 therefore hold.

Models. The full grid remained out of reach, but we repeated the comparison of Fig. 3a (chemistry and math-contradiction, teacher rollouts) on a third model family, Gemma-4-12B, with two seeds (new App. D.7). On math-contradiction, Gemma behaves like Qwen: switching to forward KL costs retention, so the coupling remains the axis to tune first. On chemistry, the differences stay within Gemma's much larger variation across seeds.

We hope these additions address your remaining concerns. Thank you for your time reviewing our paper.

Best regards, The authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global answer summary

Dear Area Chair,

Thank you for handling our submission. To make the discussion easier to navigate, we summarize it below. Reviewers recognized the systematic three-axis study (so12, fPHz, AqZN), which makes conflicting findings comparable (so12, AqZN), its sound statistics and released code and runs (fPHz), the map of the literature in Tab. 1 (fPHz), and nuanced, practically useful results, including the comparison with SFT (so12, AqZN). After our answers, reviewer AqZN replied that the concern about the controlled model is resolved and that the paper is "a valuable contribution". The table lists the main concerns and the evidence we added (all changes are marked in the revised PDF):

ConcernReviewersRebuttal evidence
Evidence for the controlled modelso12, AqZNThe correspondence between contradictory tasks and the controlled model is now checked on the LLMs (App. F.1.1), through the teacher's utility on the student's path. It is lower on the contradictory spatial task than on its ordinary version, as in the model. The model is now presented as an interpretation
Other privileged contextsso12, AqZNThree contexts added (App. D.6): the final answer as a hint, external feedback and self-feedback. The best coupling rate also depends on the context, as it does on the task and model (now stated in Sec. 6.2)
Model familiesso12, AqZNGemma-4-12B added in one setting (App. D.7): on a contradictory task, forward KL costs retention on Gemma, as on Qwen (now stated in Sec. 6.3)
Seeds and checkpointsso12, AqZNThird seed for Figs. 1a and 2a (App. D.5.1): the best coupling rate is unchanged on all five tasks. For the KL direction, every significant effect keeps its sign in each seed (Tab. 7). With acquisition measured at the final checkpoint, every significant effect keeps its sign as well (App. D.4).
Selection on the evaluation setfPHzNew runs are evaluated on an unused test split (App. D.5.2). Test acquisition is never lower than validation acquisition (0.1 to 0.9 pp higher), so selecting on the validation set does not overstate it (now stated in Sec. 6.3).
Overstated claims, presentationfPHzThe objective (Eq. 2, unchanged) covers JSD and the trust-region teacher, the two cases raised by the reviewer, and Sec. 3 now states it. The restriction to forward and reverse KL with an EMA applies to our experiments only. Secs. 1 and 2 now state what GKD, Decoupling KL and SDPO already show. Acquisition and retention are defined in the abstract, and the contributions are stated in Sec. 1.

On whether the results are expected (fPHz): prior studies report opposite effects for the same choice, and our grid shows when each one holds. A moving teacher is reported both to help and to hurt; we find that it helps up to a rate that depends on the task, the model and the context, and hurts beyond it. Studies also disagree on which KL direction to use; we find that the retention cost of forward KL depends on the model. Teacher rollouts are reported both to help and to collapse; with the full reference, we find that they help mostly where the new task contradicts the pretrained behavior.

Best regards, The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.