- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper provides an empirical assessment of self-distillation with privileged context. The authors organize the existing methods along three axes: (1) on- or off-policy setup, (2) frozen or moving teacher, and (3) distillation loss (forward vs. reverse KL). All combinations of these three axes are trained on Qwen2.5-7B and Ministral-3-3B, tested on five tasks, two of which are in contradiction with the model's pre-trained behavior. The authors measure the improvements on new tasks (acquisition) and retention on general benchmarks.
There are three main findings: off-policy rollouts improve acquisition mostly on the contradictory tasks, and have small effects on retention; a moving teacher can help up to a certain point and then degrades learning beyond it, which is task and model dependent; the cost of switching the direction of the loss (forward vs. reverse) also depends on the model.
Critical strengths
-
The paper does a good job at reconciling conflicting findings in different papers that cover self-distillation with privileged information. By training every combination across three axes, the results can be compared within the same setup. The corresponding findings seem practically useful. For instance, off-policy rollouts seem to be most useful when the new task contradicts the model's base behavior, without substantially hurting retention.
-
Acquisition and retention are evaluated together and compared against SFT trained on the same demonstrations. The authors find that most self-distillation settings achieve a better trade-off (acquisition vs. retention) than SFT, but SFT still achieves the highest acquisition on most tasks.
Critical weaknesses
-
Conclusions about when each axis matters often rely on just a few cases (e.g., just two contradictory tasks, two models, and one type of privileged context). It remains unclear whether the findings would hold for setups with different privileged information (e.g., feedback or hints).
-
The explanatory role of the controlled model is overstated. The authors state that the model explains the trade-off between acquisition and retention, but the parameters are not measured on the LLMs and are instead selected to match the trends. This only shows that the trends can come from this mechanism, and not that it explains the results.
Questions for authors
How do setups with other types of privileged context behave (e.g., feedback, hints) and do the same findings hold?
4 comments
Response to Reviewer AqZN (2/2)
W2. The explanatory role of the controlled model is overstated. The authors state that the model explains the trade-off between acquisition and retention, but the parameters are not measured on the LLMs and are instead selected to match the trends. This only shows that the trends can come from this mechanism, and not that it explains the results.
We agree. The controlled model reproduces the three trends by varying three quantities: initial-policy concentration, context strength, and feature overlap. We do not measure these quantities on the LLMs. The revised paper says so in the conclusion, and the abstract and Sec. 6 now describe the model as a way to interpret and reproduce the trends. Estimating the three quantities from a model and a task before training is listed as future work.
Until then, we can relate the rollout-source and coupling effects to differences we observe between tasks. These differences show how one task compares with another, without measuring the initial-policy concentration or the context strength themselves.
Rollout source. For this effect, we consider spatial and spatial-contradiction: they share the same inputs and the same form of privileged context, and only the expected output of spatial-contradiction contradicts the pretrained behavior. The table below gives, for the initial models and on both tasks, a. the validation set accuracy without context (student accuracy), b. with context (teacher accuracy), and c. a new measurement, the teacher utility on the student's path (defined below). We also report d. the advantage of teacher rollouts after training, i.e. their acquisition minus that of student rollouts (median over matched pairs, Tab. 5).
The teacher utility on the student's path is defined as follows (details in the new App. F.1.1). For the spatial tasks, a solution is a sequence of moves (such as (+1, 0); a reference solution is shown in App. C.5). We take the solution written in the ordinary convention, the convention the initial student follows. Before each move along this solution, we ask the teacher to predict the next move, and record the probability it gives to the correct one (which is the ordinary one for spatial, and the rotated one for spatial-contradiction). We then average those probabilities over the moves from 64 problems. A value of 1 would mean that the teacher always predicts the correct move from its context, regardless of the moves already written on the student's path.
| initial model | task | student accuracy | teacher accuracy | teacher utility on the student's path | advantage of teacher rollouts |
|---|---|---|---|---|---|
| Qwen | spatial | 26.2% | 58.0% | 0.48 | +2.4 pp |
| Ministral | spatial | 19.6% | 65.4% | 0.40 | +2.5 pp |
| Qwen | spatial-contradiction | 1.5% | 28.2% | 0.15 | +4.9 pp |
| Ministral | spatial-contradiction | 0.8% | 20.0% | 0.16 | +4.4 pp |
Two things happen on the contradictory task. First, the student's rollouts almost never contain the new behavior (accuracy near 1%), while the teacher's do (20.0-28.2%).
Second, the teacher utility on the student's path falls from 0.40-0.48 on the ordinary task to 0.15-0.16 on the contradictory one, even though its context has the same form in both tasks. We observe a larger advantage of teacher rollouts (e.g., +4.9 vs +2.4 pp for Qwen) where this utility is lower.
This is the behavior of the controlled model when the initial-policy concentration increases: the teacher utility on the student's path falls (new App. F.1.1), and the advantage of teacher rollouts grows (Fig. 1b). With respect to the ordinary task, the contradictory task thus behaves like a larger .
Coupling. The quantity is the context strength , how far the context alone moves the initial model toward the new task. Its closest observable is the acquisition reached with a frozen teacher. On Qwen (Fig. 2a), a frozen teacher reaches about 90% of the best SFT gain on tool-alpaca and chemistry, where it is nearly the best coupling, and 2% on the two spatial tasks (whose best rate is ). In the model, the best rate also increases when decreases (Fig. 2b). On Ministral this relation is less clear (Fig. 5).
The model was built to vary these two quantities separately, which the LLM tasks cannot do.
Thank you for the response and for running additional experiments. The new setup with two spatial tasks is really nice. Together with the revisions, it resolves my second concern about the role of the controlled model.
I have two minor reservations. First, the benefit of coupling seems to depend on the type of context, as it no longer helps on math-contradiction with the hint. The other reservation is about the hint experiment, where only the teacher rollouts are used on Qwen, so I'm not sure whether the finding about the rollout source would hold with a different conext.
Overall, I don't see these issues as critical and I think that the paper is a valuable contribution.
Dear Reviewer AqZN,
Thank you for your reply and for your positive assessment. We are glad that the comparison of the two spatial tasks resolves your concern about the controlled model.
Coupling and context. We agree that the benefit of coupling depends on the context. In the main grid, the best coupling rate already depends on the task and the model; the hint and the new feedback runs show that it also depends on the context (see Feedback below). Sec. 6.2 now states this dependence.
Rollout source and context. As you note, the paper compares the two rollout sources with the full reference only. Tab. 11 suggests what to expect with the hint on math-contradiction: the teacher's own responses then contain no calculations in base 9, so its rollouts would carry no more of the new behavior than the student's. Two follow-ups would test this directly, and we leave them to future work: measuring the teacher utility on the student's path (App. F.1.1) under other contexts, which requires extending it beyond the move-by-move solutions of the spatial tasks, and varying the context gradually within one task, from the final answer to the full reference.
Since our first answer, the revised paper also includes the feedback experiments and the third seed we announced, together with experiments on a third model family (Gemma).
Feedback. App. D.6.2 adds feedback as privileged context, on Qwen with student rollouts, since feedback describes the student's response. The feedback is written either by GPT-5 mini, on chemistry, or by the student itself (self-feedback), on chemistry and math-contradiction. As in the main text, acquisition still peaks at one coupling rate and falls beyond it, here at a slower rate (frozen on chemistry and on math-contradiction, against with the full reference).
Seeds. We trained a third seed for every run of Figs. 1a and 2a, and compare the three-seed averages with the two-seed averages of the main text in the new App. D.5.1. With three seeds, teacher rollouts still win every pair of the contradictory tasks by a wide margin, the retention differences barely change, and the best coupling rate of Fig. 2a is unchanged on all five tasks. The conclusions of Secs. 6.1 and 6.2 therefore hold.
Models. The full grid remained out of reach, but we repeated the comparison of Fig. 3a (chemistry and math-contradiction, teacher rollouts) on a third model family, Gemma-4-12B, with two seeds (new App. D.7). On math-contradiction, Gemma behaves like Qwen: switching to forward KL costs retention, so the coupling remains the axis to tune first. On chemistry, the differences stay within Gemma's much larger variation across seeds.
Thank you again for your constructive comments and your time reviewing our paper.
Best regards, The authors
Response to Reviewer AqZN (1/2)
Thank you for raising the role of privileged context and the interpretation of the controlled model. We added runs with a hint as privileged context (new App. D.6) and a measurement relating the LLM tasks to the controlled model (new App. F.1.1), and we revised how the paper presents the controlled model.
We extended the evaluation on the privileged context and on the seeds.
Final-answer hint as privileged context. In the main grid, the privileged context is fixed to the full reference response, so that the rollout-source, coupling and KL comparisons all share the same supervision. To test whether the findings depend on this choice, we gave the teacher only the final answer as a hint: the answer letter on chemistry and the base-9 result on math-contradiction, one ordinary and one contradictory task. We repeated the coupling and KL-direction comparisons on Qwen with teacher rollouts (new App. D.6).
On chemistry, both comparisons give the same results as in the main text. The coupling curve keeps its shape: a frozen teacher is already close to the best rate, and the fastest rate falls below the untrained accuracy. With a frozen teacher and at the best coupling rate, switching to forward KL changes acquisition by at most about one point and retention by less than half a point. With less information in the context, acquisition peaks lower than with the full reference.
On math-contradiction, the coupling curve is now flat as well: coupling no longer helps. The hint contains no intermediate calculations, so the teacher has to write them itself. Tab. 11 (new, App. D.6) reports whether the teacher and the student write these calculations in base 9, as the task requires, or in base 10. With the full reference, the share written in base 9 rises with the coupling rate, from 7% with a frozen teacher to 89% at α=0.05. With the hint, it stays below 4%: neither the teacher nor the student writes calculations in base 9, at any coupling rate. The gain from coupling that the main text reports on math-contradiction therefore comes with calculations written in base 9, which appear with the full reference and do not appear with the hint.
Feedback. Experiments with feedback as privileged context are running, and we will add them to App. D.6 before the end of the discussion period. Feedback raises a point that we now state as a limitation in the conclusion. The controlled model represents the effect of the privileged context as a fixed shift of the teacher toward the target behavior. Feedback depends on the current attempt and changes as the student's mistakes change, so the controlled model may not describe this case.
Seeds. We trained two new seeds for the five configurations of the math-contradiction panel of Fig. 3a, which reproduce its conclusions (App. D.5). Tab. 7 now reports the KL-direction effects seed by seed: no significant effect changes sign between seeds. A third seed for some of the main figures is running, and we will add it to the paper before the end of the discussion period.
Models. The experiments cover two models from different families and sizes, and the conclusion states this scope. Adding a model requires the full grid of 600 runs, which we could not run during the rebuttal.