acceptodds
Under review as a conference paper at ICLR 2027

What Does Privileged Information Add to On-Policy Self-Distillation?

Abstract

On-policy self-distillation (OPSD) lets a language model learn from a frozen copy of itself that sees an answer or a worked solution. Giving the teacher this extra information seems to offer the student more to learn, but how much does it add beyond distillation itself? To isolate that contribution, we construct AMPLE-Math, a reusable suite of 5,319 mathematical problems with six reasoning views that share the same answer, and compare each view with matched reference-free distillation. With a thinking-enabled teacher supervising direct-response rollouts, reference-free distillation accounts for much of Qwen3-1.7B's improvement under thinking-enabled evaluation, both in domain and on external benchmarks. Evidence for an additional reference benefit is modest in Qwen, strongest for a polished solution, whereas complete traces add two percentage points in SmolLM3-3B at step 50. Yet these benefits depend on the student being trained. At the same checkpoint, replacing short direct-response rollouts with long thinking-enabled rollouts turns gains into losses in both families while the problems, references, and evaluation stay fixed. Teacher profiles and matched loss interventions in Qwen further show that changing token-level supervision need not change student behavior. Together, these findings suggest that OPSD can improve access to existing reasoning capabilities through parameters shared by direct-response and thinking-enabled inference. The value of a privileged reference is what it adds to this cross-mode transfer, not how much of the solution it reveals.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy self-distillation, privileged information, knowledge distillation, LLM reasoning, LLM post-training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NLadOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies what privileged references add beyond reference-free on-policy self-distillation. Using AMPLE-MATH’s six answer-matched views, experiments in two model families show that reference benefits vary with the model and checkpoint, while changing student training trajectories can reverse transfer.

Critical strengths

The reference-free control and six answer-matched views provide a focused comparison of what privileged references add beyond distillation itself. Comparisons across model families and matched checkpoints substantiate how reference contributions vary, offering useful empirical guidance for evaluating privileged supervision.

Critical weaknesses

The direct-response versus thinking-enabled comparison changes reasoning mode, rollout length, and supervision coverage simultaneously. Although the loss-window interventions address part of this confounding, they do not fully disentangle these factors, limiting the mechanistic interpretation of the observed transfer reversal.

Questions for authors

To what extent do the existing controls separate the effects of reasoning mode, rollout length, and supervision coverage on the observed transfer reversal?

5 comments
AuthorsOpenReview ↗

Response one (1/2)

Thank you for the focused question. It is the right one to ask of Section 3.2. The existing design and two new runs show that rollout length beyond the loss window cannot change the update, that no supervision coverage we tested removes the reversal, and that mode matching alone does not explain it.

FactorControlResult
Rollout lengthThe loss covers the first 1,024 tokens in both configurationsLength beyond the window changes compute, not the update
Supervision coverageWider windows, and the whole response (new)None removes the reversal
Teacher–student mode matchA teacher with thinking disabled scoring direct-response rollouts (new)These students gain, while mode-matched thinking-enabled rollouts lose

The paper studies two model families, Qwen3-1.7B and SmolLM3-3B. Unless stated otherwise, results here are for Qwen3-1.7B, evaluated with thinking enabled under the paper's protocol (Appendix A.2).

Rollout length. The loss covers the first 1,024 completion tokens in both configurations (Eq. 1, App. A.1). Tokens past that window never enter the loss, and the teacher's score at each position depends only on the prefix. So for a fixed supervised prefix and loss mask, rollout length beyond the window changes compute, not the update. The supervised span is also nearly matched.

  • Direct-response rollouts reach the cap in 61–69% of cases early in training and 76–86% late, with about 0.93K supervised tokens on average.
  • Thinking-enabled rollouts fill the window almost always.

Supervision coverage. Within thinking-enabled rollouts we varied how much is supervised and where (Full Trace, accuracy in %, base 80.21%).

Rollouts, loss onStep 25Step 50Step 100
Thinking-enabled, first 1,024 tokens (paper)78.2675.1372.79
Thinking-enabled, first 4,096 tokens79.2374.2871.22
Thinking-enabled, four 256-token windows across the reasoning79.2375.6571.48
Thinking-enabled, the whole response (new)78.6575.3369.99
Direct response, the whole rollout (paper default)82.5582.4282.62

No window removes the penalty. Every thinking-rollout variant scores below the base at every step, while direct-response training stays above it, and none is resolved above the paper's window. The whole-response loss contains the final answer in 87.0% of training rollouts, against 0.1% for the paper's window, yet it is lower at step 100 (−2.80 [−5.27, −0.33] before Holm correction). So the reversal persists under every window we tested, and extending supervision to the final answer is not enough to remove it in these runs.

Reasoning mode. Two factors remain tied to the student's mode, what the supervised tokens are and whether teacher and student share a mode. The new control addresses the second. Its teacher has thinking disabled, sees the reference and scores direct-response rollouts, so teacher and student share a mode. The table gives each student's gain over the base at step 100, with three seeds per cell in the direct-response rows, except four for the paper's Full Trace.

Student rollouts, teacherSame modeClean SolutionFull Trace
Direct response, teacher thinks (paper)No+3.10 [1.54, 4.73]+2.38 [0.91, 3.86]
Direct response, teacher does not think (new)Yes+1.30 [−0.30, 3.02]+2.99 [1.43, 4.67]
Thinking enabled, teacher thinks (Section 3.2)Yes−2.93 [−5.01, −0.78]−7.42 [−9.77, −5.14]

With teacher and student in the same mode, Full Trace gains +2.99 with direct-response rollouts and loses 7.42 with thinking-enabled rollouts. So mode matching alone does not explain the reversal. Nor does the second pass. On the external benchmarks, evaluated in a single pass, thinking-rollout Full Trace trails direct-response Full Trace by 9.44 points [5.83, 13.24] (Table 13). The reversal also holds in SmolLM3-3B. The Full Trace trained with thinking-enabled rollouts scores 77.99% at step 50, against 84.83% for direct-response training and 82.16% for the base (two seeds), and an Answer Only student trained this way falls to 36.13% by step 100.

What remains is the student's mode itself, that is, what the supervised tokens are, which still changes together with the teacher's mode. One measurement isolates the supervised tokens. Table 16 holds the teacher in thinking mode and varies only the prefix type, and for Full Trace the teacher's shifts align with correctness on direct-response prefixes and reverse on thinking-enabled prefixes (+0.0131 vs −0.0123 nats per token). The trained students' behavior fits this. After a wrong first candidate answer in the thinking, the thinking-rollout Full Trace and Clean Solution students end correct in 13–14% of cases, against 29% for the base.

We will revise Section 3.2 to state which factors each control isolates and which remain coupled, with the summary table above, and make the section easier to follow.

AuthorsOpenReview ↗

Response one (2/2)

Other new results. These strengthen the paper's main comparisons, each with three training seeds per student.

  • The shared-parameter account is tested directly. The same-mode students above are trained with thinking disabled for both student and teacher, yet their thinking-enabled accuracy rises (+2.15 [0.63, 3.75], pooled).
  • A non-OPSD baseline. LoRA SFT on the model's own correct answers, with matched steps and about 1.2 times OPSD's supervised tokens, gains nothing resolved. Reference-free OPSD exceeds it by +5.09 [2.99, 7.38] on AIME 2024/2025 and HMMT 2025.
  • Reference content, three seeds. Genuine references beat same-view references from another problem by +1.28 [0.13, 2.47] for Key Points and +1.22 [0.26, 2.19] for Clean Solution (both survive Holm correction).
  • Evaluation protocol. The paper's second pass regenerates capped responses from the beginning (Appendix A.2). We found that this gives models that are capped more often what is in effect an extra attempt. In a single pass of up to 32,512 tokens, Clean Solution's advantage over reference-free training is +2.41 [1.32, 3.56]. The revised paper will add the single-pass results.

Thank you again. We hope the controls above answer your question as far as the current runs allow, and that the new results show the paper's comparisons are sound. If so, we would be grateful if you would consider raising your score.

AuthorsOpenReview ↗

Response two

Thank you again for your question about which factors the controls separate. Before the discussion closes, we would like to add one result on the factor our reply left coupled, and one correction.

Supervised tokens. Our reply noted that the student's mode still changes together with what the supervised tokens are. To separate these tokens from the teacher's signal, we fine-tuned the model on its own correct thinking-enabled traces, with the same optimizer, learning rate and number of steps (three seeds). Training on thinking tokens without the privileged teacher does not reduce accuracy (−0.59 [−2.15, 0.98] compared with the base, two-pass protocol). In contrast, the OPSD students trained on thinking-enabled rollouts lose 2.93 to 7.42 points (one seed). Therefore, the reversal is tied to the teacher's signal on thinking-enabled prefixes, rather than to training on thinking tokens alone.

Correction. We rebuilt the other-problem controls using references from training problems only, since the earlier ones could come from any data split. Genuine Clean Solution still outperforms its other-problem control (+1.43 [0.41, 2.47], Holm p = 0.013, two-pass protocol). For Key Points, the difference has the same sign (+1.06 [−0.09, 2.24]), but its interval now includes zero.

In the revised paper, Section 3.2 will state which factors each control separates, with the summary table in the appendix. If our replies address your question, we would be grateful if you would consider updating your score.

Reviewer_7gfNOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper aims to understand how much the privileged information that the teacher has access to contributes in the on-policy self-distillation (OPSD) training. They construct 6 different variants (called "views") of the additional information such as reasoning trace, summary, gist, etc. the teacher can be conditioned on with respect to the same problem. Their first contribution was to construct a dataset called "Ample-Math" that has math problems, with 6 different types of reasoning views associated with the answer. They conducted OPSD experiments where a copy of the model in thinking mode is initialized as the teacher and another copy in non-thinking mode as the student. After training, they evaluate the "thinking-enabled" mode of the trained students, and their results show that additional references do not contribute much to model training, and most of the gains come from the fact that a reasoning-enabled teacher is teaching a non-reasoning model. Subsequently, they show that different student responses get different supervision from the teacher and the OPSD gains vary. Finally, they show that while references do not add anything additional compared to reference-free training, using wrong references will hurt the student's performance.

Critical strengths
  1. The paper aims to investigate an interesting question of what exactly references are contributing to OPSD training. If we can narrow this down, it could be an important contribution to OPSD training and inform the training setup design.
Critical weaknesses
  1. There is a major confounding factor in their experiments, which is that they train the student in the non-thinking mode and then evaluate in thinking mode. Their assumption is that since the same parameters are shared, OPSD will enable stronger thinking-enabled reasoning. The authors do not provide any justification or results showing why this is a reasonable assumption. Their results are also questionable because of this. After OPSD training, the trained models in thinking mode are all in the same statistical band compared to the base model. This might be due to the fact that their assumption about enabling stronger reasoning is wrong.

  2. In experiments where the teacher and student models are in reasoning modes, the trained models perform worse than the base model. The authors should address why this is happening. The fact that providing the teacher model with the correct answer worsens students' performance seems very counterintuitive. The authors should have a deeper analysis of the teacher's performance given the different views and try to isolate why the student's perform worse after training.

  3. The authors conclude that the references do not provide any significant improvements in OPSD training. Despite the confounding factor in their evaluation, this would only be true in settings where the same model acts as the teacher in thinking mode and student in no think mode. For other cases, where the model is in the same mode but teacher is conditioned on additional information such as environment feedback or in-context examples, it is known that additional "privileged" information helps in OPSD training [1,2], The authors should narrow their claims in light of this evidence.

  4. Related works section needs to improve. The authors should connect how, given the existing work, this work is different and emphasize what problem they are trying to solve that has been overlooked by the prior work.

References -

  1. Hübotter, Jonas, et al. "Reinforcement learning via self-distillation." arXiv preprint arXiv:2601.20802 (2026).
  2. Shenfeld, Idan, et al. "Self-distillation enables continual learning." arXiv preprint arXiv:2601.19897 (2026).
Questions for authors
  1. Why did the authors choose to only supervise the first 1k tokens and not provide feedback on the entire student-generated response?
  2. What was the performance of the trained models when evaluated in non-reasoning mode, and did any references help with knowledge transfer? Since the students are trained to mimic the distribution of a thinking teacher, the trained models also become reasoning models in no-think mode.
7 comments
AuthorsOpenReview ↗

Response one (1/4)

Thank you for the detailed review. We answer your points in order. The main results are these.

  • The configuration is OPSD's default, but not the only one we test. It is held fixed in every reference comparison, so it does not confound the reference's added value, which is what the paper measures. The paper also varies the student's rollouts (Section 3.2), and we have now added a teacher with thinking disabled.
  • We tested the shared-parameter assumption directly. With neither the student's rollouts nor the teacher's scoring in thinking mode, thinking-enabled accuracy still improves. In this same-mode setting, closer to SDPO and SDFT, the reference improves thinking-enabled accuracy, in line with the work you cite.
  • References do add value, selectively. A polished solution adds +2.41 [1.32, 3.56] over matched reference-free training in a single pass, and genuine references beat another problem's references with three seeds.

Terms. The paper studies two model families, Qwen3-1.7B and SmolLM3-3B. Unless stated otherwise, results here are for Qwen3-1.7B, and the trained students are evaluated with thinking enabled.

  • Two-pass protocol. This is the paper's in-domain evaluation (Appendix A.2). Each response may use up to 16,384 tokens. If it reaches this limit, we discard it and generate a new response from the start with up to 32,512 tokens.
  • Single pass. One response of up to 32,512 tokens, graded as is. Point 3 explains why we added it.
  • 16K. Grading only the first 16,384 tokens.
  • External benchmarks. AIME 2024/2025 and HMMT 2025, reported as Avg@12, with each response generated in a single pass of up to 38,912 tokens, as in the released OPSD evaluation (Zhao et al., 2026).

1. Why this configuration (W1). This is OPSD's default, so the question we ask is the one practitioners face. The original method (Zhao et al., 2026) uses student_thinking=False, teacher_thinking=True, 1,024-token rollouts and thinking-enabled evaluation at 38,912 tokens, and at least ten later OPSD papers use this mode configuration as their default, for example Li et al. (2026), Kara and Ersoy (2026), Lin et al. (2026), Tian et al. (2026), Wu et al. (2026) and Zheng and Dou (2026). With the released code and data we reproduce the published gain, +5.56 [2.13, 9.26] (Table 8 bootstraps by benchmark and prints [2.22, 9.17]).

OPSD results are usually reported as gains over the base model, which mix self-distillation itself with the reference the teacher sees. Anyone deciding what to show the teacher needs the second, and this paper measures it. The configuration does not confound this measurement, because privileged and reference-free students share the teacher's mode, rollouts, recipe, checkpoint and evaluation. Only the reference differs. This paired design addresses the attribution concern that Ichihara et al. (2026) raise about the mode asymmetry.

2. The shared-parameter assumption, tested (W1). The paper offered this as an interpretation, and we have now tested it directly. The two modes share one set of weights, selected by a chat-template flag (Yang et al., 2025). We trained students whose teacher has thinking disabled and sees the reference (three seeds each for Clean Solution and Full Trace), so neither the rollouts nor the teacher's scoring use thinking mode. Without a reference this teacher would equal the student and leave it at the base (we checked that the loss and its gradient are exactly zero). With the paper's runs, this crosses the teacher's thinking with the reference (gains over the base, two-pass protocol).

Teacher's referenceTeacher thinks (paper)Teacher does not think (new)
None+1.80 [0.20, 3.47]0 (the base)
Clean Solution+3.10 [1.54, 4.73]+1.30 [−0.30, 3.02]
Full Trace+2.38 [0.91, 3.86]+2.99 [1.43, 4.67]
  • The reference without the teacher's thinking improves the student by +2.15 [0.63, 3.75], pooled over both references. Answer Only, which contains no reasoning trace, also gains +3.06 [1.04, 5.14].
  • Under direct-response evaluation the same students fall below the base (−4.62 [−7.13, −2.14]), so the gain is specific to thinking-enabled inference.
  • The teacher's thinking gives +1.80 [0.20, 3.47] without a reference, and adds +1.80 [0.67, 2.95] to the same Clean Solution reference.

This supports the assumption. Such training improves thinking-enabled accuracy through the weights the two modes share, and a thinking-enabled teacher is not needed for it. The teacher's thinking and a reference are two ways to make the teacher differ from the student, and each can improve thinking-enabled accuracy on its own. We will revise the abstract accordingly and add the table to the main text.

AuthorsOpenReview ↗

Response one (2/4)

3. The statistical claim (W1). Under the paper's two-pass protocol, every Qwen condition in Table 1 has a paired interval for its gain over the base above zero. The main comparisons use three or four training seeds per student, and every interval is a paired 95% problem-cluster bootstrap over the same test problems, with per-problem results averaged over seeds. Two quantities should be kept apart.

  • Gains over the base.
    • In domain, every Qwen view and both controls gain between +1.67 and +3.10 (Table 1).
    • On the external benchmarks, the reference-free student gains +4.07 [1.73, 6.54], and Answer Only, Clean Solution and the wrong-answer student also have intervals above zero. Full Trace does not (+2.29 [−0.16, 4.88]).
    • A matched SFT baseline, trained on the model's own correct samples with thinking disabled as in OPSD training, gains nothing resolved, although it receives about 1.2 times as many supervised tokens. Reference-free OPSD exceeds it by +1.76 [0.50, 3.04] under the two-pass protocol and by +5.09 [2.99, 7.38] externally. In a single pass in domain the two are not separated.
  • The reference's added contribution, which is what the paper measures. Clean Solution adds +2.41 [1.32, 3.56] over reference-free training in a single pass and +1.30 [0.20, 2.41] under the two-pass protocol. Genuine references also beat another problem's references (point 5).

A protocol check. Appendix A.2 regenerates capped responses from the beginning with the same sampling seed. We found that reusing the seed does not reproduce the first response (they depart at a median of token 149), so a capped response gets a second, independent attempt, and students, which are capped more often (19.9–27.9% against 16.6%), benefit more. Even graded at the first 16K tokens of the new response, the second pass raises the base's accuracy by +3.71 [2.60, 4.82]. We therefore evaluated the same checkpoints in a single pass.

DifferenceSingle passTwo-pass protocol, same seeds
Reference-free − base+0.54 [−0.69, 1.74]+1.80 [0.20, 3.47]
Clean Solution − base+2.95 [1.76, 4.17]+3.10 [1.54, 4.73]
Full Trace − base+1.87 [0.67, 3.06]+2.76 [1.30, 4.28]
Same-mode Full Trace − base+1.26 [0.15, 2.37]+2.99 [1.43, 4.67]
Clean Solution − reference-free+2.41 [1.32, 3.56]+1.30 [0.20, 2.41]

Every row uses three training seeds. In a single pass the base was evaluated three times with different sampling seeds (78.9%, 80.3% and 79.6%), and we compare against the mean. Table 1 averages Full Trace over four seeds (+2.38 [0.91, 3.86]). Students trained with a reference keep clear gains without regeneration, while the reference-free gain is not resolved. We will clarify A.2, and Table 1 will add the single-pass results.

Budget. The gains depend on the budget. Our reproduction of the original recipe gains +1.48 [−1.11, 4.26] over the base at 16K and +5.56 [2.13, 9.26] at the official 38,912 tokens, a paired increase of +4.07 [0.83, 7.78] within the same responses. In domain at 16K, no student in Table 1 has a resolved gain. The revision will add a Budget section to the main text.

AuthorsOpenReview ↗

Response one (3/4)

4. Why thinking-enabled training rollouts hurt (W2). Sharing a mode does not by itself cause the loss. The new control adds a second same-mode configuration, and the two behave in opposite ways. The table gives each student's gain over the base at step 100 (two-pass protocol), with three seeds per cell in the direct-response rows, except four for the paper's Full Trace.

Student rollouts, teacherSame modeClean SolutionFull Trace
Direct response, teacher thinks (paper)No+3.10 [1.54, 4.73]+2.38 [0.91, 3.86]
Direct response, teacher does not think (new)Yes+1.30 [−0.30, 3.02]+2.99 [1.43, 4.67]
Thinking enabled, teacher thinks (Section 3.2)Yes−2.93 [−5.01, −0.78]−7.42 [−9.77, −5.14]

In the same mode, Full Trace gains +2.99 with direct-response rollouts and loses 7.42 with thinking-enabled rollouts, so mode matching alone does not explain the loss, although the teacher's mode also changes between the two rows. Nor does the second pass. On the external benchmarks, evaluated in a single pass, thinking-rollout Full Trace trails direct-response Full Trace by 9.44 points [5.83, 13.24] (Table 13).

As you asked, we analyzed the teacher. With any view it answers 97.1–98.8% of the profiling problems correctly, against 81.2% without one, so the loss occurs despite high teacher accuracy. High accuracy does not guarantee useful token-level guidance on the student's own prefixes, as Kaur et al. (2026) also report for thinking models. For Full Trace, the teacher–student log-probability shifts align with correctness on direct-response prefixes and reverse on thinking-enabled prefixes (+0.0131 vs −0.0123, Table 16). The trained students also stop thinking a median of about 220–240 tokens after a wrong first candidate answer, against about 2,100 for the base, and end correct in 13–14% of such cases against 29%. Alignment is not the whole account, since Answer Only keeps positive alignment and still loses. Wider supervision does not remove the penalty either (Q1).

5. Scope (W3). The paper does not conclude that references add nothing. Measured against matched reference-free training, they add value selectively.

  • Clean Solution adds over reference-free training (point 3).
  • Full Trace adds +2.00 [0.82, 3.17] in SmolLM3 at step 50 and +3.06 [1.71, 4.41] at step 100.
  • Genuine references outperform same-view references from another problem by 1.2–1.3 points under the two-pass protocol and, for Clean Solution, by +1.43 [0.43, 2.50] in a single pass (three seeds each). At 16K the order reverses within the same responses, so what genuine content adds depends on the budget.

The new control also tests a setting closer to SDPO (Hübotter et al., 2026) and SDFT (Shenfeld et al., 2026), in which the teacher sees privileged information and shares the student's mode. Without the reference this teacher would equal the student, and with it the student's thinking-enabled accuracy improves (+2.15 [0.63, 3.75], point 2), in line with the evidence you cite. We will scope the claims to fixed mathematical references, compared against matched reference-free training, in the configurations we test.

6. What this paper adds (W4). In same-mode self-distillation of the kind SDPO and SDFT study, the context is what makes the teacher differ from the student, so the gain over the base measures the context, and their ablations (such as SDPO and SDFT) ask which parts of it matter. In cross-mode OPSD the teacher differs from the student even without a reference, so the gain over the base no longer isolates the reference. That is the gap this paper addresses. Concurrent analyses also study attribution (Shrestha and Tessier, 2026; Ichihara et al., 2026; Adhinarta et al., 2026; Tian et al., 2026), which shows the question is timely. Our contribution is a controlled study of how the reference's representation, the training configuration and the inference budget shape what the reference adds.

  • The answer is held fixed. AMPLE-MATH gives each of 5,319 problems six views that share the same answer, from the answer alone to the full trace, and is reusable for other privileged-information methods.
  • Every reference has a matched control. All six Qwen views have a matched reference-free student, Answer Only and Full Trace are also tested in SmolLM3, and wrong-answer and other-problem controls test the content.
  • The training configuration is varied. Switching the student's rollouts from direct responses to thinking-enabled ones, with problems, references and evaluation fixed, turns gains into losses in both families.
  • The budget is measured. What genuine content adds changes sign with the budget within the same responses, and the original recipe's gain rises by +4.07 [0.83, 7.78] between 16K and the official budget (point 3).

Section 6 of the paper will be rewritten around this question and these contributions.

Reviewer_cJWiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies what privileged information actually adds in on-policy self-distillation (OPSD), where a frozen copy of the model sees the answer or a worked solution and scores the student's own rollouts. Many existing work reports improvement over the base model, it is not clear how much of the gain comes from the reference itself versus from distillation. So, the authors build AMPLE-MATH, 5,319 math problems with six answer-matched views ranging from answer-only to the full reasoning trace. They compare each view against a matched reference-free control: a thinking-enabled teacher scores direct-response student rollouts without any privilege. Experiments on Qwen3-1.7B and SmolLM3-3B show that much of the gain is already there without a reference, and more complete solutions do not consistently help. Switching student rollouts from short direct responses to long thinking-enabled ones turns gains into losses in both families. Teacher profiles and targeted loss interventions barely change student behavior, and the apparent gain from wider windows mostly comes from picking an earlier checkpoint. The conclusions is that OPSD mainly makes existing reasoning more accessible through parameters shared by the two inference modes.

Critical strengths
  • The research question is important. Prior OPSD work mostly compares against the base model, which cannot separate the reference from distillation itself. Here every view is compared against a matched reference-free run, together with wrong-answer and other-problem references.
  • The statistical analysis seems careful, and null results are reported honestly instead of being oversold. The evaluation also handles truncation properly by regenerating capped outputs and grading saved prefixes.
  • Some findings can be valuable for the field. The reversal from direct-response to thinking-enabled rollouts holds in both families and survives excluding capped samples. The checkpoint-matched analysis in Fig 5 is also a nice reminder that a lot of "improvement" from loss design can simply come from stopping earlier, although it is just one seed.
Critical weaknesses
  • In my opinion, the main issue is that the paper's explanation for where the gains come from is never tested. The abstract attributes the gains to a thinking-enabled teacher scoring direct-response rollouts, which then transfer through parameters shared by both modes. But every condition, including the reference-free control, keeps this mode mismatch, so no condition isolates it. The same problem applies to Section 3.2: thinking rollouts also make teacher and student the same mode, so the reversal is not only about "student training trajectories". In addition, the wrong-answer control improves almost as much as reference-free, and I couldn't find a non-OPSD baseline at matched steps, so I'm not sure whether the gain depends on the teacher's signal at all. Without these controls, the last two sentences of the abstract should be tempered.
  • Part of the reported "transfer" may just be a change in response length and when the model stops? In SmolLM3, direct-response outputs grow from about 800 to 11K tokens, and about 40% still give no answer at step 100. In Qwen, the direct-response gain is +8.7 at full length but −6.0 if we only grade the first 4K tokens.
  • I'm also not fully convinced by the claim that reference content matters. Section 7 argues that "reference-free gains do not make reference content irrelevant" based on the other-problem controls. But these are single-seed comparisons, the paper's own seed-to-seed variation is up to 1.9 points, and they only pass Holm within a two-test family. The authors should either add two more seeds for these two cells or soften this statement.

Let me know if I miss anything here.

Questions for authors
  1. Can you add one Qwen run with a thinking-disabled teacher that sees the reference and scores direct-response rollouts, plus a non-OPSD baseline at matched steps (e.g., SFT on the model's own correct traces)? If the results support the shared-parameter explanation, I would raise my score.
  2. What are the no-answer rates for each condition at steps 50 and 100, in both models and both evaluation modes? Is the SmolLM3 drop at step 100 mostly from responses that never finish? What fraction of direct-response training rollouts hit the 1,024-token cap, and does it grow during training? These should be available from the existing runs.
  3. Is the "reference content matters" conclusion meant to be a firm claim? If so, more seeds for the other-problem controls would help.
7 comments
AuthorsOpenReview ↗

Response one (1/4)

Thank you for the careful review and for naming the experiments that would change your assessment. We ran both, and answer your points in order.

  1. The explanation is now tested. With neither the rollouts nor the teacher's scoring in thinking mode, the student still improves with thinking enabled. This supports transfer through the weights the two modes share, the core of the shared-parameter explanation, and corrects one claim. A thinking-enabled teacher is not needed. Matched SFT does not reproduce the gain under the paper's protocol or on the external benchmarks.
  2. The gains depend on the budget, as you suspected. We also found that the paper's second pass favors models that reach the length cap more often. In a single pass, students trained with a reference keep clear gains, and Clean Solution adds +2.41 [1.32, 3.56] over reference-free training.
  3. Reference content matters, now with three seeds per cell, and for Clean Solution also in a single pass.

Terms. The paper studies two model families, Qwen3-1.7B and SmolLM3-3B. Unless stated otherwise, results here are for Qwen3-1.7B, and the trained students are evaluated with thinking enabled. The main comparisons use three or four training seeds per student, and every interval is a paired 95% problem-cluster bootstrap over the same test problems, with per-problem results averaged over seeds.

  • Two-pass protocol. This is the paper's in-domain evaluation (Appendix A.2). Each response may use up to 16,384 tokens. If it reaches this limit, we discard it and generate a new response from the start with up to 32,512 tokens.
  • Single pass. One response of up to 32,512 tokens, graded as is. Point 2 explains why we added it.
  • 16K. Grading only the first 16,384 tokens.
  • External benchmarks. AIME 2024/2025 and HMMT 2025, reported as Avg@12, with each response generated in a single pass of up to 38,912 tokens, as in the released OPSD evaluation (Zhao et al., 2026).

1. Is the explanation tested? (W1, Q1) You are right that the paper's direct-response comparisons all kept the teacher in thinking mode. We have now varied it, as you asked.

1a. A thinking-disabled teacher. The teacher has thinking disabled and sees the reference, and the student's rollouts are direct responses, so neither the rollouts nor the teacher's scoring use thinking mode. Without a reference, teacher and student would be identical and training would leave the student at the base (we checked that the loss and its gradient are exactly zero). We trained three seeds each with the Clean Solution and Full Trace references. With the paper's runs, this crosses the teacher's thinking with the reference (gains over the base, two-pass protocol).

Teacher's referenceTeacher thinks (paper)Teacher does not think (new)
None+1.80 [0.20, 3.47]0 (the base)
Clean Solution+3.10 [1.54, 4.73]+1.30 [−0.30, 3.02]
Full Trace+2.38 [0.91, 3.86]+2.99 [1.43, 4.67]
  • The reference without the teacher's thinking improves the student by +2.15 [0.63, 3.75], pooled over both references. In a single pass, Full Trace gains +1.26 [0.15, 2.37]. The Full Trace reference is itself a reasoning trace, but Answer Only, which contains no reasoning, also gains +3.06 [1.04, 5.14].
  • The teacher's thinking without a reference gives +1.80 [0.20, 3.47]. With the same Clean Solution reference, the thinking teacher's student is +1.80 [0.67, 2.95] ahead. For Full Trace the difference is −0.62 [−1.56, 0.31], unresolved.
  • With the thinking teacher, a reference adds +1.30 [0.20, 2.41] for Clean Solution and +0.58 [−0.47, 1.64] for Full Trace over reference-free training.
  • Evaluated with thinking disabled, the new students fall below the base (−4.62 [−7.13, −2.14]).

This is the direct test the paper lacked. The new students are trained only in direct-response mode, and their direct-response accuracy falls, yet their thinking-enabled accuracy rises, so the improvement reaches thinking mode through the shared weights. This supports the core of the shared-parameter explanation. It also corrects one claim. A thinking-enabled teacher is not needed. The teacher's thinking and a reference are two ways to make the teacher differ from the student, and each can improve thinking-enabled accuracy on its own.

AuthorsOpenReview ↗

Response one (2/4)

1b. Section 3.2. You are right that thinking-enabled rollouts also put teacher and student in the same mode. The new control adds a second same-mode configuration. The table gives each student's gain over the base at step 100 (two-pass protocol), with three seeds per cell in the direct-response rows, except four for the paper's Full Trace.

Student rollouts, teacherSame modeClean SolutionFull Trace
Direct response, teacher thinks (paper)No+3.10 [1.54, 4.73]+2.38 [0.91, 3.86]
Direct response, teacher does not think (new)Yes+1.30 [−0.30, 3.02]+2.99 [1.43, 4.67]
Thinking enabled, teacher thinks (Section 3.2)Yes−2.93 [−5.01, −0.78]−7.42 [−9.77, −5.14]

In the same mode, Full Trace gains +2.99 with direct-response rollouts and loses 7.42 with thinking-enabled rollouts, so mode matching alone does not explain the reversal. Nor does the second pass. On the external benchmarks, evaluated in a single pass, thinking-rollout Full Trace trails direct-response Full Trace by 9.44 points [5.83, 13.24] (Table 13). The teacher's mode also changes between the two same-mode rows, so we will describe Section 3.2 as a change of rollout configuration, and this table will go in the main text.

1c. Does the gain depend on the teacher's signal at all? You asked this because the wrong-answer control improves almost as much as reference-free training (Table 1), and the paper had no baseline without a teacher.

Gain over the baseIn domainExternal
Reference-free+1.80 [0.20, 3.47]+4.07 [1.73, 6.54]
Wrong answer+1.67 [0.07, 3.32]+4.14 [1.33, 7.07]

Both students learn from a teacher that differs from them, because the teacher thinks and the student does not. If that difference drives the gain, the two should be similar, so their similarity does not show that the teacher is irrelevant. We therefore added the non-OPSD comparison you suggested, LoRA SFT on the base model's own correct direct-response samples for 1,024 of the 1,536 problems. The samples are generated with thinking disabled, the mode in which the OPSD student trains, so the two differ in the training signal rather than the mode. Adapter, optimizer, learning rate, batch size and steps match OPSD, with three seeds, and if anything the setup favors SFT, which receives about 1.2 times as many supervised tokens.

DifferenceTwo-pass protocolSingle passExternal
SFT − base+0.04 [−1.43, 1.52]+0.37 [−0.67, 1.35]−1.02 [−3.02, 0.99]
Reference-free OPSD − SFT+1.76 [0.50, 3.04]+0.17 [−1.15, 1.45]+5.09 [2.99, 7.38]
Clean Solution − SFT—+2.58 [1.35, 3.84]—
Full Trace − SFT—+1.50 [0.28, 2.69]—
Same-mode Full Trace − SFT+2.95 [1.65, 4.28]+0.89 [−0.22, 2.00]—

SFT does not reproduce the gain under the paper's protocol or on the external benchmarks. It gains nothing resolved in any evaluation, and neither does a fourfold learning rate, which moves the weights further than OPSD does (+0.33 [−1.43, 2.08]). Reference-free OPSD exceeds SFT under the two-pass protocol and externally, on each of the three benchmarks. In a single pass in domain the two are not separated, while the Clean Solution and Full Trace students stay ahead. The rows for students trained with a reference are post hoc.

This also supports how the paper measures references. Reference-free self-distillation is strong on its own, 5.09 points above SFT on the external benchmarks, so what a reference adds has to be measured against it, not against the base. SFT will be a row of Table 1, and the external comparison will be a main-text result.

1d. The abstract. As you asked, its last two sentences will no longer attribute the gains to the thinking-enabled teacher, and will cite the new control as the evidence for transfer through shared weights. The abstract will lead with the paper's finding on references, along these lines.

Privileged references make a selective contribution to self-distillation. A polished solution can improve accuracy beyond matched reference-free training, and the benefit of a reference depends on its representation, the training and evaluation configuration, and the inference budget.

AuthorsOpenReview ↗

Response one (3/4)

2. Length and stopping (W2, Q2) In part, yes. The gains depend on the budget and come with changes in when the model answers and stops, but forcing the base to think longer does not reproduce the Clean Solution student's gain. We first answer your three questions.

Final no-answer rates, after regeneration under the paper's two-pass protocol, in percent. Ranges are over the trained students. For SmolLM3 these are the reference-free, Answer Only and Full Trace students, and Full Trace is lowest in every cell.

Model, evaluationBaseStep 50Step 100
Qwen, thinking enabled0.50.4–1.40.2–1.2
Qwen, direct response0.00.0–0.30.0–0.2
SmolLM3, thinking enabled0.01.3–5.16.4–9.7
SmolLM3, direct response0.63.1–22.014.4–42.0
  • The SmolLM3 drop at step 100 comes from missing answers. 42.0% of the reference-free student's direct responses give no answer, against 0.6% for the base. About half of these are still running at 32,512 tokens, and the rest stop without a final answer. When the student does answer, it is more accurate than the base (51.5% vs 40.5%).
  • Training rollouts. The share that reaches the 1,024-token cap grows during training, from 61–69% early to 76–86% late in Qwen, and from 18–22% to 72–94% in SmolLM3.

The budget can decide the sign of the gain. Your Qwen example (−5.95 at 4K, +8.68 at full length, Table 15) holds in both modes and for the original OPSD recipe.

  • Qwen, thinking-enabled evaluation. Students in Table 1 reach the 16K cap in 19.9–27.9% of responses, against 16.6% for the base. At 16K none has a resolved gain, and several are below the base (reference-free −4.90 [−6.88, −2.97]).
  • The original OPSD recipe, external benchmarks. Our reproduction gains +1.48 [−1.11, 4.26] at 16K and +5.56 [2.13, 9.26] at the official 38,912 tokens, a paired increase of +4.07 [0.83, 7.78] within the same responses.

A protocol check. Appendix A.2 regenerates capped responses from the beginning with the same sampling seed. While measuring cap rates we found that reusing the seed does not reproduce the first response, and the new response departs from it at a median of token 149. A capped response therefore gets a second, independent attempt, and students, which are capped more often, benefit more. Even when we grade only the first 16K tokens of the new response, the second pass raises the base's accuracy by +3.71 [2.60, 4.82]. We therefore evaluated the same checkpoints in a single pass.

DifferenceSingle passTwo-pass protocol, same seeds
Reference-free − base+0.54 [−0.69, 1.74]+1.80 [0.20, 3.47]
Clean Solution − base+2.95 [1.76, 4.17]+3.10 [1.54, 4.73]
Full Trace − base+1.87 [0.67, 3.06]+2.76 [1.30, 4.28]
Same-mode Full Trace − base+1.26 [0.15, 2.37]+2.99 [1.43, 4.67]
Clean Solution − reference-free+2.41 [1.32, 3.56]+1.30 [0.20, 2.41]

Every row uses three training seeds. In a single pass the base was evaluated three times with different sampling seeds (78.9%, 80.3% and 79.6%), and we compare against the mean. Table 1 averages Full Trace over four seeds (+2.38 [0.91, 3.86]).

Students trained with a reference keep clear gains without regeneration, and Clean Solution's advantage over reference-free training is resolved under both protocols, with a larger point estimate in a single pass. The reference-free gain is not resolved. We will clarify Appendix A.2, and Table 1 will add the single-pass results.

When the model stops. The trained students state the correct answer about 1,000–1,500 tokens later than the base, and after a wrong first candidate answer they recover more often (33–36% against 29%, pooled +5.3 [1.0, 9.5]). For Clean Solution the gain is not only a length effect. Replacing the base's first end-of-thinking token with "Wait" (budget forcing, Muennighoff et al., 2025) lengthens its responses by 334 tokens without a detectable gain, and the Clean Solution student stays +2.91 [1.22, 4.64] ahead of this forced base. The gains do need a long budget, since at short budgets the same students fall below the base. As one student can show a loss, no gain or a gain depending on the budget, we think this is a useful lesson for evaluating OPSD in general. The revision will add a Budget section to the main text and an appendix table with no-answer, cap and length rates for every condition.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Response two (1/2)

Thank you for the follow-up. We agree that same-mode training reduces accuracy in the mode used for training. Our runs show that this loss comes from showing the reference to the teacher, rather than from the training recipe.

1. Scope. The configuration is not specific to one paper. Specifically, it is OPSD's released default (i.e., a student with thinking disabled and a thinking teacher), and the later OPSD papers we listed also use it as their default. Moreover, its published results reproduce. With the released code and data, our base model scores 36.76% on the three external benchmarks, compared with 37.10% in the released results. The released gain at step 50 (+4.4 points) is also within our interval for the same step (+5.56 [2.13, 9.26]). Beyond the default, the paper compares each reference with a matched reference-free student in two configurations where teacher and student share a mode (i.e., both with thinking disabled, or both with thinking enabled). The revised abstract will state these three configurations as the scope.

2–3. Source of the loss under same-mode training. In both same-mode configurations, the student loses accuracy in the mode it was trained in. The loss is −4.62 [−7.13, −2.14] with thinking disabled (three seeds) and −2.93 to −7.42 with thinking enabled (one seed), under the two-pass protocol. Part of our explanation follows directly from the design, and part is an association.

  • The reference causes the loss. When teacher and student share a mode, the reference is the only difference between them. Without it, the teacher receives exactly the student's input, and the update is zero. Therefore, the loss in these configurations is the effect of showing the reference to the teacher.
  • The training recipe does not cause the loss. We train with the released OPSD hyperparameters, which reproduce the published result (point 1). To test the recipe directly, we have now also fine-tuned the model on its own correct thinking-enabled traces, in the same mode and with the same optimizer, learning rate and number of steps. This student does not lose accuracy (−0.59 [−2.15, 0.98], three seeds, two-pass protocol), and reference-free OPSD outperforms it by +2.39 [1.11, 3.69]. As a result, training in thinking mode with these settings does not reduce accuracy by itself. In the default configuration, direct-response accuracy also increases without a reference (+8.68 [6.16, 11.20]). Consistent with our results, Kaur et al. (2026) report the same loss when thinking models are self-distilled. In their runs, students adopt the privileged teacher's shorter responses but do not gain its accuracy.
  • A likely route (an association). We hypothesize that a teacher that sees the answer favors committing to it, and that the student learns to commit earlier in the mode it is trained in. With thinking disabled, the student's responses become shorter and less accurate as the teacher sees more of the solution. Specifically, they average 4,993 tokens and 51.84% without a reference, and 1,738 tokens and 41.03% with Full Trace. The same-mode students write 1,454–1,531 tokens, fewer than the base's 1,701. With thinking enabled, the students trained on thinking rollouts end their thinking about 220–240 tokens after a wrong first answer, compared with about 2,100 for the base. After such a wrong first answer, they also end with the correct answer less often (13–14% compared with 29%). However, length alone does not account for the loss, since the thinking-rollout Answer Only student writes longer responses than the base and still loses. The revised paper marks which parts of this account are causal.
AuthorsOpenReview ↗

Response two (2/2)

4. Gap between teacher and student. The 97–99% is the teacher's accuracy with the answer in its context. No student without the answer can reach this level, which is why we compare each reference with a matched reference-free student. The relevant target is the thinking-enabled model without a reference (80.21%). With thinking enabled, each student in the default configuration outperforms it (+1.80 to +3.10, two-pass protocol). With thinking disabled, we agree with your reading. Students trained with a reference remain close to the base (41.03–46.35% compared with 43.16%), while the reference-free student gains +8.68. These results show that training does change this mode, and that the reference reduces the gain. We also agree that the teacher's distribution may be too hard to learn without the reference. This difficulty is known as the imitation gap (Weihs et al., 2021; Warrington et al., 2021), where the teacher acts on information that the student does not see. Our hypothesis above is one form of it. Within the standard objective, wider loss windows, up to the whole response, do not remove the loss with thinking-enabled rollouts. The revised discussion names an objective that corrects for the imitation gap as the next step.

Correction. We rebuilt the other-problem controls (i.e., a teacher given the reference of a different problem) using references from training problems only, since the earlier ones could come from any data split. Genuine Clean Solution still outperforms its control, by +1.43 [0.41, 2.47] under the two-pass protocol (Holm p = 0.013) and +2.02 [1.00, 3.08] in a single pass. For Key Points, the difference has the same sign (+1.06 [−0.09, 2.24]), but its interval now includes zero. The rebuilt wrong-answer control still gains over the base, by +2.43 [0.89, 4.06] in domain (two-pass protocol) and +4.01 [1.36, 6.91] on the external benchmarks.

We hope these results address your concerns about the soundness of the training.

AuthorsOpenReview ↗

Summary of contributions, discussion and revised PDF

Dear AC and reviewers,

Here we provide a summary of our rebuttal.

What this paper contributes. On-policy self-distillation (OPSD) provides the teacher with privileged information, such as a reference solution, and is quickly becoming a common recipe for improving reasoning models. However, prior OPSD work mostly compares with the base model, which cannot separate what the reference adds from self-distillation itself. Our paper measures this quantity directly and makes three contributions.

  • AMPLE-Math, a new resource. We construct AMPLE-Math, a reusable suite of 5,319 mathematical problems with six reference views per problem that share the same answer, from the answer alone to the complete reasoning trace, with audits and frozen splits. We will release it with the camera-ready version.
  • Controlled comparisons. In two model families, we compare each reference with matched reference-free self-distillation, wrong-answer and other-problem controls, and SFT. Much of the gain is present without any reference, a polished solution adds a selective increment, and the same references turn gains into losses when the student's rollouts or the generation budget change.
  • Teacher profiles and matched interventions. In Qwen, they show that changing the token-level supervision can leave the student's behavior largely unchanged, and that stopping time explains most of an apparent gain from broader loss coverage.

These results change how OPSD should be used and evaluated, and AMPLE-Math lets future work measure what a reference adds while holding the answer fixed. The reviewers describe the question as important (cJWi, 7gfN) and the controls as useful guidance for evaluating privileged supervision (NLad).

What the discussion added, and where it is in the revised PDF. Major changes are marked in blue, and "16K → 32K retry" is the two-pass protocol of our replies.

  • Same-mode teacher (Section 3.2, Table 2). A teacher with thinking disabled that sees the reference improves thinking-enabled accuracy by +2.15 [0.63, 3.75] points under 16K → 32K retry. Without the reference, this teacher receives exactly the student's input, and the update is zero. Therefore, the same-mode gains and losses are effects of the reference.
  • SFT baselines (Section 3.1, Table 1). SFT on the model's own correct direct responses or thinking traces does not improve over the base. In contrast, reference-free self-distillation outperforms direct-response SFT by +5.09 [2.99, 7.38] points on the external benchmarks and thinking-trace SFT by +2.39 [1.11, 3.69] points under 16K → 32K retry.
  • Budget (Section 3.3, Figure 4, Table 1). The paper reports accuracy by budget and per token, and each in-domain gain with and without the retry. On the external benchmarks, the gain of the released OPSD recipe increases by +4.07 [0.83, 7.78] points from 16K tokens to the official budget (Appendix A.2).
  • Scope (abstract, Sections 6 and 7). The abstract states the three configurations studied, the related work positions the paper with respect to SDPO and SDFT, and the discussion names objectives that correct for the imitation gap as a next step.
  • Appendix. It defines the evaluation protocols (A.2), describes the setups of the new controls, and reports no-answer, cap and length rates for each condition.

Corrections. The other-problem controls now use references from training problems only. Clean Solution still outperforms its control (+1.43 [0.41, 2.47], Holm p = 0.013). For Key Points, the difference has the same sign, but its interval now includes zero (+1.06 [−0.09, 2.24]). As a result, the paper states the content claim firmly for Clean Solution only. The rebuilt wrong-answer control still gains over the base.

Thank you again for your careful reviews and the discussion.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.