- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies what privileged references add beyond reference-free on-policy self-distillation. Using AMPLE-MATH’s six answer-matched views, experiments in two model families show that reference benefits vary with the model and checkpoint, while changing student training trajectories can reverse transfer.
Critical strengths
The reference-free control and six answer-matched views provide a focused comparison of what privileged references add beyond distillation itself. Comparisons across model families and matched checkpoints substantiate how reference contributions vary, offering useful empirical guidance for evaluating privileged supervision.
Critical weaknesses
The direct-response versus thinking-enabled comparison changes reasoning mode, rollout length, and supervision coverage simultaneously. Although the loss-window interventions address part of this confounding, they do not fully disentangle these factors, limiting the mechanistic interpretation of the observed transfer reversal.
Questions for authors
To what extent do the existing controls separate the effects of reasoning mode, rollout length, and supervision coverage on the observed transfer reversal?
5 comments
Response one (2/2)
Other new results. These strengthen the paper's main comparisons, each with three training seeds per student.
- The shared-parameter account is tested directly. The same-mode students above are trained with thinking disabled for both student and teacher, yet their thinking-enabled accuracy rises (+2.15 [0.63, 3.75], pooled).
- A non-OPSD baseline. LoRA SFT on the model's own correct answers, with matched steps and about 1.2 times OPSD's supervised tokens, gains nothing resolved. Reference-free OPSD exceeds it by +5.09 [2.99, 7.38] on AIME 2024/2025 and HMMT 2025.
- Reference content, three seeds. Genuine references beat same-view references from another problem by +1.28 [0.13, 2.47] for Key Points and +1.22 [0.26, 2.19] for Clean Solution (both survive Holm correction).
- Evaluation protocol. The paper's second pass regenerates capped responses from the beginning (Appendix A.2). We found that this gives models that are capped more often what is in effect an extra attempt. In a single pass of up to 32,512 tokens, Clean Solution's advantage over reference-free training is +2.41 [1.32, 3.56]. The revised paper will add the single-pass results.
Thank you again. We hope the controls above answer your question as far as the current runs allow, and that the new results show the paper's comparisons are sound. If so, we would be grateful if you would consider raising your score.
Response two
Thank you again for your question about which factors the controls separate. Before the discussion closes, we would like to add one result on the factor our reply left coupled, and one correction.
Supervised tokens. Our reply noted that the student's mode still changes together with what the supervised tokens are. To separate these tokens from the teacher's signal, we fine-tuned the model on its own correct thinking-enabled traces, with the same optimizer, learning rate and number of steps (three seeds). Training on thinking tokens without the privileged teacher does not reduce accuracy (−0.59 [−2.15, 0.98] compared with the base, two-pass protocol). In contrast, the OPSD students trained on thinking-enabled rollouts lose 2.93 to 7.42 points (one seed). Therefore, the reversal is tied to the teacher's signal on thinking-enabled prefixes, rather than to training on thinking tokens alone.
Correction. We rebuilt the other-problem controls using references from training problems only, since the earlier ones could come from any data split. Genuine Clean Solution still outperforms its other-problem control (+1.43 [0.41, 2.47], Holm p = 0.013, two-pass protocol). For Key Points, the difference has the same sign (+1.06 [−0.09, 2.24]), but its interval now includes zero.
In the revised paper, Section 3.2 will state which factors each control separates, with the summary table in the appendix. If our replies address your question, we would be grateful if you would consider updating your score.
Response one (1/2)
Thank you for the focused question. It is the right one to ask of Section 3.2. The existing design and two new runs show that rollout length beyond the loss window cannot change the update, that no supervision coverage we tested removes the reversal, and that mode matching alone does not explain it.
The paper studies two model families, Qwen3-1.7B and SmolLM3-3B. Unless stated otherwise, results here are for Qwen3-1.7B, evaluated with thinking enabled under the paper's protocol (Appendix A.2).
Rollout length. The loss covers the first 1,024 completion tokens in both configurations (Eq. 1, App. A.1). Tokens past that window never enter the loss, and the teacher's score at each position depends only on the prefix. So for a fixed supervised prefix and loss mask, rollout length beyond the window changes compute, not the update. The supervised span is also nearly matched.
Supervision coverage. Within thinking-enabled rollouts we varied how much is supervised and where (Full Trace, accuracy in %, base 80.21%).
No window removes the penalty. Every thinking-rollout variant scores below the base at every step, while direct-response training stays above it, and none is resolved above the paper's window. The whole-response loss contains the final answer in 87.0% of training rollouts, against 0.1% for the paper's window, yet it is lower at step 100 (−2.80 [−5.27, −0.33] before Holm correction). So the reversal persists under every window we tested, and extending supervision to the final answer is not enough to remove it in these runs.
Reasoning mode. Two factors remain tied to the student's mode, what the supervised tokens are and whether teacher and student share a mode. The new control addresses the second. Its teacher has thinking disabled, sees the reference and scores direct-response rollouts, so teacher and student share a mode. The table gives each student's gain over the base at step 100, with three seeds per cell in the direct-response rows, except four for the paper's Full Trace.
With teacher and student in the same mode, Full Trace gains +2.99 with direct-response rollouts and loses 7.42 with thinking-enabled rollouts. So mode matching alone does not explain the reversal. Nor does the second pass. On the external benchmarks, evaluated in a single pass, thinking-rollout Full Trace trails direct-response Full Trace by 9.44 points [5.83, 13.24] (Table 13). The reversal also holds in SmolLM3-3B. The Full Trace trained with thinking-enabled rollouts scores 77.99% at step 50, against 84.83% for direct-response training and 82.16% for the base (two seeds), and an Answer Only student trained this way falls to 36.13% by step 100.
What remains is the student's mode itself, that is, what the supervised tokens are, which still changes together with the teacher's mode. One measurement isolates the supervised tokens. Table 16 holds the teacher in thinking mode and varies only the prefix type, and for Full Trace the teacher's shifts align with correctness on direct-response prefixes and reverse on thinking-enabled prefixes (+0.0131 vs −0.0123 nats per token). The trained students' behavior fits this. After a wrong first candidate answer in the thinking, the thinking-rollout Full Trace and Clean Solution students end correct in 13–14% of cases, against 29% for the base.
We will revise Section 3.2 to state which factors each control isolates and which remain coupled, with the summary table above, and make the section easier to follow.