acceptodds
Under review as a conference paper at ICLR 2027

The Right Direction at the Right State: Dissecting On-Policy Self-Distillation

Abstract

On-policy self-distillation (OPSD) offers a route to improved reasoning by letting a context-augmented copy of the model supervise its own generations. What makes this additional context useful for learning? We investigate this question through controlled studies of mathematical reasoning with Qwen3 thinking models. Solutions to other problems and a single fixed reference can produce useful training effects, while target-specific solutions outperform other-problem references at both evaluated scales. The central finding is that useful guidance depends on the student's reasoning so far. Exchanging reference-induced changes in token preferences between attempts at the same problem reduces continuation accuracy across both scales, with the amount of distributional change matched. On shared Base histories, thinking-only use of the trained policy retains about 90% of its observed full-policy gain. On MATH-500, alignment improves correct completion within 4k tokens by 4.30-4.80 points and reduces mean token use under a 32k cap by 10.7-12.5% relative to exchanged guidance. Together, these findings distinguish the reference a teacher reads from the guidance a student can use. Reference content shapes the learning benefit; the student's evolving reasoning determines the usefulness of the induced guidance. This trajectory-conditioned view connects local supervision during training to successful reasoning completion.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-policy distillation, On-policy self-distillation, LLM post-training, state-conditioned supervision, mathematical reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_evqyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies why on-policy self-distillation is effective, with a focus on how the reference provided to the teacher interacts with the student's current reasoning state. Through controlled experiments on mathematical reasoning, the authors show that references from other problems can also be useful, while references specific to the target problem provide larger gains. More importantly, exchanging reference-induced guidance between different reasoning trajectories reduces its training benefit, even when the trajectories are for the same problem. The paper further shows that most of the improvement is expressed through the thinking stage rather than final-answer generation. The results suggest that matching guidance to the student's reasoning state helps the model complete reasoning more efficiently under limited token budgets.

Critical strengths
  1. The paper studies an interesting and valuable question: why on-policy self-distillation works. Understanding the role of the reference and its interaction with the student's reasoning state is useful for this line of research.
  2. Some findings are insightful. In particular, the observation that using the trained policy only during the thinking stage retains most of the improvement provides a useful perspective on where the benefit of self-distillation comes from.
Critical weaknesses
  1. The model coverage is limited. The experiments only consider Qwen3 models at two relatively small scales. It is unclear whether the same conclusions hold for other model families.
  2. The evaluation mainly focuses on mathematical reasoning. Although several math datasets are included, the task coverage is still limited. It is unclear whether the observed effects also hold for other types of tasks, such as code generation or general reasoning.
Questions for authors
  1. How can the findings be used to improve the training method in practice? For example, can they guide the design of a more effective teacher reference or supervision strategy, beyond explaining the behavior of the current method?
  2. Do the authors expect different results on other model families or non-mathematical benchmarks? Additional experiments would help clarify how general these conclusions are.
  3. Could other types of teacher references also be useful? For example, instead of a worked solution to a specific problem, could the reference be a summary of the solution strategy for a class of problems?
Reviewer_TV1FOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper decomposes OPSD into its components and examines the effect of each through controlled experiments. The authors define a “teacher direction”, the centered log-probability shift that privileged context induces on top of the unaugmented teacher distribution, and use it to quantify what OPSD training changes. With this tool, they study the effect of the reference (Section 4) and of the student’s prefix (Section 5), and then show that the result of OPSD training is carried mainly in the reasoning (thinking) stage (Section 6).

Critical strengths
  • Clear formulation, and the analysis covers diverse angles.
  • Meaningful as a study of what affects OPSD and how.
  • Well-controlled experimental design.
Critical weaknesses
  • Personally, on a first read I found it hard to follow what Sections 4 and 5 are trying to do. Since the formulation is already there, writing out the compared targets as equations in each section would make it much clearer.
  • Please see the questions below.
Questions for authors
  • The title of Section 4, “What does the reference contribute?”, promises more than the section delivers. The section measures how much accuracy changes with each source, not what the reference contributes to. Please consider a more accurate title or a clearer statement of what the section answers.
  • The contribution of Section 4 seems limited. The reference-reuse result is already reported by Ichihara et al. (2026). Could the authors clarify what the specific contribution of this section is?
  • In my understanding, all of Sections 5 and 6 use a single fixed, unrelated proof as the privileged context. I understand this is for controlling variables. However, the standard OPSD setting uses the target solution. In particular, in the same-question experiment (5.2), the target solution could have been used without confounding, since both attempts share the same problem. Why was Fixed used there, and is there evidence that the pairing effect holds under Target context?
  • Is there any bias introduced by the frozen Base-generated histories used in evaluation?
Reviewer_TYd5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies the mechanism of privileged information (PI) in on-policy self-distillation, where using seemingly less relevant context as privileged information (e.g., answers to unrelated questions) still helps supervision, and examines whether the PI serves a sharpening mechanism or whether it produces guidance specifically to the student’s current partial solution. The main analysis experiment centers on taking the per-token distribution difference in one answer between PI-teacher and student, and add it to an alternative answer's "teacher" (that has no PI) and uses that as training signal; the paper finds that the log-probability differences produced by the teacher given the alternative answer still performs better than the former. Experiments use Qwen3-1.7B and 4B with thinking on and test on AIME25/HMMT25 and MATH500, with many baselines.

Critical strengths
  • Experimental setup is rigorous and convincing.
Critical weaknesses
  • (Major) If I'm not misinterpreting the paper, the chief takeaway that the paper pushes for is that "A per-token probability difference given to a different problem is less effective than using the probability differences given to the current problem," which seems to be a trivial result. How is this surprising in any way?
Questions for authors
  • Please address W1. What is the interesting scientific takeaway?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.