acceptodds
Under review as a conference paper at ICLR 2027

Train Ahead, Distill Back: Bootstrapping On-Policy Self-Distillation for Large Language Models

20 views

Abstract

On-policy self-distillation (OPSD) improves large language models by letting a self-teacher with privileged information provide dense token-level supervision on the model's own trajectories. Yet existing methods typically construct the self-teacher from the current, initial, or slowly averaged policy state, leaving the quality of supervision constrained by the teacher's ability to exploit privileged information. We ask whether the model's own optimization progress can instead be recycled into a stronger self-teacher. In this paper, we introduce Bootstrapped On-Policy Self-Distillation (B-OPSD), which temporarily trains the policy ahead to obtain a future teacher, restores the student to the original policy state, and then uses the future teacher to supervise the restarted student. The future teacher improves supervision in two complementary ways, it can generate more reliable privileged trajectories and, conditioned on them, provide more informative token-level targets along the restarted student's on-policy trajectories. Experiments on mathematical reasoning with Qwen3-4B and Qwen3-8B show consistent improvements over standard OPSD in both settings, including gains from 27.50 to 41.30 and from 48.80 to 64.44 in the rollout-privileged setting. Our findings point to a broader principle for self-improving models that future learning progress can be distilled backward, preserving acquired knowledge while bootstrapping beyond the optimization state that produced it.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Self-Distillation, Large Language Models, Self-Improvement, Reasoning, Policy Optimization, Privileged Information

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_yWYeOpenReview ↗
Rating
3(+1)
Confidence
3(−1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes B-OPSD, which aims to improve self-distillation by constructing a better self-teacher. Specifically, B-OPSD first performs several steps of vanilla OPSD training to obtain a future version of the student as the self-teacher. It then restores the student to its initial checkpoint and uses this future self-teacher to provide privileged information and OPD-style supervision. The authors develop concrete methods for settings both with and without ground-truth answers. Experiments on Qwen3-4B and Qwen3-8B across three mathematical reasoning benchmarks show that B-OPSD consistently outperforms vanilla OPSD in both settings. The paper further investigates B-OPSD’s effectiveness and advantages through ablation studies, comparisons under matched training budgets, and visualizations of training gains. Overall, the proposed approach is interesting, demonstrates effectiveness in the mathematical reasoning domain, and offers useful insights into self-distillation.

Critical strengths
  1. Prior work on self-distillation has primarily focused on constructing suitable privileged information. B-OPSD instead explores how to build a better self-teacher, proposing to use a future version of the student as a stronger self-teacher. The motivation is compelling, and the approach does not require introducing a stronger external teacher, offering an interesting direction for further improving OPSD.
  2. B-OPSD demonstrates notable performance gains over OPSD across two model sizes, three mathematical reasoning benchmarks, and settings both with and without ground-truth answers. The experimental finding that privileged information needs to be matched to the self-teacher is also interesting, and the analysis presented in Figure 2 is intuitive.
Critical weaknesses
  1. The paper’s central finding is that a future version of the model can provide better supervision as a self-teacher. While Figure 2 offers an intuitive explanation, the results do not sufficiently clarify which properties of the supervision are improved by the future self-teacher, or why these improvements allow the restarted student to outperform continued training. A deeper mechanistic investigation or an appropriate theoretical analysis is needed. It would also be valuable to discuss the conditions under which the approach is effective and those under which it fails to provide benefits.
  2. Regarding the experimental setup, we have two concerns. First, what motivates the use of LoRA for training? Although LoRA is used in several existing OPSD studies, it remains unclear whether self-distillation would remain effective under the more conventional full-parameter fine-tuning setup, or whether its effectiveness depends on LoRA. Could the authors provide additional experiments to investigate this question? Second, the current evaluation is limited to three benchmarks in the mathematical reasoning domain. We would like to see experiments on additional benchmarks and non-mathematical domains, such as coding, to assess whether B-OPSD’s performance gains and advantages generalize more broadly.
Questions for authors

In addition to the questions raised in the weaknesses, we would appreciate clarification on the following points and are happy to revising our final score based on the authors’ responses.

  1. The paper states that thinking mode is disabled during all training and evaluation. Could the authors confirm whether thinking mode is also disabled when constructing the self-teacher? Recent work, What Does Privileged Information Add to On-Policy Self-Distillation?, suggests that the effectiveness of self-distillation may stem from a mismatch in thinking modes between the self-teacher and the student.
  2. Prior work on the failure modes of self-distillation, Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?, reports that performance can deteriorate in later stages of training, accompanied by shorter model outputs. Figure 3 also shows performance degradation for OPSD. Could the authors extend the B-OPSD performance curves to more training steps and report how output length evolves during training for both OPSD and B-OPSD?
  3. Can the proposed approach of using a future version of the model to improve its current self be applied iteratively? Recent work, RISE: Recursive Improvement via Self-Extrapolating Policy Distillation, introduces an iterative policy distillation procedure. We are curious whether B-OPSD could support a similar process of iterative improvement. For example, could the checkpoint at step 100 of the blue curve in Figure 3 serve as a second-generation future self-teacher, with the student reset to its initial checkpoint for another round of B-OPSD?
  4. B-OPSD involves several conditional rules for selecting the self-teacher and the privileged information. How would performance change if these rules were removed and the future self-teacher were consistently used to provide both privileged information and supervision?
Reviewer_KMydOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents B-OPSD, which first trains a future teacher with OPSD, then restarts the student and distills from that teacher using its own privileged trajectories. Experiments on Qwen3-4B/8B show improvements on mathematical reasoning, supported by extensive ablations.

Critical strengths
  1. Training ahead and distilling back is an interesting contribution to OPSD. Reusing optimization progress as supervision for an earlier checkpoint gives the method a distinctive design.
  2. The experimental analysis is thorough. The ablations examine why the future teacher matters, why pairing it with its own rollout helps, and why restarting can outperform continued OPSD even after counting the lookahead updates.
Critical weaknesses
  1. The motivation is less convincing than the experiments. The introduction explains why a stronger teacher may help, but the step from this observation to restarting the student is not well developed. The specific limitation of existing OPSD, and its relationship to learning from privileged context, should be made clearer.
  2. The teacher-training strategy deserves a more direct ablation. Table 5 compares Future(Base) and Future(EMA), but these configurations also change the second-stage fallback teacher. Their substantial performance gap therefore does not isolate the effect of how the future teacher is trained.
Questions for authors
  1. What limitation of standard OPSD specifically motivates distilling back into the initial checkpoint, beyond obtaining a stronger teacher?
  2. With Stage II and the lookahead budget fixed, how do different OPSD strategies for training the teacher affect the final student? Please report the corresponding teacher performance alongside the student results, extending Table 9.
Reviewer_6nvCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

B-OPSD first runs a short self-distillation phase to obtain a more capable future checkpoint, then restarts the student from its initial parameters and uses that checkpoint for privileged-information-conditioned supervision. The implementation uses the future teacher during an early restart window, followed by the specified fallback teacher. A reference-answer-free variant relies on majority agreement to choose privileged information and teacher supervision. Experiments on Qwen3-4B and Qwen3-8B across three competition-math benchmarks show substantial gains over OPSD. Ablations examine teacher quality, privileged-information source, restart timing, and base versus EMA teacher choices.

Critical strengths
  1. B-OPSD substantially improves over OPSD on both model sizes. In the eight-candidate setting, average scores rise from 27.50 to 41.30 for 4B and 48.80 to 64.44 for 8B.

  2. Ablations support combining the future teacher with its privileged information. Future-generated information alone does not help the base teacher, making this interaction more informative than simply using a stronger teacher.

Critical weaknesses
  1. Comparisons focus on OPSD variants and direct OPD, without GRPO or another representative verifier-based RL baseline. In the answer-available setting, these are relevant alternatives; the current evidence establishes gains within distillation, but not the trade-off against RL.

  2. Matching updates does not establish equal total compute, even with preliminary teacher training included. Extra teacher generation and inference may materially change cost; matched-budget comparisons and training-seed uncertainty are needed to assess the practical benefit.

Questions for authors
  1. Can you add GRPO and another representative RL baseline in the answer-available setting, using the same backbone, training problems, and evaluation protocol? Please specify verifier access and any differences in reference-solution supervision; these comparisons should not introduce gold-answer supervision into the reference-free setting.

  2. How do B-OPSD, OPSD, and the RL baselines compare at matched total compute? Please count preliminary teacher training, rollout generation, and teacher inference, and report variability across training seeds.

  3. Which optimizer states are reset, and how does the existing Future-Continue control compare with Restart on the full benchmarks? This would help isolate the benefit of restarting beyond the small-set reward-surface analysis.

Comments on the paper

2 comments

Sorry for the late reply. Thanks for your detailed response, and I will raise my score from 2 to 3.

Just as a friendly reminder (which does not affect my score or overall evaluation): the two links you provided in response to Q2 are inaccessible to me (and maybe other reviewers), so I would suggest updating them if possible.

P.S. If any of you are also serving as reviewers for other submissions that are still waiting for a response, I hope you can also take a moment to reply to those authors! :-D

AuthorsOpenReview ↗

Appreciation for Your Review and Feedback

We sincerely appreciate your recognition of our work and your decision to raise your score.

Thank you also for your insightful and constructive suggestions, which have helped improve the depth and completeness of our paper. We have already incorporated the additional findings and discussions into the revised manuscript.

We appreciate your friendly reminder regarding the two links in Q2. Since access can vary across regions, we have replaced them with alternative links below (and updated our earlier response accordingly): "We extend training for both OPSD and B-OPSD (see figure: https://ibb.co/5WJGC2CT) and report the corresponding evolution of output length (see figure: https://ibb.co/GfXGH3vj)."

We fully agree with your point about reviewer engagement. In fact, all coauthors serving as reviewers responded to the rebuttals for their assigned submissions as soon as possible. We consider timely and thoughtful engagement an important responsibility to the research community, and we sincerely hope that every author's dedicated effort is met with the same care, fairness, and encouragement.

Thank you again for your careful review and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.