acceptodds
Under review as a conference paper at ICLR 2027

How Should Teachers Be Prepared? RL on Student-Induced States for On-Policy Distillation

Abstract

On-policy distillation (OPD) improves the reasoning capabilities of small language models through token-level teacher supervision on student-generated trajectories. Yet can teachers that excel at solving problems independently also guide student reasoning effectively? Prior work shows that when student prefixes follow reasoning paths that differ from the teacher's own or contain errors, teachers can be less accurate when continuing from these prefixes than when solving problems independently. To this end, we propose Prep-OPD, which uses reinforcement learning (RL) before distillation to train the teacher to adapt to the student's existing reasoning state and correct course when errors arise. Training optimizes teacher continuations from fixed student prefixes using final-answer correctness as the reward. The prepared teacher then trains the student through trajectory guidance and token-level supervision. We evaluate Prep-OPD on eight mathematical reasoning benchmarks, using Qwen3-4B-Instruct-2507 as the teacher and Qwen3-0.6B and Qwen3-1.7B as students. With the 4B teacher and 1.7B student, Prep-OPD improves average accuracy over standard OPD and the strongest baseline, Relay-OPD, by 8.28 and 2.30 percentage points, respectively. Controlled experiments further show that teacher RL conditioned on student-generated prefixes yields higher student accuracy than problem-start teacher RL with and without handoff on Qwen3-1.7B. Reusing the same prepared teacher also improves Qwen3-0.6B.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, knowledge distillation, teacher preparation, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_baf1OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the issue of noisy teacher signals in On-Policy Distillation (OPD) when the teacher is conditioned on student-generated prefixes. To address this issue, the authors propose Prep-OPD, which fine-tunes the teacher using RLVR on student-generated prefixes before distillation. Experimental results show that Prep-OPD improves accuracy over existing baselines.

Critical strengths
  • The proposed approach is simple and intuitive while achieving strong empirical performance.
  • The paper provides strong and rigorous empirical results, convincingly demonstrating that Prep-OPD outperforms existing baselines while maintaining better training efficiency.
Critical weaknesses
  • My main concern is that while Prep-OPD improves training efficiency during student training, it introduces additional computational costs for teacher fine-tuning. This overhead could be substantial, especially for larger teacher model. Especially in the context of OPD, potentially undermining one of the main motivations for using OPD instead of RLVR: achieving competitive performance at a substantially lower training cost.
  • The paper also appears somewhat incremental. The issue of teacher failures on student-generated prefixes has been explored in prior work, including findings that teachers become less accurate when continuing from student-generated prefixes [3]. Furthermore, the idea of learning from suffixes is not new [1,2].
Questions for authors
  • Could the authors report the total training compute (teacher fine-tuning + student training) and compare it against other baselines?
  • Could the authors also provide the training dynamics during RL fine-tuning of the teacher? Specifically, does the teacher achieve higher rewards when conditioned on student-generated prefixes, and if so, to what extent?

[1] Reuse your FLOPs: Scaling RL on Hard Problems by Conditioning on Very Off-Policy Prefixes. ICML 2026.
[2] Learning to Generate Better than your Large Language Models.
[3] Pass the Baton: Trajectory-Relayed On-Policy Distillation.
1 comment

Thank you for your response. Could the authors clarify why Standard OPD, Relay-OPD, FastOPD, and IW-OPD all have exactly 55.54 hours of training? These methods do not require RLVR training on student-generated prefixes, so it is unclear why their exists computational costs. Furthermore, why does Prep-OPD require only approximately two additional hours compared to these baselines? Given that the teacher model is substantially larger than the student model, I would expect the teacher's computational cost to be significantly higher. Could the authors provide a more detailed breakdown of the computational costs and explain these results? Additionally reporting training FLOPs could also be helpful.

Reviewer_7dQwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Prep-OPD, a two-stage framework that improves on-policy distillation (OPD) by preparing a teacher to continue reasoning from student-generated prefixes rather than relying solely on its independent problem-solving ability. In the first stage, the teacher is trained using GRPO on fixed student-generated prefixes, with final-answer correctness as the reward. In the second stage, the prepared teacher is frozen and used for both bounded trajectory interventions and token-level distillation supervision. Experiments across eight mathematical reasoning benchmarks show that Prep-OPD improves student accuracy over standard OPD and Relay-OPD. Additional experiments demonstrate that student-conditioned teacher preparation improves continuation accuracy, benefits student learning with and without handoffs, and can transfer to a smaller student.

Critical strengths
  1. The paper identifies a limitation of OPD: strong independent teacher performance does not necessarily translate into reliable supervision on student-generated reasoning states. The distinction between problem-solving and continuation capabilities is well motivated and supported by controlled experiments.

  2. Training the teacher directly on student-induced prefixes is an intuitive approach that integrates naturally with existing OPD methods. The prepared teacher can be reused for both trajectory guidance and token-level supervision without adding inference-time overhead.

  3. Experiments cover eight mathematical reasoning benchmarks, two student sizes, and multiple competitive baselines. Controlled comparisons of teacher preparation, handoffs, and teacher roles provide evidence for the method's effectiveness. Three-seed results further support the improvement over Relay-OPD.

Critical weaknesses
  1. The core approach applies standard GRPO to student-generated reasoning prefixes before performing existing OPD with teacher handoffs. While effective, the individual components are established techniques, and the main novelty lies in their combination rather than a fundamentally new learning mechanism.

  2. The efficiency analysis emphasizes student training GPU-hours but excludes the additional 128-step teacher RL preparation stage. Without reporting total end-to-end training cost, it is difficult to determine whether Prep-OPD is more computationally efficient than existing OPD methods.

  3. Teacher preparation uses prefixes from a fixed Qwen3-1.7B student and is evaluated primarily on mathematical reasoning with Qwen3 models. Though reuse on a 0.6B student is promising, the paper does not establish robustness across different model families, stronger students, or non-mathematical reasoning tasks.

Questions for authors

Q1: What is the total computational cost of the 128-step teacher preparation stage? How does Prep-OPD compare with Relay-OPD under matched end-to-end training budgets, including teacher preparation and student distillation?

Q2: Does teacher preparation specifically improve recovery from incorrect reasoning, or does it mainly improve continuation from unfamiliar but otherwise valid prefixes? Could the authors separate these cases and report recovery accuracy for each?

Q3: How sensitive is the prepared teacher to the student model used to generate its training prefixes? Would preparation on Qwen3-0.6B prefixes, mixed student prefixes, or prefixes from different model families yield different results?

Q4: Why is the teacher preparation distribution fixed using the initial student rather than updated as the student improves? Could the prepared teacher become less effective as student-generated reasoning states shift during distillation?

Reviewer_myxoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Prep-OPD, which trains a teacher using GRPO on student-generated reasoning prefixes before on-policy distillation. The prepared teacher then provides trajectory guidance and token-level supervision. Experiments on eight mathematical reasoning benchmarks show improvements over existing methods, including Relay-OPD.

Critical strengths
  1. The teacher comparison in Section 4.3 is convincing. Ordinary RL improves independent problem-solving accuracy but provides little improvement in student-prefix continuation, while the prepared teacher achieves better continuation accuracy. This directly supports the paper's motivation.
  2. The experiments provide useful controls. Replacing the teacher improves both Standard OPD and Relay-OPD on the 1.7B student, and the additional 0.6B and multi-seed results support the main findings.
Critical weaknesses
  1. The necessity and generality of the teacher preparation method remain unclear. The main idea is relatively straightforward: applying GRPO to student-generated prefixes. A simpler SFT baseline using successful teacher continuations is missing, so the advantage of RL preparation is not established. The prepared teacher is also not tested with other competitive distillation methods, such as IW-OPD or FastOPD.
  2. Training dynamics and compute comparisons are insufficiently documented. Appendix A.3 states that the best checkpoint is chosen from eight candidates using a 256-problem development set, but does not explain how the candidates or development problems were selected or show validation curves. The selected checkpoint for the Qwen3-1.7B student GRPO baseline is not reported. Moreover, Appendix A.5 excludes teacher preparation from the reported student-training compute, and there is no longer-trained or compute-matched student GRPO comparison. These omissions limit the interpretation of training stability and end-to-end efficiency.
Questions for authors
  1. How does GRPO-based teacher preparation compare with SFT on successful student-prefix continuations? Does the prepared teacher also improve IW-OPD or FastOPD?
  2. How were the 256 development problems and eight candidate checkpoints selected? Could the authors provide validation curves and the selected checkpoint for the 1.7B student GRPO baseline?
  3. Could the authors report end-to-end compute costs, including teacher preparation, and compare against a longer-trained or compute-matched student GRPO baseline?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.