acceptodds
Under review as a conference paper at ICLR 2027

On the off-policy teacher in On-Policy Distillation

103 views

Abstract

On-policy distillation (OPD) has recently emerged as a promising post-training paradigm in which the student learns from trajectories generated by its own policy under dense teacher supervision. However, OPD introduces a fundamental asymmetry: although the sampled trajectories are on-policy for the student, they are off-policy for the teacher. The teacher is typically optimized to continue from prefixes generated by its own policy, but during OPD it must instead supervise prefixes generated by the student. Empirically, we find that its continuation performance degrades as these prefixes grow longer. To address this issue, we propose Student-COnditioned Updates of the Teacher (SCOUT), a co-training framework that adapts the teacher to student-generated prefixes. Alongside standard OPD updates, SCOUT periodically optimizes the teacher's conditional ability using reinforcement learning with verifiable rewards, where the teacher generates continuations from student prefixes and learns from outcome rewards. Controlled experiments show that SCOUT improves the teacher's ability to continue from student-generated prefixes, supporting the intended mechanism of student-conditioned teacher adaptation. Across multiple teacher–student configurations, model scales, and reasoning domains, SCOUT also consistently improves the effectiveness of on-policy distillation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, On-policy distillation, post-training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_nWmZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies teacher-side distribution shift in on-policy distillation: although trajectories are generated by the current student, their prefixes may differ substantially from those typically encountered by the teacher. The proposed method, SCOUT, periodically adapts the teacher through reinforcement learning on continuations of student-generated prefixes, while retaining the standard student-side distillation objective. Experiments cover three teacher–student configurations for mathematical reasoning and an additional code-generation setting, supported by analyses of teacher adaptation, update frequency, computational cost, and compatibility with other distillation objectives.

Critical strengths
  • The paper offers a useful perspective on on-policy distillation. The distinction between trajectories being on-policy for the student and off-policy for the teacher is clearly articulated. Treating teacher adaptation as an optimization target provides a meaningful direction beyond filtering or reweighting supervision from a fixed teacher.

  • The method is simple and compatible with existing distillation pipelines. SCOUT retains the student-side objective and adds periodic teacher updates conditioned on student prefixes. Its combination with OPTR produces further improvements on both math and code, supporting the usefulness of teacher adaptation alongside changes to the distillation loss.

  • The empirical improvements are consistent across the evaluated configurations. SCOUT improves average mathematical reasoning accuracy over standard OPD by 2.2, 2.6, and 1.2 points across the three teacher–student pairs, and improves code-generation accuracy from 56.6 to 59.7. Reporting three independent training runs and detailed per-run results strengthens the evidence beyond a single favorable experiment.

  • The analysis includes meaningful controls for alternative explanations. Evaluating the initial and adapted teachers on identical student prefixes helps separate teacher improvement from changes in student quality. The additional-teacher-GRPO control examines whether generic teacher training explains the gains, while the extended-OPD experiment tests whether additional student training on the same data closes the gap. These controls make the empirical argument more convincing.

Critical weaknesses
  • The connection between continuation performance and token-level supervision quality remains indirect. Lower continuation accuracy and higher entropy on longer student prefixes are consistent with teacher-side distribution shift, but they do not isolate it from accumulated reasoning errors or increasingly difficult-to-recover commitments in those prefixes. Similarly, improving final-answer accuracy when the teacher completes a prefix does not directly establish that its next-token supervision becomes more useful for the student. The fixed-prefix control is valuable, but the paper would benefit from analyses stratified by prefix correctness or recoverability, together with a more direct evaluation of supervision quality. This would strengthen the proposed explanation for why teacher adaptation improves distillation.

  • The practical performance–cost advantage is not yet fully established. The improvements over OPD are consistent but moderate, while SCOUT requires additional teacher rollouts and parameter updates. Appendix C.4 reports allocations of 24–32 GPUs for SCOUT, compared with 16 for frozen-teacher OPD. The comparable-compute experiment is useful, but covers only one mathematical reasoning configuration and compares primarily against longer OPD training. Moreover, direct student GRPO achieves a higher mean code-generation score than SCOUT in Table 1 (62.2 versus 59.7). Since SCOUT already requires verifiable rewards, comparisons against OPD combined with student-side outcome-reward updates, under matched compute and verifier budgets, would clarify when adapting the teacher is the preferable investment.

Questions for authors
  • Can the authors directly measure whether teacher adaptation improves token-level supervision, and distinguish this effect from improved recovery after incorrect student reasoning?

  • How does SCOUT compare with combining frozen-teacher OPD and student-side GRPO under matched computational and reward-evaluation budgets?

3 comments
AuthorsOpenReview ↗

Response to Reviewer nWmZ

Thank you for your thoughtful review. We appreciate your positive feedback on how we frame the problem and how SCOUT fits into existing distillation pipelines. We hope our responses below address your remaining concerns.

W1 & Q1: The connection between continuation performance and token-level supervision quality is indirect. The paper would benefit from analyses stratified by prefix correctness or recoverability, together with a more direct evaluation of supervision quality. Can the authors directly measure whether teacher adaptation improves token-level supervision, and distinguish this effect from improved recovery after incorrect student reasoning?

A1: Thank you for this insightful question.

To address your concern about direct evaluation of supervision quality, we conduct a next-token steering experiment that restricts the teacher's intervention to a single token. Specifically, given a student’s incorrect answer, each teacher steers just one token at its first reflection point, after which the same student generates the remaining response.

Across 676 originally incorrect student trajectories, the SCOUT-adapted teacher achieves a recovery rate of 20.40%, compared to 14.86% with the frozen teacher, an absolute improvement of 5.54%. Since the teacher contributes only a single token rather than a complete continuation, these results provide more direct empirical evidence that teacher adaptation improves the effectiveness of local next-token supervision, beyond merely improving the teacher's own continuation ability.

We kindly refer you to “A2” of the General Response for the detailed experiment design and our full discussion of how we assess supervision quality from direct, supporting, and downstream perspectives.

We would also be happy to examine other measures of supervision quality you would find informative.

W2 & Q2: The practical performance–cost advantage is not yet fully established. Appendix C.4 reports allocations of 24–32 GPUs for SCOUT, compared with 16 for frozen-teacher OPD. The comparable-compute experiment is useful, but covers only one mathematical reasoning configuration and compares primarily against longer OPD training. Moreover, since SCOUT already requires verifiable rewards, comparisons against OPD combined with student-side outcome-reward updates, under matched compute and verifier budgets, would clarify when adapting the teacher is the preferable investment. How does SCOUT compare with combining frozen-teacher OPD and student-side GRPO under matched computational and reward-evaluation budgets?

A2: Thank you for the valuable questions. Regarding the concern about the cost of SCOUT, we clarify that the computational FLOPs are counted on all GPUs to guarantee a fair comparison. We have added the clarification to Section 5 to avoid misunderstanding.

Regarding your suggestion of a comparable-compute experiment using OPD plus student-side outcome-reward updates (GRPO), we conduct a corresponding experiment to test whether the additional budget is better spent adapting the teacher (SCOUT) or directly training the student (OPD + GRPO).

We kindly refer you to the “A1” in General Response to All Reviewers cell, which details the experiment settings and results. At matched computation, SCOUT outperforms the OPD + GRPO method by +1.2 on math tasks and +2.8 on code tasks on average. The results empirically suggest that it is more cost-effective to spend computation on teacher-adaptation than on student-side RL.

We have added this comparison to Section 5, Analysis (RQ2) of our revised paper.

Thank you for your response. I have no further questions, and I will maintain my positive rating.

We’re glad our rebuttal addressed your questions, and we appreciate the time and care you put into reviewing our paper.

Reviewer_bwKCOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies teacher-side distribution shift in on-policy distillation: a teacher must supervise prefixes generated by a different policy. SCOUT periodically updates the teacher with outcome-reward GRPO on continuations from student prefixes, while retaining the usual student OPD update. Across three math teacher–student pairs, it improves mean accuracy over frozen-teacher OPD by 1.2–2.6 points; the code-generation gain is 3.1 points. Analyses examine teacher adaptation, update frequency and training compute.

Critical strengths
  1. Adapting the teacher to student states is a useful direction, and the controls support it. Figure 3 holds student prefixes fixed when evaluating teacher improvement. Table 4 compares against teacher GRPO without student-prefix conditioning. Together, these provide evidence beyond simply training a stronger teacher.
  2. Improvements over OPD recur across three teacher–student pairs and both math and code. Three independent training runs, detailed per-run results, the longer-OPD compute comparison and the OPTR combination make the empirical case reasonably convincing (Tables 1–5; Appendix D).
Critical weaknesses
  1. The comparison with alternatives using the same feedback remains incomplete. SCOUT adds verifiable rewards and substantial teacher-training compute; Figure 5 matches compute only against longer OPD. A fixed-teacher OPD method combined with student-side reward optimization would test whether that budget is better spent adapting the teacher. This matters particularly on code, where plain GRPO has a higher reported mean than SCOUT, 62.2 versus 59.7 (Table 1), although the uncertainty does not establish a clear difference. The results support improvement over frozen-teacher OPD more strongly than the choice of where to spend the additional reward-based training budget.
  2. The claimed generalization across model families is broader than the evaluated backbone diversity (Contributions; Section 4.2). All tested models are Qwen-derived. Skywork-OR1-Math-7B is further trained from DeepSeek-R1-Distill-Qwen-7B, and its student is DeepSeek-R1-Distill-Qwen-1.5B; both derive from Qwen2.5-Math (Skywork report, Section 8.1; DeepSeek model card). The results support transfer across Qwen generations and post-training recipes, but provide limited evidence for generalization to unrelated backbone families.

The teacher-adaptation idea is useful, and the controlled experiments support consistent gains over OPD. The missing reward-aware compute comparison limits the practical conclusion, but does not negate the demonstrated contribution.

Questions for authors
  1. Under matched total compute and verifier access, how does SCOUT compare with fixed-teacher OPD plus student-side GRPO?
  2. On fixed student prefixes, is there evidence that teacher adaptation improves the local distillation targets, beyond improving final-answer accuracy after a teacher continuation? Figures 1 and 3 measure recovery ability more directly than the quality of next-token supervision.
  3. Can the authors narrow the cross-family contribution claim to the evaluated Qwen-derived lineages, or provide evidence using an unrelated backbone family?
6 comments
AuthorsOpenReview ↗

Response to Reviewer bwKC (1/2)

Thank you for your thoughtful review. We are glad you found teacher adaptation a useful direction and appreciated the control experiments and the consistency of our results. We hope our responses below address your remaining concerns.

W1 & Q1: The comparison is incomplete. A fixed-teacher OPD method combined with student-side reward optimization would test whether that budget is better spent adapting the teacher. Under matched total compute and verifier access, how does SCOUT compare with fixed-teacher OPD plus student-side GRPO?

A1: Thank you for this insightful suggestion. We conduct an experiment combining OPD with student-side GRPO to investigate whether the computational budget is better spent adapting the teacher (SCOUT) or directly training the student (OPD + GRPO).

We kindly refer you to our “A1” in the “General Response to All Reviewers” for detailed experiment settings and results. At matched computation, SCOUT outperforms the OPD + GRPO method by +1.2 on math tasks and +2.8 on code tasks on average. These results provide empirical evidence that allocating computation to teacher adaptation is more effective than student-side RL in our evaluated settings.

We have added this comparison to Section 5, Analysis (RQ2) of our revised paper.

W2 & Q3: All tested models are Qwen-derived (Skywork-OR1-MATH-7B and DeepSeek-R1-Distill-Qwen-1.5B derive from Qwen2.5-Math). Can the authors narrow the cross-family contribution claim to the evaluated Qwen-derived lineages, or provide evidence using an unrelated backbone family?

A2: Thank you for highlighting the shared Qwen lineage among our evaluated models, which may limit the evidence for SCOUT's generalizability across model families. To address this concern, we conduct additional experiments using a model pair from the Llama family, independent of the Qwen family.

Specifically, we use Llama-3.1-Nemotron-Nano-4B-v1.1 as the teacher, and Llama-3.2-3B-Instruct as the student. Importantly, these two models are separately derived from Llama3.1-8B, so they share the vocabulary while retaining necessary model diversity. We adopt the same training and evaluation recipe as in other math domain experiments in our paper.

The results are presented below:

MethodAIME24AIME25AMC23HMMTOlympiadMATH500MEAN6
Llama-3.2-3B-Instruct2.0 ± 0.40.1 ± 0.112.0 ± 0.90.3 ± 0.28.7 ± 0.629.1 ± 0.88.7 ± 0.2
Nemotron-Nano-4B-v1.18.8 ± 0.79.2 ± 1.046.4 ± 1.03.7 ± 0.541.1 ± 1.174.0 ± 0.430.5 ± 0.3
OPD seed16.4 ± 0.70.6 ± 0.330.9 ± 0.90.4 ± 0.224.1 ± 1.055.6 ± 0.819.6 ± 0.3
OPD seed27.2 ± 0.61.0 ± 0.432.2 ± 1.00.2 ± 0.123.9 ± 0.156.1 ± 0.420.1 ± 0.2
OPD seed37.3 ± 0.52.3 ± 0.432.2 ± 1.10.2 ± 0.124.7 ± 0.355.8 ± 0.320.4 ± 0.2
OPD mean (n=3)6.9 ± 0.51.3 ± 0.931.7 ± 0.80.3 ± 0.124.2 ± 0.455.8 ± 0.320.1 ± 0.4
SCOUT seed19.9 ± 0.63.0 ± 0.640.9 ± 1.01.0 ± 0.431.9 ± 0.664.5 ± 0.525.2 ± 0.3
SCOUT seed210.2 ± 0.52.3 ± 0.639.2 ± 1.01.2 ± 0.331.0 ± 0.563.0 ± 0.624.5 ± 0.3
SCOUT seed39.3 ± 0.81.9 ± 0.437.0 ± 0.90.5 ± 0.229.1 ± 0.461.2 ± 0.323.2 ± 0.2
SCOUT mean (n=3)9.8 ± 0.52.4 ± 0.639.0 ± 2.00.9 ± 0.330.7 ± 1.462.9 ± 1.724.3 ± 1.0

SCOUT consistently outperforms OPD across all six math benchmarks, improving average accuracy from 20.1 to 24.3 (+4.2). These results provide further empirical evidence that SCOUT's effectiveness extends beyond Qwen-derived models to the Llama family.

We have included the experiment results in Table 6 and Table 24 of our revised paper.

AuthorsOpenReview ↗

Response to Reviewer bwKC (2/2)

Q2: On fixed student prefixes, is there evidence that teacher adaptation improves the local distillation targets, beyond improving final-answer accuracy after a teacher continuation? Figures 1 and 3 measure recovery ability more directly than the quality of next-token supervision.

A3: Thank you for the insightful question. To address your concern, we provide additional evidence of improved next-token supervision quality, supported by the following analyses.

  1. Direct evidence: Next-token steering demonstrates improved supervision quality.

We design a next-token steering experiment to isolate the effect of local teacher supervision from the teacher's ability to generate a complete continuation. Specifically, given a student-generated incorrect answer, a teacher modifies only a single token, after which the student continues to complete the new answer. We evaluate how effectively the next-token intervention can steer the student toward correct answers.

Across 676 originally incorrect student trajectories, the SCOUT teacher achieves a 20.40% recovery rate, 5.54% more than the frozen teacher’s 14.86% recovery rate. These results provide direct empirical evidence that teacher adaptation improves the effectiveness of local next-token supervision, beyond merely improving the teacher's continuation ability. Please refer to “A2” in our “General Response to All Reviewers” for the detailed experiment setup and complete results.

  1. Supporting evidence: Teacher adaptation reduces uncertainty on student-generated prefixes.

We repeat the entropy analysis in Figure 1 using the SCOUT-adapted teacher. As shown in Figure 4 of our revised paper, the adapted teacher exhibits substantially lower entropy on student-generated prefixes, with its entropy curve approaching that observed on its own generated responses. This suggests that SCOUT mitigates the off-policy teacher issue by making the teacher more confident on student-generated contexts, providing further supporting evidence for more reliable local supervision.

  1. Downstream evidence: Improved teacher supervision translates into better student performance.

We clarify that SCOUT and standard OPD use the same student training data, distillation objective, and number of student update steps. The key difference lies in the source of supervision. Under this controlled comparison, SCOUT consistently improves average accuracy, suggesting better supervision quality from the adapted teacher.

Thanks for the additional experiments. The next-token steering result is helpful. Just one question, how is the intervention token selected? Also since OPD learns from the teacher's next-token distribution rather than a single token, I'm still not quite sure how much this result can tell about the quality of the distillation signal itself.

Reviewer_xF7jOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies the problem of on-policy distillation (OPD) where a student collects trajectory and is trained on the answers from the teacher. The problem in OPD is that gradually the teacher may become very off-policy with respect to the trajectories generated from the student -- it sees the prompts / prefixes from the student which might not match a distribution which was used to train the teacher. It empirically may result in a decrease in performance.

This paper basically fixes this problem by slowly updating the teacher on the prefixes generated from the student and teacher own responses by optimizing a verifiable reward function. This allows the teacher to be less off-policy compared to the student.

The results in the paper provide consistent improvements in performance and the method is simply combined with other approaches.

Critical strengths

The authors identify the problem and provide analysis / diagnostic of it in Figure 1.

The authors propose a simple method to improve / modify on-policy distillation framework by updating the teacher.

The results are very consistent. There is a consistent performance improvement.

Table 4 demonstrates that the gain does not come only from improving teacher itself, but from using their method.

I really like Figure 4 -- it suggests there is a good regime where this method helps the best over OPD.

Critical weaknesses

I think paper is overall easy to follow and well-written. I think the authors need to clearly discuss the limitations of the method:

  1. First, is the ability to finetune the teacher, which is not always the case for OPD

  2. Availability of the verifiable reward for the teacher finetuning. This makes the OPD scenario broader and authors need to clearly discuss in which regimes this assumption is realistic.

I think Table 4 demonstrates that the gain comes from their method rather than from improving the teacher. However, I wonder if they can also add another comparison where they do OPD and do student GRPO with a well-tuned reward weight. In a way, when you do both RL + OPD, one could argue that the negative effects of the teacher off-policyness will be less drastic since there is an additional reward signal from the reward function. Also, this comparison is justified because the authors already assume the availability of the reward function.

Questions for authors

See above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer xF7j

Thank you for your thoughtful and constructive review. We appreciate your recognition of the simplicity of our method and the consistency of the improvements across experiments. We hope our responses below address your concerns.

W1: The authors need to clearly discuss the limitations of the method.

A1: Thank you for the valuable suggestion, which helps us better clarify the scope of our method. We acknowledge that fine-tuning the teacher is not always feasible for OPD due to limited access to teacher model parameters or insufficient devices to support teacher model training. We also agree that the availability of verifiable rewards limits the teacher adaptation of SCOUT. However, recent advances in RL offer promising directions for extending teacher adaptation to less readily verifiable domains, potentially relaxing this limitation in future work.

We have discussed both limitations in Appendix A: Limitations and Future Work in our revised paper.

W2: I wonder if they can also add another comparison where they do OPD and do student GRPO with a well-tuned reward weight.

A2: Thank you for raising this insightful question. We agree that directly applying the additional reward signal to the student model is an important comparison. Accordingly, we conduct an experiment combining OPD with student-side GRPO on both math and code tasks, with a carefully picked reward weight.

We kindly refer you to our “A1” in the “General Response to All Reviewers” cell, which details the experiment settings and results. At matched student update steps, SCOUT outperforms the OPD + GRPO method by +0.9 on math tasks and +1.9 on code tasks on average. The advantage extends to +1.2 and +2.8 when restricting their training computation at the same level. The results empirically suggest that it is more cost-effective to spend computation on teacher-adaptation than on student-side RL.

We have added this comparison to Section 5, Analysis (RQ2) of our revised paper.

Thank you for the rebuttal.

I have no more concerns and I maintain my current score.

We’re glad our rebuttal addressed your questions, and we appreciate the time and care you put into reviewing our paper.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers (1/2)

Dear Reviewers,

We sincerely appreciate the time and effort you have dedicated to reviewing our paper. Your insight is invaluable in refining our work. Many of your positive feedbacks inspired us, such as,

  1. A useful direction for OPD and a simple method. We appreciate the reviewers’ recognition of adapting the teacher to address the off-policy teacher problem as a “useful” and “meaningful” direction (Reviewers bwKC and nWmZ). We are also glad that Reviewers xF7j and nWmZ found our method simple and easy to combine with existing distillation approaches.
  2. Consistent improvements supported by solid experiments. Reviewers highlighted the consistency of our gains over standard OPD across the evaluated math and coding settings (Reviewers xF7j, bwKC, and nWmZ). Three independent training runs and detailed per-run results make it “reasonably convincing”. (Reviewers bwKC and nWmZ)
  3. Meaningful controls and analysis. All three reviewers highlighted our teacher-training control to show where SCOUT helps the most over OPD, which provides “evidence beyond simply training a stronger teacher” (Reviewer bwKC).

These encouraging comments not only acknowledged our work but also motivated us to refine the paper.

The reviewers' concerns and questions also prompted us to dig deeper into the topic. Below, we address some common concerns raised across reviews. Our individual responses to each reviewer are self-contained and address their specific concerns point by point.

Q1. Given access to verifiable rewards, how does adapting the teacher (SCOUT) compare with applying those rewards directly to the student through GRPO while keeping the OPD teacher frozen? (Reviewers xF7j, bwKC, and nWmZ)

A1. We thank the reviewers for suggesting this important comparison. Since SCOUT uses verifiable rewards to adapt the teacher, a natural question is whether applying the same feedback directly to the student would be equally effective. We thus compare SCOUT with a baseline that keeps the teacher frozen and jointly optimizes the student’s OPD and GRPO losses on math and code tasks with Qwen3-4B-Instruct-2507 → Qwen3-1.7B model pair.

Selecting the student-side reward weight. Following prior work [1, 2], we jointly optimize the student’s OPD and GRPO losses using L_hybrid = L_OPD + α L_GRPO, with eight student rollouts per prompt. We sweep α from {0.25, 0.5, 1, 4, 8, 16} and select the best-performing value. Due to limited computational resources, the experiments are conducted only on the Math task. The brief results are shown below, and the complete data are presented in Appendix C.5, Table 12 in our revised paper.

Exp 1, search for the best α:

GRPO weight α0.250.514816
MEAN649.4650.0949.9950.1849.9949.16

The average accuracy across the six math benchmarks (MEAN6) varies only slightly across the weight sweep. We select α = 4, which achieves the best observed average performance, for the following comparison experiments.

Matched-FLOPs and matched-step comparisons. We compare SCOUT and OPD + GRPO at matched total training FLOPs. We also report a separate comparison at the same number of student update steps. For both comparisons, we report the mean and standard deviation across three independent seeds.

Exp 2.a, Math:

MethodAIME24AIME25AMC23HMMTOlympiadMATH500MEAN6
OPD33.8 ± 3.024.3 ± 1.873.9 ± 1.916.0 ± 0.860.2 ± 1.687.0 ± 1.149.2 ± 1.5
OPD + GRPO, matched FLOPs34.7 ± 0.624.4 ± 0.876.0 ± 0.817.8 ± 0.560.7 ± 0.687.3 ± 0.250.2 ± 0.3
OPD + GRPO, matched steps35.6 ± 0.826.6 ± 1.175.7 ± 1.517.5 ± 0.260.1 ± 0.987.3 ± 0.250.5 ± 0.3
SCOUT35.8 ± 0.528.8 ± 0.675.8 ± 0.617.2 ± 1.962.4 ± 0.888.4 ± 0.351.4 ± 0.6

Exp 2.b, Code:

MethodLCB v5HE+MBPPMEAN3
OPD37.6 ± 0.169.7 ± 0.762.5 ± 0.356.6 ± 0.3
OPD + GRPO, matched FLOPs37.5 ± 0.467.0 ± 0.963.3 ± 0.456.9 ± 0.3
OPD + GRPO, matched steps39.0 ± 0.370.9 ± 0.163.5 ± 0.157.8 ± 0.1
SCOUT40.6 ± 2.573.4 ± 0.965.2 ± 2.659.7 ± 1.9

SCOUT achieves higher average accuracy on both math and code under both comparison settings. Incorporating student-side GRPO improves frozen-teacher OPD and narrows the performance gap with SCOUT. Nevertheless, SCOUT maintains an advantage of 1.2 points on math and 2.8 points on code at matched training FLOPs, and 0.9 and 1.9 points at matched student update counts.

AuthorsOpenReview ↗

General Response to All Reviewers (2/2)

Q2: Does teacher adaptation improve the supervision the student receives during on-policy distillation? (Reviewers bwKC and nWmZ)

A2: We thank reviewers for raising this important question. To address this concern, we provide evidence that teacher adaptation improves supervision quality from three complementary perspectives.

  1. Direct evidence from the single-token steering experiment.

We conduct a next-token steering experiment to test whether a teacher can help the student correct an incorrect solution through a single-token intervention.

For each teacher, we identify the earliest position on the student’s original trajectory where the teacher’s top-1 prediction is a reflection token and none of the student’s top-5 predictions is a reflection token. At that position, we replace the student’s next token with the teacher’s top prediction and let the same student generate the rest of the solution. This separates the effect of the teacher’s one-token intervention from its ability to recover by generating the full continuation. We then measure how often the student reaches the correct answer.

We use the reflection token list from Relay-OPD [3], which includes tokens such as “Wait,” “But,” and “Hmm.”

On the math reasoning evaluation set, we collect 676 incorrect student answers, and experiment with the fixed and trained teacher models.

MetricFixed TeacherSCOUT-Trained Teacher
Correction Rate14.86%20.40%

By steering just one token, the SCOUT-trained teacher corrects about 20.40% of wrong student answers, 5.54% more than the fixed teacher. This experiment provides strong evidence that student-context-conditioned training improves the teacher’s supervision quality.

  1. Supporting evidence from teacher uncertainty on student-generated prefixes.

We repeat the entropy analysis in Figure 1 using the SCOUT-adapted teacher: compare the teacher’s next-token entropy on its own prefixes and on student-generated prefixes. As shown in Figure 4 of our revised paper, before adaptation, the teacher’s entropy decreases as it progresses through its own solutions but remains high on longer student-generated prefixes. After adaptation, the teacher’s entropy also decreases at later positions on student-generated solutions, narrowing the gap between the two curves. This indicates that the adapted teacher is more confident about its next-token predictions on student-generated contexts, providing further supporting evidence for more reliable local supervision.

  1. Downstream evidence from SCOUT's consistent performance improvements.

SCOUT and standard OPD use the same student training data, distillation objective, and number of student update steps, with the key difference being whether the teacher is adapted. SCOUT consistently achieves higher average accuracy than standard OPD across all evaluated settings, suggesting that the adapted teacher provides more effective supervision.

Reference:

[1] Xu, H., et al. (2025). KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning. arXiv:2506.02208.

[2] Lin, W., et al. (2026). On-policy Distillation with Verifiable Reward. arXiv:2608.24696.

[3] Xu, H., et al. (2026). Pass the Baton: Trajectory-Relayed On-Policy Distillation. arXiv:2607.26057.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.