- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies teacher-side distribution shift in on-policy distillation: although trajectories are generated by the current student, their prefixes may differ substantially from those typically encountered by the teacher. The proposed method, SCOUT, periodically adapts the teacher through reinforcement learning on continuations of student-generated prefixes, while retaining the standard student-side distillation objective. Experiments cover three teacher–student configurations for mathematical reasoning and an additional code-generation setting, supported by analyses of teacher adaptation, update frequency, computational cost, and compatibility with other distillation objectives.
Critical strengths
-
The paper offers a useful perspective on on-policy distillation. The distinction between trajectories being on-policy for the student and off-policy for the teacher is clearly articulated. Treating teacher adaptation as an optimization target provides a meaningful direction beyond filtering or reweighting supervision from a fixed teacher.
-
The method is simple and compatible with existing distillation pipelines. SCOUT retains the student-side objective and adds periodic teacher updates conditioned on student prefixes. Its combination with OPTR produces further improvements on both math and code, supporting the usefulness of teacher adaptation alongside changes to the distillation loss.
-
The empirical improvements are consistent across the evaluated configurations. SCOUT improves average mathematical reasoning accuracy over standard OPD by 2.2, 2.6, and 1.2 points across the three teacher–student pairs, and improves code-generation accuracy from 56.6 to 59.7. Reporting three independent training runs and detailed per-run results strengthens the evidence beyond a single favorable experiment.
-
The analysis includes meaningful controls for alternative explanations. Evaluating the initial and adapted teachers on identical student prefixes helps separate teacher improvement from changes in student quality. The additional-teacher-GRPO control examines whether generic teacher training explains the gains, while the extended-OPD experiment tests whether additional student training on the same data closes the gap. These controls make the empirical argument more convincing.
Critical weaknesses
-
The connection between continuation performance and token-level supervision quality remains indirect. Lower continuation accuracy and higher entropy on longer student prefixes are consistent with teacher-side distribution shift, but they do not isolate it from accumulated reasoning errors or increasingly difficult-to-recover commitments in those prefixes. Similarly, improving final-answer accuracy when the teacher completes a prefix does not directly establish that its next-token supervision becomes more useful for the student. The fixed-prefix control is valuable, but the paper would benefit from analyses stratified by prefix correctness or recoverability, together with a more direct evaluation of supervision quality. This would strengthen the proposed explanation for why teacher adaptation improves distillation.
-
The practical performance–cost advantage is not yet fully established. The improvements over OPD are consistent but moderate, while SCOUT requires additional teacher rollouts and parameter updates. Appendix C.4 reports allocations of 24–32 GPUs for SCOUT, compared with 16 for frozen-teacher OPD. The comparable-compute experiment is useful, but covers only one mathematical reasoning configuration and compares primarily against longer OPD training. Moreover, direct student GRPO achieves a higher mean code-generation score than SCOUT in Table 1 (62.2 versus 59.7). Since SCOUT already requires verifiable rewards, comparisons against OPD combined with student-side outcome-reward updates, under matched compute and verifier budgets, would clarify when adapting the teacher is the preferable investment.
Questions for authors
-
Can the authors directly measure whether teacher adaptation improves token-level supervision, and distinguish this effect from improved recovery after incorrect student reasoning?
-
How does SCOUT compare with combining frozen-teacher OPD and student-side GRPO under matched computational and reward-evaluation budgets?
3 comments
Thank you for your response. I have no further questions, and I will maintain my positive rating.
We’re glad our rebuttal addressed your questions, and we appreciate the time and care you put into reviewing our paper.
Response to Reviewer nWmZ
Thank you for your thoughtful review. We appreciate your positive feedback on how we frame the problem and how SCOUT fits into existing distillation pipelines. We hope our responses below address your remaining concerns.
A1: Thank you for this insightful question.
To address your concern about direct evaluation of supervision quality, we conduct a next-token steering experiment that restricts the teacher's intervention to a single token. Specifically, given a student’s incorrect answer, each teacher steers just one token at its first reflection point, after which the same student generates the remaining response.
Across 676 originally incorrect student trajectories, the SCOUT-adapted teacher achieves a recovery rate of 20.40%, compared to 14.86% with the frozen teacher, an absolute improvement of 5.54%. Since the teacher contributes only a single token rather than a complete continuation, these results provide more direct empirical evidence that teacher adaptation improves the effectiveness of local next-token supervision, beyond merely improving the teacher's own continuation ability.
We kindly refer you to “A2” of the General Response for the detailed experiment design and our full discussion of how we assess supervision quality from direct, supporting, and downstream perspectives.
We would also be happy to examine other measures of supervision quality you would find informative.
A2: Thank you for the valuable questions. Regarding the concern about the cost of SCOUT, we clarify that the computational FLOPs are counted on all GPUs to guarantee a fair comparison. We have added the clarification to Section 5 to avoid misunderstanding.
Regarding your suggestion of a comparable-compute experiment using OPD plus student-side outcome-reward updates (GRPO), we conduct a corresponding experiment to test whether the additional budget is better spent adapting the teacher (SCOUT) or directly training the student (OPD + GRPO).
We kindly refer you to the “A1” in General Response to All Reviewers cell, which details the experiment settings and results. At matched computation, SCOUT outperforms the OPD + GRPO method by +1.2 on math tasks and +2.8 on code tasks on average. The results empirically suggest that it is more cost-effective to spend computation on teacher-adaptation than on student-side RL.
We have added this comparison to Section 5, Analysis (RQ2) of our revised paper.