- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3(+2)
- Soundness
- 3(+1)
Read review
Paper summary
This paper argues that in the on-policy distillation process, a pre-query routing technique analyzing only the student-generated rollouts can dramatically reduce teacher computation costs. When a student model generates multiple rollouts for a single problem, it selects only mixed-outcome prompts where both correct and incorrect answers are present. Hidden-state trajectories are aligned along a normalized progress axis to measure the persistent-departure area (PDA) of a failed rollout from the trajectory of a successful one. Measured by the area relative to the estimated teacher-input tokens, the method selects the single failed answer with the highest efficiency. The student is then updated by calculating token-level advantages under the guidance of the teacher model only for the selected answer. Consequently, SR-OPD maintains a reasoning performance comparable to Vanilla-OPD across three teacher-student pairs and six mathematical benchmarks while using only about 3% to 5% of the teacher-input tokens required by Vanilla-OPD.
Critical strengths
- Impressive Performance-Cost Trade-off: The method achieves remarkable efficiency gains, maintaining reasoning performance while consuming only a minimal fraction of tokens.
- High Exensibility: The method is easy to plug into existing pipelines.
Critical weaknesses
- High Background Dependency: The paper's structure assumes a strong familiarity with advanced on-policy distillation making it unfriendly to readers without prior context.
- Omission of Training Details and Hyperparameters: The paper and its appendix lack essential training details such as learning rates, optimizer configurations, maximum token lengths, etc. Omitting these critical hyperparameters severely hinders reproducibility.
- Narrow Domain Generalization: Evaluation is exclusively restricted to mathematical reasoning benchmarks
Questions for authors
Question on Base Model Performance and Distillation Efficacy: Table 1 compares various routing methods (Vanilla OPD, TA-OPD, TLR-OPD, and SR-OPD), but it lacks the performance baseline of the initial student model prior to on-policy distillation, as well as the performance of the teacher model. Could authors provide the zero-shot reasoning performance of the base student models across the six benchmarks?
Question on System-Level Overhead and Wall-Clock Time: While the paper successfully demonstrates a dramatic reduction in teacher-input tokens, the overall pipeline introduces additional student-side overhead such as generating multiple rollouts, extracting hidden states, and computing trajectory alignments. How much do the student-side routing computations offset the computational savings gained from reducing teacher queries?
2 comments
Thanks to the authors for additional experiments about cross-domain generalization and base models. As a result of the authors’ response, I have revised the soundness and presentation score.
Author Response to Reviewer f82c
We thank the reviewer for recognizing SR-OPD's efficiency and ease of integration. We address each concern below.
W1. OPD Background and Clarity
Like on-policy RL, OPD often samples multiple rollouts per prompt from the current student. For a math problem, the student generates K solutions. The teacher provides token-level probability distributions for these student-generated reasoning contexts. In our Vanilla OPD setup, all K rollouts receive teacher supervision, which is costly. SR-OPD routes rollouts before teacher queries: it first keeps prompts with both correct and incorrect answers, then selects one failed rollout using persistent-departure area (PDA) relative to teacher-input cost. Both methods use K=4; SR-OPD needs no extra rollouts. We will add a workflow figure.
W2. Training Details
We will add the missing settings:
W3. Cross-Domain Generalization
We evaluated the math-trained Qwen 8K checkpoint on GPQA-Diamond (science), IFEval (instruction following), and HumanEval (code). Values are Avg@16 / Pass@16 (%); IFEval uses prompt-level strict accuracy.
Against the initial student, SR-OPD improves Avg@16 by 5.37 points on GPQA-Diamond and 1.16 points on IFEval. HumanEval Avg@16 is unchanged, while Pass@16 rises by 1.22 points. Compared with Vanilla OPD, SR-OPD is similar on GPQA-Diamond and IFEval but lower on HumanEval. Performance on these non-math tasks is thus maintained relative to the initial student.
Q1. Initial Models and Distillation Results
We combine initial teacher/student baselines with distillation results. Scores are Avg@16 (%); Avg. is the six-benchmark mean. Input is teacher-input tokens in millions (M), with ratios relative to Vanilla OPD; – means not applicable.
SR-OPD closely matches Vanilla OPD using only 3–5% of its teacher-input tokens.
Q2. System Overhead and Wall-Clock Time
SR-OPD adds hidden-state alignment, similarity calculation, and PDA scoring. For fixed rollout count and available hidden states, scoring scales approximately linearly with compared tokens and needs no additional teacher queries.
Qwen pair on 8×H20:
Including routing overhead, SR-OPD reduces total GPU compute by 15.6% and end-to-end time by 12.7%. Student Actor rollout generation remains the main time cost. Moving some teacher resources to the Actor, subject to memory needs, may improve throughput. We will clarify teacher-query, routing, and end-to-end costs in the revision.
We thank the reviewer for the helpful comments.