- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper tackles a fairly practical problem in long-sequence on-policy distillation (OPD): the longer the student's self-generated prefix gets, the more it drifts away from the reasoning path the teacher would have written on its own. So the teacher's token-level feedback on later tokens degrades, and updating on that feedback reinforces the very deviations.
The authors propose ATI-OPD: split the student response into 128-token segments and score teacher-student disagreement per segment with a clipped K3 estimator; when the first segment's mean crosses a threshold, trigger an intervention — throw away that segment and everything after it, have the teacher write a short replacement (at most 128 tokens) from the retained prefix, then let the student continue generating from this new starting point. After generation, scan the new continuation in 64-token windows; if two consecutive windows both exceed an adaptive threshold (dynamically calibrated as the 90th percentile of a pool built from originally-correct responses), cut off the distillation loss there.
Critical strengths
The good parts first. The problem is real, and the paper documents the instability phenomena (gap growing with position, entropy rebound, length inflation) very solidly — the motivation section reads well.
The appendix is unusually meticulous about mask boundaries, paired-window statistics, and even "which numbers cannot be combined with which." The ablations are thorough: OPD vs SFT vs Masked on the teacher segment, Full vs Prune, teacher segment length L_T, trigger threshold τ, skip-incorrect — each piece gets taken apart.
Critical weaknesses
The core one: the headline gains are small. Roughly +1.2 points on average, with one training run per configuration and zero information on cross-seed variance, so a reader cannot tell whether the improvement is outside training noise.
Second, the method is essentially a combination of existing parts — segment-level divergence triggering, bounded teacher takeover (the Relay-OPD / speculative KD line), and windowed cutoffs— so novelty rests mostly on the specific recipe (earliest-segment trigger + short replacement + cutoff calibrated on correct responses).
Also, cost is never quantified: a full student rollout + teacher scoring + teacher decoding + full regeneration — how much more expensive per step than vanilla OPD (wall-clock, token counts) is nowhere given in a table. Scope-wise it's math-only, essentially 1.5B, two pairs; and AMO / HMMT scores are so low (around 2.7% and 24%) that those two tasks contribute more form than substance to the average.
Questions for authors
see the weakness above
3 comments
W3: The contribution of combining trajectory repair with supervision-range selection for subsequent student continuations.
We thank the reviewer for pointing out the connections between our work and existing methods and for suggesting a more precise characterization of our contribution. We acknowledge that disagreement detection, brief teacher takeover, and selective supervision each have precedents. Relay-OPD [1] already includes bounded teacher segments and student continuation when the budget permits; AC-OPD [3] retains the original student trajectory and selects the effective supervision length of an appended teacher branch; Prune-OPD [2] adjusts supervision weights and rollout budgets for subsequent training steps; KAT [4] terminates rollouts early based on sustained low-KL agreement.
We do not present each component as being introduced for the first time. The specific question addressed in this paper is: after repairing a bounded segment in an already deviated student trajectory, how far should supervision extend into the regenerated student continuation? To this end, we perform a bounded replacement at the first high-disagreement segment, allow the student to regenerate its continuation from the repaired prefix, and then calibrate a threshold using original correct responses to select the supervision range for this student continuation. This specific design concerns the trajectory being repaired, the regenerated continuation, and the range of tokens receiving supervision; we position our contribution as this design and its validation through long-sequence behavior.
To further distinguish the effects of supervision selection and trajectory repair, we added a Selection-only ablation. Selection-only retains the original student trajectory without teacher generation or student continuation, and selects the supervision range within the student continuation after the original triggering segment ends. This differs from the Masked ablation in the paper: Masked still inserts a teacher segment and regenerates the subsequent student trajectory, but does not directly compute loss on teacher tokens.
| Method | AIME24 | AIME25 | AMC23 | AMO-Bench | HMMT | Average |
|---|---|---|---|---|---|---|
| OPD | 47.29 | 36.46 | 88.13 | 3.69 | 22.92 | 39.70 |
| Selection-only | 47.71 | 36.04 | 87.19 | 2.72 | 22.54 | 39.24 |
| ATI-OPD | 50.00 | 37.29 | 87.66 | 2.08 | 23.86 | 40.18 |
In this run, Selection-only's average accuracy is 0.46% lower than OPD's, while ATI-OPD's is 0.94% higher than Selection-only's.
We further compared performance on long responses:
| Method | Proportion of Long Incorrect Responses | Proportion of Long Correct Responses |
|---|---|---|
| OPD | 26.99 | 3.71 |
| Selection-only | 24.71 | 2.58 |
| ATI-OPD | 24.96 | 3.65 |
Selection-only also reduces long incorrect responses, but it simultaneously reduces long correct responses and the coverage of successful solutions across multiple samples. Compared with Selection-only, ATI-OPD has a 1.07% higher proportion of long correct responses, with a 95% paired confidence interval of [+0.22%, +1.97%]. Meanwhile, pass@16 increases from 54.06% to 59.48%, a gain of 5.42%, with a 95% CI of [+1.18%, +9.95%]. This further shows that both methods reduce long incorrect responses, while ATI-OPD, which incorporates trajectory repair, preserves more long correct responses and greater coverage of solved problems. This supports distinguishing between the roles of “selecting only the supervision range on the original trajectory” and “selecting the supervision range after repairing the trajectory” in ATI-OPD.
W4: Computational costs of ATI-OPD and OPD, and their performance under the same computational resources.
We thank the reviewer for suggesting a complete accounting of computational costs. We compare the arithmetic workload of the two methods by estimating model FLOPs, including the initial student rollout, student continuation, teacher generation and scoring, actor log-probability recomputation, and forward and backward passes, as well as student generations that are ultimately discarded. We also count input, cached input, and output tokens during training.
The table below reports cumulative costs over training. FLOPs are measured in units of , and token counts are in millions (M).
| Method | Estimated FLOPs | Student Input | Student Input (Cached) | Student Output | Teacher Input | Teacher Input (Cached) | Teacher Output | Cumulative Tokens Processed in Actor Updates | Average Performance |
|---|---|---|---|---|---|---|---|---|---|
| OPD | 12.94 | 6.277 | 0.629 | 448.484 | 909.521 | 0 | 0 | 454.761 | 39.70 |
| ATI-OPD | 12.76 | 65.328 | 57.055 | 504.607 | 1,054.823 | 37.508 | 1.720 | 428.128 | 40.18 |
As shown above, ATI-OPD improves task accuracy by 0.48%, reduces FLOPs by approximately 1.41%, and reduces the cumulative number of tokens processed in actor updates by 5.86%, with the average number of tokens processed per actor update decreasing from 2.096M to 1.973M. This improvement primarily results from shorter final training trajectories, consistent with the reduced proportion of extremely long responses observed in Figure 3 of the paper. Therefore, under similar estimated model FLOPs, ATI-OPD achieves a modest accuracy gain while reducing the number of tokens processed in actor updates.
W5: Retaining difficult tasks while disclosing where gains are concentrated and reporting negative results.
We thank the reviewer for drawing attention to absolute performance on difficult tasks and the interpretation of average scores. We agree that the low absolute scores on AMO-Bench and HMMT limit the conclusions that can be drawn about reasoning ability on these tasks. We retain these tasks to present the limits of the method's capabilities and explicitly report results showing either no improvement or a decline.
As a sensitivity analysis, in the additional experiments with multiple seeds, we exclude AMO-Bench and HMMT and compute an equally weighted average over AIME24, AIME25, and AMC23 only. The average gain of ATI-OPD over OPD is +2.16%. Therefore, the positive average change is not driven by the two low-scoring tasks.
[1] Xu H, Xu X, Hong H, et al. Pass the baton: Trajectory-relayed on-policy distillation[J]. arXiv preprint arXiv:2607.26057, 2026. [2] Yang Z, Guo Z, Song Y, et al. Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning[J]. arXiv preprint arXiv:2605.07804, 2026. [3] Zhou J, Zhou Y, Guo H, et al. Teacher Should Think Ahead: Adaptive Continuations for Reliable On-Policy Distillation[J]. arXiv preprint arXiv:2609.22254, 2026. [4] Xin H, Zhao A, Sun Y, et al. Escaping the KL Agreement Trap in On-Policy Distillation[J]. arXiv preprint arXiv:2606.09471, 2026.
We sincerely thank reviewer #NNrz for the careful and constructive feedback. Our responses are as follows.
We thank the reviewer for reminding us to accurately characterize the practical significance of the performance gains. Beyond average accuracy, we further analyzed response length and correctness during independent model inference. Following the original 12,000-token threshold, we define responses whose full length exceeds this threshold as long responses and the remaining responses as short, and divide them into four categories based on the final evaluation results.
Compared with OPD, ATI-OPD changes the proportion of long incorrect responses by -2.74%, with a 95% paired confidence interval of [-4.07, -1.44]; meanwhile, the proportion of short incorrect responses increases by 1.77%, with a 95% confidence interval of [+0.50, +3.06]. The proportion of long correct responses remains essentially unchanged, while the proportion of short correct responses increases by 0.98%.
Therefore, the more direct behavioral benefit of ATI-OPD is that it produces fewer lengthy responses that are ultimately judged incorrect, alongside an observed modest gain in average accuracy.
We thank the reviewer for highlighting the importance of repeated training runs and statistical validation. Under the JustRL-DeepSeek-1.5B teacher -> DeepSeek-R1-Distill-Qwen-1.5B student setting, we conducted two additional training runs, with seed=0 and 42, for each of OPD and ATI-OPD. Both methods were evaluated on the same five benchmarks, with 16 samples per problem. The table below summarizes the mean accuracy across the two additional runs, with standard deviations in subscripts; Average denotes the mean across the five tasks.
The average gains of ATI-OPD over OPD in the two runs were +1.47% and +0.48%, respectively, with a mean gain of +0.98% and a sample standard deviation of 0.70%. Both additional runs showed a positive trend in average gains, indicating that this trend is not confined to the original single training run.