acceptodds
Under review as a conference paper at ICLR 2027

ATI-OPD: Adaptive Teacher Intervention for Stable Long-Sequence On-Policy Distillation

10 views

Abstract

On-policy distillation (OPD) trains a student on its own responses using token-level feedback from a teacher. As responses grow longer, the student's prefix can depart from the teacher's own reasoning, weakening its feedback on later tokens. Updates based on this feedback can reinforce the same deviations in later responses and destabilize training. Prior approaches such as length control and loss reweighting limit learning from problematic responses but leave their prefixes unchanged. Fixed cutoffs can also discard later reasoning. We propose Adaptive Teacher Intervention for On-Policy Distillation (ATI-OPD) to improve subsequent teacher guidance by revising the student's reasoning prefix. ATI-OPD replaces the earliest high-disagreement segment with a short teacher segment, then lets the student resume its own reasoning from the revised text. To handle renewed disagreement, it excludes persistently divergent suffixes from the distillation loss while retaining useful student continuation. Training diagnostics show less teacher-student disagreement, fewer gradient spikes, and fewer extremely long responses. Across five mathematical reasoning benchmarks and two teacher-student pairs, ATI-OPD improves average student-only accuracy over standard OPD.

Primary area:
reinforcement learning
Keywords:
large language models, knowledge distillation, training stability

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_NNrzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tackles a fairly practical problem in long-sequence on-policy distillation (OPD): the longer the student's self-generated prefix gets, the more it drifts away from the reasoning path the teacher would have written on its own. So the teacher's token-level feedback on later tokens degrades, and updating on that feedback reinforces the very deviations.

The authors propose ATI-OPD: split the student response into 128-token segments and score teacher-student disagreement per segment with a clipped K3 estimator; when the first segment's mean crosses a threshold, trigger an intervention — throw away that segment and everything after it, have the teacher write a short replacement (at most 128 tokens) from the retained prefix, then let the student continue generating from this new starting point. After generation, scan the new continuation in 64-token windows; if two consecutive windows both exceed an adaptive threshold (dynamically calibrated as the 90th percentile of a pool built from originally-correct responses), cut off the distillation loss there.

Critical strengths

The good parts first. The problem is real, and the paper documents the instability phenomena (gap growing with position, entropy rebound, length inflation) very solidly — the motivation section reads well.

The appendix is unusually meticulous about mask boundaries, paired-window statistics, and even "which numbers cannot be combined with which." The ablations are thorough: OPD vs SFT vs Masked on the teacher segment, Full vs Prune, teacher segment length L_T, trigger threshold τ, skip-incorrect — each piece gets taken apart.

Critical weaknesses

The core one: the headline gains are small. Roughly +1.2 points on average, with one training run per configuration and zero information on cross-seed variance, so a reader cannot tell whether the improvement is outside training noise.

Second, the method is essentially a combination of existing parts — segment-level divergence triggering, bounded teacher takeover (the Relay-OPD / speculative KD line), and windowed cutoffs— so novelty rests mostly on the specific recipe (earliest-segment trigger + short replacement + cutoff calibrated on correct responses).

Also, cost is never quantified: a full student rollout + teacher scoring + teacher decoding + full regeneration — how much more expensive per step than vanilla OPD (wall-clock, token counts) is nowhere given in a table. Scope-wise it's math-only, essentially 1.5B, two pairs; and AMO / HMMT scores are so low (around 2.7% and 24%) that those two tasks contribute more form than substance to the average.

Questions for authors

see the weakness above

3 comments

We sincerely thank reviewer #NNrz for the careful and constructive feedback. Our responses are as follows.

W1: The modest average accuracy gain of ATI-OPD over OPD.

We thank the reviewer for reminding us to accurately characterize the practical significance of the performance gains. Beyond average accuracy, we further analyzed response length and correctness during independent model inference. Following the original 12,000-token threshold, we define responses whose full length exceeds this threshold as long responses and the remaining responses as short, and divide them into four categories based on the final evaluation results.

MethodLong Incorrect ResponsesShort Incorrect ResponsesLong Correct ResponsesShort Correct Responses
OPD27.85±1.2227.85_{\pm 1.22}33.01±0.4333.01_{\pm 0.43}3.47±0.343.47_{\pm 0.34}35.66±0.4535.66_{\pm 0.45}
ATI-OPD25.10±0.2025.10_{\pm 0.20}34.78±0.1134.78_{\pm 0.11}3.47±0.253.47_{\pm 0.25}36.65±0.1636.65_{\pm 0.16}

Compared with OPD, ATI-OPD changes the proportion of long incorrect responses by -2.74%, with a 95% paired confidence interval of [-4.07, -1.44]; meanwhile, the proportion of short incorrect responses increases by 1.77%, with a 95% confidence interval of [+0.50, +3.06]. The proportion of long correct responses remains essentially unchanged, while the proportion of short correct responses increases by 0.98%.

Therefore, the more direct behavioral benefit of ATI-OPD is that it produces fewer lengthy responses that are ultimately judged incorrect, alongside an observed modest gain in average accuracy.

W2: Lack of experiments with multiple training seeds, variance, and other statistical measures.

We thank the reviewer for highlighting the importance of repeated training runs and statistical validation. Under the JustRL-DeepSeek-1.5B teacher -> DeepSeek-R1-Distill-Qwen-1.5B student setting, we conducted two additional training runs, with seed=0 and 42, for each of OPD and ATI-OPD. Both methods were evaluated on the same five benchmarks, with 16 samples per problem. The table below summarizes the mean accuracy across the two additional runs, with standard deviations in subscripts; Average denotes the mean across the five tasks.

MethodAIME24AIME25AMC23AMO-BenchHMMTAverage
OPD46.04±1.7746.04_{\pm 1.77}36.25±0.2936.25_{\pm 0.29}87.27±1.2287.27_{\pm 1.22}3.69±0.003.69_{\pm 0.00}22.44±0.6722.44_{\pm 0.67}39.14±0.7939.14_{\pm 0.79}
ATI-OPD51.15±1.6251.15_{\pm 1.62}37.40±0.1537.40_{\pm 0.15}87.50±0.2287.50_{\pm 0.22}2.08±0.002.08_{\pm 0.00}22.44±2.0122.44_{\pm 2.01}40.11±0.0940.11_{\pm 0.09}

The average gains of ATI-OPD over OPD in the two runs were +1.47% and +0.48%, respectively, with a mean gain of +0.98% and a sample standard deviation of 0.70%. Both additional runs showed a positive trend in average gains, indicating that this trend is not confined to the original single training run.

W3: The contribution of combining trajectory repair with supervision-range selection for subsequent student continuations.

We thank the reviewer for pointing out the connections between our work and existing methods and for suggesting a more precise characterization of our contribution. We acknowledge that disagreement detection, brief teacher takeover, and selective supervision each have precedents. Relay-OPD [1] already includes bounded teacher segments and student continuation when the budget permits; AC-OPD [3] retains the original student trajectory and selects the effective supervision length of an appended teacher branch; Prune-OPD [2] adjusts supervision weights and rollout budgets for subsequent training steps; KAT [4] terminates rollouts early based on sustained low-KL agreement.

We do not present each component as being introduced for the first time. The specific question addressed in this paper is: after repairing a bounded segment in an already deviated student trajectory, how far should supervision extend into the regenerated student continuation? To this end, we perform a bounded replacement at the first high-disagreement segment, allow the student to regenerate its continuation from the repaired prefix, and then calibrate a threshold using original correct responses to select the supervision range for this student continuation. This specific design concerns the trajectory being repaired, the regenerated continuation, and the range of tokens receiving supervision; we position our contribution as this design and its validation through long-sequence behavior.

To further distinguish the effects of supervision selection and trajectory repair, we added a Selection-only ablation. Selection-only retains the original student trajectory without teacher generation or student continuation, and selects the supervision range within the student continuation after the original triggering segment ends. This differs from the Masked ablation in the paper: Masked still inserts a teacher segment and regenerates the subsequent student trajectory, but does not directly compute loss on teacher tokens.

MethodAIME24AIME25AMC23AMO-BenchHMMTAverage
OPD47.2936.4688.133.6922.9239.70
Selection-only47.7136.0487.192.7222.5439.24
ATI-OPD50.0037.2987.662.0823.8640.18

In this run, Selection-only's average accuracy is 0.46% lower than OPD's, while ATI-OPD's is 0.94% higher than Selection-only's.

We further compared performance on long responses:

MethodProportion of Long Incorrect ResponsesProportion of Long Correct Responses
OPD26.993.71
Selection-only24.712.58
ATI-OPD24.963.65

Selection-only also reduces long incorrect responses, but it simultaneously reduces long correct responses and the coverage of successful solutions across multiple samples. Compared with Selection-only, ATI-OPD has a 1.07% higher proportion of long correct responses, with a 95% paired confidence interval of [+0.22%, +1.97%]. Meanwhile, pass@16 increases from 54.06% to 59.48%, a gain of 5.42%, with a 95% CI of [+1.18%, +9.95%]. This further shows that both methods reduce long incorrect responses, while ATI-OPD, which incorporates trajectory repair, preserves more long correct responses and greater coverage of solved problems. This supports distinguishing between the roles of “selecting only the supervision range on the original trajectory” and “selecting the supervision range after repairing the trajectory” in ATI-OPD.

W4: Computational costs of ATI-OPD and OPD, and their performance under the same computational resources.

We thank the reviewer for suggesting a complete accounting of computational costs. We compare the arithmetic workload of the two methods by estimating model FLOPs, including the initial student rollout, student continuation, teacher generation and scoring, actor log-probability recomputation, and forward and backward passes, as well as student generations that are ultimately discarded. We also count input, cached input, and output tokens during training.

The table below reports cumulative costs over training. FLOPs are measured in units of 101810^{18}, and token counts are in millions (M).

MethodEstimated FLOPsStudent InputStudent Input (Cached)Student OutputTeacher InputTeacher Input (Cached)Teacher OutputCumulative Tokens Processed in Actor UpdatesAverage Performance
OPD12.946.2770.629448.484909.52100454.76139.70
ATI-OPD12.7665.32857.055504.6071,054.82337.5081.720428.12840.18

As shown above, ATI-OPD improves task accuracy by 0.48%, reduces FLOPs by approximately 1.41%, and reduces the cumulative number of tokens processed in actor updates by 5.86%, with the average number of tokens processed per actor update decreasing from 2.096M to 1.973M. This improvement primarily results from shorter final training trajectories, consistent with the reduced proportion of extremely long responses observed in Figure 3 of the paper. Therefore, under similar estimated model FLOPs, ATI-OPD achieves a modest accuracy gain while reducing the number of tokens processed in actor updates.

W5: Retaining difficult tasks while disclosing where gains are concentrated and reporting negative results.

We thank the reviewer for drawing attention to absolute performance on difficult tasks and the interpretation of average scores. We agree that the low absolute scores on AMO-Bench and HMMT limit the conclusions that can be drawn about reasoning ability on these tasks. We retain these tasks to present the limits of the method's capabilities and explicitly report results showing either no improvement or a decline.

As a sensitivity analysis, in the additional experiments with multiple seeds, we exclude AMO-Bench and HMMT and compute an equally weighted average over AIME24, AIME25, and AMC23 only. The average gain of ATI-OPD over OPD is +2.16%. Therefore, the positive average change is not driven by the two low-scoring tasks.

[1] Xu H, Xu X, Hong H, et al. Pass the baton: Trajectory-relayed on-policy distillation[J]. arXiv preprint arXiv:2607.26057, 2026. [2] Yang Z, Guo Z, Song Y, et al. Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning[J]. arXiv preprint arXiv:2605.07804, 2026. [3] Zhou J, Zhou Y, Guo H, et al. Teacher Should Think Ahead: Adaptive Continuations for Reliable On-Policy Distillation[J]. arXiv preprint arXiv:2609.22254, 2026. [4] Xin H, Zhao A, Sun Y, et al. Escaping the KL Agreement Trap in On-Policy Distillation[J]. arXiv preprint arXiv:2606.09471, 2026.

Reviewer_hF1eOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies training instability in long-sequence on-policy distillation (OPD), where teacher feedback is computed on student-generated reasoning trajectories. In this paper, the authors argue that as the student generates longer responses, its reasoning may diverge from the teacher's preferred behavior, resulting in unreliable feedback, repetitive generations, and unstable optimization. To address this, the paper proposes Adaptive Teacher Intervention for On-Policy Distillation (ATI-OPD), consisting of two components. First, Adaptive Teacher Intervention monitors teacher-student disagreement using a sampled-token divergence estimator and replaces the earliest high-disagreement segment with a short teacher-generated continuation, after which the student resumes generation. Second, Adaptive Supervision Selection identifies persistent disagreement in the regenerated student continuation and excludes the corresponding suffix from the distillation loss. Experiments on two 1.5B teacher-student pairs across five mathematical reasoning benchmarks show that ATI-OPD improves average student-only accuracy over vanilla OPD by 1.20 and 1.35 percentage points.

Critical strengths
  1. It is an important and well-motivated problem. The paper addresses an important limitation of long-sequence OPD. As reasoning trajectories become longer, teacher feedback conditioned on student-generated prefixes may become less useful, potentially reinforcing undesirable behaviors.
  2. It is an intuitive combination of local intervention and adaptive supervision.
  3. Empirically, there are improvements across multiple mathematical reasoning benchmarks. ATI-OPD improves average student-only accuracy over standard OPD on both primary teacher-student pairs. The improvement is relatively consistent across the evaluated benchmarks, particularly for the Nemotron pair, where all five tasks improve. The exploratory 7B experiment also shows a positive average improvement, suggesting that the method may extend beyond the primary student scale.
  4. Beyond final benchmark performance, the paper analyzes teacher-student disagreement, token coverage, entropy, gradient norms, and response-length distributions.
Critical weaknesses
  1. The paper's central mechanism substantially overlaps with recent approaches that detect problematic student prefixes, allow the teacher to intervene, and return generation to the student. In particular, Relay-OPD already introduces short teacher-generated segments followed by student continuation, while MOTAB studies divergence-triggered backtracking and corrective teacher intervention. AC-OPD further considers adaptive supervision horizons for teacher-generated continuations.
  2. Relatively small performance improvements without sufficient statistical validation.
  3. TI-OPD requires generating a complete initial student response, evaluating teacher-student disagreement, discarding part of the original trajectory, generating a teacher replacement, and regenerating the remaining student continuation. The resulting mixed trajectory is subsequently scored again for supervision selection. These operations can introduce substantial overhead. Furthermore, adaptive supervision selection only changes the loss mask after the complete continuation has already been generated, so it does not directly reduce decoding costs.
  4. Although the paper includes useful ablations, several important controls are missing. (1) there is no clear baseline that applies the same adaptive supervision-selection rule directly to the original student trajectories without teacher intervention. (2) The paper does not compare divergence-triggered intervention against random or alternative intervention positions under matched teacher-token and generation budgets. (3) the reported teacher-segment OPD advantage over masking is relatively small and the SFT alternative uses a fixed auxiliary loss coefficient rather than a tuned comparison.
  5. The main evaluation focuses on two 1.5B teacher-student pairs trained on the same mathematical reasoning dataset. Although the exploratory 7B results are encouraging, they cover only four tasks and do not establish generality across different model architectures or application domains.
Questions for authors
  1. Can the authors provide direct, matched-budget comparisons with Relay-OPD, AC-OPD, or other recent teacher-intervention methods?
  2. Can the authors report results across multiple training seeds and provide paired per-question confidence intervals for the improvements over vanilla OPD?
  3. What are the total student-generated tokens, teacher-generated tokens, teacher-scoring costs, and wall-clock training times for ATI-OPD compared with vanilla OPD?
  4. Can the authors recompute the main absolute-gap and top-16 coverage diagnostics using only student-generated tokens, ideally with matched response positions or prefix lengths?
3 comments

We sincerely thank reviewer #hF1e for the careful and constructive feedback. Our responses are as follows.

W1: Comparisons with closely related work, such as Relay-OPD and Prune-OPD.

We thank the reviewer for suggesting more direct comparisons with related methods. We added Relay-OPD [1], which is the closest to our local teacher takeover mechanism, using the same JustRL-DeepSeek-1.5B teacher -> DeepSeek-R1-Distill-Qwen-1.5B student, training data, seed, maximum response budget, and hardware configuration, while retaining each method's specific trigger rules and training objective. As shown below, ATI-OPD improves average accuracy over Relay-OPD by 0.96%.

MethodStepAIME24AIME25AMC23AMO-BenchHMMTAverage
OPD21747.2936.4688.133.6922.9239.70
ATI-OPD21750.0037.2987.662.0823.8640.18
Relay-OPD21749.1735.0086.412.2423.3039.22

We also compare with Prune-OPD [2], which is closely related to our work in terms of supervision selection. We report the absolute gains of each paper's method over its own OPD baseline. The table below uses the original results from both papers, each under the JustRL-DeepSeek-1.5B teacher -> DeepSeek-R1-Distill-Qwen-1.5B student setting.

MethodAIME24 GainAIME25 GainAMC23 GainAverage
ATI-OPD+2.92-0.42+1.56+1.35
Prune-OPD+1.70+0.80+0.90+1.13

The improvements of the two methods over their respective OPD baselines are of similar magnitude, with ATI-OPD's within-paper average gain being 0.22% higher.

We acknowledge the common foundations of these methods: Relay-OPD already includes brief teacher takeover and student continuation when the budget permits, while Prune-OPD allocates supervision and computation through reliability decay and dynamic rollout budgets. Our specific contribution is to combine bounded replacement of the first high-disagreement segment with adaptive supervision selection for the regenerated student suffix, calibrating the supervision threshold using original correct responses and examining how this design affects long-sequence training behavior.

Regarding the maximum training response length, our setting uses 23,552 tokens, compared with 16,384 tokens for Relay-OPD and 8,192–12,288 tokens for Prune-OPD. This further situates our contribution in long-sequence trajectory repair and supervision-range selection.

W2: Lack of experiments with multiple training seeds, variance, and other statistical measures.

We thank the reviewer for highlighting the importance of repeated training runs and statistical validation. Under the JustRL-DeepSeek-1.5B teacher -> DeepSeek-R1-Distill-Qwen-1.5B student setting, we conducted two additional training runs, with seed=0 and 42, for each of OPD and ATI-OPD. Both methods were evaluated on the same five benchmarks, with 16 samples per problem. The table below summarizes the mean accuracy across the two additional runs, with standard deviations in subscripts; Average denotes the mean across the five tasks.

MethodAIME24AIME25AMC23AMO-BenchHMMTAverage
OPD46.04±1.7746.04_{\pm 1.77}36.25±0.2936.25_{\pm 0.29}87.27±1.2287.27_{\pm 1.22}3.69±0.003.69_{\pm 0.00}22.44±0.6722.44_{\pm 0.67}39.14±0.7939.14_{\pm 0.79}
ATI-OPD51.15±1.6251.15_{\pm 1.62}37.40±0.1537.40_{\pm 0.15}87.50±0.2287.50_{\pm 0.22}2.08±0.002.08_{\pm 0.00}22.44±2.0122.44_{\pm 2.01}40.11±0.0940.11_{\pm 0.09}

The average gains of ATI-OPD over OPD in the two runs were +1.47% and +0.48%, respectively, with a mean gain of +0.98% and a sample standard deviation of 0.70%. Both additional runs showed a positive trend in average gains, indicating that this trend is not confined to the original single training run.

W3: Computational costs of ATI-OPD and OPD, and their performance under the same computational resources.

We thank the reviewer for suggesting a complete accounting of computational costs. We compare the arithmetic workload of the two methods by estimating model FLOPs, including the initial student rollout, student continuation, teacher generation and scoring, actor log-probability recomputation, and forward and backward passes, as well as student generations that are ultimately discarded. We also count input, cached input, and output tokens during training.

The table below reports cumulative costs over training. FLOPs are measured in units of 101810^{18}, and token counts are in millions (M).

MethodEstimated FLOPsStudent InputStudent Input (Cached)Student OutputTeacher InputTeacher Input (Cached)Teacher OutputCumulative Tokens Processed in Actor UpdatesAverage Performance
OPD12.946.2770.629448.484909.52100454.76139.70
ATI-OPD12.7665.32857.055504.6071,054.82337.5081.720428.12840.18

As shown above, ATI-OPD improves task accuracy by 0.48%, reduces FLOPs by approximately 1.41%, and reduces the cumulative number of tokens processed in actor updates by 5.86%, with the average number of tokens processed per actor update decreasing from 2.096M to 1.973M. This improvement primarily results from shorter final training trajectories, consistent with the reduced proportion of extremely long responses observed in Figure 3 of the paper. Therefore, under similar estimated model FLOPs, ATI-OPD achieves a modest accuracy gain while reducing the number of tokens processed in actor updates.

W4: Recomputing diagnostic metrics, such as the teacher-student gap and top-16 coverage, using student-generated tokens.

We thank the reviewer for suggesting further controls for token source and response position. Following this suggestion, we added new training metrics. Specifically, we exclude the inserted teacher tokens from the final trajectories and retain all valid student tokens, including subsequent student tokens excluded by the loss mask. We then divide responses into 512-token bins based on absolute response position and compute gap metrics on the same set of shared training steps, as shown below.

MethodStudent-only gap AUCStudent-only top-16 coverage (%)
OPD0.2274±0.00060.2274_{\pm 0.0006}94.76±0.0094.76_{\pm 0.00}
ATI-OPD0.1943±0.00590.1943_{\pm 0.0059}96.27±0.3496.27_{\pm 0.34}

ATI-OPD achieves a relative reduction of 14.52% in student-only gap AUC and an increase of 1.51% in top-16 coverage. Therefore, after excluding teacher tokens, retaining student tokens that do not contribute to the loss, and controlling for the response-position distribution, the trend of lower gap and higher coverage for ATI-OPD persists.

W5: The modest average accuracy gain of ATI-OPD over OPD.

We thank the reviewer for reminding us to accurately characterize the practical significance of the performance gains. Beyond average accuracy, we further analyzed response length and correctness during independent model inference. Following the original 12,000-token threshold, we define responses whose full length exceeds this threshold as long responses and the remaining responses as short, and divide them into four categories based on the final evaluation results.

MethodLong Incorrect ResponsesShort Incorrect ResponsesLong Correct ResponsesShort Correct Responses
OPD27.85±1.2227.85_{\pm 1.22}33.01±0.4333.01_{\pm 0.43}3.47±0.343.47_{\pm 0.34}35.66±0.4535.66_{\pm 0.45}
ATI-OPD25.10±0.2025.10_{\pm 0.20}34.78±0.1134.78_{\pm 0.11}3.47±0.253.47_{\pm 0.25}36.65±0.1636.65_{\pm 0.16}

Compared with OPD, ATI-OPD changes the proportion of long incorrect responses by -2.74%, with a 95% paired confidence interval of [-4.07, -1.44]; meanwhile, the proportion of short incorrect responses increases by 1.77%, with a 95% confidence interval of [+0.50, +3.06]. The proportion of long correct responses remains essentially unchanged, while the proportion of short correct responses increases by 0.98%.

Therefore, the more direct behavioral benefit of ATI-OPD is that it produces fewer lengthy responses that are ultimately judged incorrect, alongside an observed modest gain in average accuracy.

W6: Additional ablations to distinguish the effects of supervision selection and trajectory repair.

We thank the reviewer for suggesting more direct ablations to distinguish the roles of the two components. We added a Selection-only experiment that retains the original student trajectory without teacher generation or student continuation, and selects the supervision range within the student continuation after the original triggering segment ends. This differs from the Masked ablation in the paper: Masked still inserts a teacher segment and regenerates the subsequent student trajectory, but does not directly compute loss on teacher tokens.

MethodAIME24AIME25AMC23AMO-BenchHMMTAverage
OPD47.2936.4688.133.6922.9239.70
Selection-only47.7136.0487.192.7222.5439.24
ATI-OPD50.0037.2987.662.0823.8640.18

In this run, Selection-only's average accuracy is 0.46% lower than OPD's, while ATI-OPD's is 0.94% higher than Selection-only's.

We further compared performance on long responses:

MethodProportion of Long Incorrect ResponsesProportion of Long Correct Responses
OPD26.993.71
Selection-only24.712.58
ATI-OPD24.963.65

Selection-only also reduces long incorrect responses, but it simultaneously reduces long correct responses and the coverage of successful solutions across multiple samples. Compared with Selection-only, ATI-OPD has a 1.07% higher proportion of long correct responses, with a 95% paired confidence interval of [+0.22%, +1.97%]. Meanwhile, pass@16 increases from 54.06% to 59.48%, a gain of 5.42%, with a 95% CI of [+1.18%, +9.95%]. This further shows that both methods reduce long incorrect responses, while ATI-OPD, which incorporates trajectory repair, preserves more long correct responses and greater coverage of solved problems. This supports distinguishing between the roles of “selecting only the supervision range on the original trajectory” and “selecting the supervision range after repairing the trajectory” in ATI-OPD.

[1] Xu H, Xu X, Hong H, et al. Pass the baton: Trajectory-relayed on-policy distillation[J]. arXiv preprint arXiv:2607.26057, 2026. [2] Yang Z, Guo Z, Song Y, et al. Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning[J]. arXiv preprint arXiv:2605.07804, 2026.

Reviewer_RanaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the instability of on-policy distillation (OPD) in long reasoning responses: as the student’s prefix gradually deviates from the content the teacher would generate, the quality of the teacher’s feedback on subsequent tokens declines, leading to entropy rebound, gradient abrupt changes, and an increase in maximum-length responses.

ATI-OPD:

  • Monitors discrepancies between the teacher and student at the segment level using a token-sampling-based K3 estimator.
  • Replaces the segment where high discrepancy first occurs (tau = 0.15) with a short segment generated by the teacher, then allows the student to continue generating.
  • For outputs across two consecutive windows, if they exceed a threshold calibrated based on the original correct response, their subsequent content is masked from the loss.

Results: On the DAPO-Math-17K dataset, using two sets of teacher-student pairs (JustRL-DeepSeek-1.5B and R1-Distill-Qwen-1.5B; JustRL-Nemotron-1.5B to OpenMath-Nemotron-1.5B), the average accuracy across five math benchmarks increased from 38.45 to 39.65 and from 52.72 to 54.07, respectively, outperforming the standard OPD method; At the same time, the AUC gap decreased by 47.8%, and gradient abrupt changes and responses of extreme length were also significantly reduced. Experimental results for the 7B student model are provided in the appendix.

Critical strengths
  1. Clearly identified issues, supported by mechanistic evidence: a 47.8% decrease in gap AUC, along with reductions in gradient spikes, entropy rebound, and extreme length responses, while still maintaining inference capabilities for thousands of tokens.

  2. The method is simple and preserves the standard OPD update mechanism; its two components address different failure modes; it demonstrates consistent performance improvements across both benchmark sets (4 out of 5 benchmarks and 5 out of 5 benchmarks). The appendix includes ablation experiment results and test results for the 7B model.

Critical weaknesses

The comparison is incomplete and does not account for costs.

  • Although closely related methods (Relay-OPD, AC-OPD, KAT, Prune-OPD, TRD) are discussed, they were not tested; Baselines include GRPO, standard OPD, and ESR, as well as published performance gains from TOPD/POPD under various configurations.
  • The method requires additional generation, teacher scoring, and regeneration processes, as well as several hyperparameters, but no computationally comparable comparisons were conducted; the study was conducted on math reasoning with 1.5B-parameter students.
Questions for authors
  1. How many training seeds were run, and what is the variance in the average gain?

  2. What is the computational overhead compared to standard OPD? Given equal computational resources, can OPD narrow this gap?

  3. Can it be compared with Relay-OPD, AC-OPD, and KAT-style baseline models under the same experimental setup?

2 comments

We sincerely thank reviewer #Rana for the careful and constructive feedback. Our responses are as follows.

W1: Lack of experiments with multiple training seeds, variance, and other statistical measures.

We thank the reviewer for highlighting the importance of repeated training runs and statistical validation. Under the JustRL-DeepSeek-1.5B teacher -> DeepSeek-R1-Distill-Qwen-1.5B student setting, we conducted two additional training runs, with seed=0 and 42, for each of OPD and ATI-OPD. Both methods were evaluated on the same five benchmarks, with 16 samples per problem. The table below summarizes the mean accuracy across the two additional runs, with standard deviations in subscripts; Average denotes the mean across the five tasks.

MethodAIME24AIME25AMC23AMO-BenchHMMTAverage
OPD46.04±1.7746.04_{\pm 1.77}36.25±0.2936.25_{\pm 0.29}87.27±1.2287.27_{\pm 1.22}3.69±0.003.69_{\pm 0.00}22.44±0.6722.44_{\pm 0.67}39.14±0.7939.14_{\pm 0.79}
ATI-OPD51.15±1.6251.15_{\pm 1.62}37.40±0.1537.40_{\pm 0.15}87.50±0.2287.50_{\pm 0.22}2.08±0.002.08_{\pm 0.00}22.44±2.0122.44_{\pm 2.01}40.11±0.0940.11_{\pm 0.09}

The average gains of ATI-OPD over OPD in the two runs were +1.47% and +0.48%, respectively, with a mean gain of +0.98% and a sample standard deviation of 0.70%. Both additional runs showed a positive trend in average gains, indicating that this trend is not confined to the original single training run.

W2: Computational costs of ATI-OPD and OPD, and their performance under the same computational resources.

We thank the reviewer for suggesting a complete accounting of computational costs. We compare the arithmetic workload of the two methods by estimating model FLOPs, including the initial student rollout, student continuation, teacher generation and scoring, actor log-probability recomputation, and forward and backward passes, as well as student generations that are ultimately discarded. We also count input, cached input, and output tokens during training.

The table below reports cumulative costs over training. FLOPs are measured in units of 101810^{18}, and token counts are in millions (M).

MethodEstimated FLOPsStudent InputStudent Input (Cached)Student OutputTeacher InputTeacher Input (Cached)Teacher OutputCumulative Tokens Processed in Actor UpdatesAverage Performance
OPD12.946.2770.629448.484909.52100454.76139.70
ATI-OPD12.7665.32857.055504.6071,054.82337.5081.720428.12840.18

As shown above, ATI-OPD improves task accuracy by 0.48%, reduces FLOPs by approximately 1.41%, and reduces the cumulative number of tokens processed in actor updates by 5.86%, with the average number of tokens processed per actor update decreasing from 2.096M to 1.973M. This improvement primarily results from shorter final training trajectories, consistent with the reduced proportion of extremely long responses observed in Figure 3 of the paper. Therefore, under similar estimated model FLOPs, ATI-OPD achieves a modest accuracy gain while reducing the number of tokens processed in actor updates.

W3: Comparisons with closely related work, such as Relay-OPD and Prune-OPD.

We thank the reviewer for suggesting more direct comparisons with related methods. We added Relay-OPD, which is the closest to our local teacher takeover mechanism, using the same JustRL-DeepSeek-1.5B teacher -> DeepSeek-R1-Distill-Qwen-1.5B student, training data, seed, maximum response budget, and hardware configuration, while retaining each method's specific trigger rules and training objective. As shown below, ATI-OPD improves average accuracy over Relay-OPD by 0.96%.

MethodStepAIME24AIME25AMC23AMO-BenchHMMTAverage
OPD21747.2936.4688.133.6922.9239.70
ATI-OPD21750.0037.2987.662.0823.8640.18
Relay-OPD21749.1735.0086.412.2423.3039.22

To further understand how different methods affect long-sequence behavior, we retain the original 12,000-token threshold and divide responses from independent evaluation into long incorrect, short incorrect, long correct, and short correct responses. The proportions below are equally weighted averages across the five benchmarks.

MethodAverage Output LengthLong Incorrect ResponsesShort Incorrect ResponsesLong Correct ResponsesShort Correct Responses
OPD10,02826.99%33.32%3.71%35.98%
ATI-OPD9,73224.96%34.86%3.65%36.53%
Relay-OPD9,71324.65%36.12%2.69%36.53%

Both intervention methods reduce long incorrect responses. Relay-OPD has a slightly lower proportion of long incorrect responses, but its proportion of long correct responses also decreases. With nearly identical average output lengths and essentially the same proportions of short correct responses, ATI-OPD preserves more long correct responses than Relay-OPD: the proportion of long correct responses is higher by 0.95%, with a 95% confidence interval of [+0.18, +1.76]. This difference corresponds to almost the entire gap in average accuracy between the two methods.

This observation also reflects the difference in supervision range between the two methods: Relay-OPD terminates the current training trajectory after exhausting its teacher takeover budget; ATI-OPD regenerates the student continuation after local repair and adaptively selects the subsequent supervision range based on disagreement. Our additional results are consistent with the design goal of this paper: reducing long, unproductive generation while preserving long responses that successfully solve the problem.

We also compare with Prune-OPD [2], which is closely related to our work in terms of supervision selection. We report the absolute gains of each paper's method over its own OPD baseline. The table below uses the original results from both papers, each under the JustRL-DeepSeek-1.5B teacher -> DeepSeek-R1-Distill-Qwen-1.5B student setting.

MethodAIME24 GainAIME25 GainAMC23 GainAverage
ATI-OPD+2.92-0.42+1.56+1.35
Prune-OPD+1.70+0.80+0.90+1.13

The improvements of the two methods over their respective OPD baselines are of similar magnitude, with ATI-OPD's within-paper average gain being 0.22% higher.

We acknowledge the common foundations of these methods: Relay-OPD already includes brief teacher takeover and student continuation when the budget permits, while Prune-OPD allocates supervision and computation through reliability decay and dynamic rollout budgets. Our specific contribution is to combine bounded replacement of the first high-disagreement segment with adaptive supervision selection for the regenerated student suffix, calibrating the supervision threshold using original correct responses and examining how this design affects long-sequence training behavior.

Regarding the maximum training response length, our setting uses 23,552 tokens, compared with 16,384 tokens for Relay-OPD and 8,192–12,288 tokens for Prune-OPD. This further situates our contribution in long-sequence trajectory repair and supervision-range selection.

[1] Xu H, Xu X, Hong H, et al. Pass the baton: Trajectory-relayed on-policy distillation[J]. arXiv preprint arXiv:2607.26057, 2026. [2] Yang Z, Guo Z, Song Y, et al. Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning[J]. arXiv preprint arXiv:2605.07804, 2026.

Comments on the paper

1 comment

General Response

We sincerely thank the three reviewers for their careful reading and constructive feedback, and for recognizing the value of our research problem, method design, and empirical analysis.

  • The reviewers strongly recognized the importance of our research problem and its motivation. #hF1e described the problem addressed in this paper as an important and well-motivated problem; #NNrz explicitly stated that The problem is real and positively assessed our analysis of instability in long-sequence OPD.
  • The reviewers recognized the simplicity and targeted design of our method. #Rana noted that The method is simple and preserves the standard OPD update mechanism and acknowledged that its two components address different failure modes; #hF1e described it as an intuitive combination of local intervention and adaptive supervision.
  • All three reviewers acknowledged the paper's empirical evidence and analytical details. #Rana highlighted Clearly identified issues, supported by mechanistic evidence; #hF1e noted that The improvement is relatively consistent across the evaluated benchmarks; #NNrz commented that The appendix is unusually meticulous and The ablations are thorough.

We also take the reviewers' shared concerns seriously: the connections to and differences from recent related methods need to be clarified, the modest accuracy gains require additional training runs and statistical validation, and the additional costs of generation, scoring, and continuation need to be fully reported. To address these concerns, we conducted additional training runs with more seeds for both OPD and ATI-OPD under the JustRL-DeepSeek-1.5B setting, and report sample standard deviations across runs and confidence intervals based on paired per-question comparisons. We also added a selection-only ablation to further clarify the roles of ATI-OPD's components. In addition, we report more training metrics, including the teacher-student gap, token usage, and computational cost in FLOPs. Regarding related methods, we provide comparisons with Relay-OPD and Prune-OPD to demonstrate the performance advantages of ATI-OPD.

In the revised manuscript, we will further clarify the relationship between our work and existing methods: disagreement detection, local teacher takeover, and selective supervision all have precedents. Our specific contribution is to repair a deviated student trajectory and select an effective supervision range for the regenerated student continuation, evaluating this design through training dynamics and additional ablations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.