acceptodds
Under review as a conference paper at ICLR 2027

Flash-OPD: Fast On-Policy Distillation

Abstract

On-policy distillation (OPD) provides dense teacher supervision on student-generated trajectories, but generating and evaluating long rollouts incurs substantial training cost. Existing acceleration methods reduce this cost through open-loop rollout schedules or closed-loop horizon adaptation. However, supervision compatibility can vary substantially across trajectories, making a single rollout horizon difficult to match their heterogeneous reliable lengths: an overly short horizon may truncate useful supervision, while an overly long one wastes computation beyond reliable regions. Our key insight is that the trajectory-specific reliability boundary need not be predicted before generation. By viewing reliability as the first-passage of accumulated low teacher–student compatibility events, the boundary is inherently unknown before sampling, yet whether it has been reached can be determined exactly from the observed prefix. Building on this insight, we propose _Flash-OPD_, which shifts from rollout-horizon control to adaptive trajectory-level boundary verification. _Flash-OPD_ interleaves cached generation with teacher verification and independently stops each trajectory according to its observed compatibility events. To reduce verification overhead, the recent event rate is used only to schedule the next verification point, while the actual stopping decision always relies on the exact cumulative count. This separation prevents estimation errors from causing premature termination while enabling efficient verification during generation. Extensive experiments across diverse datasets and teacher–student settings show that _Flash-OPD_ achieves – speedups over full OPD while maintaining or improving accuracy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Efficient Machine Learning, On-Policy Distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tHrPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Flash-OPD accelerates on-policy distillation by stopping each student trajectory when its cumulative count of low teacher–student top-k overlap events reaches a budget. Generation alternates with teacher verification; an estimated event rate schedules checks, while observed counts determine stopping. Experiments report 2.23–7.52× training speedups over full OPD across four teacher–student pairs, with broadly comparable accuracy and a larger improvement for the Qwen3-4B student.

Critical strengths
  1. The measured savings are substantial and include verification overhead. For Qwen3-4B, average accuracy rises from 15.85 to 19.20 while training time falls from 33.09 to 4.40 hours (Table 2). This is a useful practical result.
  2. The implementation and cost analysis are unusually concrete. Appendices B–C explain exact boundary localization, EOS handling and caching; Tables 6–7 quantify verification costs and decoding overshoot. These details make the reported acceleration easier to assess.
Critical weaknesses
  1. The experiments do not isolate the benefit of trajectory-specific stopping from simply using much shorter rollouts. TOPD uses 4,096 tokens and Prune-OPD has a 1,024-token minimum, whereas Flash-OPD retains only 304–585 tokens on average in three settings (Appendix D.3; Table 7). A similarly short fixed-horizon baseline, or a control preserving the stopping-length distribution while removing its association with each trajectory, is needed to establish the central contribution. Appendix F acknowledges this gap.
  2. The cumulative overlap budget is not independently validated as a boundary of useful supervision. Equation (4) permanently accumulates low-overlap events, without distinguishing persistent drift from recoverable disagreement. Figure 2 measures horizon mismatch against this same criterion-defined boundary, while the stopping propositions establish correct boundary detection. Neither directly establishes that the discarded suffix provides less useful supervision. Appendix F acknowledges this limitation, but the reliability interpretation remains insufficiently supported.
  3. The contribution of adaptive verification scheduling is not established separately from early stopping. With exact boundary localization, fixed-interval verification would retain the same prefix on a fixed trajectory (Appendix B.2). Figure 8 varies the proposed scheduler's discount factor but does not compare against fixed verification blocks. The reported speedups therefore do not establish whether event-rate extrapolation improves the verification-cost/decoding-overshoot trade-off over a simpler cached implementation.

The practical speedups are promising, but the missing matched-budget comparison leaves the value of trajectory-specific allocation unresolved. The interpretation of the cumulative overlap threshold as a useful-supervision boundary also needs direct validation.

Questions for authors
  1. At matched retained-token or time budgets, how does Flash-OPD compare with fixed horizons near 300–600 tokens or with stopping lengths reassigned across trajectories?
  2. With the same stopping criterion and prefix caching, how does event-rate scheduling compare with fixed verification blocks in verification cost and decoding overshoot?
  3. What evidence from the existing full rollouts links the Equation (4) crossing to reduced suffix supervision utility, rather than temporary or recoverable teacher–student disagreement?
3 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Part 1)

Dear Reviewer tHrP,

We sincerely thank you for your constructive feedback and the effort invested in reviewing our work. We have carefully addressed each of your concerns below: (We wrote this rebuttal entirely by ourselves, word by word, without using any AI assistance. We hope that this can reciprocate your effort in taking the time to review it. Thank you!). Specifically:


Experiment-related

  • W1 & W4 & Q1: Trajectory-specific stopping vs. simply shorter rollouts

    • This raises a central question: do the gains stem from adaptive allocation, or merely from aggressive truncation? First, to clarify: shorter retained lengths were not chosen arbitrarily, but are jointly determined by the compatibility criterion and budget. The exact same setting yields markedly different budgets across pairs: 2,7082,708 / 585585 / 304304 / 569569 tokens on G1–G4, spanning a 9×9\times range. Within a single step, retained lengths also span 3.1–7.2×3.1\text{–}7.2\times (Table 7, max/min).
    • Following your suggestion, we added controlled comparisons across two student–teacher pairs and five benchmarks (2×52 \times 5 settings), where TOPD uses rollout lengths matching the average retained tokens of our method, under default settings. We have observed that:
      • As retained rollout length decreases, overall performance tends to drop, which is particularly pronounced in the first pair—ranging from a 1.7%1.7\% to 28.1%28.1\% relative drop compared to TOPD's default 40964096 setting.
      • When token counts are matched, TOPD still lags behind Flash-OPD. We attribute this primarily to batch-level fixed truncation, which can induce the feasible-length mismatch illustrated in Figure 1 of the paper. By contrast, our trajectory-level adaptive handling preserves feasible lengths and maintains competitive accuracy.
      • Although TOPD achieves the shortest training time under extreme retention settings by skipping intermediate overhead, it incurs severe performance degradation, whereas Flash-OPD achieves a superior overall performance–efficiency trade-off.

Student / Teacher: Qwen3-1.7B-Base / Qwen3-4B (Non-thinking)

MethodRetained tokens / trajAMC23AIME24AIME25HMMT24HMMT25AvgTime ↓\downarrow
OPDN/A27.948.123.333.330.218.5911.06h
TOPD4096 (default)28.357.853.132.150.288.354.80h
TOPD304 (match)20.185.212.921.460.216.001.31h
Flash-OPD (Ours)304 (avg)29.296.465.003.330.428.901.55h

Student / Teacher: Qwen3-4B-Base / Qwen3-4B (Non-thinking)

MethodRetained tokens / trajAMC23AIME24AIME25HMMT24HMMT25AvgTime ↓\downarrow
OPDN/A43.0014.5812.506.252.9215.8533.09h
TOPD4096 (default)42.0915.0012.505.213.1215.587.81h
TOPD586 (match)43.4516.2512.083.751.0415.312.63h
Flash-OPD (Ours)586 (avg)48.7218.7515.837.085.6219.204.40h
  • W3 & Q2: Separating the stopping rule from the adaptive scheduling of Eq. (5)

    • We believe there may be a methodological misunderstanding here. By Proposition 2, the crossover point lies within verified chunks, so the retained prefix is a function solely of the stopping rule and is independent of exact checkpoint placement. We verified this by replaying position-by-position measured event sequences from real trajectories back into the controller across seven scheduling policies:
    • Retained lengths are identical across all seven schedules on all four pairs. Consequently, accuracy and generated token counts are by construction unchanged, which addresses the accuracy and token columns in Q2.
    • What the scheduling policy changes is the cost frontier, where Eq. (5) dominates: at matched verification cost, adaptive scheduling produces 2.8×2.8\times to 6.8×6.8\times less decoding overshoot than any fixed-block schedule (2.8×2.8\times on G1, 5.3×5.3\times on G2, 5.9×5.9\times on G3, and 6.8×6.8\times on G4). This directly speaks to the trade-off between verification cost and overshoot raised in your question, and motivates Eq. (5) as a core component of our method.
ScheduleRetained lenVerif. roundsAmplificationOvershootOvershoot (%)
Adaptive ρ=1.0\rho = 1.0 (ours)232.72.332.20×2.20\times5.12.2%
Adaptive ρ=0.7\rho = 0.7232.73.283.29×3.29\times4.51.9%
Fixed block 6464232.73.323.14×3.14\times17.26.9%
Fixed block 128128232.72.372.28×2.28\times34.813.0%
Fixed block 256256232.71.801.71×1.71\times46.916.8%
Fixed block 512512232.71.752.00×2.00\times138.037.2%

AuthorsOpenReview ↗

Rebuttal by Authors (Part 2)

Method-related

  • W2 & W4 & Q3: Is the crossing a boundary of useful supervision?

    • We think there is a framing mismatch worth making explicit: Eq. (4) is a budget on accumulated unreliability, not a change-point detector, and it is not intended to classify a disagreement as persistent or recoverable. The operative question for a cost-control rule is not "will this trajectory recover?" but "at the point the budget is exhausted, does continuing to supervise buy accuracy?" That question has a direct answer, and it is the measurement you ask for in Q3.
    • The allowance BB is a dose knob on exactly the discarded tokens, so sweeping it is the utility measurement. On G3, the curve saturates around the boundary, as shown in the table below:
      • Doubling the retained length beyond the boundary yields a +0.02+0.02 improvement, which falls within the ±0.9\pm 0.9 random fluctuation range observed in our measurements. Full OPD represents the limit for this dosage: it supervises 13.6×13.6\times to 21.9×21.9\times more tokens on G2–G4, yet is not more accurate overall (Table 2 in our paper).
      • Therefore, in the only measurable sense, namely its effect on training, supervision beyond the critical point is not a factor on which accuracy relies.
G3 SettingRetained Tokens / TrajectoryAverage AccuracyΔ\Delta vs Default
B=100B = 1001451458.688.68−0.22-0.22
B=200B = 200 (default)3043048.908.90—
B=400B = 4006426428.928.92+0.02+0.02

Thank you again for your constructive suggestions, which have helped us substantially sharpen the positioning of this work. If possible, we kindly request that you reconsider the score, and we would be very happy to discuss any remaining questions.

Best regards,

Authors

AuthorsOpenReview ↗

Kindly Request for Reviewer's Feedback

Dear Reviewer tHrP,


We hope this message finds you well. As the author–reviewer discussion period approaches its end, we wanted to gently follow up and thank you once again for your thoughtful and constructive review. Your insightful comments regarding trajectory-specific stopping versus fixed truncation, the utility of supervision beyond the crossing boundary, and separating the stopping rule from adaptive scheduling helped us substantially strengthen the paper. We have provided detailed analyses, allowance-sweeping measurements, and new matched-length experimental results in our response above to address each point.

We truly appreciate your valuable time and guidance. If our responses and new results have satisfactorily resolved your concerns, we would be deeply grateful if you would consider raising your score. Please also let us know if any further clarification would be helpful.


Warm regards,

The Authors

Reviewer_3QXLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Flash-OPD aims to reduce the training cost of long rollouts in on-policy distillation. Instead of having the entire batch share the same rollout length, it determines when to stop based on the cumulative number of “teacher–student top-k low overlap” events per trajectory. It also adaptively schedules validation points based on the recent event rate and caches prefixes to avoid redundant generation. Experiments covered four sets of student–teacher combinations, primarily in mathematical reasoning, achieving a 2.2×–7.5× acceleration in training speed compared to vanilla OPD, while maintaining accuracy and even showing improvements in some settings.

Critical strengths
  1. The problem approach is valuable. Reviewers noted that the paper accurately identifies the limitations of batch-level horizon control—namely, the significant variability in the reliability of supervision across different trajectories—making the shift to per-trajectory stopping a natural direction. The design of “approximate verification point scheduling combined with precise stopping criteria” also prevents estimation errors in event rates from directly causing premature termination.
  2. The empirical results are impressive. Qwen3-1.7B and 4B-Base achieved speedups of 7.14× and 7.52×, respectively, with the average accuracy of the 4B model improving from 15.85 to 19.20. Analyses such as time decomposition, length statistics, OOD evaluation, and the 32B teacher model are also quite comprehensive.
Critical weaknesses
  1. The average length retained by Flash-OPD (585/304/569 tokens for G2–G4, respectively) is much shorter than that of vanilla OPD (7,961/6,647/11,278), while TOPD uses a fixed length of 4,096 and Prune-OPD uses a minimum of 1,024. Without comparisons that align for length or computational resources, it is impossible to determine whether the gains stem from “adaptive allocation” or simply “aggressive truncation.” The configurations for G1 are also unequal: Flash-OPD has an upper limit of 4,400, while the baseline is 12,288, and approximately 96 trajectories per batch terminate due to the length limit.
  2. The paper does not separate the individual contributions of the “cumulative event-count-based stopping rule” and the “adaptive validation scheduling in Eq(5).”.
Questions for authors

Q1: Please provide a comparison between fixed horizons (e.g., 300–600 tokens) and batch-level controllers under matched average length and computational cost, while including sample-level baselines such as PG-OPD. Q2: Please provide a comparison of fixed validation intervals (64/128/256/512) under the same stopping rule, reporting accuracy, number of generated tokens, number of validation rounds, overhead from prefix repetition handling, and actual runtime. Q3: Whether the 4,400 limit for G1 includes the prompt, and how it relates to the maximum retention length of 3,509 and the 96 truncated trajectories in Table 7

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Part 1)

Dear Reviewer 3QXL,

We sincerely thank you for your constructive feedback and the effort invested in reviewing our work. We have carefully addressed each of your concerns below: (We wrote this rebuttal entirely by ourselves, word by word, without using any AI assistance. We hope that this can reciprocate your effort in taking the time to review it. Thank you!). Specifically:


Experiment-related

  • W1 & Q1: Matched-length and matched-compute comparisons

    • We agree this is a central question: do the gains stem from adaptive allocation, or merely from aggressive truncation? First, to clarify: shorter retained lengths were not chosen arbitrarily, but are jointly determined by the compatibility criterion and budget. The exact same setting yields markedly different budgets across pairs: 2,7082,708 / 585585 / 304304 / 569569 tokens on G1–G4, spanning a 9×9\times range. Within a single step, retained lengths also span 3.1–7.2×3.1\text{–}7.2\times (Table 7, max/min).
    • Following your suggestion, we added controlled comparisons across two student–teacher pairs and five benchmarks (2×52 \times 5 settings), where TOPD uses rollout lengths matching the average retained tokens of our method, under default setting. We have observed that:
      • As retained rollout length decreases, overall performance tends to drop, which is particularly pronounced in the first pair—ranging from a 1.71.7% to 28.128.1% relative drop compared to TOPD's default 40964096 setting.
      • When token counts are matched, TOPD still lags behind Flash-OPD. We attribute this primarily to batch-level fixed truncation, which can induce the reliable-length mismatch illustrated in Figure 1 of the paper. By contrast, our trajectory-level adaptive handling preserves feasible lengths and maintains competitive accuracy.
      • Although TOPD achieves the shortest training time under extreme retention settings by skipping intermediate overhead, it incurs severe performance degradation, whereas Flash-OPD achieves a superior overall performance–efficiency trade-off.

Student / Teacher: Qwen3-1.7B-Base / Qwen3-4B (Non-thinking)

MethodRetained tokens / trajAMC23AIME24AIME25HMMT24HMMT25AvgTime ↓\downarrow
OPDN/A27.948.123.333.330.218.5911.06h
TOPD4096 (default)28.357.853.132.150.288.354.80h
TOPD304 (match)20.185.212.921.460.216.001.31h
Flash-OPD (Ours)304 (avg)29.296.465.003.330.428.901.55h

Student / Teacher: Qwen3-4B-Base / Qwen3-4B (Non-thinking)

MethodRetained tokens / trajAMC23AIME24AIME25HMMT24HMMT25AvgTime ↓\downarrow
OPDN/A43.0014.5812.506.252.9215.8533.09h
TOPD4096 (default)42.0915.0012.505.213.1215.587.81h
TOPD586 (match)43.4516.2512.083.751.0415.312.63h
Flash-OPD (Ours)586 (avg)48.7218.7515.837.085.6219.204.40h

  • Because PG-OPD has no open-source implementation, for a fair comparison we directly report comparable results from its original paper (selecting its best performance–efficiency configuration among three settings) in the table below. Flash-OPD remains optimal on the performance–efficiency frontier, achieving comparable average accuracy while providing a further 2.77×2.77\times speedup over PG-OPD.

Student / Teacher: DeepSeek-R1-Distill-Qwen-1.5B / DeepSeek-R1-Distill-Qwen-7B

MethodAMC23AIME24AIME25HMMT24HMMT25AvgTime ↓\downarrowSpeedup
OPD64.9133.3322.9215.8314.7930.3610.95h—
PG-OPD73.1226.4624.3811.0418.1230.635.70h1.92×1.92\times
Flash-OPD (Ours)66.2730.4224.1713.3314.5829.752.06h5.32×5.32\times
AuthorsOpenReview ↗

Rebuttal by Authors (Part 2)

  • W2 & Q2: Separating the stopping rule from the adaptive scheduling of Eq. (5)

    • We believe there may be a methodological misunderstanding here. By Proposition 2, the crossover point lies within verified chunks, so the retained prefix is a function solely of the stopping rule and is independent of exact checkpoint placement. We verified this by replaying position-by-position measured event sequences from real trajectories back into the controller across seven scheduling policies (see the table below):
    • Retained lengths are identical across all seven schedules on all four pairs. Consequently, accuracy and generated token counts are by construction unchanged, which addresses the accuracy and token columns in your comment.
    • What the scheduling policy changes is the cost frontier, where Eq. (5) dominates: at matched verification cost, adaptive scheduling produces 2.8×2.8\times to 6.8×6.8\times less decoding overshoot than any fixed-block schedule (2.8×2.8\times on G1, 5.3×5.3\times on G2, 5.9×5.9\times on G3, and 6.8×6.8\times on G4). This directly speaks to the trade-off between verification cost and overshoot raised in your question, and motivates Eq. (5) as a core component of our method.
ScheduleRetained lenVerif. roundsAmplificationOvershootOvershoot (%)
Adaptive ρ=1.0\rho = 1.0 (ours)232.72.332.20×2.20\times5.12.2%
Adaptive ρ=0.7\rho = 0.7232.73.283.29×3.29\times4.51.9%
Fixed block 6464232.73.323.14×3.14\times17.26.9%
Fixed block 128128232.72.372.28×2.28\times34.813.0%
Fixed block 256256232.71.801.71×1.71\times46.916.8%
Fixed block 512512232.71.752.00×2.00\times138.037.2%

Method-related

  • Q3: The L_max = 4400 limit on G1

  • The Lmax⁡=4400L_{\max} = 4400 cap applies strictly to the response (data.max_response_length); prompts have a separate 10241024-token budget, and the engine context window is their sum. Thus, 44004400 does not include the prompt.

  • 35093509 is not a single-trajectory maximum; it is the maximum per-batch average retained length across 203203 steps (Table 7 reports min / mean / max of per-batch averages: 830830 / 27082708 / 35093509). Because it is averaged over 256256 trajectories, being below 44004400 is expected and not a contradiction.

  • 9696 is a per-batch stopping-branch count rather than a failure count: among 256256 trajectories per batch, 130130 stop at the allowance, 9696 hit Lmax⁡L_{\max}, and 3030 hit EOS (130+96+30=256130 + 96 + 30 = 256).

  • Regarding fairness: a cap can only remove supervision signal, never add it, making it a handicap rather than an advantage. Moreover, on G1, Flash-OPD still reaches 40.3740.37 using 2.3×2.3\times fewer tokens than OPD (40.0940.09). Raising the cap is a cost choice—Table 8 shows accuracy stays flat for Lmax⁡L_{\max} between 40004000 and 60006000, while step time correlates almost perfectly with it (r=+1.00r = +1.00).


Thank you for your valuable feedback, which has helped us strengthen our manuscript! We hope the above responses address your concerns, reinforce your appreciation and confidence in our work, and we look forward to further discussion.

Best regards,

Authors

AuthorsOpenReview ↗

Kindly Request for Reviewer's Feedback

Dear Reviewer 3QXL,


We hope this message finds you well. As the author–reviewer discussion period approaches its end, we wanted to gently follow up and thank you once again for your thoughtful and constructive review. Your insightful comments regarding matched-length and matched-compute comparisons, separating the stopping rule from adaptive scheduling, and clarifying the Lmax⁡L_{\max} budget helped us substantially strengthen the paper. We have provided detailed theoretical clarifications, replay analyses across scheduling policies, and new matched-length experimental results in our response above to address each point.

We truly appreciate your valuable time and guidance. If our responses and new results have satisfactorily resolved your concerns, we would be deeply grateful if you would consider raising your score. Please also let us know if any further clarification would be helpful.


Warm regards,

The Authors

Reviewer_yTUPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Flash-OPD, a method designed to reduce the training costs of on-policy distillation (OPD). It assesses local compatibility using the top-k overlap between teacher and student outputs, defining the "reliability boundary" as the point where the cumulative count of low-compatibility events first reaches a pre-set budget. During training, the approach alternates between cached generation and teacher verification; while the recent event rate determines the timing of the next verification step, the decision to terminate is based strictly on the precise cumulative count of low-compatibility events. This ensures that trajectories are not prematurely halted due to estimation errors. Experiments across four teacher–student pairs demonstrate a training speedup of approximately 2.23× to 7.52× compared to standard OPD. In most cases, the average accuracy is either comparable or improved, though performance across different tasks warrants further investigation.

Critical strengths

igure 2 demonstrates, based on empirical measurements, that the usable length E varies significantly across different trajectories and evolves throughout the training process. Methods that employ a batch-shared horizon(TOPD and Prune-OPD)tend to either over-decode or truncate sequences prematurely; this mismatch cannot be resolved simply by refining the horizon controller. Table 1 further categorizes existing rollout-length control methods based on their decision criteria and granularity, thereby clarifying this paper’s focus on sequence-level, continuous-length control.

Flash-OPD decouples the decision of "when to verify" from the decision of "whether to stop". An inaccurate estimate \hat{\lambda} merely results in the generation of a few extra tokens and does not lead to the premature truncation of the reliable prefix. Furthermore, by leveraging the properties of Locality and Permanence, the interleaved process of generation and verification ensures that the established prefix supervision remains unaltered.

Critical weaknesses

Under certain settings, the accuracy is lower than that of vanilla OPD; therefore, the summary claim of "maintaining or improving accuracy" is too strong. In Table 2, for G2 (R1-1.5B ← R1-7B), the average score for Flash-OPD is 29.75—lower than the 30.36 achieved by OPD—with a decline of 2.91 points on AIME24 (30.42 vs. 33.33) and 2.50 points on HMMT24. Similarly, in Table 8, G1 shows a decline on both Olympiad-Bench (-2.90) and GPQA (-2.54).

There are indications that the hyperparameters were tuned on a per-pair basis, which somewhat contradicts the claims of a "fixed configuration" and the ability to "transfer without per-pair tuning." As shown in Tables 4 and 5 of the appendix, G1 employs distinct parameters—k0=W=1024k_0=W=1024, kmin=512k_{min}=512, and Lmax=4400L_{max}=4400—justified by the claim that "its trusted prefix is ​​typically longer." This, in itself, constitutes pair-specific hyperparameter tuning. Notably, G1 represents the scenario where Flash-OPD offers the least advantage, underscoring the fact that sensitivity to hyperparameters cannot be overlooked.

Training is conducted exclusively on the DAPO-Math-17K dataset, with student models up to 4B parameters and teacher models up to 32B parameters, utilizing a single 8×H20 node. Long-chain mathematical reasoning represents the scenario where teacher-student compatibility decays most significantly as sequence length increases; in other domains—such as coding, general-purpose conversation, or multi-turn agent interactions—the dynamics between B and performance/success rates may differ entirely.

Questions for authors

Can you provide a direct measure of the utility of token supervision inside and outside the boundary? The method defines the available length as cτ ≥ B, but the paper does not directly measure the KL of the discarded tokens, the teacher-student divergence, or their contribution to the reduction of loss.

What is the relative contribution of "truncation" versus "eliminating the log-prob pass via logit reuse" to the efficiency gains? Table 6 shows that eliminating the log-prob pass contributes significantly (e.g., from 45.6s to 0 in G4); this technique is independent of adaptive truncation. An ablation study—specifically one that isolates "truncation without reuse" or "reuse without truncation"—is needed to disentangle the contributions of the core method from those of these system-level optimizations.

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Part1)

Dear Reviewer yTUP,

We sincerely thank you for your time and effort in providing insightful feedback on our paper. We have carefully considered each of your comments to address your concerns (We wrote this rebuttal entirely by ourselves, word by word, without using any AI assistance. We hope this can reward you for taking the time to review the content on your own, rather than using AI-generated material. Thank you!). Specifically:


Presentation-related

  • W1: The claim of "maintaining or improving accuracy"
    • We would like to emphasize that the results in Table 2 show the average performance over multiple runs on several datasets. Therefore, these differences fall within the normal fluctuation range of benchmark testing and do not represent a systematic loss of accuracy. What is important is the significant improvement in training speed, ranging from 2.23 to 7.52 times.
    • We understand that the specific performance drops you mentioned may stem from unfamiliarity with the benchmarks. For example, AIME24 and HMMT24 each consist of only 30 questions (each question is worth approximately 3.33 points). Therefore, decreases of -2.91 and -2.50 correspond to a gap of less than one question. Thus, the slight performance differences shown in Table 2 actually reflect minimal differences in whether a question is solved or not.
    • To formally verify this, we conducted a paired bootstrap test. For each benchmark, we resampled questions with replacement 10,000 times and scored both methods on the same resampled questions. This measures whether the difference persists if another set of problems of the same size were drawn. As shown in the table below, all confidence intervals include zero (p≥0.18p \ge 0.18), so we can consider this as fluctuation at the baseline level.
    • However, we agree that "maintaining or improving accuracy" is too coarse a summary. We will revise it to the precise statement in the final version.
Flagged differenceΔ\Delta (Flash − OPD)95% CIpp
G2 AIME24−2.91[−7.08, +1.05]0.18
G2 HMMT24−2.50[−7.08, +2.29]0.32
G2 Average (5 benchmarks)−0.61[−2.13, +0.97]0.45

Method-related

  • W2: "Fixed configuration" and the G1 setting

    • Great observation; we believe there may be some misunderstanding here. In fact, there are two types of controllers:
      • Standard knobs for defining boundaries: η=0.7\eta=0.7, B=200B=200, and ρ=1.0\rho=1.0. These knobs are identical across all data pairs.
      • Scheduling knobs (k0,W,kmin⁡k_0, W, k_{\min}) cannot alter any retained prefix. According to Algorithm 1, these parameters only change the scheduling rate and therefore can only affect the number of verification rounds. We further verified that under k0∈{64,128,1024}k_0 \in \{64, 128, 1024\}, kmin⁡∈{1,16,512}k_{\min} \in \{1, 16, 512\}, W∈{64,128,1024}W \in \{64, 128, 1024\}, ρ∈{0.35,0.7,1}\rho \in \{0.35, 0.7, 1\}, and fixed chunks of 16–1,024 tokens, the retained lengths are completely consistent. Therefore, G1 uses larger values simply because its boundaries are 5–9 times longer (average 2,708 tokens); this is a cost choice, not accuracy tuning.
  • W3: Scope beyond reasoning and model scale

    • We agree that a broader range of domains would strengthen the paper's persuasiveness. However, we would like to emphasize that Flash-OPD only assumes token-level white-box teacher supervision under locality and persistence conditions, which is the same as the assumptions of OPD itself. Furthermore, the top-kk compatibility criterion does not use any math-specific content, so our method is conceptually general.
    • Long-chain reasoning is a domain where unfolding costs dominate in OPD, and it is currently a reasonable default setting in many classical papers. In fact, we tested 5 different pairings * 9 benchmarks, and also verified scalability up to 32B.
    • We emphasized in the conclusion that agent tasks are left as future work. If you have better suggestions for the setup, you should point them out directly. Thank you.

AuthorsOpenReview ↗

Rebuttal by Authors (Part2)

Experiment-related

  • Q1: Direct measure of the utility of supervision inside vs. beyond the boundary
    • Regarding the KL divergence metric, we would like to help clarify a common misunderstanding: A large KL divergence only indicates how much disagreement a teacher has over a state, not whether following that disagreement is beneficial. In states where the student has drifted, the teacher's own conditional distribution becomes a less reliable target. Empirically, at positions visited by the student, the two are nearly orthogonal, with correlations ranging from −0.011 (G1) to −0.041 (G4) across 0.30M–1.85M positions.
    • Thus, we define utility based on training effectiveness and report divergence as a descriptive statistic, rather than assuming it as a proxy. The budget value BB is precisely a control parameter applied to the discarded tokens, so scanning over it constitutes a measurement of utility. On G3, the curve saturates near the boundary: doubling the retention length from B=200B=200 yields only a +0.02 improvement, which falls within the measured run-to-run variance of ±0.9\pm 0.9.
    • Full OPD represents the limit of this dosage: it supervises 13.6×–21.9× more tokens on G2–G4 without being systematically more accurate (Table 2). Hence, supervision beyond the boundary is not a factor upon which accuracy depends, in any measurable sense.
G3 SettingRetained Tokens / TrajectoryAverage AccuracyΔ\Delta vs Default
B=100B=1001458.68−0.22
B=200B=200 (default)3048.90—
B=400B=4006428.92+0.02
  • Q2: Truncation vs. reuse of verification outputs
    • We first clarify that reuse is a by-product of boundary verification, not an independent system trick: because training stops exactly at a verified position, every retained token has already been scored by the student and teacher passes inside the loop (Locality/Permanence), so the post-rollout passes become redundant. Vanilla OPD has no verification pass whose outputs could be reused.
    • To quantify the two sources, we decompose Table 6. "Truncation only" re-adds OPD's log-prob and teacher-pass cost, rescaled to the retained tokens; "reuse only" is an upper bound that removes OPD's entire log-prob pass for free:
    • The speedup comes almost entirely from per-trajectory truncation; reuse alone could yield at most 1.09×1.09\times.
PairReportedTruncation only (no reuse)Reuse share of saved timeReuse only (upper bound)
G12.20×1.90×13.3%≤1.09×\le 1.09\times
G25.32×4.80×2.5%≤1.07×\le 1.07\times
G37.13×6.70×1.0%≤1.07×\le 1.07\times
G47.52×7.09×0.9%≤1.08×\le 1.08\times

Thank you again for your constructive suggestions, which have helped us substantially sharpen the positioning of this work. If possible, we kindly request that you reconsider the score, and we would be very happy to discuss any remaining questions.

Best regards,

Authors

AuthorsOpenReview ↗

Kindly Request for Reviewer's Feedback

Dear Reviewer yTUP,


We hope this message finds you well. As the author–reviewer discussion period approaches its end, we wanted to gently follow up and thank you once again for your thoughtful and constructive review. Your insightful comments regarding the statistical precision of our accuracy claims, the decoupling of boundary versus scheduling knobs, and the utility and speedup decomposition of boundary truncation helped us substantially strengthen the paper. We have provided paired bootstrap tests, parameter-invariance checks, budget-sweeping utility analyses, and a breakdown of truncation versus reuse gains in our response above to address each point.

We truly appreciate your valuable time and guidance. If our responses and new analyses have satisfactorily resolved your concerns, we would be deeply grateful if you would consider raising your score. Please also let us know if any further clarification would be helpful.


Warm regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to AC and All Reviewers

Dear AC and Reviewers,

We are truly grateful for the invaluable time and detailed feedback provided by all the reviewers. It is encouraging to see that all reviewers have recognized the positive aspects of our manuscript, such as a valuable and practical problem (Reviewers 3QXL, tHrP), clear identification of batch-level horizon mismatch (Reviewers yTUP, 3QXL), principled decoupling of verification scheduling and stopping (Reviewers yTUP, 3QXL), concrete implementation and cost analyses (Reviewers 3QXL, tHrP), and substantial speedups with strong empirical gains (Reviewers yTUP, 3QXL, tHrP).


We have responded to each reviewer in detail. Below we summarize the main clarifications and additional experimental analyses provided during the rebuttal.

  • Matched-length, matched-compute, and sample-level baseline comparisons. In response to feedback from Reviewers 3QXL and tHrP:
    • ① We clarified that retained lengths are not manually fixed, but emerge dynamically from our top-kk compatibility criterion (η=0.7\eta = 0.7) and budget (B=200B = 200), spanning a 9×9\times range across pairs (304304–2,7082,708 tokens) and 3.1–7.2×3.1\text{–}7.2\times within a single step.
    • ② We added 2×52 \times 5 controlled experiments matching TOPD's rollout length to Flash-OPD's average retained tokens across two student–teacher pairs and five benchmarks; Flash-OPD consistently outperforms length-matched TOPD (8.908.90 vs. 6.006.00 and 19.2019.20 vs. 15.3115.31 average accuracy), avoiding feasible-length mismatch via trajectory-level adaptation.
    • ③ We included a direct comparison against PG-OPD's best reported configuration, showing that Flash-OPD achieves comparable accuracy while delivering an additional 2.77×2.77\times speedup (5.32×5.32\times over OPD).
  • Decoupling the stopping rule, adaptive scheduling, and boundary supervision utility. In response to Reviewers yTUP, 3QXL, and tHrP:
    • ① We verified via position-by-position replay across seven scheduling policies that retained prefixes depend solely on the stopping rule, keeping accuracy and generated token counts invariant across scheduling knobs (k0,W,kmin⁡,ρk_0, W, k_{\min}, \rho).
    • ② We demonstrated that adaptive scheduling (Eq. 5) dominates the cost frontier, reducing decoding overshoot by 2.8×2.8\times to 6.8×6.8\times compared to fixed-block schedules at matched verification cost.
    • ③ We quantified supervision utility via allowance-sweeping experiments (B∈{100,200,400}B \in \{100, 200, 400\}), showing that accuracy saturates at the boundary (doubling BB yields only +0.02+0.02, well within the ±0.9\pm 0.9 run-to-run variance) and explaining why token-level KL divergence is nearly orthogonal to downstream utility.
  • Statistical rigor, speedup decomposition, and Lmax⁡L_{\max} clarifications. Following feedback from Reviewers yTUP and 3QXL:
    • ① We conducted 10,000-resample paired bootstrap tests on the flagged benchmark differences, confirming that all 95%95\% confidence intervals include zero (p≥0.18p \ge 0.18) and reflect single-question variance rather than systematic degradation, while committing to refine our accuracy phrasing in the camera-ready version.
    • ② We decomposed the end-to-end speedup into truncation versus verification reuse, showing that per-trajectory truncation drives nearly the entire acceleration (up to 7.09×7.09\times), whereas reuse alone contributes at most 1.07×–1.09×1.07\times\text{–}1.09\times.
    • ③ We clarified the separation between response (Lmax⁡=4400L_{\max} = 4400) and prompt (10241024) budgets on G1, as well as the per-batch average and stopping-branch breakdowns.

The expertise of all reviewers has greatly helped us strengthen our manuscript! We have made sincere efforts to address all the issues raised and are deeply grateful for the recognition and suggestions from all reviewers. We still respectfully welcome further discussions from all reviewers.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.