acceptodds
Under review as a conference paper at ICLR 2027

PRUNE-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning

Abstract

On-policy distillation (OPD) provides dense teacher feedback on student-generated responses, but its rollout budget does not account for how well the student and teacher match along those responses. We identify this unused mismatch information as an opportunity to accelerate OPD: long, weakly compatible suffixes can be curtailed while retaining the supervision needed for learning. We introduce PruneOPD, which uses top-k overlap to measure local compatibility, reweight rewards after cumulative mismatches, and adaptively truncate training rollouts. Within configured length bounds, the algorithm determines and updates the truncation length from compatibility feedback, without a manually specified cutoff or length schedule for each pair. Across four student–teacher pairs, Prune-OPD reduces reported training time by 37.6%–68.0% while broadly preserving accuracy on AMC, AIME, and HMMT. In the high-compatibility DeepSeek-7B / Skywork7B pair, Prune-OPD retains long supervision and stays close to OPD accuracy, avoiding the larger losses from fixed-4K truncation. Prune-OPD thus saves training computation where compatible supervision is short, while retaining longer rollouts where they are needed to preserve performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
OPD, reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cfBYOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Prune-OPD, which uses teacher–student prediction overlap to adapt the computational budget of on-policy distillation. At each generated position, cumulative low-overlap events determine a declining reliability score. This score both reweights the distillation signal and estimates an effective training length. A batch-level controller then adjusts the maximum length of future rollouts, expanding the budget when reliable supervision reaches the current cap and contracting it after repeated evidence that the cap is unnecessarily long.

Experiments cover five teacher–student pairs on mathematical reasoning tasks. Across four pairs, the method reduces training time by 37.6%–68.0% while broadly preserving accuracy. For a high-compatibility pair, it retains longer rollouts and remains close to baseline accuracy. Component ablations, fixed-budget comparisons, and parameter sweeps examine the contributions of weighting and continued length adaptation.

Critical strengths

The most interesting contribution is the use of compatibility feedback to adjust rollout length during training. This avoids committing to a fixed cutoff that may become inappropriate as the student changes. Figure 2 provides compelling evidence that the method reaches comparable accuracy in substantially less training time across several teacher–student pairs. Large accuracy improvements are not necessary for this contribution: preserving performance while reducing computation is a meaningful result. The high-compatibility experiment further strengthens the case, showing that the controller retains long supervision when aggressive fixed truncation would hurt accuracy.

The experiments distinguish reward weighting from adaptive truncation and show that truncation supplies most of the speedup. The fixed-initial-budget comparison also demonstrates why continued adaptation matters: a short initial cap can become restrictive later in training. Together with the threshold sweeps and repeated runs for one pair, these analyses make the empirical contribution substantially clearer than a comparison of the complete method against standard OPD alone.

Critical weaknesses

Evaluation is limited to mathematical reasoning. Although the paper considers several teacher–student pairs, both training and evaluation remain within mathematics. It is therefore unclear how well the overlap-based controller transfers to coding, general instruction following, or other tasks with different response structures. This limits the demonstrated scope, but does not undermine the reported efficiency gains.

Some controller choices remain insufficiently justified. In particular, the reliability decrement of 0.01 makes the score reach zero after 100 low-overlap events. This directly affects effective training lengths and subsequent rollout budgets, yet its selection and sensitivity are not established by the reported overlap-threshold and base-weight sweeps. Understanding this choice would help assess how much tuning is needed when applying the method to a new setting.

Questions for authors

Parameter selection and sensitivity. How was the reliability decrement of 0.01 chosen? Does the accuracy–time trade-off remain stable when this decrement changes, or when it is varied jointly with the overlap threshold? This would help determine whether the controller transfers readily across settings without additional tuning.

Transfer beyond mathematics. Are any results available on a non-mathematical training task using the same controller settings? If not, please clarify which characteristics of mathematical reasoning the current design may depend on.

Improvement beyond the cutoffs. Figure 2 shows impressive learning efficiency in the early stages of training. Does the method offers sustained improvement? If not, why couldn't it?

7 comments
AuthorsOpenReview ↗

Response to Reviewer cfBY

Thank you for your thoughtful review and positive assessment of our work. We address your three questions below.

AuthorsOpenReview ↗

1. Reliability decrement: interpretation and sensitivity

Thank you for highlighting the decrement’s role. The default wdrop=0.01w_{\mathrm{drop}}=0.01 corresponds to a tolerance of 100 accumulated low-overlap events before reliability reaches zero. This accumulates evidence across repeated disagreements rather than exhausting reliability after a few isolated mismatches. It is not a fixed 100-token cutoff: compatible positions do not consume this tolerance, so the resulting effective length depends on the trajectory.

Our existing sensitivity experiments quantify how the reliability parameters affect the accuracy–efficiency trade-off. On the JustRL pair, the overlap-threshold sweep in Appendix A.10 gives:

Overlap threshold γ\gammaMean accuracy (%)Training time (h)
0.639.84.8
0.7 — default39.74.1
0.838.73.1
0.938.22.4
OPD baseline39.16.9

The default offers a favorable trade-off: it retains nearly the same Mean as γ=0.6\gamma=0.6 at lower cost, whereas more aggressive thresholds shorten training further but reduce accuracy. Appendix A.7.4 additionally varies wbasew_{\mathrm{base}} over 0, 0.5, and 1.0, obtaining Mean accuracies of 38.9%, 39.7%, and 39.6%, respectively.

For the decrement specifically, our evidence concerns transfer of a shared setting: the same wdrop=0.01w_{\mathrm{drop}}=0.01 and controller configuration are used across all five mathematical model pairs, producing short, growing, or sustained long budgets. The additional coding experiment below also retains these settings without retuning, reducing training time by 36.3% with a 0.2-percentage-point score decrease.

We therefore regard 0.01 as a practical, transferable default in the evaluated settings, not a theoretically optimal value. We agree that the existing sweeps and transfer results do not replace a direct decrement sweep or a joint sweep with γ\gamma, and will make that distinction explicit in the revision.

AuthorsOpenReview ↗

2. Transfer beyond mathematical reasoning

We add a code-generation distillation experiment, training on TACO [1] and evaluating on the newly added problems in LiveCodeBench v6 [2].

We use DeepSeek-R1-Distill-Qwen-1.5B as the student and DeepCoder-1.5B-Preview [3] as the teacher. Importantly, all controller settings are unchanged from the mathematics experiments, without coding-specific retuning. Both methods use ** training steps** and the same evaluation protocol.

MethodLiveCodeBench v6 new subset: Score (%)Time reduction
OPD25.1—
Prune-OPD24.936.3%

Prune-OPD reduces training time by 36.3% while retain almost the same accuracy. This directly evaluates the controller on non-mathematical training, rather than only cross-domain evaluation of a mathematics-trained model. The result supports transfer of the existing controller settings to coding. We will incorporate it while retaining broader instruction-following and multi-turn transfer as limitations.

Reviewer_crEfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Prune-OPD, which accelerates the OPD process for long-horizon reasoning by cutting long suffixes that are identified as weakly compatible. It leverages the mismatch between the teacher and student (via top-k candidate overlap) as a reliability signal. Based on this overlap, the authors propose token-level reward reweighting and an adaptive rollout budget controller. Experiments on five mathematical reasoning benchmarks across multiple student-teacher pairs show that Prune-OPD significantly decreases training time while preserving baseline OPD accuracy.

Critical strengths
  1. The motivation is clear and the idea is intuitive.
  2. Experiments show that Prune-OPD effectively decreases training time.
Critical weaknesses
  1. The method design is overly heuristic. Leveraging unweighted top-k overlap to represent supervision quality is heuristic and ignores the actual probability mass. Taking it as a reliability score is a plausible guess, but it remains doubtful whether it is a generally optimal choice. Deeper investigation is warranted.
  2. The experimental setup uses relatively weak teachers, which are either of the same size as the student or only slightly larger without dedicated reasoning RL optimization. Evaluating the method under stronger and larger teacher setups is necessary to validate its practical utility.
  3. The effectiveness of reward reweighting appears marginal, and the accuracy gains look more like optimization noise. To show that it truly works, statistical analysis across multiple runs is needed to validate its effectiveness.
  4. A more detailed wall-clock speedup breakdown should be reported, so that readers can better understand how much of the speedup comes from mitigating rollout stragglers versus increasing microbatch parallelism under the specific token budget.
Questions for authors
  1. Have you tried using probability-aware divergence metrics (e.g., top-k KL or JSD) or other alternatives as a proxy for the reliability score?
  2. Does the conclusion generalize to larger and stronger reasoning teachers?
  3. Have you profiled a more detailed stage-wise breakdown of the wall-clock speedup?
  4. Can this method generalize to other domains beyond mathematics?
10 comments
AuthorsOpenReview ↗

Response to Reviewer crEf

Thank you for recognizing the clear motivation and training-time reductions. We address your concerns about the compatibility metric, teacher strength, reward reweighting, runtime breakdown, and transfer beyond mathematics below.

AuthorsOpenReview ↗

1. Compatibility metric and probability-aware alternatives

Our contribution is to use student–teacher mismatch to guide OPD computation, rather than to establish top-kk overlap as a universally optimal reliability measure. Overlap measures agreement between high-ranked continuations at student-generated prefixes using candidate information already available in the OPD pipeline. It is a compatibility proxy, not a correctness probability or a direct measurement of gradient usefulness.

The original manuscript already evaluates an alternative signal: whether the sampled student token belongs to the teacher’s visible top-pp candidate set. On DeepSeek-R1-Distill-Qwen-1.5B / JustRL-DeepSeek-1.5B, both signals yield substantial savings:

MethodMean accuracy (%)Time (h)Time reduction
OPD39.16.9—
Prune-OPD (top-pp)39.04.337.7%
Prune-OPD (overlap)39.74.140.6%

Across the four pairs in Table 1, overlap is faster than the evaluated top-pp variant in every pair and matches or exceeds its accuracy in 18 of 20 benchmark comparisons. These results support the chosen configuration while demonstrating that the efficiency opportunity is not exclusive to overlap.

We agree that unweighted overlap omits probability mass. The top-pp variant uses teacher probabilities to construct the acceptance set, but it is not a substitute for a direct KL/JSD comparison. We have not established superiority over those alternatives and will make this scope explicit. Our empirical claim is that compatibility feedback provides an effective signal for computational allocation, not that the current metric is generally optimal.

AuthorsOpenReview ↗

2. Teacher strength and reasoning specialization

We would like to clarify that our experiments are not restricted to teachers without dedicated reasoning-RL optimization. JustRL-DeepSeek-1.5B and Skywork-OR1-7B are explicitly reasoning-RL-scaling models. Their training procedures are documented in the corresponding releases and technical reports. (arxiv.org)

The original experiments cover both reasoning specialization and a teacher–student size gap:

StudentTeacherSetting
DeepSeek-R1-Distill-Qwen-1.5BJustRL-DeepSeek-1.5BSame-size, reasoning-RL teacher
DeepSeek-R1-Distill-Qwen-1.5BDeepSeek-R1-Distill-Qwen-7BLarger teacher
DeepSeek-R1-Distill-Qwen-7BSkywork-OR1-7BReasoning-RL teacher; high compatibility

The Skywork setting is particularly informative. Prune-OPD retains long rollouts and stays within 0.6 percentage points of OPD on every benchmark, whereas fixed-4K truncation loses accuracy on all five benchmarks, including 3.7 points on AIME25. Its time reduction is only 2.9% because little supervision is removed.

This is the intended behavior: Prune-OPD does not force aggressive truncation when the compatibility feedback supports retaining long supervision. Nevertheless, we distinguish these results from validation with substantially larger teachers beyond the evaluated scale, which remains an extension rather than an established claim.

Reviewer_kAqoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Prune-OPD, which uses top-k overlap to identify teacher-student compatibility, and adjust rewards weight or training rollout lengths dynamically. Experiments on five student–teacher pairs show substantial training-time reductions on four pairs while broadly preserving or improving accuracy.

Critical strengths
  1. The method adjusts training rollout lengths online based on student–teacher compatibility, making the supervision horizon responsive to learning progress while directly reducing generation and teacher-scoring costs.
  2. The method achieves substantial training-time savings while broadly preserving or improving accuracy across multiple student–teacher pairs.
Critical weaknesses
  1. The core design would benefit from a more direct connection to the training objective: it addresses a supervision problem indirectly through agreement between teacher and student predictions. This gap leaves both the definition of overlap and the subsequent parameter choices heuristic and empirically justified.
  2. Although fixed-length baselines are included, a finer sweep of truncation lengths would provide a more comprehensive picture. Comparison with simple heuristic length scheduling method such as [1] should also be included.

[1] Fast and Effective On-Policy Distillation from Reasoning Prefixes. Dongxu Zhang et al. Findings of ACL, 2026.

Questions for authors

See weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer kAqo

Thank you for your constructive comments. We have expanded the fixed-length comparisons and implemented the suggested prefix schedule. Prune-OPD achieves both higher Mean accuracy and lower training time than the tested linear schedule on both model pairs, strengthening the evidence for compatibility-guided adaptation.

AuthorsOpenReview ↗

1. Connection between compatibility and the training objective

Our contribution is not simply a particular overlap formula: it is that student–teacher mismatch exposes an actionable opportunity to improve OPD efficiency. OPD computes teacher feedback at student-generated prefixes, but a fixed rollout budget does not use the agreement between these distributions to decide how far to generate and train. Prune-OPD uses this information to adapt the supervision horizon.

Two sets of experiments support this principle beyond one metric or domain.

An alternative compatibility signal is already evaluated in the manuscript. Table 1 and Appendix A.4 include both top-kk overlap and an alternative based on whether the sampled student token belongs to the teacher’s visible top-pp candidate set. On DeepSeek-R1-Distill-Qwen-1.5B / JustRL-DeepSeek-1.5B, both substantially reduce training time while broadly retaining accuracy:

MethodAMC23AIME24AIME25HMMT24HMMT25MeanTime (h)Time reduction
OPD77.543.532.723.518.139.16.9—
Prune-OPD (top-pp)77.344.030.823.319.839.04.337.7%
Prune-OPD (overlap)78.445.233.522.718.839.74.140.6%

Additional coding results demonstrate transfer without controller retuning. We train DeepSeek-R1-Distill-Qwen-1.5B with DeepCoder-1.5B-Preview on TACO and evaluate on the newly added LiveCodeBench v6 subset. All controller settings are unchanged from the mathematics experiments.

MethodLiveCodeBench v6 new subset: Score (%)Time reduction
OPD25.1—
Prune-OPD24.936.3%

The alternative metric shows that the efficiency opportunity is not exclusive to the overlap definition. The coding result extends the evidence beyond mathematics: the same controller settings reduce training time by 36.3% with a 0.2-percentage-point score decrease.

We agree that overlap is not a direct measure of gradient usefulness, and the specific controller involves heuristic choices. However, these results support a broader contribution than an isolated heuristic: compatibility feedback can guide computational allocation across different signals and tasks. They demonstrate practical transfer of the shared settings, not parameter optimality. We will make this distinction clearer in the revision.

AuthorsOpenReview ↗

2. Advantages over fixed truncation and prefix scheduling

Thank you for pointing out Fast and Effective On-Policy Distillation from Reasoning Prefixes [1]. Its demonstration that short prefixes can support effective distillation, together with progressive prefix expansion, is a valuable contribution closely related to our work. We appreciate you identifying this omission and will add the citation and an explicit comparison.


Expanded fixed-length comparisons. The original manuscript already includes fixed-12K OPD and fixed-4K OPD. We now add fixed-1K and fixed-512 OPD on DeepSeek-R1-Distill-Qwen-1.5B / JustRL-DeepSeek-1.5B. All four fixed-length baselines use standard OPD without reward reweighting, varying only the rollout cap.

MethodRollout capAMC23AIME24AIME25HMMT24HMMT25MeanTime (h)
OPD (fixed 12K)12,28877.543.532.723.518.139.16.9
OPD (fixed 4K)4,09677.843.131.022.717.938.54.8
OPD (fixed 1K)1,02473.142.331.321.516.937.02.8
OPD (fixed 0.5K)51270.341.330.020.617.135.91.9
Prune-OPD (ours)Adaptive78.445.233.522.718.839.74.1

Scores are percentages; Mean equally averages the five benchmarks. Fixed-12K, fixed-4K, and Prune-OPD results are from the original manuscript.

None of the tested fixed cutoffs matches Prune-OPD’s accuracy. Prune-OPD is both more accurate and faster than fixed-12K and fixed-4K. More aggressive truncation is faster, but fixed-1K and fixed-512 reduce Mean accuracy to 37.0% and 35.9%, compared with 39.7% for Prune-OPD.

These results expose the limitation of a single cutoff: shorter budgets save computation but sacrifice performance, while longer budgets retain supervision at a higher cost throughout training. Prune-OPD adapts the budget instead of committing to this same choice for the entire run.


Direct comparison with linear prefix scheduling. We implement the cited work’s linear growth rule within our OPD setup:

Lt=min⁡(1+256(t−1),Mmax⁡),Mmax⁡=12,288.L_t=\min\left(1+256(t-1),M_{\max}\right),\qquad M_{\max}=12{,}288.

The scheduling baseline retains the standard OPD objective and uses the same training and evaluation setup.

Student / TeacherMethodMean accuracy (%)Time (h)
DeepSeek-R1-Distill-Qwen-1.5B / JustRL-DeepSeek-1.5BOPD39.16.9
OPD + linear prefix scheduling [1]39.36.1
Prune-OPD39.74.1
Qwen3-4B-Base / Qwen3-4B (Non-thinking)OPD15.328.5
OPD + linear prefix scheduling [1]15.424.7
Prune-OPD17.213.5

The linear schedule improves efficiency over full-budget OPD, confirming the value of progressive prefix training. Prune-OPD improves on this baseline on both pairs:

  • JustRL: Mean accuracy increases from 39.3% to 39.7%, with 32.8% less training time.
  • Qwen3-4B: Mean accuracy increases from 15.4% to 17.2%, with 45.3% less training time.

The key distinction is feedback-guided adaptation rather than prescribed growth. Progressive expansion is consistent with the growing effective lengths observed on JustRL. However, effective lengths remain short on the low-compatibility Qwen3 pair, where automatically increasing the budget need not be appropriate. Prune-OPD uses the same compatibility-based controller to respond to both situations, instead of increasing length regardless of the observed supervision.

We acknowledge that further tuning a fixed cutoff or schedule may find other favorable trade-offs than the original setting in paper[1]. Prune-OPD addresses precisely the question of what information should guide those choices: it uses evolving student–teacher compatibility to select and adjust rollout budgets, rather than requiring a cutoff or growth trajectory to be prescribed beforehand. The new comparisons demonstrate the practical benefit of this distinction over the tested fixed cutoffs and linear schedule.

We hope these additional results and clarifications address your concerns about the method’s empirical basis and its advantage over simpler length-control strategies.


Reference

[1] Zhang et al. Fast and Effective On-Policy Distillation from Reasoning Prefixes. Findings of ACL, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.