- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes Prune-OPD, which uses teacher–student prediction overlap to adapt the computational budget of on-policy distillation. At each generated position, cumulative low-overlap events determine a declining reliability score. This score both reweights the distillation signal and estimates an effective training length. A batch-level controller then adjusts the maximum length of future rollouts, expanding the budget when reliable supervision reaches the current cap and contracting it after repeated evidence that the cap is unnecessarily long.
Experiments cover five teacher–student pairs on mathematical reasoning tasks. Across four pairs, the method reduces training time by 37.6%–68.0% while broadly preserving accuracy. For a high-compatibility pair, it retains longer rollouts and remains close to baseline accuracy. Component ablations, fixed-budget comparisons, and parameter sweeps examine the contributions of weighting and continued length adaptation.
Critical strengths
The most interesting contribution is the use of compatibility feedback to adjust rollout length during training. This avoids committing to a fixed cutoff that may become inappropriate as the student changes. Figure 2 provides compelling evidence that the method reaches comparable accuracy in substantially less training time across several teacher–student pairs. Large accuracy improvements are not necessary for this contribution: preserving performance while reducing computation is a meaningful result. The high-compatibility experiment further strengthens the case, showing that the controller retains long supervision when aggressive fixed truncation would hurt accuracy.
The experiments distinguish reward weighting from adaptive truncation and show that truncation supplies most of the speedup. The fixed-initial-budget comparison also demonstrates why continued adaptation matters: a short initial cap can become restrictive later in training. Together with the threshold sweeps and repeated runs for one pair, these analyses make the empirical contribution substantially clearer than a comparison of the complete method against standard OPD alone.
Critical weaknesses
Evaluation is limited to mathematical reasoning. Although the paper considers several teacher–student pairs, both training and evaluation remain within mathematics. It is therefore unclear how well the overlap-based controller transfers to coding, general instruction following, or other tasks with different response structures. This limits the demonstrated scope, but does not undermine the reported efficiency gains.
Some controller choices remain insufficiently justified. In particular, the reliability decrement of 0.01 makes the score reach zero after 100 low-overlap events. This directly affects effective training lengths and subsequent rollout budgets, yet its selection and sensitivity are not established by the reported overlap-threshold and base-weight sweeps. Understanding this choice would help assess how much tuning is needed when applying the method to a new setting.
Questions for authors
Parameter selection and sensitivity. How was the reliability decrement of 0.01 chosen? Does the accuracy–time trade-off remain stable when this decrement changes, or when it is varied jointly with the overlap threshold? This would help determine whether the controller transfers readily across settings without additional tuning.
Transfer beyond mathematics. Are any results available on a non-mathematical training task using the same controller settings? If not, please clarify which characteristics of mathematical reasoning the current design may depend on.
Improvement beyond the cutoffs. Figure 2 shows impressive learning efficiency in the early stages of training. Does the method offers sustained improvement? If not, why couldn't it?
7 comments
1. Reliability decrement: interpretation and sensitivity
Thank you for highlighting the decrement’s role. The default corresponds to a tolerance of 100 accumulated low-overlap events before reliability reaches zero. This accumulates evidence across repeated disagreements rather than exhausting reliability after a few isolated mismatches. It is not a fixed 100-token cutoff: compatible positions do not consume this tolerance, so the resulting effective length depends on the trajectory.
Our existing sensitivity experiments quantify how the reliability parameters affect the accuracy–efficiency trade-off. On the JustRL pair, the overlap-threshold sweep in Appendix A.10 gives:
| Overlap threshold | Mean accuracy (%) | Training time (h) |
|---|---|---|
| 0.6 | 39.8 | 4.8 |
| 0.7 — default | 39.7 | 4.1 |
| 0.8 | 38.7 | 3.1 |
| 0.9 | 38.2 | 2.4 |
| OPD baseline | 39.1 | 6.9 |
The default offers a favorable trade-off: it retains nearly the same Mean as at lower cost, whereas more aggressive thresholds shorten training further but reduce accuracy. Appendix A.7.4 additionally varies over 0, 0.5, and 1.0, obtaining Mean accuracies of 38.9%, 39.7%, and 39.6%, respectively.
For the decrement specifically, our evidence concerns transfer of a shared setting: the same and controller configuration are used across all five mathematical model pairs, producing short, growing, or sustained long budgets. The additional coding experiment below also retains these settings without retuning, reducing training time by 36.3% with a 0.2-percentage-point score decrease.
We therefore regard 0.01 as a practical, transferable default in the evaluated settings, not a theoretically optimal value. We agree that the existing sweeps and transfer results do not replace a direct decrement sweep or a joint sweep with , and will make that distinction explicit in the revision.
2. Transfer beyond mathematical reasoning
We add a code-generation distillation experiment, training on TACO [1] and evaluating on the newly added problems in LiveCodeBench v6 [2].
We use DeepSeek-R1-Distill-Qwen-1.5B as the student and DeepCoder-1.5B-Preview [3] as the teacher. Importantly, all controller settings are unchanged from the mathematics experiments, without coding-specific retuning. Both methods use ** training steps** and the same evaluation protocol.
| Method | LiveCodeBench v6 new subset: Score (%) | Time reduction |
|---|---|---|
| OPD | 25.1 | — |
| Prune-OPD | 24.9 | 36.3% |
Prune-OPD reduces training time by 36.3% while retain almost the same accuracy. This directly evaluates the controller on non-mathematical training, rather than only cross-domain evaluation of a mathematics-trained model. The result supports transfer of the existing controller settings to coding. We will incorporate it while retaining broader instruction-following and multi-turn transfer as limitations.
Response to Reviewer cfBY
Thank you for your thoughtful review and positive assessment of our work. We address your three questions below.