acceptodds
Under review as a conference paper at ICLR 2027

Dr. OPD: Learning What to Follow for Optimal On-Policy Distillation of Large Language Models

25 views

Abstract

On-policy distillation (OPD) trains a student on its own generated responses using dense, token-level supervision from a stronger teacher. Vanilla OPD treats all teacher signals equally, assuming that the teacher's supervision is equally important for every token. However, teacher signals at different tokens may have very different effects on the student's performance: some correct important reasoning errors, while others have little effect on the final answer. Motivated by this observation, we introduce **Dr. OPD** (*OPD Done Right*), which defines the optimal weighted OPD to maximize the student's performance. We formulate Dr. OPD as a bilevel optimization problem in which the student learns from weighted teacher supervision, while the weights are selected to maximize the expected reward of the resulting student. To solve Dr. OPD, we develop an efficient iterative solver that updates the token weights and student policy alternatively. At each round, it updates weights in closed form and then takes one gradient step on the resulting weighted OPD objective. Under regularity conditions, we show that this weighted update achieves a higher expected reward than a vanilla OPD update. Empirically, across strong-to-weak and same-size distillation on math and code, Dr. OPD consistently outperforms all evaluated baselines. In particular, in the strong-to-weak distillation setting, Dr. OPD improves average math performance by points over vanilla OPD, and enables the smaller student to surpass its larger teacher.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Optimal On-policy Distillation, Reinforcement Learning, Bilevel Optimization, Token-level Credit Assignment

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WTyhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to weight teacher signals in on-policy distillation according to their usefulness for student performance. It formulates a bilevel objective and derives a clipped weighting rule from a local reward bound for a one-step student update. Token credits measure alignment between teacher-correction and reward-gradient directions. The practical algorithm approximates this direction and uses Jacobian–vector products, reward momentum, optimizer scaling, and normalization. Experiments evaluate cross-scale Qwen3 distillation on mathematics and code, and same-size distillation on mathematics. The method achieves the highest reported average scores among the compared methods in each evaluated setting, with students exceeding their teachers in some mathematical settings.

Critical strengths

The paper adopts a meaningful objective for assessing teacher supervision. Matching a teacher's output distribution more closely does not necessarily improve the student's task performance. Using outcome rewards to determine the strength of individual teacher corrections is a reasonable direction. The proposed token-level weighting retains fine-grained teacher supervision while providing a concrete mechanism for incorporating task feedback into distillation.

The experiments report substantial gains across several teacher–student configurations. Table 1 (p. 8) reports an increase in Qwen3-1.7B's average mathematics score from 25.3 under vanilla OPD to 35.0, and its code score from 43.4 to 48.4. Same-size distillation also shows positive gains. The gains across both cross-scale and same-size settings support the method’s potential practical value. The JVP implementation avoids explicitly constructing a parameter-dimensional gradient for each token, making the proposed credit computation practically implementable.

Critical weaknesses

Model selection and incomplete attribution limit how confidently the gains can be interpreted.

Section 5.1 (p. 7) selects the best checkpoint after evaluation every ten steps on the evaluated datasets, and §5.3 and Table 3 (pp. 8–9) select λ = 0.4 after a sweep. An independent validation-based selection procedure is not described. Table 5 (p. 15) lists fixed coefficients or defaults for competing methods without reporting comparable searches, and the main tables do not report uncertainty across independent training runs. The extent to which selection, tuning, and run variation affect the reported gains therefore remains unclear. OPD+GRPO and GRPD are useful reward-aware controls, but do not fully isolate the contribution of token-credit estimation from simpler weighting, reward momentum, or optimizer scaling. The paper reports neither these component ablations nor direct comparisons with TrOPD and IW-OPD. These gaps leave the size of the estimator’s independent benefit unresolved, despite the substantial improvements over the evaluated configurations.

The contribution and guarantee should be delimited more precisely relative to prior work and the implementation.

Theorem 5 (p. 5) establishes a conditional guarantee for an idealized one-step update. The implementation instead uses an approximate reward direction (p. 6), lagged momentum, optimizer scaling, and PPO-style clipping (p. 17). These choices are disclosed, and Appendix D.3 (p. 20) explicitly states that global convergence is not established. The theoretical result therefore provides a local justification, while its correspondence to the implemented update requires further assessment. The “optimal weighted OPD” framing should preserve this distinction. The discussion of gradient-based reweighting and Unmasking OPD (pp. 9 and 14) appropriately identifies relevant precedents, but the specific benefit of the derived weighting rule and parameter-space estimator needs stronger isolation. Component ablations would help establish that benefit and the relevance of the local analysis. Similarly, a single JVP avoids naive per-token gradient construction, but does not quantify the claimed small end-to-end overhead (p. 6).

Questions for authors

Selection protocol and uncertainty: What data were used to select checkpoints and λ, and was one checkpoint used across benchmarks? Can the authors provide validation-only selection results, or fixed-step results if no validation set was used, together with uncertainty across independent training runs? This would clarify the stability of the main gains.

Added value of token credits: Beyond the existing reward-aware baselines, can the authors compare with simple response-reward weighting or shuffle token weights while preserving their distribution? Please also isolate reward momentum and optimizer scaling to determine which components account for the improvement.

Closest comparisons and tuning: Can the authors provide comparable tuning budgets and a direct comparison with a closely related weighting method, such as TrOPD or IW-OPD, or explain material incompatibilities? Please distinguish the contribution from prior gradient-based reweighting and alignment diagnostics, which the paper already acknowledges.

Theory–implementation relationship: Which implementation steps retain the assumptions of Theorem 5, and which should be understood as heuristics motivated by the local analysis? Can a small-scale comparison with the more direct reward-direction estimate assess the approximation? Please state the scope of the guarantee separately from the empirical justification for these approximations.

Measured cost and teacher comparisons: Can the authors report JVP and reward-gradient overhead, peak memory, and total generated tokens, and compare methods at similar measured compute budgets? Please also provide paired per-question uncertainty for small teacher–student differences, distinguishing these claims from the larger improvements over vanilla OPD.

5 comments

Thank you for the detailed review. Our additional experiments in the Qwen3-1.7B student / Qwen3-4B teacher setting address both weaknesses and all five questions:

  • W1 / Q1 — Independent selection and uncertainty. We hold out 1,000 DeepMath questions, excluded from training and testing, to select λ and each method’s checkpoint. Each selected checkpoint is used across all five math benchmarks. Across three training seeds, Dr. OPD improves average accuracy by 7.01–7.90 pp over OPD, GRPD, OPD+GRPO, and ExOPD; every paired 95% CI is above zero.
  • W1 / Q2 — Added value of token credits. Matched single-run ablations show that shuffling weights within responses while preserving their distribution, or replacing token credits with response-level reward weights, reduces generation-weighted test accuracy by 6.71 and 7.17 pp. Removing reward momentum costs 3.46 pp; removing optimizer scaling yields no significant difference. These controls identify token-specific assignment and reward momentum as key contributors.
  • W1 / Q3 — Closest comparisons and tuning. With matched data order, a 50-update budget, and independent validation-based checkpoint selection, Dr. OPD outperforms IW-OPD by 6.99 pp in generation-weighted test accuracy, with a strictly positive paired 95% CI. We use published baseline defaults and reuse our once-selected λ. Our credits optimize the updated student’s reward, distinguishing them from discrepancy weighting, validation-loss example reweighting, and alignment diagnostics.
  • W2 / Q4 — Theory and implementation. “Optimal weighted OPD” defines the bilevel optimization target; the theorem guarantees a local, single-step reward improvement over vanilla OPD under its stated conditions. Empirically, the practical reward direction outperforms a direct current-policy estimate without momentum or scaling by 2.15 pp (95% CI: [1.13, 3.17]), supporting the implementation’s direction estimate.
  • W2 / Q5 — Computational cost and teacher comparisons. On eight H100 GPUs, the run time per step increases from 156.4 to 246.3 seconds, with almost all overhead attributable to credit computation. Dr. OPD’s step-20 checkpoint outperforms every baseline checkpoint through step 50 in all three seeds. Paired teacher comparisons favor Dr. OPD on AIME24 and the teacher on AMC, with unresolved differences elsewhere; gains over OPD are significant on all five math benchmarks.

Moreover, during the rebuttal period, we added Llama-3.2-3B→1B experiments on non-verifiable tasks. Dr. OPD improves on the initial student across all three benchmarks, while every baseline degrades it, supporting effectiveness beyond the Qwen model family and verifiable tasks (updated manuscript, Section E.1.2, p. 23).

These additional results directly address the weaknesses and questions raised and further strengthen the evidence for our method’s superiority over competing methods across benchmarks. We would greatly appreciate your consideration of raising the score.

Next, we answer each weakness and answer in order with more detailed explanations.

Weakness 1 (Q1–Q3): Model selection and attribution. Checkpoints and were selected on the test benchmarks without run-level uncertainty, and the benefit of token credits was not isolated from simpler variants or the closest methods.

Q1. Selection protocol and uncertainty

Independent validation and one checkpoint across benchmarks. In the Qwen3-1.7B student / Qwen3-4B teacher setting, we held out 1,000 DeepMath questions, excluded from training and all final test sets. The validation sweep in Table 1 selects λ=0.4\lambda=0.4, with the highest validation accuracy of 56.80% at step 30, matching the value used in the paper.

Table 1. Validation accuracy (%) across and training steps.

1020304050
0.246.4049.2050.1653.2953.15
0.348.8650.1254.9454.5555.57
0.447.4455.7956.8053.6053.52
0.546.2647.8355.7655.6753.76
0.650.5454.9456.2952.9953.81

We fix =0.4 for the repeated runs. For every method and run, we select the checkpoint with the highest accuracy on this same validation set among steps 10, 20, 30, 40, and 50. That single checkpoint is used for all five test benchmarks.

Multi-seed results and confidence intervals. We repeated Dr. OPD and all four baselines with three seeds in this main setting, using eight responses per test question. Table 2 reports mean accuracy and standard errors; Table 3 reports 95% CIs for the gains.

Table 2. Math accuracy (%), mean ± SE over three seeds.

MethodAIME24AIME25AMCMinervaOlympiadBenchAverage
Dr. OPD23.89±1.3319.31±0.1454.12±0.4327.42±0.3743.42±0.1133.63±0.31
OPD18.61±0.5012.36±1.0045.33±0.9420.88±0.5335.91±0.5726.62±0.12
GRPD15.14±0.5613.05±1.4144.58±0.5221.36±0.1535.97±0.4126.02±0.10
OPD+GRPO15.42±0.4810.28±0.3745.08±0.2021.52±0.2936.38±0.2825.73±0.01
ExOPD17.64±0.7411.25±0.4845.48±1.0921.26±0.3436.32±0.0626.39±0.39

Table 3. 95% CIs for accuracy gains.

Comparison / benchmark95% CI (pp)
A. Average accuracy: Dr. OPD versus each baseline
Dr. OPD − OPD[+5.26,+8.76]
Dr. OPD − GRPD[+5.87,+9.35]
Dr. OPD − OPD+GRPO[+6.51,+9.28]
Dr. OPD − ExOPD[+6.27,+8.21]
B. Per-benchmark accuracy: Dr. OPD versus OPD
AIME24 (30 questions)[+2.92,+8.33]
AIME25 (30 questions)[+4.17,+10.00]
AMC (83 questions)[+6.17,+11.60]
Minerva (272 questions)[+4.87,+8.32]
OlympiadBench (673 questions)[+6.67,+8.40]

Dr. OPD achieves the highest mean accuracy on every benchmark. All reported gain intervals are strictly positive, supporting a significant improvement over all four baselines on average and over OPD on each benchmark under validation-only selection.

We also added three-seed Llama-3.2-1B/3B-Instruct experiments on non-verifiable tasks, where Dr. OPD significantly outperforms all four baselines, while vanilla OPD and the other baselines degrade the initial student, consistent with prior findings on low-quality or repetitive OPD outputs (Gu et al., 2024; Cui et al., 2026).

Given the rebuttal time constraints, we focused repeated runs on this main setting and will extend the multi-seed evaluation to the remaining settings in the camera-ready version.

Q2. Added value of token credits

We added four controlled ablations in the Qwen3-1.7B student / Qwen3-4B teacher setting. Table 4 reports paired question-bootstrap 95% CIs for differences from full Dr. OPD. The results highlight the importance of token-specific credit assignment and reward momentum, with more details in Section E.1.3 (p. 24) of the updated manuscript.

Table 4. Component ablations with 95% CIs.

VariantValidation difference (pp, 95% CI)Test difference (pp, 95% CI)Test accuracy (%)
Full Dr. OPD——33.02
Token shuffle−14.05[−15.88,−12.24]−6.71[−7.76,−6.00]26.55
Response reward weighting−13.81[−15.65,−12.05]−7.17[−8.24,−6.12]25.95
No reward momentum−7.11[−8.62,−5.61]−3.46[−4.40,−2.54]30.10
No optimizer scaling+0.41[−0.98,+1.81]+0.83[−0.14,+1.78]34.17
OPD−14.85[−16.78,−12.99]−6.70[−7.73,−5.70]26.79

Q3. Closest comparisons and tuning

Thank you for this suggestion. We added a direct comparison with IW-OPD, a closely related token-weighting method, in the Qwen3-1.7B student / Qwen3-4B teacher setting.

1. Direct comparison with IW-OPD. The comparison matches the training seed, data order, and 50-update budget; each method selects one checkpoint using the same independent validation set. We preserve IW-OPD's original prefix-discrepancy weighting and default γ=0.5\gamma=0.5. Table 5 reports paired question-bootstrap 95% CIs for differences from Dr. OPD.

Table 5. IW-OPD comparison with 95% CIs.

MethodValidation difference (pp, 95% CI)Test difference (pp, 95% CI)Test accuracy (%)
Dr. OPD——33.02
IW-OPD−14.11[−15.96,−12.29]−6.99[−7.97,−6.03]26.34
OPD−14.85[−16.78,−12.99]−6.70[−7.73,−5.70]26.79

Dr. OPD significantly outperforms IW-OPD on both validation and test performance.

A faithful comparison with TrOPD requires teacher-generated prefixes, student continuations, and forward-KL guidance; its FKL outlier variant also uses teacher top-64 probabilities. This requires extending our student-only rollout/scoring pipeline beyond token weighting. We therefore report the completed IW-OPD comparison under the shared pipeline; a full TrOPD evaluation is not included here.

2. Hyperparameter protocol. For Dr. OPD, we select once on the independent 1,000-question DeepMath validation set in the Qwen3-1.7B student / Qwen3-4B teacher setting. We then reuse λ=0.4\lambda=0.4 across all other settings. For the baselines, we retain the authors' reported method-specific coefficients under our shared training protocol. These coefficients have already been tuned by the original authors in closely related Qwen3 distillation settings (RL-trained 4B teachers in ExOPD and OPDVR, and a Qwen3-4B-Instruct-2507 teacher in IW-OPD), so we use them as is rather than re-tuning them.

3. Distinction from gradient-based reweighting and diagnostics. Prior gradient-based reweighting, such as Ren et al. (2018), selects example weights using a clean validation-loss objective. Dr. OPD selects weights for individual teacher-token corrections using the updated student's outcome reward. Our local reward lower bound yields bounded weights around vanilla OPD, and a shared parameter-space reward direction enables efficient credit estimation through JVP. While Unmasking OPD studies token-level alignment as a training-free diagnostic, we show that for any positive weighting function, the weighted OPD objective remains a valid distillation objective with the same teacher-matching target. We then use reward-sensitive credits to select token weights according to their effect on the updated student's outcome reward.

Weakness 2 (Q4, Q5): Scope of the guarantee. Theorem 5 covers an idealized one-step update, so the “optimal weighted OPD” framing, its link to the implementation, and the JVP overhead need clarification.

Thank you for highlighting this distinction. We clarify the optimization target, the result proved for the solver, and its scope.

Optimization target. “Optimal weighted OPD” defines the bilevel objective: choose feasible positive weights to maximize the outcome reward of the student trained with that weighted distillation objective.

Local optimality and the guarantee. Our solver replaces full inner training with one student update and constructs a quadratic local lower bound on the updated student's reward. The clipped rule

wt+=clip⁡(1+λct,wmin⁡,wmax⁡)w_t^+=\operatorname{clip}(1+\lambda c_t,w_{\min},w_{\max})

exactly maximizes this one-step surrogate over the bounded weights. Under Theorem 5's exact-credit, regularity, smoothness, and step-size conditions, it guarantees a nonnegative improvement in the true expected reward over a vanilla OPD update from the same student.

This is a local single-step guarantee. Global optimality of the bilevel problem and convergence of repeated practical updates are outside its scope. The practical reward-direction choices are assessed empirically through the direction ablations.

Reviewer_NRuROpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The author proposes an extension of On-Policy Distillation to cases where the reward function is available and comes up with an algorithm in which, during distillation, each token has its own weight. The weights are calculated with respect to the token's influence on the task reward, which is assumed to exist.

Critical strengths
  • The question of how to combine teachers' supervision with reward is still an open one, and the paper proposes an appealing solution that clearly outperforms the baselines.

  • Although it requires a few simplifications and assumptions, the final algorithm is simple and easy to implement.

Critical weaknesses

Looking at the experiment section, I noticed that most training curves end after 50 steps. Moreover, Dr. OPD is downtrending starting at step ~30. This makes me wonder if the algorithm is indeed better, or just has a stronger influence in the first few gradient steps. I think running the experiments for much longer (a few hundred steps) will help make a stronger claim here.

Questions for authors
  • What is the extra computational cost of adding a JVP for every step?

  • I would recommend adding CI for all the results in the table. Some of these datasets are very small (like AIME) and even with avg@16, a few points of improvement can be due to noise.

4 comments

Thank you for the positive assessment of reward-guided supervision. Our additional experiments in the Qwen3-1.7B student / Qwen3-4B teacher setting address the training-horizon, computational-cost, and uncertainty concerns:

  • Weakness — Training beyond the early updates. Across three training seeds, Dr. OPD outperforms every baseline at matched steps 20, 30, 40, and 50. We also extend one Dr. OPD run to 200 steps and evaluate eight checkpoints on a separate held-out 1,000-question validation set. Validation avg@8 is 53.52% at step 50, reaches 57.63% at step 80, and is 55.14% at step 200, showing that accuracy fluctuates without a consistent downward trend.
  • Q1 — Measured computational cost. On eight H100 GPUs, the run time increases from 156.4 s/step for OPD to 246.3 s/step for Dr. OPD, a 57.5% increase. Token-credit computation accounts for 88.4 s of the 89.9 s overhead, including 52.7 s for JVP. The remaining stages take 157.9 s, close to OPD’s 156.4 s, showing that almost all additional runtime is attributable to reward-sensitive credit computation.
  • Q2 — Repeated runs and confidence intervals. We evaluate three training seeds for Dr. OPD and all four baselines, selecting one checkpoint per run on an independent 1,000-question DeepMath validation set and generating eight responses per test question. Dr. OPD achieves 33.63 ± 0.31% average math accuracy, compared with 26.62 ± 0.12% for OPD (mean ± SE), and the highest mean accuracy on every benchmark. Paired 95% CIs are strictly positive against all four baselines on average and against OPD on each benchmark, including both 30-question AIME sets.

Moreover, during the rebuttal period, we added Llama-3.2-3B→1B experiments on non-verifiable tasks. Dr. OPD improves on the initial student across AlpacaEval, WritingBench, and WildBench, while every baseline degrades it, supporting effectiveness beyond the Qwen model family and verifiable tasks (updated manuscript, Section E.1.2, p. 23).

These additional results and clarifications directly address the concerns raised and strengthen the evidence for our method. We would appreciate your consideration of a higher score in light of these findings. Thank you.

Weakness: Training horizon and early stopping. Training curves end at 50 steps and Dr. OPD declines after step ~30, so the gain may come only from the early updates.

Thank you for raising this point. Table 1 shows every seed's validation curve in the Qwen3-1.7B student / Qwen3-4B teacher setting, using the same independent 1,000-question DeepMath validation set.

Table 1. Validation accuracy (%) by seed and training step.

MethodSeed1020304050
Dr. OPD047.4455.7956.8053.6053.52
149.3056.5156.2054.7553.11
250.0050.1456.0455.4352.76
OPD041.9541.3041.7541.5941.66
142.1641.0543.2943.1041.90
240.5540.1542.5242.9140.85
GRPD049.9542.2449.7844.6048.55
149.7040.7748.3645.7345.80
248.7044.5146.6544.1944.92
OPD+GRPO045.8645.0245.5744.5245.32
145.1046.8546.6243.9943.92
243.4846.2746.1445.8045.20
ExOPD041.8542.9144.5243.5843.01
144.2741.7644.0044.2242.89
241.1742.8444.0943.7442.83

By 50 steps, validation performance has largely plateaued or declined: all 15 runs reach their best observed accuracy by step 40, and none sets a new best at step 50. For a fair comparison, every method selects its best checkpoint using the same independent validation set, with one selected checkpoint per run used across all test benchmarks.

The advantage also holds at matched training steps: Dr. OPD outperforms every baseline at steps 20, 30, 40, and 50 in all three seeds. At step 50, its mean validation accuracy remains 53.13%, versus 41.47% for OPD and 46.42% for the strongest baseline, GRPD.

This behavior supports validation-based early stopping. ExOPD (Yang et al., 2026, Appendix B) also reports that extending distillation can reduce generalization through overfitting.

In an additional single-run study, we trained Dr. OPD for 200 steps and evaluated eight checkpoints on a separate held-out 1,000-question validation set. Table 2 below shows that Dr. OPD’s validation accuracy fluctuates over 200 training steps without exhibiting a consistent downward trend.

Table 2. Validation accuracy through 200 steps.

Checkpoint405080100120150160200
Validation avg@8 (%)53.6053.5257.6356.1853.8153.3453.7055.14

Q1. Measured JVP and training cost. What is the extra cost of the per-step JVP?

We evaluate training times in the Qwen3-1.7B / Qwen3-4B setting on eight H100 GPUs (Table 3). Dr. OPD adds 89.9 s per step, a 57.5% increase.

Table 3. Training time comparison.

MethodTime (s/step)
OPD156.4
Dr. OPD246.3(1.575×)

Table 4 shows that almost all additional time comes from token-credit computation.

Table 4. Per-step time breakdown.

ComponentFitted time (s/step)
Non-credit stages157.9
Token-credit computation (including JVP: 52.7 s)88.4
Total246.3

The credit block accounts for 88.4 s of the 89.9 s overhead, including 52.7 s for JVP. The remaining stages total 157.9 s, close to OPD's 156.4 s.

Q2. Confidence intervals on small benchmarks. Add CIs for all results, since small benchmarks such as AIME are noisy even with avg@16.

Thank you for this suggestion. We conducted three-seed runs for Dr. OPD and all four baselines in our main math setting: Qwen3-1.7B student / Qwen3-4B teacher. Each run selects one checkpoint on an independent 1,000-question DeepMath validation set and evaluates it with eight responses per question. Table 5 reports mean accuracy and standard errors; Table 6 reports 95% confidence intervals for the performance gains.

Table 5. Math accuracy (%), mean ± SE over three seeds.

MethodAIME24AIME25AMCMinervaOlympiadBenchAverage
Dr. OPD23.89±1.3319.31±0.1454.12±0.4327.42±0.3743.42±0.1133.63±0.31
OPD18.61±0.5012.36±1.0045.33±0.9420.88±0.5335.91±0.5726.62±0.12
GRPD15.14±0.5613.05±1.4144.58±0.5221.36±0.1535.97±0.4126.02±0.10
OPD+GRPO15.42±0.4810.28±0.3745.08±0.2021.52±0.2936.38±0.2825.73±0.01
ExOPD17.64±0.7411.25±0.4845.48±1.0921.26±0.3436.32±0.0626.39±0.39

Table 6. 95% CIs for accuracy gains.

Comparison / benchmark95% CI (pp)
A. Average accuracy: Dr. OPD versus each baseline
Dr. OPD − OPD[+5.26,+8.76]
Dr. OPD − GRPD[+5.87,+9.35]
Dr. OPD − OPD+GRPO[+6.51,+9.28]
Dr. OPD − ExOPD[+6.27,+8.21]
B. Per-benchmark accuracy: Dr. OPD versus OPD
AIME24 (30 questions)[+2.92,+8.33]
AIME25 (30 questions)[+4.17,+10.00]
AMC (83 questions)[+6.17,+11.60]
Minerva (272 questions)[+4.87,+8.32]
OlympiadBench (673 questions)[+6.67,+8.40]

Dr. OPD achieves the highest mean accuracy on every benchmark. All reported gain intervals are strictly positive: the improvement is significant over all four baselines on average and over OPD on each benchmark, including both AIME sets.

We also added three-seed Llama-3.2-1B/3B-Instruct experiments on non-verifiable tasks, where Dr. OPD significantly outperforms all four baselines, while vanilla OPD and the other baselines degrade the initial student, consistent with prior findings on low-quality or repetitive OPD outputs (Gu et al., 2024; Cui et al., 2026).

Given the rebuttal time constraints, we focused repeated runs on this main setting. We will extend the multi-seed evaluation to the remaining settings in the camera-ready version.

I thank the authors for their response and for the details about the extra time required to run their algorithm.

I still find it worrisome that all the learning happened within 20 steps, and after that, the performance just fluctuates. Looking at the results, it doesn't seem unique to your algorithm, which makes me think the experimental setting the authors chose is insufficient to properly test their algorithm. Just for context, OPD is a well-established algorithm, with previous papers [1,2] reporting increasing improvements over hundreds, if not thousands, of gradient steps.

[1] Agarwal, Rishabh, et al. "On-policy distillation of language models: Learning from self-generated mistakes." International Conference on Learning Representations. Vol. 2024. 2024. [2] Lu, Kevin and Thinking Machines Lab, "On-Policy Distillation", Thinking Machines Lab: Connectionism, Oct 2025.

Thank you for raising this concern. OPD’s convergence horizon depends on the student initialization, teacher–student pair, and batch size. In Qwen3-based OPD studies, rapid gains within the first 20–30 steps followed by plateaus or fluctuations have been repeatedly reported, including in ExOPD and IW-OPD, consistent with our observations. We also observe early gains followed by fluctuations in our Llama experiments on non-verifiable tasks, extending this observation beyond the Qwen3 family and mathematical reasoning.

We clarify the relevant experimental differences below, and we hope these clarifications address your concerns about the training horizon and the adequacy of our experimental setting.

For Agarwal et al. (2024) GKD

  • Initialization. They use pretrained, supervised-fine-tuned T5/FLAN-T5 models. We start from post-trained Qwen3 checkpoints, and Qwen3 is also widely used in subsequent OPD studies, including ExOPD and IW-OPD.
  • Batch size. Their XSum/GSM8K experiments use batches of 32 per step. Our updates use 128 prompts × 8 rollouts = 1,024 responses per step. Therefore, raw gradient-step counts alone do not provide a comparable training budget.
  • Training dynamics. Moreover, in their paper, they report results after 40,000 distillation steps, but do not report learning curves demonstrating continuous improvement throughout this horizon. It therefore remains unclear whether performance had already plateaued or begun fluctuating much earlier in training.

For Thinking Machines Lab TML

  • Reported rapid convergence. Their controlled comparison explicitly reports that “the AIME score is recovered in under 10 gradient steps, versus 70 steps for RL.” Their Figure 14 further shows that OPD performance plateaus around steps 10-20 and fluctuates thereafter.
  • Reported training dynamics. Their Figure 9 reports an AIME’24 score of 70% at approximately 150 steps. The accompanying figure shows only a few intermediate points plotted against compute FLOPS, without specifying their corresponding training steps, providing limited detail about the step-wise convergence trajectory.

Additional Evidence

  • ExOPD’s Figure 5(a) shows training reward rising rapidly within approximately 10–20 steps and subsequently fluctuating; Appendix B also reports that longer training can hurt generalization. IW-OPD’s Figure 2(a) similarly shows early gains followed by a plateau, while standard OPD declines after approximately step 30. Our additional single-run study extends Dr. OPD to 200 steps and shows validation performance fluctuating without a sustained decline.

Finally, our setting uses a fixed teacher. Longer-horizon improvements might be more often observed in self-distillation with evolving teachers, where the supervision itself changes throughout training, leading to different learning dynamics.SDPO, §4.3, SDFT, Figure 3.

Reviewer_Wci1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates policy distillation in large-scale models, pointing out that uniformly matched teacher distribution may not be optimal for students' final task performance. It proposes Dr. OPD: scoring each teacher's correction signal based on the improvement direction of student task rewards, thereby strengthening or weakening the supervisory weights of the corresponding tokens. The method derives pruned weights based on local reward lower bounds and efficiently computes credits using JVP. Theoretically, it provides a reward improvement guarantee under ideal single-step updates (not a global convergence guarantee). Experiments on multiple student models and mathematical and coding tasks show that this method outperforms the compared OPD baseline in terms of average task scores.

Critical strengths
  1. This paper clearly distinguishes between "approximating the teacher distribution" and "improving student task performance," and links the choice of teacher supervision weights to the student's outcome reward after training. This perspective is valuable: it transforms token weights from empirical rules into optimization variables with explicit performance objectives, providing a unified analytical framework for reward-guided distillation.

  2. This paper establishes a connection between weighted divergence and sampled token gradients, and derives closed-form weight updates using reward smoothness. Under precise credit and specified step size conditions, the reward improvement theorem provides a non-negative gain lower bound for the true expected reward relative to the vanilla OPD update in one step, rather than just an improvement in the surrogate objective. This provides a clear and testable theoretical basis for weight design.

  3. Calculating the inner product of parameter gradients per token is costly for large models. In engineering implementation, the paper utilizes the teacher-corrected structure to transform these inner products into log-probability directional derivatives, and simultaneously calculates the credits of multiple tokens using JVP, avoiding explicit construction of parameter dimension gradients per token.

Critical weaknesses
  1. Paper selects the best checkpoint on the final evaluation benchmarks and chooses its main hyperparameter using results on the same benchmarks. This protocol introduces a risk of optimistic selection bias. The paper also does not report means, standard deviations, or confidence intervals across training seeds, so the stability of the reported improvements remains unclear. Applying the same checkpoint-selection protocol to all methods does not necessarily eliminate this concern, since their evaluation variance and training trajectories may differ.

  2. It demonstrates short-horizon improvements across several initialization and distillation settings, but the experiments remain confined to one model family, relatively small models, and tasks with verifiable rewards. The evidence does not yet establish whether the advantage persists transfers to other model families, larger scales, or tasks without verifiable outcomes.

Questions for authors
  1. Refer to weakness 1, I recommended to use an independent validation set to select checkpoints and hyperparameters, and to report multiple sub-outcomes and confidence intervals for performance differences.
  2. Refer to weakness 2, I suggest supplementing the model family with more experiments on non-verifiable tasks to demonstrate its generalization ability.

If you resolve my confusion, I will consider raising my rating accordingly.

5 comments

Thank you for recognizing the reward-guided distillation formulation. We address the weaknesses and questions through the following additional results:

  • W1 / Q1 — Independent validation-based selection. In the Qwen3-1.7B student / Qwen3-4B teacher setting, we hold out 1,000 DeepMath questions, excluded from training and testing, to select λ and each method’s checkpoint. The validation sweep selects λ=0.4\lambda = 0.4, which we reuse across settings. Each method’s selected checkpoint is evaluated across all five math benchmarks.
  • W1 / Q1 — Repeated runs and uncertainty. We repeat Dr. OPD, OPD, GRPD, OPD+GRPO, and ExOPD with three training seeds, reporting mean accuracy, standard errors, and paired 95% confidence intervals. Dr. OPD achieves the highest mean accuracy on every math benchmark and improves average accuracy by 7.01–7.90 pp over the four baselines. Every paired 95% CI for these average gains is above zero, establishing significant improvements under independent validation-based selection.
  • W2 / Q2 — Another model family and non-verifiable tasks. We distill Llama-3.2-3B-Instruct into Llama-3.2-1B-Instruct on UltraChat using a learned reward model. Three-seed external-reward evaluation and a separate single-checkpoint evaluation with benchmark-specific protocols both show that Dr. OPD improves on the initial student across AlpacaEval, WritingBench, and WildBench, while all four baselines degrade it. Under benchmark-specific evaluation with a shared custom judge, gains over OPD are 5.00 pp in AlpacaEval’s raw weighted win rate, 2.30 WritingBench score points, and 7.11 WildBench score points; every paired 95% CI is above zero. These results support effectiveness beyond the Qwen model family and verifiable tasks (updated manuscript, Section E.1.2, p. 23).

These additional results and clarifications directly address the concerns raised and strengthen the evidence for our method. We would appreciate your consideration of a higher score in light of these findings. Thank you.

Weakness 1 (and Q1): Independent validation and uncertainty. Checkpoints and selected on the test benchmarks, and multi-seed means or confidence intervals.

We now use an independent validation set to select Dr. OPD's hyperparameter and each method's checkpoint. Across three independent runs, we report mean accuracy with standard errors and paired 95% confidence intervals for the gains. These results show consistent, statistically significant improvements over all four baselines.

Independent validation for hyperparameter and checkpoint selection. Following your suggestion, we held out 1,000 DeepMath questions in the Qwen3-1.7B student / Qwen3-4B teacher setting, excluding them from training and all final test sets. We use this independent validation set to select for Dr. OPD and the best checkpoint for every method. Each method's validation-selected checkpoint is then evaluated on all five math benchmarks. Table 1 shows the validation sweep. It selects λ=0.4\lambda=0.4, the same value used in the paper.

Table 1. Validation accuracy (%) across and training steps.

1020304050
0.246.4049.2050.1653.2953.15
0.348.8650.1254.9454.5555.57
0.447.4455.7956.8053.6053.52
0.546.2647.8355.7655.6753.76
0.650.5454.9456.2952.9953.81

Repeated runs, standard errors and confidence intervals. We repeated this setting with three seeds for Dr. OPD and all four baselines. Table 2 reports mean accuracy with standard error across these runs. Dr. OPD achieves the highest mean accuracy on every math benchmark. Table 3 reports paired 95% confidence intervals for the average accuracy gains over each baseline. Every interval is strictly positive, supporting the improvement over all four baselines under validation-based selection.

Table 2. Math accuracy (%), mean ± SE over three seeds.

MethodAIME24AIME25AMCMinervaOlympiadBenchAverage
Dr. OPD23.89±1.3319.31±0.1454.12±0.4327.42±0.3743.42±0.1133.63±0.31
OPD18.61±0.5012.36±1.0045.33±0.9420.88±0.5335.91±0.5726.62±0.12
GRPD15.14±0.5613.05±1.4144.58±0.5221.36±0.1535.97±0.4126.02±0.10
OPD+GRPO15.42±0.4810.28±0.3745.08±0.2021.52±0.2936.38±0.2825.73±0.01
ExOPD17.64±0.7411.25±0.4845.48±1.0921.26±0.3436.32±0.0626.39±0.39

Table 3. Average accuracy gains with 95% CIs.

ComparisonMean gain95% CI
Dr. OPD − OPD+7.01[+5.26,+8.76]
Dr. OPD − GRPD+7.61[+5.87,+9.35]
Dr. OPD − OPD+GRPO+7.90[+6.51,+9.28]
Dr. OPD − ExOPD+7.24[+6.27,+8.21]

Given the rebuttal time constraints, we prioritized repeated runs for this main setting. We will extend the multi-seed evaluation to the remaining settings in the camera-ready version.

Weakness 2 (and Q2): Another model family and non-verifiable tasks. Experiments cover only one model family, relatively small models, and verifiable-reward tasks.

We have added experiments distilling Llama-3.2-3B-Instruct into Llama-3.2-1B-Instruct on open-ended instruction following. Dr. OPD improves over the initial student and all four baselines.

We train on UltraChat prompts using the learned reward model Skywork-Reward-V2-Llama-3.2-3B, with a separate 1,000-prompt validation set for checkpoint selection. We evaluate all methods on AlpacaEval (805 prompts), WritingBench (1,000 prompts), and WildBench (1,024 prompts), using both an external reward model and the benchmarks' evaluation protocols.

External reward evaluation. Table 4 reports scores from Skywork-Reward-V2-Llama-3.1-8B. Dr. OPD improves over the initial student on all three benchmarks, while all four baselines score below the initial student.

Table 4. External reward scores, mean ± SD over three seeds.

MethodAlpacaEvalWritingBenchWildBench
Initial student12.9915.0434.295
Frozen teacher21.08512.99214.019
Dr. OPD16.098±0.7718.500±0.5367.873±0.238
OPD12.413±0.0233.714±0.0543.154±0.056
GRPD12.547±0.0833.789±0.0553.245±0.073
OPD+GRPO12.595±0.0863.949±0.1283.345±0.075
ExOPD12.015±0.0523.106±0.0292.575±0.100

Evaluation with benchmark-specific protocols. We also use the benchmarks' evaluation prompts, rubrics, parsing, and aggregation, with GPT-5.6 Luna as a shared custom judge. AlpacaEval reports the raw weighted win rate against vanilla OPD; WritingBench and WildBench use their checklist-based scores. On these non-verifiable tasks, vanilla OPD underperforms the initial student on all three benchmarks, consistent with prior reports that teacher probabilities on student rollouts can favor repetitive or low-quality outputs (Gu et al., 2024; Cui et al., 2026). The other three baselines likewise underperform the initial student. Dr. OPD improves on the initial student across all three benchmarks and achieves the highest scores among the distilled models (Table 5).

Table 5. Benchmark-specific scores.

MethodAlpacaEval (%)WritingBenchWildBench
Initial student53.1424.54−41.81
Frozen teacher79.5631.60−24.77
Dr. OPD55.0025.92−36.97
OPD50.0023.62−44.08
GRPD48.1823.81−43.54
OPD+GRPO49.8123.68−42.01
ExOPD44.5522.86−44.39

Table 6. Benchmark-specific gains over OPD with 95% CIs.

BenchmarkGain over OPD95% CI
AlpacaEval+5.00 pp[+1.79,+8.17] pp
WritingBench+2.30[+1.92,+2.68]
WildBench+7.11[+5.92,+8.32]

Table 6 shows significant improvements over OPD on all three benchmarks, with every 95% confidence interval strictly above zero. These results extend Dr. OPD's effectiveness to a new model family and non-verifiable tasks, including a setting where the baseline methods degrade the student.

The independent validation-based selection, multi-seed results, and additional Llama experiments address most of my original concerns. I am inclined to raise my rating.

But, please clarify my last question: Reconcile the main-text claims (+9.7 pp and surpassing the teacher) with the updated three-seed results (+7.01 pp; 33.63% vs. 34.1%), and state the selection protocol underlying each main table.

Comments on the paper

2 comments

Dear Reviewers,

Following up on personal messages to each of you.

If you haven't done already: please please read the reviewers' response and take a minute to write an acknowledgment and confirm your final evaluation.

Kind reminder that the deadline of the discussion period is Nov. 18 please do this as soon as possible.

Thanks a lot for your help.

AuthorsOpenReview ↗

Summary of Rebuttal

We thank the reviewers and Area Chair for their constructive feedback.

Dr. OPD introduces a principled formulation of optimal OPD: among all valid weighted OPD objectives, select the one that maximizes the resulting student's task performance. The paper proves that any positive token weighting yields a valid OPD objective, formulates this optimal selection as a bilevel optimization problem, and develops an efficient iterative solver with closed-form weight updates. Experiments demonstrate substantial gains over the evaluated baselines across verifiable math/code and non-verifiable instruction-following tasks.

During rebuttal, we have fully addressed the weaknesses and questions raised by all three reviewers, as summarized below:

  • R1 (Wci1) — Reliability and generalization: Three-seed runs show gains of +7.01 pp on math and +4.52 pp on code over vanilla OPD. Additional Llama experiments on non-verifiable instruction-following tasks demonstrate statistically significant improvements over all four baselines across all three benchmarks.

  • R2 (NRuR) — Training dynamics, uncertainty, and cost: We report checkpoint trajectories across three seeds and confidence intervals for the gains. An additional 200-step run shows that Dr. OPD's validation accuracy fluctuates after step 30 without a consistent downward trend. Our training-time analysis shows Dr. OPD's per-step runtime at 1.58× that of vanilla OPD, with nearly all additional cost coming from token-credit computation, including JVP and reward-gradient calculations.

  • R3 (WTyh) — Credit attribution and theoretical scope: In the controlled study, token shuffling and response-level weighting reduce generation-weighted test accuracy by 6.71 and 7.17 pp, demonstrating the value of token-specific credit assignment. Dr. OPD also outperforms IW-OPD by 6.99 pp. We clarify what the single-step guarantee covers and use direction ablations to assess the practical approximation.

The revised manuscript incorporating these updates has been uploaded. We respectfully ask that the contributions and additional evidence be considered in the final assessment.

Best, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.