- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to weight teacher signals in on-policy distillation according to their usefulness for student performance. It formulates a bilevel objective and derives a clipped weighting rule from a local reward bound for a one-step student update. Token credits measure alignment between teacher-correction and reward-gradient directions. The practical algorithm approximates this direction and uses Jacobian–vector products, reward momentum, optimizer scaling, and normalization. Experiments evaluate cross-scale Qwen3 distillation on mathematics and code, and same-size distillation on mathematics. The method achieves the highest reported average scores among the compared methods in each evaluated setting, with students exceeding their teachers in some mathematical settings.
Critical strengths
The paper adopts a meaningful objective for assessing teacher supervision. Matching a teacher's output distribution more closely does not necessarily improve the student's task performance. Using outcome rewards to determine the strength of individual teacher corrections is a reasonable direction. The proposed token-level weighting retains fine-grained teacher supervision while providing a concrete mechanism for incorporating task feedback into distillation.
The experiments report substantial gains across several teacher–student configurations. Table 1 (p. 8) reports an increase in Qwen3-1.7B's average mathematics score from 25.3 under vanilla OPD to 35.0, and its code score from 43.4 to 48.4. Same-size distillation also shows positive gains. The gains across both cross-scale and same-size settings support the method’s potential practical value. The JVP implementation avoids explicitly constructing a parameter-dimensional gradient for each token, making the proposed credit computation practically implementable.
Critical weaknesses
Model selection and incomplete attribution limit how confidently the gains can be interpreted.
Section 5.1 (p. 7) selects the best checkpoint after evaluation every ten steps on the evaluated datasets, and §5.3 and Table 3 (pp. 8–9) select λ = 0.4 after a sweep. An independent validation-based selection procedure is not described. Table 5 (p. 15) lists fixed coefficients or defaults for competing methods without reporting comparable searches, and the main tables do not report uncertainty across independent training runs. The extent to which selection, tuning, and run variation affect the reported gains therefore remains unclear. OPD+GRPO and GRPD are useful reward-aware controls, but do not fully isolate the contribution of token-credit estimation from simpler weighting, reward momentum, or optimizer scaling. The paper reports neither these component ablations nor direct comparisons with TrOPD and IW-OPD. These gaps leave the size of the estimator’s independent benefit unresolved, despite the substantial improvements over the evaluated configurations.
The contribution and guarantee should be delimited more precisely relative to prior work and the implementation.
Theorem 5 (p. 5) establishes a conditional guarantee for an idealized one-step update. The implementation instead uses an approximate reward direction (p. 6), lagged momentum, optimizer scaling, and PPO-style clipping (p. 17). These choices are disclosed, and Appendix D.3 (p. 20) explicitly states that global convergence is not established. The theoretical result therefore provides a local justification, while its correspondence to the implemented update requires further assessment. The “optimal weighted OPD” framing should preserve this distinction. The discussion of gradient-based reweighting and Unmasking OPD (pp. 9 and 14) appropriately identifies relevant precedents, but the specific benefit of the derived weighting rule and parameter-space estimator needs stronger isolation. Component ablations would help establish that benefit and the relevance of the local analysis. Similarly, a single JVP avoids naive per-token gradient construction, but does not quantify the claimed small end-to-end overhead (p. 6).
Questions for authors
Selection protocol and uncertainty: What data were used to select checkpoints and λ, and was one checkpoint used across benchmarks? Can the authors provide validation-only selection results, or fixed-step results if no validation set was used, together with uncertainty across independent training runs? This would clarify the stability of the main gains.
Added value of token credits: Beyond the existing reward-aware baselines, can the authors compare with simple response-reward weighting or shuffle token weights while preserving their distribution? Please also isolate reward momentum and optimizer scaling to determine which components account for the improvement.
Closest comparisons and tuning: Can the authors provide comparable tuning budgets and a direct comparison with a closely related weighting method, such as TrOPD or IW-OPD, or explain material incompatibilities? Please distinguish the contribution from prior gradient-based reweighting and alignment diagnostics, which the paper already acknowledges.
Theory–implementation relationship: Which implementation steps retain the assumptions of Theorem 5, and which should be understood as heuristics motivated by the local analysis? Can a small-scale comparison with the more direct reward-direction estimate assess the approximation? Please state the scope of the guarantee separately from the empirical justification for these approximations.
Measured cost and teacher comparisons: Can the authors report JVP and reward-gradient overhead, peak memory, and total generated tokens, and compare methods at similar measured compute budgets? Please also provide paired per-question uncertainty for small teacher–student differences, distinguishing these claims from the larger improvements over vanilla OPD.
5 comments
Weakness 1 (Q1–Q3): Model selection and attribution. Checkpoints and were selected on the test benchmarks without run-level uncertainty, and the benefit of token credits was not isolated from simpler variants or the closest methods.
Q1. Selection protocol and uncertainty
Independent validation and one checkpoint across benchmarks. In the Qwen3-1.7B student / Qwen3-4B teacher setting, we held out 1,000 DeepMath questions, excluded from training and all final test sets. The validation sweep in Table 1 selects , with the highest validation accuracy of 56.80% at step 30, matching the value used in the paper.
Table 1. Validation accuracy (%) across and training steps.
| 10 | 20 | 30 | 40 | 50 | |
|---|---|---|---|---|---|
| 0.2 | 46.40 | 49.20 | 50.16 | 53.29 | 53.15 |
| 0.3 | 48.86 | 50.12 | 54.94 | 54.55 | 55.57 |
| 0.4 | 47.44 | 55.79 | 56.80 | 53.60 | 53.52 |
| 0.5 | 46.26 | 47.83 | 55.76 | 55.67 | 53.76 |
| 0.6 | 50.54 | 54.94 | 56.29 | 52.99 | 53.81 |
We fix =0.4 for the repeated runs. For every method and run, we select the checkpoint with the highest accuracy on this same validation set among steps 10, 20, 30, 40, and 50. That single checkpoint is used for all five test benchmarks.
Multi-seed results and confidence intervals. We repeated Dr. OPD and all four baselines with three seeds in this main setting, using eight responses per test question. Table 2 reports mean accuracy and standard errors; Table 3 reports 95% CIs for the gains.
Table 2. Math accuracy (%), mean ± SE over three seeds.
| Method | AIME24 | AIME25 | AMC | Minerva | OlympiadBench | Average |
|---|---|---|---|---|---|---|
| Dr. OPD | 23.89±1.33 | 19.31±0.14 | 54.12±0.43 | 27.42±0.37 | 43.42±0.11 | 33.63±0.31 |
| OPD | 18.61±0.50 | 12.36±1.00 | 45.33±0.94 | 20.88±0.53 | 35.91±0.57 | 26.62±0.12 |
| GRPD | 15.14±0.56 | 13.05±1.41 | 44.58±0.52 | 21.36±0.15 | 35.97±0.41 | 26.02±0.10 |
| OPD+GRPO | 15.42±0.48 | 10.28±0.37 | 45.08±0.20 | 21.52±0.29 | 36.38±0.28 | 25.73±0.01 |
| ExOPD | 17.64±0.74 | 11.25±0.48 | 45.48±1.09 | 21.26±0.34 | 36.32±0.06 | 26.39±0.39 |
Table 3. 95% CIs for accuracy gains.
| Comparison / benchmark | 95% CI (pp) |
|---|---|
| A. Average accuracy: Dr. OPD versus each baseline | |
| Dr. OPD − OPD | [+5.26,+8.76] |
| Dr. OPD − GRPD | [+5.87,+9.35] |
| Dr. OPD − OPD+GRPO | [+6.51,+9.28] |
| Dr. OPD − ExOPD | [+6.27,+8.21] |
| B. Per-benchmark accuracy: Dr. OPD versus OPD | |
| AIME24 (30 questions) | [+2.92,+8.33] |
| AIME25 (30 questions) | [+4.17,+10.00] |
| AMC (83 questions) | [+6.17,+11.60] |
| Minerva (272 questions) | [+4.87,+8.32] |
| OlympiadBench (673 questions) | [+6.67,+8.40] |
Dr. OPD achieves the highest mean accuracy on every benchmark. All reported gain intervals are strictly positive, supporting a significant improvement over all four baselines on average and over OPD on each benchmark under validation-only selection.
We also added three-seed Llama-3.2-1B/3B-Instruct experiments on non-verifiable tasks, where Dr. OPD significantly outperforms all four baselines, while vanilla OPD and the other baselines degrade the initial student, consistent with prior findings on low-quality or repetitive OPD outputs (Gu et al., 2024; Cui et al., 2026).
Given the rebuttal time constraints, we focused repeated runs on this main setting and will extend the multi-seed evaluation to the remaining settings in the camera-ready version.
Q2. Added value of token credits
We added four controlled ablations in the Qwen3-1.7B student / Qwen3-4B teacher setting. Table 4 reports paired question-bootstrap 95% CIs for differences from full Dr. OPD. The results highlight the importance of token-specific credit assignment and reward momentum, with more details in Section E.1.3 (p. 24) of the updated manuscript.
Table 4. Component ablations with 95% CIs.
| Variant | Validation difference (pp, 95% CI) | Test difference (pp, 95% CI) | Test accuracy (%) |
|---|---|---|---|
| Full Dr. OPD | — | — | 33.02 |
| Token shuffle | −14.05[−15.88,−12.24] | −6.71[−7.76,−6.00] | 26.55 |
| Response reward weighting | −13.81[−15.65,−12.05] | −7.17[−8.24,−6.12] | 25.95 |
| No reward momentum | −7.11[−8.62,−5.61] | −3.46[−4.40,−2.54] | 30.10 |
| No optimizer scaling | +0.41[−0.98,+1.81] | +0.83[−0.14,+1.78] | 34.17 |
| OPD | −14.85[−16.78,−12.99] | −6.70[−7.73,−5.70] | 26.79 |
Q3. Closest comparisons and tuning
Thank you for this suggestion. We added a direct comparison with IW-OPD, a closely related token-weighting method, in the Qwen3-1.7B student / Qwen3-4B teacher setting.
1. Direct comparison with IW-OPD. The comparison matches the training seed, data order, and 50-update budget; each method selects one checkpoint using the same independent validation set. We preserve IW-OPD's original prefix-discrepancy weighting and default . Table 5 reports paired question-bootstrap 95% CIs for differences from Dr. OPD.
Table 5. IW-OPD comparison with 95% CIs.
| Method | Validation difference (pp, 95% CI) | Test difference (pp, 95% CI) | Test accuracy (%) |
|---|---|---|---|
| Dr. OPD | — | — | 33.02 |
| IW-OPD | −14.11[−15.96,−12.29] | −6.99[−7.97,−6.03] | 26.34 |
| OPD | −14.85[−16.78,−12.99] | −6.70[−7.73,−5.70] | 26.79 |
Dr. OPD significantly outperforms IW-OPD on both validation and test performance.
A faithful comparison with TrOPD requires teacher-generated prefixes, student continuations, and forward-KL guidance; its FKL outlier variant also uses teacher top-64 probabilities. This requires extending our student-only rollout/scoring pipeline beyond token weighting. We therefore report the completed IW-OPD comparison under the shared pipeline; a full TrOPD evaluation is not included here.
2. Hyperparameter protocol. For Dr. OPD, we select once on the independent 1,000-question DeepMath validation set in the Qwen3-1.7B student / Qwen3-4B teacher setting. We then reuse across all other settings. For the baselines, we retain the authors' reported method-specific coefficients under our shared training protocol. These coefficients have already been tuned by the original authors in closely related Qwen3 distillation settings (RL-trained 4B teachers in ExOPD and OPDVR, and a Qwen3-4B-Instruct-2507 teacher in IW-OPD), so we use them as is rather than re-tuning them.
3. Distinction from gradient-based reweighting and diagnostics. Prior gradient-based reweighting, such as Ren et al. (2018), selects example weights using a clean validation-loss objective. Dr. OPD selects weights for individual teacher-token corrections using the updated student's outcome reward. Our local reward lower bound yields bounded weights around vanilla OPD, and a shared parameter-space reward direction enables efficient credit estimation through JVP. While Unmasking OPD studies token-level alignment as a training-free diagnostic, we show that for any positive weighting function, the weighted OPD objective remains a valid distillation objective with the same teacher-matching target. We then use reward-sensitive credits to select token weights according to their effect on the updated student's outcome reward.
Weakness 2 (Q4, Q5): Scope of the guarantee. Theorem 5 covers an idealized one-step update, so the “optimal weighted OPD” framing, its link to the implementation, and the JVP overhead need clarification.
Thank you for highlighting this distinction. We clarify the optimization target, the result proved for the solver, and its scope.
Optimization target. “Optimal weighted OPD” defines the bilevel objective: choose feasible positive weights to maximize the outcome reward of the student trained with that weighted distillation objective.
Local optimality and the guarantee. Our solver replaces full inner training with one student update and constructs a quadratic local lower bound on the updated student's reward. The clipped rule
exactly maximizes this one-step surrogate over the bounded weights. Under Theorem 5's exact-credit, regularity, smoothness, and step-size conditions, it guarantees a nonnegative improvement in the true expected reward over a vanilla OPD update from the same student.
This is a local single-step guarantee. Global optimality of the bilevel problem and convergence of repeated practical updates are outside its scope. The practical reward-direction choices are assessed empirically through the direction ablations.
Thank you for the detailed review. Our additional experiments in the Qwen3-1.7B student / Qwen3-4B teacher setting address both weaknesses and all five questions:
Moreover, during the rebuttal period, we added Llama-3.2-3B→1B experiments on non-verifiable tasks. Dr. OPD improves on the initial student across all three benchmarks, while every baseline degrades it, supporting effectiveness beyond the Qwen model family and verifiable tasks (updated manuscript, Section E.1.2, p. 23).
These additional results directly address the weaknesses and questions raised and further strengthen the evidence for our method’s superiority over competing methods across benchmarks. We would greatly appreciate your consideration of raising the score.
Next, we answer each weakness and answer in order with more detailed explanations.