- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies instability in on-policy distillation and proposes DKLD, which separates supervision of region probability masses from supervision within each region. It retains forward KL on the region masses and matched candidates, and uses student-mass-weighted reverse KL within the mismatched region. The method combines this loss with GRPO and advantage-weighted distillation (AWD) on correct rollouts. Experiments report improvements across three Qwen3 student sizes and five math benchmarks, with an additional coding study.
Critical strengths
(1) The math results are consistent across student sizes. Table 2 reports average gains over ToDi of 2.4, 2.5, and 2.7 points for the 0.6B, 1.7B, and 4B models. The use of three training seeds and validation-selected checkpoints gives these comparisons some substance. The gains on the harder math sets are particularly encouraging.
(2) The ablations make a useful case for the loss design. Removing the binary term reduces Hard_mini150 accuracy from 73.1 to 60.7 (Table 3). Table 8 also shows that DKLD remains effective without AWD, whereas FKL collapses even with AWD. This is meaningful evidence that the stability improvement comes at least partly from the distillation objective.
Critical weaknesses
Major concerns
(1) The benefit of AWD is not fully separated from a reduction in distillation strength. Its weight is at most 0.5 on correct rollouts, while the main baselines use weight 1 at the same nominal KD coefficient. Tables 3 and 8 help address this for the fixed-direction losses, but the strongest alternatives, AKL and ToDi, are not tested with AWD or a tuned constant weight. Such controls would make the claimed margin over existing adaptive losses more convincing.
(2) The gradient interpretation in Section 3.1 needs correction or clarification. For a fixed partition, let a and b denote the teacher and student masses in G, and D the conditional reverse KL there. Differentiating Eq. (10) along a common shift of all student logits in G gives b - a + b(1 - b)D. Thus the total push toward G is not unchanged unless the student-mass multiplier is detached. This does not invalidate the empirical gains, but it matters for the proposed explanation of them.
(3) The coding experiment uses held-out test cases from problems that are also used for training (Section 4.1). This measures whether solutions pass additional tests on familiar problems; it does not establish transfer to unseen coding problems. I would give this result limited weight in assessing generality. A problem-disjoint split would be considerably more informative.
Minor concerns
(1) Algorithm 1 renormalizes the teacher top-256 logits but leaves the student normalization unclear. Please specify whether student probabilities are renormalized on the same support and how omitted vocabulary mass is handled, since both affect the binary term and region assignment.
(2) Table 4 mixes fixed absolute thresholds with the main configuration, which uses a relative teacher threshold and an annealed student threshold. A sweep that holds the student schedule fixed and varies only the teacher ratio would provide a cleaner sensitivity check.
Questions for authors
(1) Does DKLD retain its advantage over AKL and ToDi when all three use AWD, or when constant KD weights are selected on validation? How much does AWD add after matching average distillation strength?
(2) Is the student-mass multiplier in Eq. (10) detached in the implementation? Please reconcile the implemented gradient with the claim in Section 3.1 that the total push toward G is unchanged.
(3) Can the authors evaluate on LiveCodeBench problems excluded from training and report numerical pass@4 and truncation rates with variation across seeds?
2 comments
Response to Reviewer cbHR (Part 2 / 2): W4–W5, Q1–Q3
W4 (Minor 1): Student normalization in Algorithm 1
Thank you for this question. The student is renormalized in the same way as the teacher. We take the student's logits at the teacher's top-256 token ids and apply a softmax over just those tokens, so the two region masses sum to one for each model, and both the binary term and the region assignment use these renormalized probabilities. We have clarified this in Algorithm 1.
W5 (Minor 2): Threshold sweep in Table 4
Thank you for raising this, and Reviewer vv2E shared the same concern.
Time during the rebuttal was tight, so instead of a full sweep we trained DKLD with two more teacher ratios, one below and one above our default. The student threshold keeps the same annealed schedule in every row, so only changes.
Table R3: 1.7B, MATH-500
| schedule | Seeds | pass@1 | ||
|---|---|---|---|---|
| DKLD | 0.05 | 1 | 81.8 | |
| DKLD (paper) | 3 | |||
| DKLD | 0.20 | 1 | 82.0 |
We have also marked the main row of Table 4 clearly, since it follows a different rule from the fixed-threshold rows.
Q1: DKLD vs. AKL and ToDi under matched AWD / KD weight
Thank you. Yes, DKLD keeps its lead in both settings (Table R1 in our response to W1). With the AWD gate on all three, DKLD beats AKL by 7.1 and 4.1 points and ToDi by 4.4 and 5.5 points on Hard_mini150 and MATH-500. With a constant weight of 1, DKLD is still the best, by a smaller margin. The gate adds about 4 points for DKLD, and a constant of 0.01, below the gate's range, also falls short of it. For AKL and ToDi the gate changes the score by at most 1.1 points.
Q2: Is the student-mass multiplier detached
Thank you for this question. No, the multiplier is not detached. The implemented gradient along a common shift in is , as you derived. We have corrected the claim in Section 3.1 (see our response to W2).
Q3: Problem-disjoint LiveCodeBench evaluation
Thank you. Yes, Table R2 in our response to W3 reports pass@4 and truncation rates on 131 LiveCodeBench problems that are fully excluded from training. Because of the time limit we could only train one seed per method during the rebuttal.
We appreciate your careful review. If any point needs more detail, please do not hesitate to tell us.
Sincerely,
Authors
Response to Reviewer cbHR (Part 1 / 2): W1–W3
We thank you for your comments and suggestions. The following are our responses to your concerns.
Thank you for this suggestion. We ran new experiments to compare AKL, ToDi and DKLD under the same weight in both settings.
Table R1: Qwen3-1.7B
Due to the time limit, we could run only one seed for AKL and ToDi with the AWD gate.
Same constant weight. At w=1 no method uses the gate, and DKLD is still the best.
Same AWD gate. Following your suggestion, we trained AKL and ToDi with the AWD gate. The gate changes their scores by at most 1.1 points, which is within the variation across seeds. Under the same gate, DKLD leads AKL by 7.1 and 4.1 points and ToDi by 4.4 and 5.5 points.
We have added these changes to the appendix.
You are right. The multiplier rGS is not detached in our implementation, so the push towards G is b−a+b(1−b)D, and our statement in Section 3.1 is not exact. We have corrected it in the revision.
Also, we measured how large this term is in our final objective. It is very small and almost negligible.
Specifically, we measured it with the released loss code on 256 training rollouts of the initial Qwen3-1.7B student with the Qwen3-8B teacher, at the first and last values of εS in our schedule.
Here a, b and b⋅D are averaged over the positions where G is non-empty, and the last column over all positions. The push towards G is therefore dominated by b−a, so the statement in Section 3.1 holds as a close approximation, and we now present it as such.
Thank you for this comment. We agree. The split in Section 4.1 is on test cases, not problems, so it does not measure transfer to unseen problems. Following your suggestion, we retrained the model under the problem split configuration. We train on 737 LiveCodeBench problems released before 2025-02-01, and test on the same 131 LiveCodeBench v6 problems as in the paper (released February to April 2025), with all of their test cases. All other settings follow Section 4.1.
Table R2: Qwen3-1.7B student, Qwen3-8B teacher, 131 LiveCodeBench v6 problems
On problems never seen in training, DKLD still gives the best pass@4 and the lowest truncation rate, while GRPO drops the most and truncates over a fifth of its responses. We have replaced the coding experiment in Section 4.1 with this split.