- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes On-policy Delta Distillation, which uses the difference in logits between the teacher and student as supervision, instead of the teacher logits directly. The algorithm shows large improvements on non-thinking Qwen3 and Gemma4 models on math and code benchmarks, and small improvements over on-policy distillation with thinking turned on. Wordmaps show that the algorithm suppresses uncertainty-related tokens, and enhances conclusion-words like hence.
Critical strengths
The gains of OPD$^2$ seem large in the non-thinking settings. It does work for thinking models as well, though the gains are much smaller.
Critical weaknesses
The following are the reasons that support rejection:
- As with other on-policy distillation methods, the gains are very small on thinking models (see Tables 7 and 8). Recent work over the past few months have all reported similar findings and this is because answer-conditioning provides very bad supervision for thought tokens by suppressing uncertainty or exploratory tokens. This is also what the wordmaps in this paper suggest.
- I have concerns about the generation budget and its effect on the reported results. In the past I have run OPD with 24k or 28k token budgets and even then many problems’ accuracies went up simply because where the model was previously always super verbose and getting truncated, now it finishes its responses in time. Table 11 provides a generation budget of 8192 tokens which is extremely insufficient for Qwen-evaluation at least. There is therefore a good chance that a lot of the gains in accuracy come from just answer shortening.
- The delta-supervision does not have a stable global optimum for the difference of the two logit distributions is by itself not a desirable distribution induced by a language model. Hence while it might locally provide useful gradients at first, I am unconvinced by the general applicability or stability of the provided algorithm.
Questions for authors
I have the following questions:
- Do you have truncation rates before and after training? Also, set C1 be the set of problems with >50% solve rate before training and C2 be the set with <50% solve rate before training but >50% after. What are the truncation rates of the model before/after OPD$^2$ for C1 and C2?
- Do you have results against a GRPO baseline as well?
Dear Reviewer knxF,
We have added new analyses and experimental results to address your concerns regarding the difference between the paper's equation and code, and the ExOPD comparison with an extended lambda range.
We would appreciate it if you could take a look at our response and share any further thoughts or questions before the discussion period ends.
Thank you again for your time and valuable feedback.