acceptodds
Under review as a conference paper at ICLR 2027

On-Policy Delta Distillation

Abstract

On-policy distillation is an alternative post-training method in reinforcement learning that alleviates the constraints imposed by reward models by providing token-level supervision from a teacher model. Although on-policy distillation has been studied and applied across various settings, its fundamental design remains underexplored. In this paper, we introduce a new distillation reward, termed delta signal, instead of directly imitating the teacher's output distribution. The delta signal is defined as the difference between the teacher model and its base model prior to instruction tuning for reasoning capability. It therefore captures the changes induced by reasoning tuning and provides a more direct signal for transferring reasoning capabilities. Using extensive empirical evidence, we show that the delta signal substantially improves on-policy distillation and refer to the new distillation method as On-Policy Delta Distillation (OPD). Experiments across mathematics, science, and code-reasoning benchmarks demonstrate that OPD consistently outperforms conventional on-policy distillation, enabling reasoning LLMs to achieve strong performance with only a short post-training period.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
on-policy distillation, delta signal, post-training, reasoning models, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kze4OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes On-policy Delta Distillation, which uses the difference in logits between the teacher and student as supervision, instead of the teacher logits directly. The algorithm shows large improvements on non-thinking Qwen3 and Gemma4 models on math and code benchmarks, and small improvements over on-policy distillation with thinking turned on. Wordmaps show that the algorithm suppresses uncertainty-related tokens, and enhances conclusion-words like hence.

Critical strengths

The gains of OPD$^2$ seem large in the non-thinking settings. It does work for thinking models as well, though the gains are much smaller.

Critical weaknesses

The following are the reasons that support rejection:

  • As with other on-policy distillation methods, the gains are very small on thinking models (see Tables 7 and 8). Recent work over the past few months have all reported similar findings and this is because answer-conditioning provides very bad supervision for thought tokens by suppressing uncertainty or exploratory tokens. This is also what the wordmaps in this paper suggest.
  • I have concerns about the generation budget and its effect on the reported results. In the past I have run OPD with 24k or 28k token budgets and even then many problems’ accuracies went up simply because where the model was previously always super verbose and getting truncated, now it finishes its responses in time. Table 11 provides a generation budget of 8192 tokens which is extremely insufficient for Qwen-evaluation at least. There is therefore a good chance that a lot of the gains in accuracy come from just answer shortening.
  • The delta-supervision does not have a stable global optimum for the difference of the two logit distributions is by itself not a desirable distribution induced by a language model. Hence while it might locally provide useful gradients at first, I am unconvinced by the general applicability or stability of the provided algorithm.
Questions for authors

I have the following questions:

  • Do you have truncation rates before and after training? Also, set C1 be the set of problems with >50% solve rate before training and C2 be the set with <50% solve rate before training but >50% after. What are the truncation rates of the model before/after OPD$^2$ for C1 and C2?
  • Do you have results against a GRPO baseline as well?
Reviewer_knxFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OPD2 uses the teacher–base log-prob difference to train a student on its own responses, with centering and gating to control the updates. The experiments cover several models and task domains and show promising gains over OPD and ExOPD, at the cost of extra training compute.

Critical strengths
  1. The method improves results across several model sizes and task domains.
  2. The ablations help explain which parts of the method matter.
Critical weaknesses

W1: The code uses a different gate from the equation. When the student matches the teacher, the equation stops the update but the code can retain a learning signal. W2: The ExOPD comparison leaves an important reward setting unexplored. After rescaling, its reward approaches Delta as the reward parameter grows, but the experiments test only one setting.

Questions for authors
  1. Did the reported runs use the gate in Eq 7 or the submitted code?
  2. Do existing ExOPD results with a wider lambda range and matched reward scales still favor OPD2?
2 comments

Dear Reviewer knxF,

We have added new analyses and experimental results to address your concerns regarding the difference between the paper's equation and code, and the ExOPD comparison with an extended lambda range.

We would appreciate it if you could take a look at our response and share any further thoughts or questions before the discussion period ends.

Thank you again for your time and valuable feedback.

Dear Reviewer knxF,

As the author–reviewer discussion period ends, we would like to thank you again for your constructive feedback.

We sincerely appreciate your decision to raise the rating. We hope our additional analyses and experiments have addressed your concerns about the gate implementation and the extended ExOPD comparison.

Thank you again for your time and valuable feedback.

Reviewer_76TqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes On-Policy Delta Distillation, which uses the token-level log-probability difference between the teacher and its base model, rather than between the teacher and the student, as the main learning signal. Across 14 math, science, and coding benchmarks, the proposed method generally outperforms standard OPD, although some existing capabilities still degrade.

Critical strengths
  1. The task related delta signal is a simple and well-motivated modification to standard OPD. The ablation study supports its effectiveness, identifying it as the main source of performance gains.
  2. Evaluated under comprehensive settings across math, code, and science domains in both thinking and non-thinking modes.
Critical weaknesses
  1. The method requires access to the teacher’s corresponding base model and an additional teacher forward pass, increasing training time by approximately 24–28% in the Qwen3 experiments.
  2. The rationale for the delta signal remains largely intuitive. The paper interprets the teacher–base difference as reflecting reasoning-related changes and provides word-cloud and token-level analyses to support this interpretation. While the empirical results support the method’s effectiveness, the logical connection between the properties highlighted in these analyses and the observed distillation gains is not sufficiently developed.
Questions for authors
  1. Could the authors plot performance against training compute (e.g., GPU-hours) for the proposed method and the baselines to provide a more comprehensive view?
  2. Could the authors strengthen the logical connection between the proposed advantages of the delta signal and the observed distillation gains through more concrete analysis or evidence?
4 comments

Dear Reviewer 76Tq,

We have added new experimental results and analyses to address your concerns regarding GPU-hour efficiency and the logical connection between the delta signal and performance improvements.

We would appreciate it if you could take a look at our response and share any further thoughts or questions before the discussion period ends.

Thank you again for your time and valuable feedback.

I thank the authors for their response. The additional GPU-budget comparison clarifies the computational trade-offs. The new analysis also suggests better preservation of existing capabilities. However, the student’s original preferences may reflect reasoning errors as well. Could the authors explain how reinforcing these preferences helps preserve useful capabilities without hindering the correction of existing errors?

Thank you for your thoughtful follow-up and for acknowledging our additional analyses.

We agree that the additional term (log⁡πθ−log⁡πbase∗)(\log \pi_\theta - \log \pi^{\ast}_{\mathrm{base}}), when considered independently, could reinforce the student’s existing reasoning errors.

However, this term is always combined with the original OPD reward (log⁡π∗−log⁡πθ)(\log \pi^{\ast} - \log \pi_\theta). As a result, the student’s original preferences cancel out, yielding the final reward (log⁡π∗−log⁡πbase∗)(\log \pi^{\ast} - \log \pi^{\ast}_{\mathrm{base}}).

Therefore, if the teacher correctly identifies the student’s errors, the original OPD signal counteracts their reinforcement, and the resulting learning signal follows the teacher rather than the student’s potentially incorrect preferences. While the magnitude of the signal suppressing these errors may be reduced, its direction does not reverse to reinforce them.

Nevertheless, we acknowledge that, as with conventional OPD, our method cannot directly correct errors that the teacher itself fails to recognize.

We hope this clarification addresses your concern, and thank you again for your valuable feedback.

Dear Reviewer 76Tq,

As the author–reviewer discussion period is ending in approximately 30 minutes, we would like to thank you again for your constructive feedback.

We hope our latest response has adequately addressed your follow-up question about preserving existing capabilities and correcting reasoning errors.

If you feel that your concerns have been sufficiently addressed, we would greatly appreciate it if you could consider raising your rating.

Thank you again for your time and valuable feedback.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Paper Revision Notification

Dear Reviewers and AC,

We have uploaded a revised version of our paper, including the additional experiments, analyses, and clarifications discussed in our rebuttal. All revisions are highlighted in yellow for easy identification.

The main revisions are summarized below:

  1. Sec. 2.2: Reward Decomposition
  2. Sec. 2.3: Correction of Eq. 7 ( renumbered as Eq. 8)
  3. Sec. 3.5: Preserving Reasoning Capabilities (Table 5)
  4. Appendix B.4: Computational Cost Analysis (Fig. 5)
  5. Appendix B.5: Extended ExOPD Comparison (Table 11)
  6. Appendix B.6: Comparison with GRPO (Table 12)
  7. Appendix B.7: Truncation Rate Analysis (Table 13)

We kindly ask you to review our responses and the revised manuscript. We welcome any additional questions or further discussion and would be happy to provide clarification.

Thank you for your time and consideration.

Area_Chair_Ao1GOpenReview ↗

AC Note: Rebuttal Received, Please Engage in Discussion

Dear authors and reviewers,

Thank you to the authors for their detailed rebuttal and for the time and care they put into addressing the reviewers' comments. I appreciate the additional clarifications and results.

Reviewers: please read the authors' responses carefully and reply to them before the end of the discussion period. Even a short acknowledgment is valuable. It is especially helpful to say whether your concerns have been addressed and, if not, what remains unresolved. If the rebuttal changes your assessment, please update your score and review accordingly, whether up or down. If your score stays the same, a brief explanation of why helps me and the senior area chairs.

Authors: I will follow the discussion closely and take your rebuttal fully into account when forming my recommendation.

Thank you all for your time and effort.

Best regards, Your Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.