acceptodds
Under review as a conference paper at ICLR 2027

Distillation with Errors: Efficient On-Policy Distillation via Sparse Unbiased Gradient Estimation

Abstract

On-policy distillation (OPD) mitigates the distribution mismatch between training and inference by using teacher supervision on trajectories generated by the student. However, standard full-distribution OPD backpropagates dense gradients over the entire vocabulary, incurring substantial computational cost in the output-head backward pass. To reduce this cost, we introduce Distillation with Errors (DWE), an efficient OPD algorithm based on sparse unbiased gradient estimation. Our key insight is that distillation learns to match the teacher distribution over the course of training, so each step need not use the exact full-vocabulary gradient as long as the full-distribution gradient is preserved in expectation. Based on this insight, DWE exploits the zero-sum structure of the logit gradient to form two sampling distributions from its positive and negative parts. Sampling one coordinate from each distribution yields a paired sparse update whose expectation matches the original dense gradient. To reduce estimation error, DWE uses output-head geometry to guide the pairing. It then applies stratified sampling over pairs to reduce sampling variance. We theoretically establish the unbiasedness of the resulting estimator. Our experiments show that DWE reduces output-head backward time by up to 96.8% and total output-head time by up to 37.6%. Over 100 training updates, DWE reduces mean recorded training-step time by 2.31% while achieving average performance comparable to standard full-distribution reverse-KL distillation across 13 mathematics, code, and science benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Knowledge Distillation, Large Language Models, Unbiased Gradient Estimation, Efficient Post-Training

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_eNnQOpenReview ↗
Rating
1(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Distillation with Errors (DWE), a variant of On-Policy Distillation (OPD), which accelerates the backward process of LLM output head through unbiased gradient estimation. DWE leverages the zero-sum structure of the logit gradient and decoupling it into positive and negative parts. It then constructs an unbiased paired sparse estimator, and uses geometry-aware pairing and stratified sampling to reduce estimation variance. Experiments show that DWE improves the efficiency of Reverse KL baseline by 2.31%, while achieving comparable testing results over 13 benchmarks (38.9% vs 38.2%).

Critical strengths
  1. The gradient construction method and the accompanying unbiasedness derivation is mathematically sound.
  2. The paper connects gradient sparsity to output-head backward efficiency, rather than reporting sparsity alone. The implementation accelerates the backward of output head substantially.
Critical weaknesses
  1. Unclear motivation. The introduction does not convincingly motivate the proposed method. I agree that each token context provides only a local view of optimization, while distillation learns behavior across contexts. However, this does not justify why a vocabulary-wise gradient sparsification is needed.
  2. Marginal efficiency improvement. The proposed method only speeds up the output head of LLM. Even if it brings 37.6% improvement on the output head, it only boosts the overall efficiency by 2.31%. Considering the fluctuations during the training process, it is hard to assess whether the gradient sparsification is actually beneficial.
  3. Unclear benefits. The experiments do not establish whether DWE achieves better downstream quality at a matched training-time budget. A useful comparison would train the reverse-KL baseline for fewer updates to match DWE’s elapsed training time and compare validation scores.
  4. Limited evidence of evaluation robustness. During evaluation, the accuracies are computed over 1 or 5 responses per question for different models. This raises a significant concern in evaluation robustness due to the LLM’s randomness.
Questions for authors
  1. Could you provide the curves of validation scores against training time?
  2. In which realistic configurations does DWE yield materially larger overall speedups? Can this be demonstrated experimentally?
  3. Can the proposed sparse gradient construction be extended to attention layers?
Reviewer_Jhi9OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces Distillation with Errors (DWE), a variant of on-policy distillation that aims to reduce computation. Specifically, DWE backpropagates only selected logit gradients, chosen as zero-sum logit-gradient pairs. The authors show that DWE preserves the dense gradient in expectation. Experiments with Qwen3-1.7B show that DWE reduces recorded training time by around 2%, with a marginal decrease in the average benchmark score..

Critical strengths
  • DWE is a rather simple method with desirable properties, such as preservation of the expected gradient and reduced computation. Its pairing of positive and negative logit gradients is an interesting design choice.
Critical weaknesses
  • Empirical Proof. The largest weakness appears to concern the overall empirical gains. While DWE dramatically reduces computation in the logit heads, its overall effect on wall-clock time appears rather marginal, at only around 2–3%. Furthermore, DWE performs worse than base OPD by up to 4% on some benchmarks. Although it shows improvements on others, I am not sure practitioners would prefer such potential performance degradation just to save 2% in computation. Using a little more compute for more stable training seems to be a better choice. Furthermore, the evaluation is limited to a single model with a single training run, which makes the evidence supporting the empirical claims unreliable, especially since all the reported differences are small. There is also no downstream quality–time ablation over K or a similar analysis showing the tradeoff between quality and time.
  • Limited Baseline Comparison. The paper compares DWE only with full-token OPD and TA-OPD. I believe a critical baseline missing from the paper is sampled-token OPD. This is especially relevant because the paper focuses on a computational tradeoff, and prior work has shown that well-tuned sampled-token OPD can achieve good performance using only a single token. Furthermore, methods such as REOPOLD[1], TrOPD[2], and vOPD[3] have shown improvements in stabilizing the single-token estimator.

[1] Scaling Reasoning Efficiently via Relaxed On-Policy Distillation
[2] Trust Region On-Policy Distillation
[3] KL for a KL: On-Policy Distillation with Control Variate Baseline

Questions for authors

Relative to top-k support OPD, can the combination of DWE on top-k OPD provide additional end-to-end runtime or peak GPU memory savings while maintaining downstream quality?

Reviewer_GEKJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes DWE, an efficient on-policy distillation method that replaces the dense output-head gradient with a sparse unbiased estimator. The method exploits the zero-sum structure of logit gradients, constructs paired positive/negative coordinate updates, and uses geometry-aware pairing and stratified sampling to reduce estimator variance. Experiments show large reductions in output-head backward time while maintaining performance close to full reverse-KL distillation.

Critical strengths
  1. Clean and principled estimator. The zero-sum decomposition leads to a simple sparse gradient estimator that preserves the full-distribution gradient in expectation, with a clear theoretical justification.

  2. Careful variance-reduction design. Geometry-aware pairing and stratified sampling are well motivated and supported by useful ablations, while downstream performance remains close to full reverse-KL distillation.

Critical weaknesses
  1. Limited end-to-end efficiency improvement. The main computational gain is concentrated in the student output-head backward. While this component is accelerated substantially, the reported training-step improvement is only 2.31%, suggesting that the optimized component is not a dominant bottleneck in the evaluated OPD setting.

  2. Limited evidence for practical scaling. The evaluation uses one main teacher/student pair and relatively short training. There is no reported peak-memory benefit, and it remains unclear whether the end-to-end advantage becomes significant for larger models, vocabularies, or longer sequences.

Questions for authors
  1. Can the authors provide a stronger motivation for targeting the LM-head backward? It is not clear that this component is a critical bottleneck in end-to-end OPD training, so the practical need for optimizing it remains somewhat unclear.

  2. What fraction of the total OPD training time is spent in the student output-head backward, and can the authors provide a full runtime breakdown?

  3. Does DWE provide any measurable peak-memory reduction?

  4. How does the end-to-end speedup scale with model size, vocabulary size, and sequence length?

  5. How stable is the performance relative to full reverse-KL over longer training and multiple random seeds?

Reviewer_wdp6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a on-policy distillation method based on sparse unbiased gradient estimation. DWE exploits the zero-sum structure of the logit gradient to form two sampling distributions from its positive and negative parts, and uses output-head geometry to guide the pairing. DWE applies stratified sampling over K pairs to reduce sampling variance. The experimental results show that DWE reduces output-head backward time by up to 96.8%

Critical strengths

Starting from addressing output-head optimization for large vocabularies, this paper proposes a sparse unbiased gradient estimation solution that is general and has a certain effect on output-head backward computation.

Critical weaknesses
  1. The actual effectiveness of DWE is exaggerated. The authors use output-head backward time to show that DWE substantially reduces the computation time of OPD. However, output-head backward time accounts for only a very small fraction of the computational overhead in LLM optimization, especially for large-scale models. In this case, the benefits brought by DWE need to be reconsidered.
  2. Generally speaking, DWE is applicable to any classification-type model with a softmax output. This paper should discuss the effectiveness of DWE from a broader perspective rather than limiting it to the OPD domain.
Questions for authors

What is the OPD performance of DWE on larger-scale models and data from other domains, such as a 32B student model and long-horizon Agent tasks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.