- Rating
- 1(−1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes Distillation with Errors (DWE), a variant of On-Policy Distillation (OPD), which accelerates the backward process of LLM output head through unbiased gradient estimation. DWE leverages the zero-sum structure of the logit gradient and decoupling it into positive and negative parts. It then constructs an unbiased paired sparse estimator, and uses geometry-aware pairing and stratified sampling to reduce estimation variance. Experiments show that DWE improves the efficiency of Reverse KL baseline by 2.31%, while achieving comparable testing results over 13 benchmarks (38.9% vs 38.2%).
Critical strengths
- The gradient construction method and the accompanying unbiasedness derivation is mathematically sound.
- The paper connects gradient sparsity to output-head backward efficiency, rather than reporting sparsity alone. The implementation accelerates the backward of output head substantially.
Critical weaknesses
- Unclear motivation. The introduction does not convincingly motivate the proposed method. I agree that each token context provides only a local view of optimization, while distillation learns behavior across contexts. However, this does not justify why a vocabulary-wise gradient sparsification is needed.
- Marginal efficiency improvement. The proposed method only speeds up the output head of LLM. Even if it brings 37.6% improvement on the output head, it only boosts the overall efficiency by 2.31%. Considering the fluctuations during the training process, it is hard to assess whether the gradient sparsification is actually beneficial.
- Unclear benefits. The experiments do not establish whether DWE achieves better downstream quality at a matched training-time budget. A useful comparison would train the reverse-KL baseline for fewer updates to match DWE’s elapsed training time and compare validation scores.
- Limited evidence of evaluation robustness. During evaluation, the accuracies are computed over 1 or 5 responses per question for different models. This raises a significant concern in evaluation robustness due to the LLM’s randomness.
Questions for authors
- Could you provide the curves of validation scores against training time?
- In which realistic configurations does DWE yield materially larger overall speedups? Can this be demonstrated experimentally?
- Can the proposed sparse gradient construction be extended to attention layers?