acceptodds
Under review as a conference paper at ICLR 2027

Look Before You Select: Rethinking Vocabulary Sparsification in On-Policy Distillation

Abstract

On-policy distillation (OPD) uses teacher correction on student-generated responses. Full-vocabulary correction can provide important corrections even for tokens that the student assigns low probability, but backpropagating through all token logits becomes memory-intensive for long sequences. Existing memory-saving approaches estimate corrections from sampled tokens or restrict supervision to the student's TopK tokens, introducing sampling noise or changing the full-vocabulary correction. We introduce **SparseOPD**, which uses full-vocabulary teacher correction to determine which corrections matter before selecting the token logits to differentiate. SparseOPD first constructs the full-vocabulary correction without retaining its backward graph, then selects tokens by correction magnitude rather than student probability. Signed residual compensation preserves the total promoting and suppressing correction mass, while correction-aware budget allocation distributes the sparse support across positions. Finally, the update backpropagates only through the selected token logits. Across six task–scale settings spanning mathematics, chemistry QA, and multimodal reasoning, SparseOPD outperforms Sampled Token and TopK in task-average accuracy and matches or exceeds Full Vocabulary. Gradient cosine similarity reaches 99% on 4B mathematics, while 8K full-parameter profiling shows 70.5% lower backward memory.

Primary area:
optimization
Keywords:
on-policy distillation, knowledge distillation, vocabulary sparsification, sparse backpropagation, memory-efficient training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Xt7eOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SparseOPD, which computes full-vocabulary reverse-KL corrections before selecting sparse logit coordinates for differentiation. It combines correction-magnitude selection, signed residual compensation, and batch-global budget allocation. Experiments across mathematical, chemical, and multimodal reasoning show improvements over sampled-token and standard TopK distillation, with reduced backward memory and high gradient similarity to full-vocabulary supervision.

Critical strengths
  1. Principled sparse correction.

The method uses full-vocabulary teacher–student discrepancies to select important corrections and preserves positive and negative correction mass. The derivations are coherent, and the released code includes useful numerical tests.

  1. Consistent improvements over the evaluated sparse baselines.

SparseOPD outperforms standard TopK across six task–scale averages. Component ablations and gradient-fidelity measurements further support the proposed mechanism.

Critical weaknesses
  1. Practical efficiency gains are less substantial than suggested.

The reported 70.5% memory reduction at 8K concerns extra backward allocation; total peak memory decreases by only 14.4%, while complete-update time increases by 14.5%. SparseOPD is also slower than TopK. Moreover, efficiency profiling uses full-parameter training rather than the LoRA setting used for accuracy evaluation, with only one measured update per configuration.

  1. Limited statistical validation of performance gains.

Improvements over TopK are relatively modest (0.58–2.2 points). Supplementary results show variability, including a five-point regression against Full Vocabulary on AIME26. Multi-seed experiments cover only SparseOPD on 1.7B mathematics, without paired baseline runs, leaving the reliability of smaller gains uncertain.

  1. The accuracy–memory trade-off needs stronger controls.

Comparisons with memory-optimized exact reverse-KL implementations (e.g., chunked or recomputed backward) are missing. The supplementary code also includes an alternative sparse projection with lower coefficient error in a unit test, but no downstream evaluation. These comparisons would better establish the benefits of the proposed approximation.

Questions for authors
  1. Can you compare against a fused or custom-backward exact reverse-KL implementation to better isolate the benefit of sparse differentiation?

  2. The supplementary code includes an orthogonal_sparseopd variant that minimizes coefficient L2 error under a weaker constraint. How does preserving separate positive and negative correction masses affect downstream accuracy?

  3. Can you report repeated complete-update timing and total peak memory, including planning, under the LoRA setting used for accuracy evaluation?

  4. Can you provide paired multi-seed comparisons against TopK and Full Vocabulary to establish the reliability of the reported improvements?

Reviewer_7ujgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SparseOPD, a memory-saving approach to full-vocabulary reverse-KL on-policy distillation. It compresses full-vocabulary logit-gradient coefficients and backpropagates through selected logits. Evaluation covers mathematical reasoning, chemistry QA, and multimodal mathematical reasoning.

Critical strengths

The paper is clearly written and logically organized.

Selecting coordinates according to their correction magnitude is well motivated. The distinction between student probability and update importance is clearly explained.

The coefficient construction and signed compensation are mathematically coherent. The error decomposition in Equation (9) helps characterize the approximation.

The component ablations and direct gradient comparisons provide supporting evidence for the design. The reported training-averaged gradient cosine similarity of 0.989 supports update fidelity in the evaluated 4B mathematical reasoning setting.

Critical weaknesses

Missing an optimized exact baseline. The paper does not compare against a memory-efficient exact reverse-KL implementation, such as Nitrobrew. Liger’s fused/chunked distillation infrastructure is also relevant. This comparison is needed to establish whether sparse approximation offers additional benefits beyond implementation-level memory savings.

Limited efficiency measurements. Each configuration is profiled for only one update. At 8K, the 70.5% reduction concerns extra backward allocation; total backward-stage peak allocation decreases by approximately 14.4%, while complete-update time, including planning, increases by 14.5%. Although the appendix discloses these distinctions, repeated steady-state measurements and whole-update peak-memory reporting would better characterize the trade-off.

Unmatched accuracy and efficiency settings. Accuracy experiments use LoRA, whereas gradient-fidelity and efficiency analyses use full-parameter updates. Matched measurements of accuracy, memory, and update time under the same adaptation regime are needed to establish simultaneous quality and resource benefits.

Questions for authors

How does SparseOPD compare against an optimized, exact, chunked reverse-KL implementation under matched hardware, precision, and adaptation settings?

Can the authors provide an accuracy–memory–time frontier across several support budgets, with planning costs included and reported separately?

Does the method preserve training quality at the 8K–16K lengths emphasized in the memory study? Table 7 reports maximum training response lengths of 2,048 tokens for mathematics and multimodal reasoning, and 512 tokens for chemistry.

I will consider raising my score if the authors can adequately elaborate on the problem.

Reviewer_f2RsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SparseOPD, a memory-efficient method for On-Policy Distillation (OPD). The approach decouples "full-vocabulary teacher correction" from "sparse backpropagation." Unlike sampled-token methods, which constrain supervision signals based on the student model’s own probabilities. SparseOPD computes the full-vocabulary gradient coefficients for the reverse KL divergence without requiring the storage of a backward computational graph. Subsequently, it constructs a sparse update signal through mechanisms such as filtering based on correction magnitude, signed residual compensation, and global budget allocation across the batch.

Critical strengths

1. Rigorous conceptualization with a logically coherent technical design.

This paper introduces a compelling perspective on vocabulary sparsification: rather than selecting tokens solely based on the probabilities predicted by the student model, it filters them according to their contribution to the "full-vocabulary correction." In my view, this conceptual approach is the paper’s most significant strength.

2. Encouraging experimental results, supported by direct validation of gradient fidelity.

Across six distinct tasks and scale settings, the proposed method consistently outperforms various sparsification baselines in terms of average task accuracy. More importantly, gradient analysis provides direct evidence that SparseOPD approximates "full-vocabulary updates" more accurately than Top-K or sampling-based methods.

Critical weaknesses
  1. Lack of comparison against stronger, more relevant baselines. In the experimental evaluation, SparseOPD is primarily compared against sampling-based distillation, traditional Top-K distillation (based on student probabilities), and full-vocabulary distillation. However, the paper fails to benchmark against more advanced "tail-aware" sparse distillation methods, including the highly relevant tail-aware Top-K OPD approach mentioned elsewhere in the text. Furthermore, regarding the claimed "efficiency advantages," the paper does not provide comparisons against memory-optimized implementations of "exact full-vocabulary distillation" (such as those utilizing vocabulary chunking, recomputation, or fused backward-pass operations). Without these comparisons, it is difficult to determine whether the proposed method truly offers an advantage over other, more robust alternatives.

  2. In terms of actual efficiency gains, the method’s performance is not as significant as the claimed memory savings might suggest. With a vocabulary size of 8K, the 70.5% reduction in memory usage mentioned in the paper refers specifically to the additional memory saved during the backpropagation phase (relative to a “pre-step baseline”), rather than a decrease in the total peak GPU memory consumption. Furthermore, with an 8K vocabulary, SparseOPD increases the time required to complete a full update from 72.04 seconds to 82.50 seconds. Consequently, the method essentially trades increased computation time for lower memory usage, rather than achieving a genuine, substantial improvement in overall efficiency. Additionally, the efficiency experiments were conducted using full-parameter training, with only a single measurement taken for each configuration, whereas the primary accuracy experiments utilized the LoRA technique.

Questions for authors

Same as weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.