acceptodds
Under review as a conference paper at ICLR 2027

1% of Tokens Can Be Enough: On Gradient Estimation in On-Policy Distillation

Abstract

Sparse on-policy distillation (OPD) allocates teacher supervision to a small subset of tokens in student-generated trajectories. However, useful teacher guidance can yield a noisy update when its gradient is estimated from a sampled next token. We study this estimation problem at a fixed prefix in information geometry and propose an information-efficiency ratio (IER) based on a signal-to-noise decomposition. IER characterizes relative gradient estimation error under an optimal scalar baseline. A candidate-set approximation enables token selection based on IER and its combination with existing usefulness scores, while retaining the sampled reverse-KL training objective. On mathematical and medical reasoning tasks, adding IER improves existing selectors in multiple settings, with sparse configurations matching or exceeding full OPD without token selection at small token budgets of 0.1%-1%. These results support accounting for both usefulness and gradient-estimation reliability when allocating sparse supervision. Our code is available at https://anonymous.4open.science/r/IER-F7FC.

Primary area:
reinforcement learning
Keywords:
On-policy distillation, Token selection, Gradient estimation, Information geometry, Knowledge distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GQaiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies gradient-estimation reliability in sampled on-policy distillation. At a fixed student-generated prefix, it derives a signal–noise decomposition in Fisher geometry and defines the information-efficiency ratio (IER) using an optimal scalar baseline. A numerically stabilized candidate-set approximation ranks token positions, either alone or combined with usefulness scores through soft OR/AND rules. Experiments on mathematical and medical reasoning show that selected configurations using nominal supervision budgets of 0.1%–1% can match or exceed sampled OPD applied at all token positions.

Critical strengths
  • A principled formulation of conditional estimation reliability. The signal–noise decomposition and the inverse-IER interpretation as relative MSE under the optimal scalar baseline are mathematically coherent. Distinguishing estimation reliability from supervision usefulness provides a meaningful perspective on sparse OPD.

  • Promising results under very sparse supervision. The evaluation spans mathematical and medical reasoning and multiple teacher–student settings. In particular, at the 0.1% budget for Qwen3, TIP+IER-AND improves over full OPD on all four mathematical benchmarks, with gains of 5.0, 3.8, 3.0, and 1.0 percentage points in Table 2. These results make the proposed criterion worth investigating.

Critical weaknesses
  • The practical score lacks direct validation. Candidate-set truncation and numerical stabilization can alter the theoretical IER, yet the paper does not verify whether the implemented score predicts relative gradient-estimation error. Its interpretation as a reliability measure therefore remains insufficiently supported.

  • The attribution and robustness of the gains remain unclear. Targeted controls, such as shuffled IER rankings under the same combination rule, are missing. Independent training-run counts and the sources of reported uncertainty are also unspecified, limiting confidence that the improvements reflect IER’s specific contribution and persist across training runs.

Questions for authors
  • Validation of the proposed interpretation: On a fixed bank of prefixes, how well does the implemented score agree with full-vocabulary IER and predict relative gradient-estimation error under the training estimator? Do these rankings remain stable when varying KK, the missing-logit rule, and the leverage floor?

  • Statistical clarification: How many independent training runs support the principal IER comparisons? What randomness do the reported standard errors and intervals cover, and how were checkpoints and OR/AND variants selected? Paired comparisons across training seeds for the principal gains would materially strengthen the evidence.

  • Attribution of the gains: Do the improvements persist against vOPD at matched supervision budgets, and against simple controls such as Euclidean SNR or shuffled IER ranks with the same combination rule? These comparisons would directly affect the assessment of the proposed criterion’s added value.

4 comments

Rebuttal: We are grateful for the encouraging review and constructive feedback! With our best efforts, we conducted additional experiments and analysis to address your concerns. Additional results and clarifications will be added to the manuscript accordingly. Below we provide the point-by-point responses. For your convenience, we provided TL;DR summaries before lengthy detailed explanations to help you quickly grasp our main points.

W1 & Q1: Score validation and sensitivity

W1: The practical score lacks direct validation. Candidate-set truncation and numerical stabilization can alter the theoretical IER, yet the paper does not verify whether the implemented score predicts relative gradient-estimation error. Its interpretation as a reliability measure therefore remains insufficiently supported.

and

Q1: Validation of the proposed interpretation: On a fixed bank of prefixes, how well does the implemented score agree with full-vocabulary IER and predict relative gradient-estimation error under the training estimator? Do these rankings remain stable when varying (K), the missing-logit rule, and the leverage floor?

TL;DR: the choice of K follows previous work and is an efficient choice. Ranking remain stable when imputation and leverage floor is changed.

As we mentioned in Section 4, the analytic IER needs full-vocabullary logits to calculate and this way is computaionally infeasible in practice.

1. Choice of K

No matter how large K is, the truncation exists and the rest of the tokens are ignored. Calculation based on top-K logits is thus a common practice in OPD. For example, vOPD applies k=20 and found that the approximation accurate. Our baseline TA-OPD compares K={8,16,32} and concludes 16 as the default. We tests before that K=32 and 64 do not bring better distillation.

We add a further comparison to other feasible approximations under memory limit (K={32,64,128}). We compare the Spearman correlations against the exact IER calculated in offline evaluation.

KSpearmanTime cost To Ours
Ours(16)0.96751.0×
320.98936.1×
640.994510.6×
1280.998322.3×

This shows that higher K in candidate set leads to higher correlations while K=16 is an efficient choice with high Spearman correlation.

2. Missing value imputation and Leverage floor

Missing value imputation: the missing values are imputed as -12 based on early observations that typically there is only 1 missing value in the top-k logits and -12 is not too high comapred to the tail logits.

Leverage floor: we set the leverage floor as 1e-8 just to avoid zero denominator in the computation, which is a normal choice dealing with similar case.

We add a comparison to solve your concern about ranking sensitivity. Spearman correlations are calculated against the exact IER in offline evaluation. These scores remain stable when the leverage floor and imputation changes.

Leverage FloorImputationSpearman
1e−8(Ours)−120.8523
1e−6−120.7704
1e−10−120.8897
1e−8−100.7717
1e−8−140.8959

To sum up, the choice of K follows previous work and is an efficient choice. Ranking remain stable when imputation and leverage floor is changed.

W2.1 and Q3: Gains Attribution and controls

W2: The attribution and robustness of the gains remain unclear. Targeted controls, such as shuffled IER rankings under the same combination rule, are missing.

and

Q3: Attribution of the gains: Do the improvements persist against vOPD at matched supervision budgets, and against simple controls such as Euclidean SNR or shuffled IER ranks with the same combination rule? These comparisons would directly affect the assessment of the proposed criterion’s added value.

Thank you for pointing this out.

TL;DR: We add additional experiments to show that IER is better than other SNRs and training with baselines does not show better performance.

1. Comparison of Different SNRs

We run the experiments of Euclidean SNR and shuffled IER for comparison in Nemotron pairs with 0.1% token budget. Table shows the average performance of three runs of different mothods.

MethodRunsAIME25AIME26HMMT25HMMT26
TAOPD(baseline)153.9257.2535.1033.87
IER (ours)354.5857.5535.8835.15
Shuffled IER352.4857.6835.0434.23
Euclidean SNR353.5657.4835.9833.93

IER has significant advantages on AIME25 and HMMT26 and outperform Shuffled IER and Euclidean SNR on average.

For a closer look at each seed, this table shows that Fisher IER generally brings better performance than Euclidean SNR.

MethodTraining/rollout seedAIME25AIME26HMMT25HMMT26
Fisher IER1234/4254.7156.6735.4936.19
Euclidean SNR1234/4253.7358.4335.7834.14
Fisher IER1235/4353.4359.3136.1834.76
Euclidean SNR1235/4353.5356.9636.2733.24
Fisher IER1236/4455.5956.6735.9834.49
Euclidean SNR1236/4453.4357.0635.8834.40

2. Training with baseline

Regarding vOPD, we present the results of training with different baselines.

MethodBaselineAIME25AIME26HMMT25HMMT26Average
TAOPDNo (ours)53.957.335.133.945.0
TAOPDFisher53.656.334.435.044.8
TAOPDEuclidean52.757.935.133.444.9
TAOPD+IER-ANDNo (ours)54.756.735.536.245.8
TAOPD+IER-ANDFisher54.658.235.034.845.7
TAOPD+IER-ANDEuclidean52.458.335.835.145.4

This table indicates that adding Fisher and Euclidean baselines does no bring better performance. By the way, selecting tokens with TAOPD+IER-AND consistently outperforms selecting tokens with TAOPD on average across different baseline plans.

Plus, vOPD with Euclidean with full tokens shows no advantage in this setting.

MethodBaselineAIME25AIME26HMMT25HMMT26
vOPDEuclidean55.258.737.034.6
OPD (Table 2)Null56.359.936.934.7

To sum up, IER is better than other SNRs and training with baselines does not show better performance.

W2.2 and Q2

W2: Independent training-run counts and the sources of reported uncertainty are also unspecified, limiting confidence that the improvements reflect IER’s specific contribution and persist across training runs.

and

Q2: Statistical clarification: How many independent training runs support the principal IER comparisons? What randomness do the reported standard errors and intervals cover, and how were checkpoints and OR/AND variants selected? Paired comparisons across training seeds for the principal gains would materially strengthen the evidence.

Thank you for pointing this out.

To clarify, there is only one independent run for each experiment because of our extensive settings (200+ training runs and ~1000 evaluations). In fact, it is a common practice in other papers [1-3]. The standard errors come from the official evaluation codes for Bayes@32 [4] and Healthbench [5]. We are committed to make the evaluation and comparison reliable, thus evaluate each sample 32 times in Math tasks and report the confidence intervals based on the official standard errors. The standard errors are often much smaller than the gains. Only checkpoints of final step are selected in the training. And we add repeated runnings in the comparison of different SNRs.

References:

[1] Yuanyi Wang, Su Lu, Yanggan Gu, Pengkai Wang, Yifan Yang, Zhaoyi Yan, Congkai Xie, Jianmin Wu, and Hongxia Yang. Not all disagreement is learnable: Token teachability in on-policy distillation. arXiv preprint arXiv:2605.26844, 2026.

[2] Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang, and Alborz Geramifard. TIP: Token importance in on-policy distillation. arXiv preprint arXiv:2604.14084, 2026.

[3] Jongwoo Ko, Sara Abdali, Young Jin Kim, Tianyi Chen, and Pashmina Cameron. Scaling reasoning efficiently via relaxed on-policy distillation. arXiv preprint arXiv:2603.11137, 2026.

[4] Mohsen Hariri, Amirhossein Samandar, Michael Hinczewski, and Vipin Chaudhary. Don't pass@k: A bayesian framework for large language model evaluation. arXiv preprint arXiv:2510.04265, 2026.

[5] Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, and Karan Singhal. HealthBench: Evaluating large language models towards improved human health. arXiv preprint arXiv:2505.08775, 2025.

Thank you for your detailed response and the additional experiments. However, I think two points still require further clarification:

  1. Under the default configuration, why do the two analyses yield different Spearman correlation coefficients (0.9675 and 0.8523, respectively)? Furthermore, what is the degree of overlap between the top 0.1%–1% of tokens selected based on the approximate IER and those selected based on the exact IER?

  2. Given that the IER is derived under the assumption of an "optimal scalar baseline"—a baseline that is not actually employed during training—can this approximate IER reliably predict the gradient estimation error for the estimators used in practice?

Reviewer_qVBwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents a method to improve the noisy updates in sparse on policy distillation setup so that sparse OPD leads to better learning outcomes for the student model. In sparse OPD, a primary question is how to select which tokens to grade - grading every token is both computationally expensive and somewhat redundant whereas an RL-like setup where the entire sequence gets a single reward can be too sparse. The paper proposes a combined metric of reliability (of gradient estimation) and usefulness (to the overall learning objective) to inform which tokens should be chosen in a top-K setting. The proposed combined metric (OR or AND of the two criteria) are evaluated for sparse OPD success across a range of student and teacher models sizes. The central claim is: If selected carefully, 1% tokens can be sufficient to provide meaningful feedback for sparse OPD by using a combination of gradient estimation reliability and usefulness.

Critical strengths

Exhaustive experimentation across a few model sizes, several baselines and token budgets.

Core idea of tracking reliability and usefulness is good but needs development. Reliability of gradient is well-defined.

Critical weaknesses

IER is signal to noise ratio. Why not call it just that? It's a standard term in information theory.

Usefulness is not sufficiently defined in Section 4. Table2 shows that the IER-AND and IER-OR are varying by a small amount. If the two metrics are not correlated, IER-AND would show a larger variation. Since the paper's core idea rests on the combination of reliability and usefulness, defining usefulness sufficiently is quite critical. Resting on entropy is not sufficient.

Figure 4: The benefit is somewhat convincing on smaller token budgets but the results in Figure 4 do not seem to support general claims about IER-AND across thinking-on vs off.

IER does not use the direction of the gradient.

The student models are fairly small and while I understand that compute availability is a common limitation for scaling the experiments, the regime of sparse distillation alongside a very small model makes the results (or gains specifically) difficult to tease apart in the results. For example, there are clear gains on HealthBench with Prefix method, but the on Qwen3 (Table2), even the best settings remain far from the teacher across token budgets.

The experimentation is weak in the sense that the results do not show a clear trend for either variant of the combined criteria (IER-AND and IER-OR seem to vary similarly) and most of the differentiation in the results comes from the existing selection methods and the IER-* methods seem to sometimes perform well and sometimes within the stated noise bounds. The best methods vary by domain and token budget and the gains are modest in comparison to the gap between the student and the teacher.

While the motivation is good, it needs further development to become useful across a wide range of settings. It is difficult as an ML practitioner to draw and applicable insights from this paper at this stage beyond the core idea and a bunch of experimental settings that should perhaps be tried.

Questions for authors

Across the tables in the appendix, at various token budgets and model sizes, a different method emerges as the recommended sparse OPD method, with consistent gains with IER-AND. If an ML practitioner were to read the paper, what would you recommend they use as a basis for selecting which algorithm to try?

Did the authors not find the direction of the gradient useful?

7 comments

Rebuttal: We are grateful for the encouraging review and constructive feedback! For your valuable questions and some misunderstandings, additional clarifications will be added to the manuscript accordingly. Below we provide the point-by-point responses. For your convenience, we provided TL;DR summaries before lengthy detailed explanations to help you quickly grasp our main points.

W1: Why not directly call IER as SNR?

W1: IER is signal to noise ratio. Why not call it just that? It's a standard term in information theory.

Thank you for this question! IER is a special Fisher SNR with sampling property, and we call it IER to distinguish from other SNRs in related work.

We emphasize that IER is a signal-to-noise ratio in many places, including but not limited to the abstract, introduction, methodolody and conclusion. We call it as information efficiency ratio instead of directly SNR for two reasons.

  1. We want to differentiate our Fisher SNR from other Euclidean SNRs mentioned in the related work, which mostly are measured in parameter space. They are not the same, which we have discussed in Sec. 3.2, line 142-145. Moreover, the target of defining a SNR is also different. We aim to find the tokens with most reliable gradient estimation while none of previous work is related to token selection in OPD.
  2. We name it as IER (information efficiency ratio) to highlight its property that it prioritizes the tokens with highest information efficiency. The Collorary 1 in Appendix B.3 shows that the reciprocal of IER is exactly the relative MSE of gradient estimation, which directly infers the sampling complexity to get an accurate estimation. A token with high IER needs less samples to be estimated accuratly, indicating it is efficient in learning signal. IER better refelcts this theoretical property.

To sum up, we call our SNR as IER to emphasize its theoretical property and to distinguish from SNRs in previous work.

W2: Definition of usefulness and Table 2 experiments

W2: Usefulness is not sufficiently defined in Section 4. Table 2 shows that the IER-AND and IER-OR are varying by a small amount. If the two metrics are not correlated, IER-AND would show a larger variation. Since the paper's core idea rests on the combination of reliability and usefulness, defining usefulness sufficiently is quite critical. Resting on entropy is not sufficient.

Thank you for rasing this quetsion. We already define usefulness in introduction and some claims of reviewer are misunderstanding.

  1. To clarify, we define usefulness in the last sentence of second paragraph in introduction part (line 41-42) with many examples (more in literature discussion): a token is considered useful if its supervision is likely to help the student. And we use several representatives for comparison, which also consider divergence (TIP) and local disagreement (TA-OPD and CA-SoftOR). Prefix methods only consider the location of tokens because research found that early tokens can be generally more useful than later tokens. Usefulness (or importance) is defined in various ways in previous distillation studies, while gradient estimation relaibility is almost ignored. So we collectively call those metrics in different papers as "Usefulness" and distinguish with IER.
  2. Regarding “Table 2 shows that the IER-AND and IER-OR are varying by a small amount”, in fact not all gains are very small since the standard errors could be much smaller. For example, Prefix+IER-AND +3.0/+3.2/+3.6 on AIME2025/AIME2026/HMMT2025 of Nemotron 1% experiment while the standard error is about 1. For Qwen3 experiment, the standard errors are much smaller (some are ~0.4) but the gains are obviously larger especially with 0.1% budget. And the gains generally exist across different settings beyond Table 2.
  3. To clarify on your point regarding "If the two metrics are not correlated, IER-AND would show a larger variation", token usefulness and gradient reliability are two dimensions, but it does not mean that they are not necessarily uncorrelated. In our analysis, we find that they are correlated (Figure1). However, there are still useful tokens with unreliable gradient (as we show at the end of Appendix). In fact, when two metrics are uncorrelated (Prefix), IER-OR is often better than IER-AND. Across six comparisons in Table 2, IER-AND only outperforms in 7/24 benchmark scores and there are three experiments that Prefix+IER-OR exceeds Prefix+IER-AND on all benchmarks. Regarding why IER-AND, which considers a token to be both useful and reliable at the same time, does not outperform IER-OR, which considers a token to be useful or reliable, IER-AND might select tokens with medium usefulness and medium gradient reliability, which might still have a high IER-AND score. On the other hand, IER-OR may choose tokens with higher gradient reliability (and maybe slightly less usefulness) instead. In this case, IER-OR would still help because a token that has less gradient reliability may mislead the model optimization toward a suboptimal direction, which may cause IER-AND to have a worse performance than IER-OR.

W3: Figure 4 experiments

W3: Figure 4: The benefit is somewhat convincing on smaller token budgets but the results in Figure 4 do not seem to support general claims about IER-AND across thinking-on vs off.

Thank you for this question! To clarify, our focus in this paper is proving sparse OPD (e.g., 0.1% and 1% of tokens) could be enough for training. Figure 4 strongly supports this general claim as you say "convincing" and our claim in fact refers to that IER makes sparse OPD successful. Thanks for reminding us that the claim could be misleading! We will make it clearer in the revised version.

W4 and Q2: Gradient direction

W4: IER does not use the direction of the gradient.

and

Q2: Did the authors not find the direction of the gradient useful?

Thank you for the great question about gradient direction.

TL;DR: IER directly quantifies the estimation relaibility of gradient direction. Gradient direction in OPD are already useful, even more combining with usefulness metrics.

  1. IER does not ignore the importance of gradient direction but directly measure how noisy the direction is estimated. IER prioritizes token gradients with less noisy estimation thus more reliable directions.
  2. Exact usfulness of gradient direction is expensive and computationally prohibited in training. In standard OPD, the gradient under sampled RKL is unbiased and naturally correct to drive student distribution moving towards the teacher distribution. However, this direction does not hint downstream success while there is no gold standard for this. The most related study is Armandpour et al. (2026) [1] we mentioned in line 100. This work calculates the cosine similarity between distillation direction and estimated direction that most benefit task success. However, this estimation needs ~100 times of sampling for each prefix, making training computationally prohibited given limited resources.
  3. Usefulness metrics are good proxies and gradient direction has been further emphasized in IER combinations. As we emphasize in Sec. 4 (line 217-219), IER measures whether the gradient of a token can be accuratly estimated but this token could be less helpful to task performance. This is indeed the motivation of combining reliability with usefulness, which heuristically measures how good a gradient direction is. Usefulness metrics (e.g., TIP and TAOPD) serve as cheap proxy to most useful gradient directon as they are found to be effective.

To sum up, IER directly measures the gradient direction relaibility and can better capture the most beneficial tokens given that the gradient direction is already right (due to sampled RKL in OPD or usefulness metrics).

W5:

W5: The student models are fairly small and while I understand that compute availability is a common limitation for scaling the experiments, the regime of sparse distillation alongside a very small model makes the results (or gains specifically) difficult to tease apart in the results. For example, there are clear gains on HealthBench with Prefix method, but the on Qwen3 (Table2), even the best settings remain far from the teacher across token budgets.

Thanks for pointing this out. Compute availability limits our training model size, which is a common fact for many OPD studies (e.g., Rethinking OPD [2], REOPOLD [3]). However, gains from IER combination remain consistent across different tasks, model pairs and thinking modes. In fact, performance improvment on different settings is often different. Big-to-small distillation is much harder than strong-to-weak experiment due to capacity gap between teacher and student, a well-known consensus since knowledge distillation studies and also supported by previous OPD studies like Rethinking OPD [2]. This is why even the best settings remain far from the teacher.

Reviewer_a58vOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks how reliable the reverse-KL gradient is in sparse on-policy distillation when it is estimated from a single sampled token. The authors derive the signal, noise and optimal baseline of this single-token estimator and define their ratio as IER. A higher IER means the token's gradient estimate is more accurate. In practice, IER is approximated on the top-16 candidate tokens of the student and teacher, and it can be used alone to select tokens or combined with existing usefulness scores. Experiments on math and medical reasoning show that some combinations match or exceed full OPD at token budgets of 0.1% to 1%.

Critical strengths
  1. The theory is simple and clear. Theorem 1 gives closed forms for the signal and noise, and Corollary 1 shows that 1/IER is the relative error of the gradient estimate, so the score has a clear meaning. It is also cheap: training time increases by only about 2%.
  2. The experiments are broad. They cover math and medical reasoning, two distillation settings, thinking on and off, and budgets from 0.1% to 80%, with comparisons against five existing selectors and Liu et al. (2026d). On HealthBench, Prefix+IER-OR at a 0.1% budget improves overall/hard from 38.30/8.68 to 44.98/19.49.
Critical weaknesses
  1. IER does not match what is used in training. IER is derived for the gradient estimator with the optimal baseline b* subtracted, but training uses the plain sampled reverse-KL without this baseline. So IER does not reflect the actual gradient noise during training. In addition, IER is approximated on only about 30 candidate tokens, and the paper does not check whether this approximation is accurate.
  2. The gains are not stable. Most improvements are only 1 to 2 points, about the size of the standard errors, and some combinations make results worse. For example, with Nemotron at a 0.1% budget, every sparse method is below full OPD, and at 0.1% and 1% budgets, Sampled RKL min from Liu et al. beats all IER combinations. The claim that 1% of tokens is enough therefore needs more support.
Questions for authors
  1. Is b* used in training? If not, why select tokens with an IER computed under b*? Also, how is the loss normalized when only a few tokens are selected? If it is averaged over the selected tokens, each token gets a much larger weight, and the gains may come from this rather than from better selection.
  2. How different is the candidate-set IER from the IER computed over the full vocabulary?
3 comments

Rebuttal: We are grateful for the encouraging review and constructive feedback! With our best efforts, we conducted additional experiments and analysis to address your concerns. Additional results and clarifications will be added to the manuscript accordingly. Below we provide the point-by-point responses. For your convenience, we provided TL;DR summaries before lengthy detailed explanations to help you quickly grasp our main points.

W1.1 & Q1.1: Why optimal baseline is not used in training?

W1: IER does not match what is used in training. IER is derived for the gradient estimator with the optimal baseline b* subtracted, but training uses the plain sampled reverse-KL without this baseline. So IER does not reflect the actual gradient noise during training.

and

Q1: Is b* used in training? If not, why select tokens with an IER computed under b*?

TL;DR: Training with baseline is another research question and shows no improvement in additioanl experiment. Fisher b* is helpful to measure training noise and rank tokens but not suitable to be added in loss term.

Thank you for raising this important point! b* is not used for three reasons.

  1. Our research question is token selection in standard OPD instead of in vOPD [1]. Samlped RKL is the default loss in standard OPD and sparse OPD for better comparisons between different selectors against full OPD. We use this loss to study token selection in standard OPD setting. Adding a baseline in training is the setting of vOPD, which changes our reseach question into token selection in vOPD.

  2. IER is reflects the real training noise by upbounding the gradient reliability. The noise term in IER theoretically measure the irreducible lower bound of noise given the optimal baseline, indicating that IER is exactly the upperbound of gradient reliability for each token in training. When the IER score of one token is very small, the gradient is unreliable no matter what baseline is applied. Thus training without a baseline does not influence the noise quantification.

  3. Fisher b* is an approximation value for variance reduction, different to the Euclidean one. vOPD run OPD with optimal baseline in Euclidean space to reduce the variance of sampled RKL. Our b* is in the Fisher space. They are different as we discuss in Sec. 3.2, line 142-145. Our baseline has approximation error, which may make the training loss noisy. But this approximation remains reliable as we clarify in the answer to next question.

To compare with training with baselines , we add an experiment under the setting of JustRL-Nemotron-1.5B → OpenMath-Nemotron-1.5B with 0.1% token budget. Here Euclidean means the baseline in vOPD [1] and Fisher refers to the baseline used in our IER.

MethodBaselineAIME25AIME26HMMT25HMMT26Average
TAOPDNo (ours)53.957.335.133.945.0
TAOPDFisher53.656.334.435.044.8
TAOPDEuclidean52.757.935.133.444.9
TAOPD+IER-ANDNo (ours)54.756.735.536.245.8
TAOPD+IER-ANDFisher54.658.235.034.845.7
TAOPD+IER-ANDEuclidean52.458.335.835.145.4

The experiment shows that adding Fisher and Euclidean baselines does not improve performance. Moreover, selecting tokens with TAOPD+IER-AND consistently outperforms selecting tokens with TAOPD on average across different baseline plans.

W1.2 & Q2: Is IER approximation accurate enough?

W1: In addition, IER is approximated on only about 30 candidate tokens, and the paper does not check whether this approximation is accurate.

and

Q2: How different is the candidate-set IER from the IER computed over the full vocabulary?

TL;DR: We decide the approximation based on previous work and our experimental tests. K=16 is an efficient choice.

As we mentioned in Section 4, the analytic IER needs full-vocabullary logits to calculate and this way is computaionally infeasible in practice. Calculation based on top-K logits is a common practice in OPD to save computation cost. For example, vOPD [1] applies k=20 and found that the approximation is accurate enough. Our baseline TA-OPD [2] compares K={8,16,32} and concludes 16 as the default. We tested K=32 and K=64 before and found that larger K didn't bring better distillation.

We add a further comparison to other feasible approximations under memory limit (K={32,64,128}). We compare the Spearman correlations against the exact IER calculated in offline evaluation.

KSpearmanTime cost To Ours
Ours(16)0.96751.0×
320.98936.1×
640.994510.6×
1280.998322.3×

This shows that higher K in candidate set leads to higher correlation to the exact IER while K=16 is an efficient choice with high Spearman correlation.

W2: Gains are not stable

W2: The gains are not stable. Most improvements are only 1 to 2 points, about the size of the standard errors, and some combinations make results worse. For example, with Nemotron at a 0.1% budget, every sparse method is below full OPD, and at 0.1% and 1% budgets, Sampled RKL min from Liu et al. beats all IER combinations. The claim that 1% of tokens is enough therefore needs more support.

Thank you for pointing this out! We further clarify with more details and takeways here and will revise the paper clearly.

TL;DR: Generally IER combinations make distillation better accross different settings. We clarify some misunderstandings here.

  1. Gains from IER combination are actually large in many settings given the exact standard errors. The standard errors in Table2 are summarized as [0.4,1] and not all gains are only 1 point. To clarify, the standard errors vary in different benchmarks and experiments and the gains are not all small. For example, Prefix+IER-AND +3.0/+3.2/+3.6 on AIME2025/AIME2026/HMMT2025 of Nemotron 1% experiment while the standard error is about 1. For Qwen3 experiment, the standard errors are much smaller (some are ~0.4) but the gains are obviously larger especially with 0.1% budget. For the experiments we present in Figure 2-4, a small overlap of confidence intervals hint significance (e.g., Qwen3 AIME2025 in Figure 3).

  2. Explanations for experiment results. Regarding the Nemorton experiment, the token budget is too small here. Before our work, traditional view about token selection is that generaly more tokens brings better performance. Small budgets limit the sufficiency of learning signals, thus even the best usefulness method is below full OPD across most of our settings. However, selecting tokens with IER-AND and IER-OR generally brings better performance in various settings and thus we found 1% can be enough. This does no hint 1% is always enough for all settings as we found the number is 5% in experiments of Figure 3, while 0.1% could match full OPD in Table1 (e.g., Entropy+IER-OR and TIP+IER-OR) and Table2 (most of methods for Qwen 3 pair). The concurrent work Liu et al. [5] raises simple yet effective methods which achieve great performance in Nemotron experiments, while we also found that they could fail in our Qwen3 experiments, where IER methods bring better results.

Q1.2: Explain the loss normalization in training

Q1: Is b* used in training? If not, why select tokens with an IER computed under b*? Also, how is the loss normalized when only a few tokens are selected? If it is averaged over the selected tokens, each token gets a much larger weight, and the gains may come from this rather than from better selection.

TL;DR: Normalization on selected tokens is the typical and correct approach. Random selection shows the gains of better selection as a control.

Thank you for the question!

We normalize the loss on selected tokens in our training, which is the typical approach for token selection studies in both knowledge distillation (e.g., AdaKD [3]) and OPD (e.g., TIP [4] and TAOPD [2] ). This normalization matches the learning signal of sparse OPD with full OPD. Moreover, when tokens are selected randomly, this loss is a unbiased estimation to full OPD. So normalizetion on selected tokens is correct. As a control, random selection often underperform other selectors in our experiments and related studies, highlighting that distillation could be poor with sparse tokens though the weights are larger. This proves that the gains come form better selection instead of weight changes.

References:

[1] Minjae Oh, Sangjun Song, Gyubin Choi, Yunho Choi, and Yohan Jo. KL for a KL: On-policy distillation with control variate baseline. arXiv preprint arXiv:2605.07865, 2026.

[2] Yuanyi Wang, Su Lu, Yanggan Gu, Pengkai Wang, Yifan Yang, Zhaoyi Yan, Congkai Xie, Jianmin Wu, and Hongxia Yang. Not all disagreement is learnable: Token teachability in on-policy distillation. arXiv preprint arXiv:2605.26844, 2026.

[3] Xurong Xie, Zhucun Xue, Jiafu Wu, Jian Li, Yabiao Wang, Xiaobin Hu, Yong Liu, and Jiangning Zhang. LLM-oriented token-adaptive knowledge distillation. arXiv preprint arXiv:2510.11615, 2025.

[4] Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang, and Alborz Geramifard. TIP: Token importance in on-policy distillation. arXiv preprint arXiv:2604.14084, 2026.

[5] Zhishuai Liu, Xingzi Xu, Mehmet Saygin Seyfioglu, Pan Xu, and Karim Bouyarmane. Extremely sparse supervision incentivizes reasoning ability. arXiv preprint arXiv:2609.04565, 2026.

Thank you for the clarifications and additional experiments, which address my concerns. I am raising my rating from 2 to 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.