- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies gradient-estimation reliability in sampled on-policy distillation. At a fixed student-generated prefix, it derives a signal–noise decomposition in Fisher geometry and defines the information-efficiency ratio (IER) using an optimal scalar baseline. A numerically stabilized candidate-set approximation ranks token positions, either alone or combined with usefulness scores through soft OR/AND rules. Experiments on mathematical and medical reasoning show that selected configurations using nominal supervision budgets of 0.1%–1% can match or exceed sampled OPD applied at all token positions.
Critical strengths
-
A principled formulation of conditional estimation reliability. The signal–noise decomposition and the inverse-IER interpretation as relative MSE under the optimal scalar baseline are mathematically coherent. Distinguishing estimation reliability from supervision usefulness provides a meaningful perspective on sparse OPD.
-
Promising results under very sparse supervision. The evaluation spans mathematical and medical reasoning and multiple teacher–student settings. In particular, at the 0.1% budget for Qwen3, TIP+IER-AND improves over full OPD on all four mathematical benchmarks, with gains of 5.0, 3.8, 3.0, and 1.0 percentage points in Table 2. These results make the proposed criterion worth investigating.
Critical weaknesses
-
The practical score lacks direct validation. Candidate-set truncation and numerical stabilization can alter the theoretical IER, yet the paper does not verify whether the implemented score predicts relative gradient-estimation error. Its interpretation as a reliability measure therefore remains insufficiently supported.
-
The attribution and robustness of the gains remain unclear. Targeted controls, such as shuffled IER rankings under the same combination rule, are missing. Independent training-run counts and the sources of reported uncertainty are also unspecified, limiting confidence that the improvements reflect IER’s specific contribution and persist across training runs.
Questions for authors
-
Validation of the proposed interpretation: On a fixed bank of prefixes, how well does the implemented score agree with full-vocabulary IER and predict relative gradient-estimation error under the training estimator? Do these rankings remain stable when varying , the missing-logit rule, and the leverage floor?
-
Statistical clarification: How many independent training runs support the principal IER comparisons? What randomness do the reported standard errors and intervals cover, and how were checkpoints and OR/AND variants selected? Paired comparisons across training seeds for the principal gains would materially strengthen the evidence.
-
Attribution of the gains: Do the improvements persist against vOPD at matched supervision budgets, and against simple controls such as Euclidean SNR or shuffled IER ranks with the same combination rule? These comparisons would directly affect the assessment of the proposed criterion’s added value.
4 comments
W2.1 and Q3: Gains Attribution and controls
W2: The attribution and robustness of the gains remain unclear. Targeted controls, such as shuffled IER rankings under the same combination rule, are missing.
and
Q3: Attribution of the gains: Do the improvements persist against vOPD at matched supervision budgets, and against simple controls such as Euclidean SNR or shuffled IER ranks with the same combination rule? These comparisons would directly affect the assessment of the proposed criterion’s added value.
Thank you for pointing this out.
TL;DR: We add additional experiments to show that IER is better than other SNRs and training with baselines does not show better performance.
1. Comparison of Different SNRs
We run the experiments of Euclidean SNR and shuffled IER for comparison in Nemotron pairs with 0.1% token budget. Table shows the average performance of three runs of different mothods.
| Method | Runs | AIME25 | AIME26 | HMMT25 | HMMT26 |
|---|---|---|---|---|---|
| TAOPD(baseline) | 1 | 53.92 | 57.25 | 35.10 | 33.87 |
| IER (ours) | 3 | 54.58 | 57.55 | 35.88 | 35.15 |
| Shuffled IER | 3 | 52.48 | 57.68 | 35.04 | 34.23 |
| Euclidean SNR | 3 | 53.56 | 57.48 | 35.98 | 33.93 |
IER has significant advantages on AIME25 and HMMT26 and outperform Shuffled IER and Euclidean SNR on average.
For a closer look at each seed, this table shows that Fisher IER generally brings better performance than Euclidean SNR.
| Method | Training/rollout seed | AIME25 | AIME26 | HMMT25 | HMMT26 |
|---|---|---|---|---|---|
| Fisher IER | 1234/42 | 54.71 | 56.67 | 35.49 | 36.19 |
| Euclidean SNR | 1234/42 | 53.73 | 58.43 | 35.78 | 34.14 |
| Fisher IER | 1235/43 | 53.43 | 59.31 | 36.18 | 34.76 |
| Euclidean SNR | 1235/43 | 53.53 | 56.96 | 36.27 | 33.24 |
| Fisher IER | 1236/44 | 55.59 | 56.67 | 35.98 | 34.49 |
| Euclidean SNR | 1236/44 | 53.43 | 57.06 | 35.88 | 34.40 |
2. Training with baseline
Regarding vOPD, we present the results of training with different baselines.
| Method | Baseline | AIME25 | AIME26 | HMMT25 | HMMT26 | Average |
|---|---|---|---|---|---|---|
| TAOPD | No (ours) | 53.9 | 57.3 | 35.1 | 33.9 | 45.0 |
| TAOPD | Fisher | 53.6 | 56.3 | 34.4 | 35.0 | 44.8 |
| TAOPD | Euclidean | 52.7 | 57.9 | 35.1 | 33.4 | 44.9 |
| TAOPD+IER-AND | No (ours) | 54.7 | 56.7 | 35.5 | 36.2 | 45.8 |
| TAOPD+IER-AND | Fisher | 54.6 | 58.2 | 35.0 | 34.8 | 45.7 |
| TAOPD+IER-AND | Euclidean | 52.4 | 58.3 | 35.8 | 35.1 | 45.4 |
This table indicates that adding Fisher and Euclidean baselines does no bring better performance. By the way, selecting tokens with TAOPD+IER-AND consistently outperforms selecting tokens with TAOPD on average across different baseline plans.
Plus, vOPD with Euclidean with full tokens shows no advantage in this setting.
| Method | Baseline | AIME25 | AIME26 | HMMT25 | HMMT26 |
|---|---|---|---|---|---|
| vOPD | Euclidean | 55.2 | 58.7 | 37.0 | 34.6 |
| OPD (Table 2) | Null | 56.3 | 59.9 | 36.9 | 34.7 |
To sum up, IER is better than other SNRs and training with baselines does not show better performance.
W2.2 and Q2
W2: Independent training-run counts and the sources of reported uncertainty are also unspecified, limiting confidence that the improvements reflect IER’s specific contribution and persist across training runs.
and
Q2: Statistical clarification: How many independent training runs support the principal IER comparisons? What randomness do the reported standard errors and intervals cover, and how were checkpoints and OR/AND variants selected? Paired comparisons across training seeds for the principal gains would materially strengthen the evidence.
Thank you for pointing this out.
To clarify, there is only one independent run for each experiment because of our extensive settings (200+ training runs and ~1000 evaluations). In fact, it is a common practice in other papers [1-3]. The standard errors come from the official evaluation codes for Bayes@32 [4] and Healthbench [5]. We are committed to make the evaluation and comparison reliable, thus evaluate each sample 32 times in Math tasks and report the confidence intervals based on the official standard errors. The standard errors are often much smaller than the gains. Only checkpoints of final step are selected in the training. And we add repeated runnings in the comparison of different SNRs.
References:
[1] Yuanyi Wang, Su Lu, Yanggan Gu, Pengkai Wang, Yifan Yang, Zhaoyi Yan, Congkai Xie, Jianmin Wu, and Hongxia Yang. Not all disagreement is learnable: Token teachability in on-policy distillation. arXiv preprint arXiv:2605.26844, 2026.
[2] Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang, and Alborz Geramifard. TIP: Token importance in on-policy distillation. arXiv preprint arXiv:2604.14084, 2026.
[3] Jongwoo Ko, Sara Abdali, Young Jin Kim, Tianyi Chen, and Pashmina Cameron. Scaling reasoning efficiently via relaxed on-policy distillation. arXiv preprint arXiv:2603.11137, 2026.
[4] Mohsen Hariri, Amirhossein Samandar, Michael Hinczewski, and Vipin Chaudhary. Don't pass@k: A bayesian framework for large language model evaluation. arXiv preprint arXiv:2510.04265, 2026.
[5] Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, and Karan Singhal. HealthBench: Evaluating large language models towards improved human health. arXiv preprint arXiv:2505.08775, 2025.
Thank you for your detailed response and the additional experiments. However, I think two points still require further clarification:
-
Under the default configuration, why do the two analyses yield different Spearman correlation coefficients (0.9675 and 0.8523, respectively)? Furthermore, what is the degree of overlap between the top 0.1%–1% of tokens selected based on the approximate IER and those selected based on the exact IER?
-
Given that the IER is derived under the assumption of an "optimal scalar baseline"—a baseline that is not actually employed during training—can this approximate IER reliably predict the gradient estimation error for the estimators used in practice?
Rebuttal: We are grateful for the encouraging review and constructive feedback! With our best efforts, we conducted additional experiments and analysis to address your concerns. Additional results and clarifications will be added to the manuscript accordingly. Below we provide the point-by-point responses. For your convenience, we provided TL;DR summaries before lengthy detailed explanations to help you quickly grasp our main points.
W1 & Q1: Score validation and sensitivity
and
TL;DR: the choice of K follows previous work and is an efficient choice. Ranking remain stable when imputation and leverage floor is changed.
As we mentioned in Section 4, the analytic IER needs full-vocabullary logits to calculate and this way is computaionally infeasible in practice.
1. Choice of K
No matter how large K is, the truncation exists and the rest of the tokens are ignored. Calculation based on top-K logits is thus a common practice in OPD. For example, vOPD applies k=20 and found that the approximation accurate. Our baseline TA-OPD compares K={8,16,32} and concludes 16 as the default. We tests before that K=32 and 64 do not bring better distillation.
We add a further comparison to other feasible approximations under memory limit (K={32,64,128}). We compare the Spearman correlations against the exact IER calculated in offline evaluation.
This shows that higher K in candidate set leads to higher correlations while K=16 is an efficient choice with high Spearman correlation.
2. Missing value imputation and Leverage floor
Missing value imputation: the missing values are imputed as -12 based on early observations that typically there is only 1 missing value in the top-k logits and -12 is not too high comapred to the tail logits.
Leverage floor: we set the leverage floor as 1e-8 just to avoid zero denominator in the computation, which is a normal choice dealing with similar case.
We add a comparison to solve your concern about ranking sensitivity. Spearman correlations are calculated against the exact IER in offline evaluation. These scores remain stable when the leverage floor and imputation changes.
To sum up, the choice of K follows previous work and is an efficient choice. Ranking remain stable when imputation and leverage floor is changed.