- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work investigates whether OPD necessarily relies on dense token-level supervision. It examines whether supervising only a very small number of tokens per reasoning trajectory can still lead to effective learning. It applies sparse masks to the distillation loss and compares strategies such as random token selection and extreme positive or negative supervision signal selection. The experimental results are evaluated on mathematical and coding reasoning tasks.
Critical strengths
The research question is clear and has independent value: In some configurations, supervising only a few tokens per trajectory significantly improves the base student, and some sparse OPD variants can even match or outperform plain OPD, which is enlightening for understanding the role of supervision density. The experiments cover nine Qwen3 teacher–student configurations, various token-selection strategies and threshold sweeps, and examine both avg@8 and pass@k, providing a more systematic description of the phenomenon.
Critical weaknesses
-
The paper argues that random single-token supervision still approximately optimizes the same underlying OPD objective. However, according to Equation (3.1), the loss scaling depends on response length, so sparse supervision may change the relative weights of different trajectories, rather than merely increasing the gradient variance. The paper lacks analysis and experimental verification of this issue.
-
The paper finds that extreme supervision signals can effectively improve model performance and attempts to explain this phenomenon through the mode-seeking property of reverse KL minimization and the student's representation capacity. However, these explanations have not been fully verified. In particular, the authors' token audit indicates that most extremely negative supervision signals come from correct-but-teacher-dispreferred tokens, rather than genuine mathematical errors. Therefore, why extreme supervision signals can provide more effective reasoning supervision still lacks a clear mechanism explanation.
-
The main experiments use Qwen3 backbones, with additional validation on Llama models in the appendix. However, broader model-family validation remains limited. It is recommended to evaluate more model backbones and conduct multiple training runs with different random seeds to verify training stability.
Questions for authors
See weaknesses.
3 comments
Q3: Additional model-family validation and multi-seed training stability validation.
Response:
Additional model family. In the revised paper, we add five-seed experiments with a Qwen2.5-1.5B-Instruct student and Qwen2.5-14B-Instruct teacher. Because the 1.5B student is weaker than the Qwen3 students used in our primary experiments, we evaluate it on MATH500, AMC23, Minerva, and OlympiadBench. Results are reported in Table 7 (avg@8, mean ± standard deviation, %):
| Method | MATH500 | AMC23 | Minerva | OlympiadBench | Mean |
|---|---|---|---|---|---|
| Teacher | 78.4 | 58.4 | 45.4 | 44.6 | 56.7 |
| Student | 51.1 | 22.5 | 18.0 | 18.8 | 27.6 |
| plain OPD | 55.4±0.6 | 30.4±2.6 | 21.7±0.5 | 23.1±0.3 | 32.7±0.6 |
| 1 token per trajectory | |||||
| rand1tok | 51.9±0.7 | 25.5±2.3 | 19.6±0.6 | 20.0±0.6 | 29.3±0.9 |
| mintok | 54.3±0.5 | 30.4±1.6 | 21.9±0.4 | 22.0±0.3 | 32.1±0.5 |
| maxtok | 37.0±0.6 | 15.6±1.0 | 13.5±0.5 | 10.8±0.2 | 19.2±0.5 |
| 2 tokens per trajectory | |||||
| minmaxtok | 54.0±0.7 | 29.6±0.7 | 20.8±0.5 | 22.0±0.3 | 31.6±0.3 |
| 0.1% tokens per trajectory | |||||
| randmask 0.1% | 52.0±0.7 | 27.5±1.6 | 20.0±0.6 | 20.1±0.4 | 29.9±0.3 |
| pctltail 0.05% per tail | 53.7±0.8 | 28.7±2.3 | 20.8±0.7 | 21.4±0.5 | 31.2±0.7 |
On Qwen2.5, mintok (32.1 ± 0.5) and minmaxtok (31.6 ± 0.3) come within 0.6 and 1.1 points, respectively, of dense OPD (32.7 ± 0.6). On Llama, mintok and pctltail 0.05% per tail generally match dense OPD (Figure 22), although the other sparse variants do not improve the base student. Across the tested configurations in which dense OPD improves the student, we observe at least one extremely sparse variant achieves performances on par with plain OPD. We treat this recurring pattern as an empirical conjecture, not a general guarantee. To the best of our knowledge, Qwen3, Qwen2.5, and Llama are the model families for which prior work has reported successful sampled-token OPD for reasoning. Our experiments now cover all three, and we observe the extremely sparse supervision phenomenon in each. We also tested Gemma 3, Gemma 4, and Mistral 3. Under our configurations, plain OPD failed to improve these students and sometimes collapsed; extremely sparse OPD also failed to improve them but degraded them less, consistent with applying substantially fewer token-level supervision signals. Taken together, these results motivate the conjecture that successful dense OPD is a precondition for observing the sparse phenomenon. We do not treat this as a universal guarantee and discuss the limitation in Section 5.
Multiple training seeds. Due to rebuttal-time compute constraints, five-seed matched comparisons and controls are limited to Family 9 (70 training runs); Qwen2.5 also uses five seeds, while the other Qwen3 configurations and Llama use one. Across all tested settings, at least one sparse variant matches or outperforms dense OPD, and the multi-seed studies show low variation. This complementary evidence supports robustness in the settings tested; we will add multi-seed validation for additional configurations in the final version.
Table 5 reports avg@8 scores (mean ± standard deviation) across five independent training seeds (seeds 0–4):
| Method | AIME24 | AIME25 | HMMT25 | MATH500 | BeyondAIME | Mean |
|---|---|---|---|---|---|---|
| plain OPD | 63.7±1.8 | 45.8±2.3 | 28.1±2.2 | 93.1±0.2 | 32.1±0.3 | 52.6±0.8 |
| rand1tok | 47.0±4.9 | 35.0±3.0 | 20.4±2.6 | 89.7±0.7 | 24.2±2.6 | 43.3±2.3 |
| mintok | 57.2±1.8 | 45.9±3.8 | 26.6±1.8 | 92.3±0.2 | 32.7±1.2 | 50.9±1.4 |
| maxtok | 63.1±1.5 | 51.6±1.0 | 29.6±2.2 | 93.0±0.2 | 34.6±0.8 | 54.4±0.6 |
| minmaxtok | 63.8±1.4 | 50.0±3.5 | 30.8±1.8 | 93.0±0.4 | 35.4±0.8 | 54.6±0.3 |
| randmask 0.1% | 59.1±3.7 | 44.7±4.8 | 28.8±1.6 | 91.5±1.0 | 30.7±2.3 | 51.0±2.3 |
| pctltail 0.05% per tail | 61.6±1.0 | 47.5±1.9 | 30.4±1.3 | 92.9±0.3 | 33.7±1.3 | 53.2±0.7 |
Across Table 5, benchmark-level standard deviations range from 0.2 to 4.9 points, while standard deviations of the five-benchmark mean range from 0.3 to 2.3 points. Maxtok and minmaxtok have higher sample means than plain OPD in Family 9. These results show that the headline gains persist across independent runs.
In summary, the revision shows that active-token normalization yields similar point estimates in Family 9, adds Qwen2.5 validation, and reports five-seed stability results for Family 9. We also narrow the mechanistic claims: the paper establishes the phenomenon and relevant empirical constraints, while treating its causal explanation as an open problem.
We hope these additions address the reviewer's concerns. We would be happy to answer any remaining questions.
Dear Reviewer cuMN,
The authors have provided a detailed reply to your review. Could you please take a look, acknowledge the rebuttal, and give your response?
Thank you for serving as a reviewer for ICLR.
Best,
AC
We thank the reviewer for their valuable time and constructive feedback. We hope the responses below address all questions and concerns.
Q1: Compare different loss scaling under sparsity.
Response: In our implementation, the loss is normalized by the total number of response tokens in the batch (
N_all). We add details to Appendix E.2 of the revised paper.The original manuscript did not make this reduction explicit; Appendix E.2 of the revision now does so.
We compare two batch-level reductions: division by
N_all(all response tokens) and division byN_kept(activated tokens). For a fixed batch, these reductions differ by a positive scalar and do not change the relative weights of trajectories within that batch. Adam also largely cancels a consistent gradient rescaling: its first-moment estimate scales linearly, whereas the square root of its second-moment estimate scales by the same factor. This invariance is approximate because the scalar varies across batches, Adam's denominator includes the numerical-stability termε, and operations such as gradient-norm clipping can respond differently to the two scales. We therefore test the choice empirically using minmaxtok over five Family 9 seeds (Table 32,avg@8, %):The two reductions yield similar performance in Family 9 (54.6 ± 0.3 and 54.8 ± 0.8).
Q2: Missing Mechanistic Explanation
Response: We agree that the paper does not establish a causal mechanism for this phenomenon. Our contribution is to establish the phenomenon systematically and report observations that constrain possible explanations. To avoid overstating our interpretation, we renamed the appendix subsection from "Why and How Extreme Token Supervision Works" to "Motivation, Potential Explanation, and Related Empirical Observations." Reverse KL's mode-seeking behavior motivated us to examine tokens with extreme positive or negative supervision signals; pushing this idea to its limit led to the one-token-supervision result. The observed relationship between student capacity and the effectiveness of positive-signal supervision is empirical. We present the mode-seeking account only as a tentative interpretation.
We would like to delimit the open question of the underlying mechanism based on our established empirical observations. We hope these observations help explain why identifying the mechanism is difficult given the current understanding of neural networks and OPD.
Existing token-selection literature analyzes sparse token selection in terms of gradients. The general idea in prior literature is to use gradients from a few tokens to recover plain OPD. However, this explanation does not work in the extremely sparse regime. From the entropy results (Figure 18) and reverse KL measurements (Tables 1–4), students trained with sparse OPD are substantially different from students trained with plain OPD: they either have higher entropy, or their reverse KL from the teacher policy increases rather than decreases, even though reducing this divergence is the objective of plain OPD. Strictly speaking, extremely sparse OPD is no longer conventional distillation and is closer to "tutoring". It represents a different training regime in which signal accuracy can matter more than signal density. To the best of our knowledge, there is currently no theory connecting parameter updates induced by such sparse signals to improvements in reasoning ability.
The reviewer makes an important observation that many negative supervision signals occur on correct-but-teacher-dispreferred tokens rather than genuine mathematical errors. This suggests that negative-token extremely sparse OPD does not work by simply correcting errors. We further test this hypothesis in an ablation: we use a frontier model to identify errors in each response, assign large negative signals to those tokens, and train only on them. The student collapses within a few steps.
Thus, developing a rigorous explanation for the extremely sparse supervision phenomenon is difficult and beyond the scope of the present empirical study. We leave this question for future investigation.