acceptodds
Under review as a conference paper at ICLR 2027

Extremely Sparse Supervision Incentivizes Reasoning Ability

Abstract

Large language models demonstrate increasingly strong reasoning capabilities through effective post-training. Yet, prevailing post-training methods optimize over massive numbers of tokens, implicitly assuming that effective learning must be token-intensive. We revisit this assumption in the on-policy distillation (OPD) setting, which naturally admits dense teacher supervision at every generated token. Using the Qwen3 family, **we discover a counter-intuitive phenomenon: reasoning can be effectively incentivized by an extremely small fraction of generated tokens—as few as one or two tokens per reasoning trajectory, corresponding to only 0.05% of all tokens**. Surprisingly, this sparse supervision in most cases matches or surpasses full-token training in improving reasoning ability, despite excluding the vast majority of generated tokens from the training objective. This phenomenon is consistently observed across nine teacher–student configurations spanning different model scales on mathematical reasoning tasks, and is further validated on coding reasoning, Llama models and Proximal Policy Optimization (PPO)-based reinforcement learning with verifiable reward (RLVR). Interestingly, such extremely sparse supervision may be closer to the natural learning process: rather than correcting every step word by word, one reflects on a few critical reasoning steps, updates one’s prior understanding, and continues the trial-and-error, avoiding micro-level corrections while remaining remarkably effective. Overall, our results challenge the assumption that effective post-training must be token-intensive and point to a new direction for understanding and designing more efficient post-training algorithms.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, post-training, reasoning, OPD

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cuMNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work investigates whether OPD necessarily relies on dense token-level supervision. It examines whether supervising only a very small number of tokens per reasoning trajectory can still lead to effective learning. It applies sparse masks to the distillation loss and compares strategies such as random token selection and extreme positive or negative supervision signal selection. The experimental results are evaluated on mathematical and coding reasoning tasks.

Critical strengths

The research question is clear and has independent value: In some configurations, supervising only a few tokens per trajectory significantly improves the base student, and some sparse OPD variants can even match or outperform plain OPD, which is enlightening for understanding the role of supervision density. The experiments cover nine Qwen3 teacher–student configurations, various token-selection strategies and threshold sweeps, and examine both avg@8 and pass@k, providing a more systematic description of the phenomenon.

Critical weaknesses
  1. The paper argues that random single-token supervision still approximately optimizes the same underlying OPD objective. However, according to Equation (3.1), the loss scaling depends on response length, so sparse supervision may change the relative weights of different trajectories, rather than merely increasing the gradient variance. The paper lacks analysis and experimental verification of this issue.

  2. The paper finds that extreme supervision signals can effectively improve model performance and attempts to explain this phenomenon through the mode-seeking property of reverse KL minimization and the student's representation capacity. However, these explanations have not been fully verified. In particular, the authors' token audit indicates that most extremely negative supervision signals come from correct-but-teacher-dispreferred tokens, rather than genuine mathematical errors. Therefore, why extreme supervision signals can provide more effective reasoning supervision still lacks a clear mechanism explanation.

  3. The main experiments use Qwen3 backbones, with additional validation on Llama models in the appendix. However, broader model-family validation remains limited. It is recommended to evaluate more model backbones and conduct multiple training runs with different random seeds to verify training stability.

Questions for authors

See weaknesses.

3 comments

We thank the reviewer for their valuable time and constructive feedback. We hope the responses below address all questions and concerns.

Q1: Compare different loss scaling under sparsity.

Response: In our implementation, the loss is normalized by the total number of response tokens in the batch (N_all). We add details to Appendix E.2 of the revised paper.

The original manuscript did not make this reduction explicit; Appendix E.2 of the revision now does so.

We compare two batch-level reductions: division by N_all (all response tokens) and division by N_kept (activated tokens). For a fixed batch, these reductions differ by a positive scalar and do not change the relative weights of trajectories within that batch. Adam also largely cancels a consistent gradient rescaling: its first-moment estimate scales linearly, whereas the square root of its second-moment estimate scales by the same factor. This invariance is approximate because the scalar varies across batches, Adam's denominator includes the numerical-stability term ε, and operations such as gradient-norm clipping can respond differently to the two scales. We therefore test the choice empirically using minmaxtok over five Family 9 seeds (Table 32, avg@8, %):

Benchmark1/N_all1/N_kept
AIME2463.8±1.466.1±1.6
AIME2550.0±3.549.7±2.3
HMMT2530.8±1.829.7±1.4
MATH50093.0±0.493.0±0.4
BeyondAIME35.4±0.835.7±0.9
Mean54.6±0.354.8±0.8

The two reductions yield similar performance in Family 9 (54.6 ± 0.3 and 54.8 ± 0.8).

Q2: Missing Mechanistic Explanation

Response: We agree that the paper does not establish a causal mechanism for this phenomenon. Our contribution is to establish the phenomenon systematically and report observations that constrain possible explanations. To avoid overstating our interpretation, we renamed the appendix subsection from "Why and How Extreme Token Supervision Works" to "Motivation, Potential Explanation, and Related Empirical Observations." Reverse KL's mode-seeking behavior motivated us to examine tokens with extreme positive or negative supervision signals; pushing this idea to its limit led to the one-token-supervision result. The observed relationship between student capacity and the effectiveness of positive-signal supervision is empirical. We present the mode-seeking account only as a tentative interpretation.

We would like to delimit the open question of the underlying mechanism based on our established empirical observations. We hope these observations help explain why identifying the mechanism is difficult given the current understanding of neural networks and OPD.

  1. Existing token-selection literature analyzes sparse token selection in terms of gradients. The general idea in prior literature is to use gradients from a few tokens to recover plain OPD. However, this explanation does not work in the extremely sparse regime. From the entropy results (Figure 18) and reverse KL measurements (Tables 1–4), students trained with sparse OPD are substantially different from students trained with plain OPD: they either have higher entropy, or their reverse KL from the teacher policy increases rather than decreases, even though reducing this divergence is the objective of plain OPD. Strictly speaking, extremely sparse OPD is no longer conventional distillation and is closer to "tutoring". It represents a different training regime in which signal accuracy can matter more than signal density. To the best of our knowledge, there is currently no theory connecting parameter updates induced by such sparse signals to improvements in reasoning ability.

  2. The reviewer makes an important observation that many negative supervision signals occur on correct-but-teacher-dispreferred tokens rather than genuine mathematical errors. This suggests that negative-token extremely sparse OPD does not work by simply correcting errors. We further test this hypothesis in an ablation: we use a frontier model to identify errors in each response, assign large negative signals to those tokens, and train only on them. The student collapses within a few steps.

Thus, developing a rigorous explanation for the extremely sparse supervision phenomenon is difficult and beyond the scope of the present empirical study. We leave this question for future investigation.

Q3: Additional model-family validation and multi-seed training stability validation.

Response:

Additional model family. In the revised paper, we add five-seed experiments with a Qwen2.5-1.5B-Instruct student and Qwen2.5-14B-Instruct teacher. Because the 1.5B student is weaker than the Qwen3 students used in our primary experiments, we evaluate it on MATH500, AMC23, Minerva, and OlympiadBench. Results are reported in Table 7 (avg@8, mean ± standard deviation, %):

MethodMATH500AMC23MinervaOlympiadBenchMean
Teacher78.458.445.444.656.7
Student51.122.518.018.827.6
plain OPD55.4±0.630.4±2.621.7±0.523.1±0.332.7±0.6
1 token per trajectory
rand1tok51.9±0.725.5±2.319.6±0.620.0±0.629.3±0.9
mintok54.3±0.530.4±1.621.9±0.422.0±0.332.1±0.5
maxtok37.0±0.615.6±1.013.5±0.510.8±0.219.2±0.5
2 tokens per trajectory
minmaxtok54.0±0.729.6±0.720.8±0.522.0±0.331.6±0.3
0.1% tokens per trajectory
randmask 0.1%52.0±0.727.5±1.620.0±0.620.1±0.429.9±0.3
pctltail 0.05% per tail53.7±0.828.7±2.320.8±0.721.4±0.531.2±0.7

On Qwen2.5, mintok (32.1 ± 0.5) and minmaxtok (31.6 ± 0.3) come within 0.6 and 1.1 points, respectively, of dense OPD (32.7 ± 0.6). On Llama, mintok and pctltail 0.05% per tail generally match dense OPD (Figure 22), although the other sparse variants do not improve the base student. Across the tested configurations in which dense OPD improves the student, we observe at least one extremely sparse variant achieves performances on par with plain OPD. We treat this recurring pattern as an empirical conjecture, not a general guarantee. To the best of our knowledge, Qwen3, Qwen2.5, and Llama are the model families for which prior work has reported successful sampled-token OPD for reasoning. Our experiments now cover all three, and we observe the extremely sparse supervision phenomenon in each. We also tested Gemma 3, Gemma 4, and Mistral 3. Under our configurations, plain OPD failed to improve these students and sometimes collapsed; extremely sparse OPD also failed to improve them but degraded them less, consistent with applying substantially fewer token-level supervision signals. Taken together, these results motivate the conjecture that successful dense OPD is a precondition for observing the sparse phenomenon. We do not treat this as a universal guarantee and discuss the limitation in Section 5.

Multiple training seeds. Due to rebuttal-time compute constraints, five-seed matched comparisons and controls are limited to Family 9 (70 training runs); Qwen2.5 also uses five seeds, while the other Qwen3 configurations and Llama use one. Across all tested settings, at least one sparse variant matches or outperforms dense OPD, and the multi-seed studies show low variation. This complementary evidence supports robustness in the settings tested; we will add multi-seed validation for additional configurations in the final version.

Table 5 reports avg@8 scores (mean ± standard deviation) across five independent training seeds (seeds 0–4):

MethodAIME24AIME25HMMT25MATH500BeyondAIMEMean
plain OPD63.7±1.845.8±2.328.1±2.293.1±0.232.1±0.352.6±0.8
rand1tok47.0±4.935.0±3.020.4±2.689.7±0.724.2±2.643.3±2.3
mintok57.2±1.845.9±3.826.6±1.892.3±0.232.7±1.250.9±1.4
maxtok63.1±1.551.6±1.029.6±2.293.0±0.234.6±0.854.4±0.6
minmaxtok63.8±1.450.0±3.530.8±1.893.0±0.435.4±0.854.6±0.3
randmask 0.1%59.1±3.744.7±4.828.8±1.691.5±1.030.7±2.351.0±2.3
pctltail 0.05% per tail61.6±1.047.5±1.930.4±1.392.9±0.333.7±1.353.2±0.7

Across Table 5, benchmark-level standard deviations range from 0.2 to 4.9 points, while standard deviations of the five-benchmark mean range from 0.3 to 2.3 points. Maxtok and minmaxtok have higher sample means than plain OPD in Family 9. These results show that the headline gains persist across independent runs.

In summary, the revision shows that active-token normalization yields similar point estimates in Family 9, adds Qwen2.5 validation, and reports five-seed stability results for Family 9. We also narrow the mechanistic claims: the paper establishes the phenomenon and relevant empirical constraints, while treating its causal explanation as an open problem.

We hope these additions address the reviewer's concerns. We would be happy to answer any remaining questions.

Dear Reviewer cuMN,

The authors have provided a detailed reply to your review. Could you please take a look, acknowledge the rebuttal, and give your response?

Thank you for serving as a reviewer for ICLR.

Best,

AC

Reviewer_9WWTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how much token-level supervision is needed in sampled-token on-policy distillation. The authors retain one or two positions per student trajectory, or roughly 0.05%–0.1% of the response, using random or signal-based masks. Across nine Qwen3 teacher–student configurations, these variants often retain much of the gain from dense OPD, and the best sparse variant sometimes performs better. Smaller studies cover coding, Llama models, and PPO.

The most striking result is that supervising a single token can remain effective across several teacher–student regimes. The paper attributes this to high redundancy in dense token-level supervision and argues that extremely sparse targets may be sufficient for OPD.

Critical strengths
  • The central empirical finding is simple and consequential: useful OPD updates can arise from an extremely small number of supervised positions, sometimes only one token per response.
  • The nine Qwen3 teacher–student configurations cover strong-to-weak, same-scale, and weak-to-strong regimes. The paper also reports teacher and student baselines, retained-token fractions, pass@k, reverse KL, response lengths, and mixed results in the Llama and PPO extensions.
Critical weaknesses
  • The experiments do not yet isolate the contribution of the proposed extreme-sparsity regime from existing token-selection methods and objective scaling. TIP already studies entropy- and divergence-based token selection in OPD and reports that a small selected fraction can approach full-token training, but the paper includes no TIP-style baseline at the same 0.05%–0.1% budget. In addition, Equation (3.1) retains the 1/|y| denominator after masking. With one uniformly sampled position, this changes the length weighting of the expected gradient unless the implementation uses a different reduction. These issues affect both the novelty claim and the interpretation of why sparse supervision works.
  • The evidence is insufficient to establish that a deployable sparse rule reliably outperforms dense OPD or improves efficiency. The discussion highlights the best result among six masks, while several dense-versus-sparse gaps are small and the reported intervals capture only inference sampling from a fixed checkpoint. Representative training seeds for a rule selected before test evaluation are needed. The efficiency claim also lacks peak-memory, throughput, step-time, and wall-clock measurements; sparse OPD retains rollout cost, and signal-based masks still require teacher scores at every position.
Questions for authors
  • Clarification: What loss reduction is used after masking: all response positions, active positions, or all active positions in the batch? Does the result persist with active-token normalization or an inverse-probability/scale-matched control?
  • Assessment-critical evidence: At the same 0.05%–0.1% supervision budget, how do TIP-style entropy or teacher–student divergence selection and a pre-specified proposed rule compare across multiple training seeds?
  • Assessment-critical evidence: What are the measured peak memory, step time, throughput, and total wall-clock costs of dense and sparse OPD? Did the reported experiments use gather-then-project?
6 comments

We thank the reviewer for their valuable time and constructive feedback. We hope the responses below address all questions and concerns.

Q1: Compare different loss normalization.

Response: In our implementation, the loss is normalized by the total number of response tokens in the batch (N_all). We add details to Appendix E.2 of the revised paper.

We compare two batch-level reductions: division by N_all (all response tokens) and division by N_kept (activated tokens). For a fixed batch, these reductions differ by a positive scalar and do not change the relative weights of trajectories within that batch. Adam also largely cancels a consistent gradient rescaling: its first-moment estimate scales linearly, whereas the square root of its second-moment estimate scales by the same factor. This invariance is approximate because the scalar varies across batches, Adam's denominator includes the numerical-stability term ε, and operations such as gradient-norm clipping can respond differently to the two scales. We therefore test the choice empirically using minmaxtok over five Family 9 seeds (Table 32, avg@8, %):

Benchmark1/N_all1/N_kept
AIME2463.8±1.466.1±1.6
AIME2550.0±3.549.7±2.3
HMMT2530.8±1.829.7±1.4
MATH50093.0±0.493.0±0.4
BeyondAIME35.4±0.835.7±0.9
Mean54.6±0.354.8±0.8

The two reductions yield similar performance in Family 9 (54.6 ± 0.3 and 54.8 ± 0.8), showing that normalization alone does not account for the result in this setting.

Q2: Comparison with TIP at matched budget across training seeds.

Response: We provide this comparison in revised Table 5. We compare all methods under identical 1-token, 2-token, and 0.1%-per-trajectory budgets across five independent training seeds. Results in Table 5 report avg@8 scores (mean ± standard deviation):

MethodAIME24AIME25HMMT25MATH500BeyondAIMEMean
plain OPD63.7±1.845.8±2.328.1±2.293.1±0.232.1±0.352.6±0.8
1 token per trajectory
rand1tok47.0±4.935.0±3.020.4±2.689.7±0.724.2±2.643.3±2.3
mintok57.2±1.845.9±3.826.6±1.892.3±0.232.7±1.250.9±1.4
maxtok63.1±1.551.6±1.029.6±2.293.0±0.234.6±0.854.4±0.6
TIP62.7±2.448.9±3.229.7±1.592.9±0.432.9±1.253.4±1.4
TIP+IER60.2±2.145.7±1.529.2±1.392.5±0.332.0±1.151.9±0.6
2 tokens per trajectory
minmaxtok63.8±1.450.0±3.530.8±1.893.0±0.435.4±0.854.6±0.3
TIP61.9±1.648.4±1.328.7±1.392.7±0.432.4±1.552.8±0.4
TIP+IER62.3±2.146.7±1.829.1±2.992.7±0.231.6±1.152.5±0.9
0.1% tokens per trajectory
randmask 0.1%59.1±3.744.7±4.828.8±1.691.5±1.030.7±2.351.0±2.3
pctltail 0.05% per tail61.6±1.047.5±1.930.4±1.392.9±0.333.7±1.353.2±0.7
TIP61.9±4.050.2±1.130.7±2.292.8±0.333.8±1.353.9±1.3
TIP+IER61.6±1.745.6±2.928.5±1.492.9±0.332.8±1.352.3±1.0

No method dominates every budget: maxtok has the highest mean at one token (54.4), minmaxtok at two tokens (54.6), and TIP at 0.1% total supervision (53.9). Thus, naive and theoretically motivated strategies occupy the same performance range, and the extremely sparse supervision phenomenon is not tied to one selection heuristic. We respectfully reiterate that our goal is to establish this interesting and unexpected phenomenon, rather than to propose a token-selection rule that consistently outperforms plain OPD. This scope is now explicitly stated in the Remark in Section 4, which we moved from the appendix to the main text in the revision. Across the broader teacher–student configurations, the minmaxtok and pctltail are the most consistently effective, while mintok and maxtok vary by regime.

All Table 5 results use five training seeds; standard deviations of the five-benchmark mean range from 0.3 to 2.3 points. Due to rebuttal-time compute constraints, this matched comparison is limited to Family 9; the additional Qwen2.5 experiment also uses five seeds, while the other Qwen3 configurations and Llama use one. The breadth across settings and repeatability in the multi-seed studies support robustness in the settings tested. We will add multi-seed validation for additional configurations in the final version.

Q3: Efficiency measurements

Response: The main experiments used mask-only sparse OPD, not gather-then-project. The original submission noted in an appendix Remark that sparse OPD retains the full rollout cost; with the additional page allowance, we moved this limitation to the end of Section 4. We also added a controlled Family 9 ablation measuring wall-clock time, peak memory, per-token step and generation time, and micro-batch scaling with and without gather-then-project (Tables 30–31).

Table 30 reports the evaluation scores from the corresponding efficiency runs, whereas Table 5 reports five-seed means.

MethodImplementationMean avg@8 (%)Wall-ClockPeak Memory (GB)Step Time (ms/1k tok)Gen Time (ms/1k tok)
plain OPDDense52.66h37m57.0293174
minmaxtokMask only54.66h43m57.1292174
minmaxtokGather-then-project54.76h34m43.7283168
pctltailMask only52.46h34m57.0286168
pctltailGather-then-project53.56h38m43.8289173

Table 31 — Micro-batch scaling (actor-update memory peak):

Micro-batchDense OPD Peak Mem (GB)Dense OPD Update Time (ms)Sparse OPD Peak Mem (GB)Sparse OPD Update Time (ms)
257.08543.785
4OOM—52.385
8OOM—62.884

Gather-then-project reduces about 23.4% memory (57.1 → 43.7 GB) compared to plain OPD. Mean avg@8 remains 54.6–54.7. It does not materially reduce step time, generation time, throughput, or total wall-clock time because sparse OPD still requires full trajectory generation and transformer forward/backward passes. The near-identical normalized times imply near-identical throughput. The benefit is therefore memory capacity rather than raw speed: dense OPD runs out of memory at micro-batch sizes 4 and 8, whereas gather-then-project sparse OPD supports both. We have revised the paper to state this scope explicitly.

In summary, the revision provides the requested active-token normalization control, budget-matched TIP/TIP+IER comparisons across five seeds, and direct measurements of memory and runtime costs. The results support training stability in Family 9 and a 13.3 GB memory reduction, while showing no material wall-clock speedup.

We hope these additions address the reviewer's concerns. We would be happy to answer any remaining questions.

Dear Reviewer 9WWT,

The authors have provided a detailed reply to your review. Could you please take a look, acknowledge the rebuttal, and give your response?

Thank you for serving as a reviewer for ICLR.

Best,

AC

Reviewer_ALdHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether dense token-level supervision is necessary for improving reasoning capabilities through on-policy distillation (OPD). The authors introduce several sparse OPD variants that supervise only one or two tokens per trajectory, selected randomly or according to extreme teacher–student supervision signals. Experiments on mathematical reasoning benchmarks show that extremely sparse supervision can consistently improve student models, with selected variants matching dense OPD. The authors further identify a non-monotonic relationship between supervision density and performance, differences in the effectiveness of positive and negative supervision signals across model configurations, and cases where reasoning improvements occur despite increased reverse KL divergence. The paper highlights potential opportunities for more memory-efficient training.

Critical strengths

1. Clarity and simplicity. The paper is clearly written and easy to follow. The proposed sparse supervision strategies are simple and intuitive, making them easy to understand and implement.

2. Empirical finding. The paper demonstrates that supervising only one or two tokens per trajectory can match or outperform dense OPD across multiple Qwen3 teacher–student configurations. This offers a promising direction for reducing training memory overhead.

Critical weaknesses

1. Limited methodological novelty. The approach relies on simple token masking and selection based on extreme teacher–student supervision signals. Given prior work on selective token supervision in OPD, the conceptual advance over existing methods is not sufficiently clear.

2. Limited generalizability. The strong results are primarily demonstrated on Qwen3 models. Validation on Llama is restricted to a single teacher–student configuration, where only two sparse variants approximately match dense OPD. This raises concerns about whether the findings generalize robustly across model families and training settings.

Questions for authors

1. Comparison with prior work. How does the proposed method compare with existing selective-token OPD methods (TIP [1] and IER [2])? Can the authors provide budget-matched comparisons at low token-retention rates to establish the advantages of selecting tokens based on extreme supervision signals?

2. Robustness of the results. How sensitive are the reported gains to random seeds and evaluation uncertainty, particularly given the limited number of benchmark problems?

3. Cross-model generalizability. Why do most sparse variants fail to improve the base student on Llama, unlike the stronger results on Qwen3? Can the authors identify conditions under which extreme sparse supervision reliably matches or outperforms dense OPD?

[1] TIP: Token Importance in On-Policy Distillation

[2] 1% of Tokens Can Be Enough: On Gradient Estimation in On-Policy Distillation

5 comments

We thank the reviewer for their valuable time and constructive feedback. We hope the responses below address all questions and concerns.

Q1: On novelty and comparison with other token-selection methods.

Response: Thanks for the question on novelty. We do not present our variants as new token-selection algorithms. To the best of our knowledge, however, this is the first work to establish the extremely sparse supervision phenomenon and investigate it systematically across model families, tasks, teacher–student regimes, and training algorithms. The central result is counterintuitive: supervising only one or two tokens per trajectory (roughly 0.01–0.1%) can retain or exceed the reasoning gains of dense OPD. Thus, our focus is not on token selection itself, but on whether plain-OPD-level performance can be achieved with only a minimal number of learning signals per trajectory. We use the different selection rules as controlled probes rather than proposed algorithms. rand1tok tests whether an arbitrary token can provide a useful update; mintok and maxtok isolate extreme negative and positive signals; minmaxtok combines both extremes; and the 0.1% masks probe an intermediate density. We moved this positioning from an appendix Remark in the original submission to a Remark in Section 4 of the revision.

That said, it is informative to compare our naive token-selection strategies with the theoretically motivated strategies from prior and concurrent work. We therefore conduct a budget-matched comparison in revised Table 5: TIP (Xu et al., 2026) predates our work, whereas TIP+IER (Sheng et al., 2026) appeared concurrently. We evaluate all methods under identical 1-token, 2-token, and 0.1%-per-trajectory budgets across five independent training seeds. Results report avg@8 scores (mean ± standard deviation):

MethodAIME24AIME25HMMT25MATH500BeyondAIMEMean
plain OPD63.7±1.845.8±2.328.1±2.293.1±0.232.1±0.352.6±0.8
1 token per trajectory
rand1tok47.0±4.935.0±3.020.4±2.689.7±0.724.2±2.643.3±2.3
mintok57.2±1.845.9±3.826.6±1.892.3±0.232.7±1.250.9±1.4
maxtok63.1±1.551.6±1.029.6±2.293.0±0.234.6±0.854.4±0.6
TIP62.7±2.448.9±3.229.7±1.592.9±0.432.9±1.253.4±1.4
TIP+IER60.2±2.145.7±1.529.2±1.392.5±0.332.0±1.151.9±0.6
2 tokens per trajectory
minmaxtok63.8±1.450.0±3.530.8±1.893.0±0.435.4±0.854.6±0.3
TIP61.9±1.648.4±1.328.7±1.392.7±0.432.4±1.552.8±0.4
TIP+IER62.3±2.146.7±1.829.1±2.992.7±0.231.6±1.152.5±0.9
0.1% tokens per trajectory
randmask 0.1%59.1±3.744.7±4.828.8±1.691.5±1.030.7±2.351.0±2.3
pctltail 0.05% per tail61.6±1.047.5±1.930.4±1.392.9±0.333.7±1.353.2±0.7
TIP61.9±4.050.2±1.130.7±2.292.8±0.333.8±1.353.9±1.3
TIP+IER61.6±1.745.6±2.928.5±1.492.9±0.332.8±1.352.3±1.0

No method has the highest mean universally: maxtok is highest at one token, minmaxtok at two tokens, and TIP at 0.1% total supervision in Family 9. The naive and theoretically motivated strategies therefore occupy the same performance range in this experiment, and the phenomenon is not tied to one selection heuristic. Our contribution is therefore not a claim that a particular sparse rule universally outperforms dense OPD. It is the finding that extremely sparse supervision remains effective across diverse settings and can be realized through multiple distinct selection strategies rather than depending on one specific heuristic.

Q2: On cross-model generalizability and conditions for extremely sparse supervision phenomenon to emerge.

Response: We add experiments with a Qwen2.5-1.5B-Instruct student and Qwen2.5-14B-Instruct teacher. Because the 1.5B student is weaker than the Qwen3 students used in our primary experiments, we evaluate it on MATH500, AMC23, Minerva, and OlympiadBench. Qwen2.5 results across five training seeds (Table 7, avg@8, mean ± standard deviation, %):

MethodMATH500AMC23MinervaOlympiadBenchMean
Teacher78.458.445.444.656.7
Student51.122.518.018.827.6
plain OPD55.4±0.630.4±2.621.7±0.523.1±0.332.7±0.6
1 token per trajectory
rand1tok51.9±0.725.5±2.319.6±0.620.0±0.629.3±0.9
mintok54.3±0.530.4±1.621.9±0.422.0±0.332.1±0.5
maxtok37.0±0.615.6±1.013.5±0.510.8±0.219.2±0.5
2 tokens per trajectory
minmaxtok54.0±0.729.6±0.720.8±0.522.0±0.331.6±0.3
0.1% tokens per trajectory
randmask 0.1%52.0±0.727.5±1.620.0±0.620.1±0.429.9±0.3
pctltail 0.05% per tail53.7±0.828.7±2.320.8±0.721.4±0.531.2±0.7

On Qwen2.5, mintok (32.1 ± 0.5) and minmaxtok (31.6 ± 0.3) come within 0.6 and 1.1 points, respectively, of dense OPD (32.7 ± 0.6). On Llama (Figure 22), mintok and pctltail 0.05% per tail generally match dense OPD, the other sparse rules do not improve the base student. As we discussed in Appendix B, for student with insufficient representation capability, which is the setting of Llama experiment, mintok works better and maxtok could be harmful. This partially explains why only sparse variants with sufficient negative signal succeed, though an clear mechanism is still unknown. We therefore do not claim that every selection rule transfers across families. Across the tested configurations in which dense OPD improves the student, we observe at least one extremely sparse variant achieves performances on par with plain OPD.

To the best of our knowledge, Qwen3, Qwen2.5, and Llama are the families for which prior works have reported successful sampled-token OPD. Our experiments now cover all three, and we observe the extremely sparse supervision phenomenon in each. We also tested Gemma 3, Gemma 4, and Mistral 3. Under our configurations, plain OPD failed to improve students. Taken together, these results motivate the conjecture that successful dense OPD is a precondition for observing the phenomenon, which means that we can find a extremely sparse variant that matches plain OPD. The effectiveness of negative or positive tokens alone varies across configurations, depending on the student representation capability. Empirically, mintok works better when the student is less representative and maxtok works better in other settings. In general, the combination, minmaxtok or pctltail, is relatively robust. We leave a systematic investigation of sparser or more targeted supervision scheme for future study.

3. Sensitivity to random seeds and limited number of benchmark problems.

Response: To reduce evaluation uncertainty, we added two benchmarks: MATH500 (500 prompts) and BeyondAIME (100 prompts). All main Qwen3 result tables now report scores across five benchmarks: AIME 2024, AIME 2025, HMMT Feb 2025, MATH500, and BeyondAIME, and the conclusion remain the same. Since we are not proposing a method that significantly outperforms plain OPD, a matching result with small gap sufficies to support the main claim of this work.

Table 5 reports five independent training seeds (0–4) for all methods. Across the table, the standard deviations of the five-benchmark mean range from 0.3 to 2.3 points. Maxtok (54.4 ± 0.6) and minmaxtok (54.6 ± 0.3) have higher sample means than plain OPD (52.6 ± 0.8). Rand1tok has a mean of 43.3 ± 2.3, 12.5 points above the base student's 30.8. These results show that the headline gains persist across independent runs. Due to rebuttal-time compute constraints, five-seed matched comparisons and normalization controls are limited to Family 9; Qwen2.5 also uses five seeds, while the other Qwen3 configurations and Llama use one. Across all tested settings, at least one sparse variant matches or outperforms dense OPD, and the multi-seed studies show low variation. This complementary evidence supports robustness to seed-randomness; we will add multi-seed validation for additional configurations in the final version.

In summary, the revision clarifies that the contribution is the phenomenon rather than a new masking rule, adds budget-matched TIP/TIP+IER comparisons, extends validation to Qwen2.5, adds 600 evaluation prompts, and reports five-seed stability results for the representative Family 9 configuration.

We hope these additions address the reviewer's concerns. We would be happy to answer any remaining questions.

Dear Reviewer ALdH,

The authors have provided a detailed reply to your review. Could you please take a look, acknowledge the rebuttal, and give your response?

Thank you for serving as a reviewer for ICLR.

Best,

AC

Comments on the paper

3 comments

Summary of Revisions and Their Implications

We thank all reviewers for their time and constructive feedback. The revised paper addresses the reviewers' main empirical concerns through the additions below and narrows the mechanistic claims where the evidence remains incomplete. All table and figure references refer to the revised manuscript uploaded to OpenReview. We have marked all revised content in color teal.

  1. Multi-seed evaluation. For Family 9 (Qwen3-8B student, Qwen3-4B-Instruct teacher), we run five independent seeds for plain OPD, all six naive sparse variants, and TIP/TIP+IER at matched 1-token, 2-token, and 0.1% budgets. Results are reported in Table 5.

  2. Budget-matched comparison with prior and concurrent methods. We compare our naive selection rules with TIP (Xu et al., 2026) and TIP+IER (Sheng et al., 2026), under identical supervision budgets. Results are reported in Table 5.

  3. Loss-normalization ablation. We compare normalization by all response tokens in a batch (N_all) with normalization by activated tokens (N_kept) for minmaxtok across five seeds. Results are reported in Table 32.

  4. Measured computational costs. We report peak memory, wall-clock time, step time, and generation time for dense OPD and sparse OPD with and without gather-then-project. We also evaluate micro-batch scaling. Results are reported in Tables 30 and 31.

  5. Broader model-family validation. We add five-seeded experiments with a Qwen2.5-1.5B-Instruct student and Qwen2.5-14B-Instruct teacher. Results are reported in Table 7.

  6. Broader benchmark coverage. We add MATH500 (500 prompts) and BeyondAIME (100 prompts) to all main Qwen3 result tables, expanding that evaluation from three to five benchmarks.

  7. Longer thinking-mode trajectories. We test Qwen3 thinking-mode models whose trajectories are 2–4× longer, further reducing the supervised-token fraction. Tables 17–18 show that sparse supervision remains effective when approximately 0.01% or less of the response tokens are supervised.

To the best of our knowledge, this is the first work to establish the extremely sparse supervision phenomenon and investigate it systematically across model families, tasks, teacher–student regimes, and training algorithms. The revision strengthens that evidence through matched comparisons with prior and concurrent methods, broader model-family and benchmark coverage, multi-seed evaluation, normalization controls, and direct measurements of memory savings. Together, these results support the central claim that dense token-level coverage is unnecessary for obtaining substantial reasoning gains in the settings studied. They also reveal empirical patterns that motivate future work on the still-open underlying mechanism. Consistent with ICLR's general theme, we believe that improving reasoning by supervising only one or two tokens per trajectory is itself a novel and consequential result.

We thank the reviewer for the further engagement in discussion.

About the first comment, sorry we did not accurately understand your point. We have deleted the sentence "Random token selection provides an approximately unbiased estimator (up to scale) of the dense OPD objective" is based on the masking objective in (3.1). " as it could be misleading and this is not our focus of this work. As for the trajectory re-weighting explanation for random one token selection, we have to leave it for future investigation. Moreover, Equation (3.1) describes the mathematical objective for an individual trajectory, where our focus is the masking and we follow the notation convention used in the GRPO paper. Batch-level aggregation is an implementation choice and is therefore documented separately in Appendix E.2.

Moreover, we are confused about several points made in the new comment.

On comment 2, could you please elaborate why the new evidence contradicts the paper's narrative? In particular:

(1) The reviewer points out that maxtok significantly outperforms both plain OPD and the teacher, yet after training its reverse KL to the teacher is 1.116, higher than the base student (0.268) and plain OPD, this is explicitly claimed in line 112-119, and is one of the main findings of the paper that the student can be improved not by mimicking the teacher, which is demonstrated by some extremely sparse OPD variants.

(2) The claim Only training on the real error tokens identified by a frontier model, the student will collapse within a few steps. also does not conflict with the analogy in abstract: "one reflects on a few critical reasoning steps", since we use the word "critical" instead of "wrong", this corresponds to only a few tokens with extreme positive or negative signal contribute to the learning signal for parameter update. "critical" does not mean "incorrect" or "error".

(3)The claim "further validated on … Llama models" still holds: though many sparse OPD variants fail, but we do observe the extremely sparse supervision phenomenon in Llama experiments achieved by mintok.

For comment 3: On Qwen2.5 experiments, plain OPD achieves mean avg@8 of 32.7±0.6, and the mintok achieves mean avg@8 of 32.1±0.5. We cannot say plain OPD significantly outperforms mintok, and thus they are on par. This supports that the extremely sparse supervision phenomenon holds for Qwen2.5. In the Nemotron experiment, though 1% budget is larger than 0.05%, it is significantly lower than the 10%-50% in existing literature. Thus 1% still falls into the extremely sparse regime and validate the phenomenon. The 0.05% of tokens we claim in abstract is explicitly restricted to the Qwen3 family (line 15-19, "Using the Qwen3 family, we discover ..."). Also the Nemotron experiments use the thinking mode by default, different from other experiment settings. We clearly explained these in the Appendix and leave a more detailed investigation of extremely sparse supervision under thinking mode for future study.

Finally, we would like to respectfully noted that we have used caution for choosing words in articulating our contribution: we are not proposing a single token-selection method that always outperforms plain OPD (this is also explicitly stated in the Remark in the end of Section 4), but we are the first to systematically establish the extremely sparse supervision phenomenon. The six token-selection strategies serve as controlled probes for testing the existence of the extremely sparse supervision phenomenon and whether it depends on a particular selection rule. This phenomenon conveys the idea: it is possible that signal accuracy itself suffices to improve the reasoning ability, while dense token-level signal is largely unnecessary. We believe this unexpected observation is independently valuable and can motivate a new research direction on how sparse, accurate feedback induces reasoning improvements.

Thank you for your response. Below are my thoughts regarding your points.

(1) Reverse KL. I do not dispute that lines 112–116 report an increase in reverse KL. My point is that the paper does more than just report a phenomenon: the introduction states that the goal is to understand the mechanism of OPD (line 91), and Appendix B provides an explanation (lines 820–845 and from line 854 onwards). However, these explanations do not align with the paper's own data:

  • Appendix B invokes the mode-seeking property of reverse KL to explain the role of extreme-signal tokens, suggesting that in settings like Family 9 the student is better able to approximate the teacher distribution. Yet in Family 9, the reverse KL after maxtok training is 1.116—roughly 4 times that of the base student (0.268) and 6 times that of plain OPD (0.184) (Table 3)—indicating that the student is actually further from the teacher distribution.

  • Appendix B posits that tokens with moderate supervision signal could largely be noise. However, randomly selected tokens are highly unlikely to be extreme-signal tokens; nevertheless, rand1tok recovers more than half of the gain that dense OPD achieves over the base student, and randmask 0.1% performs only 1.6 points below dense OPD (Tables 3 and 5).

Even if the paper's contribution were limited to establishing the phenomenon, a control that excludes teacher information is essential. If applying a signal of the same magnitude at the same positions—but without any teacher-derived information—yields a similar performance boost, then the phenomenon should be characterized as "sparse perturbations can improve reasoning scores" rather than "sparse supervision incentivizes reasoning ability." The hypothesis in your response that "signal accuracy itself suffices" likewise requires such a control for verification.

(2) "Critical" or "Error." In the response, "critical" is interpreted as extreme-signal tokens rather than as errors, but this is inconsistent with the paper itself:

  • In Appendix B (starting at line 854), the motivation for sparse supervision is illustrated by targeted interventions such as "correcting one particularly important error or reinforcing one useful behavior." Yet in the experiment mentioned in the response—where training was conducted solely on genuine error tokens identified by a frontier model—the student collapsed within just a few steps.

  • The paper's own audit (Appendix B, Tables 29–30; minmaxtok, first 20 training steps) reveals that tokens with extremely negative supervision signal are predominantly "correct-but-teacher-dispreferred," with genuine mathematical errors accounting for only a small fraction. The tables even include whitespace and markdown separators (described in Table 29 as "organizes layout, no meaning"). It is difficult to characterize such tokens as "critical reasoning steps."

  • The Remark at the end of Section 4 (line 497) states that the paper does not argue that the learning signal necessarily originates from any particular type of token; however, the analogy in the abstract suggests that the observed gains stem from "a few critical reasoning steps."

If "critical" simply means "selected by the extreme-signal rule," then this analogy offers no additional explanatory value.

(3) Llama. Appendix C (lines 910–911) shows that only mintok and pctltail 0.05% perform roughly on par with plain OPD. All other variants, including rand1tok, which serves as the starting point of Section 4, fail to improve upon the base student. Furthermore, in Figure 22, maxtok performs notably worse than the base student. The paper attributes the varying effectiveness of these variants to the student's expressiveness (lines 108–111). Does this indicate that the phenomenon is contingent upon a match between the token-selection rule and the experimental setting, rather than holding broadly?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.