- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Negative Self-Distillation, a label-free method to improve LLM reasoning. The authors mainly argue that conventional self-distillation can suppress exploration and self-correction by encouraging models to imitate overly confident teachers. On their side of approach, they constructs a negative teacher using self-generated instructions and trains the student to avoid its predictions. Experiments on three Qwen3 models across seven mathematical reasoning benchmarks show improvements over several baselines, together with analysis and ablation that supports it.
Critical strengths
I find the idea of learning from negative signals without ground-truth answers relatively interesting. The method is simple and uses the same model with different input conditions to obtain token-level training signals without requiring an external teacher (which is a plus for efficiency). The experimental results are also encouraging, where NSD achieves consistent improvements across several model sizes and mathematical reasoning benchmarks. I also appreciate the comparison of different negative conditioning strategies and the efficiency analysis, which provide a better understanding of how the method performs in practice.
Critical weaknesses
-
My main concern towards the paper is that the motivation does not clearly lead to the proposed method. The authors argue that OPSD suppresses exploration and self-correction by encouraging models to imitate overly confident teachers. However, Table 2 mainly measures reflection-token frequency, which actually does not tell us whether the model actually explores different solutions or successfully corrects its mistakes. (See Q1 below to address the concern)
-
Even if we accept this explanation of OPSD's limitations, it is unclear why moving away from a negatively conditioned teacher would address these issues. Equation (2) penalizes tokens whose probabilities increase under negative conditioning, but the paper does not establish that these tokens correspond to reasoning errors or that suppressing them helps preserve exploration and self-correction. Clarifying this together with W1 would be helpful.
-
While I find the results in Figure 4 particularly interesting since adding irrelevant Wikipedia content achieves improvements comparable to the default negative conditioning strategy, this further raise the question of whether NSD works specifically by avoiding flawed reasoning or through a more general effect of changing the teacher distribution. See Q2 below as a way to address the concern
-
While the accuracy improvements are encouraging, I think the paper needs to better justify how the proposed training objective addresses the particular failure mechanism of OPSD described in the motivation.
Questions for authors
-
Based on W1, can the authors show whether OPSD actually reduces reasoning diversity or the ability to identify and correct intermediate errors? It would be particularly useful to compare the base model, OPSD, and NSD on the same problems and examine whether these behavioral changes help explain their performance differences.
-
Based on W3, this makes me wonder whether explicitly inducing flawed reasoning is necessary for NSD to work. Does the gating mechanism actually assign higher weights to incorrect reasoning steps? An analysis of the selected tokens against annotated reasoning errors, together with a controlled comparison using neutral or random conditioning, would help distinguish learning from reasoning flaws from learning from general distribution shifts.
-
Section 5.2 studies the effect of removing KL regularization, but I would also like to understand the individual contributions of the gating mechanism and bounded unlikelihood objective. How does downstream accuracy change when the gate is removed or when bounded unlikelihood is replaced with standard unlikelihood, while keeping the other components fixed? This would help establish which parts of the proposed objective are responsible for the improvements.
-
Could the authors clarify the derivation of Equation (4)? The paper describes the reference-weighted term as a single-sample importance-weighted estimator of forward KL divergence, but the tokens are sampled from the student distribution. As written, the weighting does not appear to yield an unbiased estimator of the stated KL divergence. Is the term intended as an approximation or a different regularization objective?
7 comments
Response to Reviewer jBsG [Part 2/5]
Q2 (1): Does the gating mechanism actually assign higher weights to incorrect reasoning steps?
To prove the effectiveness and explainability of our gating mechanism, we conduct an analysis experiment and illustrate that these tokens are highly relevant to the reasoning errors.
We define error tokens during the student rollout stage as the tokens in a sentence where a reasoning error really happens. We annotate these tokens by Kimi-K3.1-Pro, on all samples in the training dataset. After finishing, we calculate the summary value of adaptive gating on these tokens. We define the enrichment factor of gating as: Where the represents all tokens, and represents the error tokens. The result is shown in the following table.
| Type | E(S) |
|---|---|
| Random sentences | 0.9374 |
| Reasoning errors | 1.4385 |
The result shows that gating function tends to allocate more value on real erroneous reasoning tokens, which means the student learns from meaningful distribution shifts. To illustrate the result more visually, we show the density distribution of the gating value of the random and error tokens in Figure 9, Appendix H.
Furthermore, if we drop the gating function, the training process will collapse, and the entropy will increase heavily because of indiscriminatively suppressing all tokens’ probability. The training dynamic is shown in Figure 10.
We also evaluate a random gating function that randomly selects 44% of all tokens as activated tokens in every sample, matching the average gate activation rate in NSD, and assigns them a gate value of 1. This variant yields only marginal improvements on the evaluation benchmarks. For a fairer comparison, we also evaluate a variant that randomly assigns gate values between 0 and the 90th percentile of the gate values in the original NSD. This variant likewise fails to yield meaningful improvements, because these settings make LLMs learn meaningless distribution shifts. The detailed experiment result is shown in the following table:
| Dataset | Qwen3-4B | NSD | Random 44% gate = 1 | gate ∈ [0 , P90 max gate] |
|---|---|---|---|---|
| AIME 2024 | 23.80 | 35.80 | 27.08 | 29.17 |
| AIME 2025 | 20.40 | 31.30 | 21.67 | 19.58 |
| AIME 2026 | 17.90 | 29.20 | 17.92 | 22.92 |
| HMMT-Feb 2025 | 10.80 | 16.30 | 12.50 | 10.83 |
| AMC 2023 | 68.80 | 76.30 | 66.56 | 67.81 |
| MATH-500 | 71.20 | 73.10 | 71.88 | 71.80 |
| OlympiadBench | 47.80 | 51.00 | 48.98 | 48.85 |
| Avg. | 37.24 | 44.71 | 38.08 | 38.71 |
Response to Reviewer jBsG [Part 3/5]
Q2 (2) Whether explicitly inducing flawed reasoning is necessary for NSD to work?
We have almost resolved this confusion in our answer of Q2 (1). For NSD to improve performance, the negative condition should induce meaningful shifts in the teacher model’s output distribution (e.g., the default negative condition strategy can induce the exact error tailored for the model); otherwise, the gains are limited (e.g., random gating yields no improvement). However, negative conditions need not be carefully crafted: even long, irrelevant Wikipedia passages can yield considerable improvements. Such noise serves a meaningful purpose by simulating conditions that induce errors during long-context reasoning, thereby improving the model’s robustness over long CoTs (it’s not a casual setting). So we explored Wikipedia noise to show that even simple methods can produce meaningful negative conditions. However, since this raised a concern for you, we conducted the following additional experiment:
We further designed an experiment to compare the effectiveness of a random condition, which is a style instruction (e.g., “Please be cute”).
| Model | AIME 2024 | AIME 2025 | AIME 2026 | HMMT-Feb 2025 | AMC 2023 | MATH-500 | OlympiadBench | Avg |
|---|---|---|---|---|---|---|---|---|
| 1.7B | 9.60 | 10.00 | 9.60 | 7.10 | 44.10 | 62.50 | 37.10 | 25.71 |
| NSD | 14.20 | 17.90 | 10.00 | 7.10 | 45.90 | 62.60 | 38.70 | 28.06 |
| Random condition | 12.08 | 10.42 | 10.83 | 5.42 | 47.19 | 62.25 | 38.13 | 26.62 |
The results show that the meaningless condition results in limited improvement. But high quality conditions lead to better results.
Response to Reviewer jBsG [Part 1/5]
We thank you for your very insightful questions on behavioral changes of NSD-ed and OPSD-ed models, the soundness of adaptive gating function, and some confusion about Eq.(4). We also thank you for considering that our overall idea is relatively interesting. We further answer your questions in detail:
In our paper, we show the reflection token frequency of OPSD in Section 4.2. Furthermore, we provide a detailed case study in Table 9, which shows that the base model prefers a symmetric-guess method (which is wrong). However, in the OPSD-ed model’s answer, there’s no self-correction but over-confident behavior that treats this minimum value as the answer with no attempt. In contrast, the NSD-ed model explicitly abandons the approach preferred by the base model and reframes the problem.
Nevertheless, we provide an additional experiment beyond single case study to solve your concerns: We analyze all the answers in AIME 2024, AIME 2025, and HMMT 2025 by prompting Kimi-K3 (96.67% acc on AIME 26) to judge the effective/ineffective reflection times in every reasoning trajectory. Kimi-K3 is also given the gold solution for reference. The result is shown in this table:
The results show that NSD achieves the highest Eff./Ineff. ratio, indicating the highest reflection efficiency. In contrast, OPSD shows only a small increase in effective reflections over the base model, while its total number of reflections is even lower. This suggests that OPSD primarily replaces incorrect reflections with correct ones guided by privileged oracle information, rather than strengthening the model’s intrinsic ability to reflect. Some recent work (https://openreview.net/forum?id=Az7guis46K, https://openreview.net/forum?id=NmHpi2SCtC) also supports that OPSD brings overconfidence because of the gold solution hint. These evidence together support our motivation.