acceptodds
Under review as a conference paper at ICLR 2027

Negative Self-Distillation: Learning to Reason by Avoiding Flaws

Abstract

On-Policy Self-Distillation (OPSD) has emerged as a popular paradigm for large language model (LLM) self-improvement, allowing models to act as their own teachers by leveraging privileged information such as ground-truth solutions. However, recent findings indicate that OPSD can severely degrade the performance of LLMs on complex reasoning tasks: By forcing the student to imitate an artificially confident reasoning trace conditioned on privileged information, OPSD inadvertently suppresses expressions of uncertainty and penalizes the exploratory, self-corrective behaviors required to solve challenging problems. To address this, we introduce Negative Self-Distillation (NSD), a new framework that optimizes LLMs by diverging from flawed reasoning rather than imitating privileged solutions. Instead of relying on ground-truth answers or external supervision, NSD uses the model itself to generate a question-specific negative condition (e.g., acting as a ”careless reasoner”) and pushes the student's distribution away from this self-generated negative teacher. Naively applying unlearning objectives to achieve this divergence is problematic, as flawed reasoning tokens are confounded with basic linguistic tokens; indiscriminately penalizing both risks catastrophically degrading the model's foundational language capabilities. We resolve this by designing a dynamic gating mechanism that automatically identifies and isolates reasoning-critical tokens, ensuring gradient updates target only behavioral flaws while preserving the model's linguistic priors. Empirically, NSD consistently outperforms OPSD and other label-free, self-bootstrapping reinforcement learning (RL) baselines. Across seven mathematical reasoning benchmarks (AIME 24/25/26, HMMT, AMC, OlympiadBench, and MATH), NSD achieves average gains of 2.3%, 7.5%, and 6.0% for 1.7B, 4B, and 8B models, respectively. Further analyses show that NSD achieves higher training efficiency while preserving the self-correction behaviors crucial for complex reasoning. Code is available at https://anonymous.4open.science/r/NSD.

Primary area:
foundation or frontier models, including LLMs
Keywords:
self-distillation, unsupervised post-training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jBsGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Negative Self-Distillation, a label-free method to improve LLM reasoning. The authors mainly argue that conventional self-distillation can suppress exploration and self-correction by encouraging models to imitate overly confident teachers. On their side of approach, they constructs a negative teacher using self-generated instructions and trains the student to avoid its predictions. Experiments on three Qwen3 models across seven mathematical reasoning benchmarks show improvements over several baselines, together with analysis and ablation that supports it.

Critical strengths

I find the idea of learning from negative signals without ground-truth answers relatively interesting. The method is simple and uses the same model with different input conditions to obtain token-level training signals without requiring an external teacher (which is a plus for efficiency). The experimental results are also encouraging, where NSD achieves consistent improvements across several model sizes and mathematical reasoning benchmarks. I also appreciate the comparison of different negative conditioning strategies and the efficiency analysis, which provide a better understanding of how the method performs in practice.

Critical weaknesses
  1. My main concern towards the paper is that the motivation does not clearly lead to the proposed method. The authors argue that OPSD suppresses exploration and self-correction by encouraging models to imitate overly confident teachers. However, Table 2 mainly measures reflection-token frequency, which actually does not tell us whether the model actually explores different solutions or successfully corrects its mistakes. (See Q1 below to address the concern)

  2. Even if we accept this explanation of OPSD's limitations, it is unclear why moving away from a negatively conditioned teacher would address these issues. Equation (2) penalizes tokens whose probabilities increase under negative conditioning, but the paper does not establish that these tokens correspond to reasoning errors or that suppressing them helps preserve exploration and self-correction. Clarifying this together with W1 would be helpful.

  3. While I find the results in Figure 4 particularly interesting since adding irrelevant Wikipedia content achieves improvements comparable to the default negative conditioning strategy, this further raise the question of whether NSD works specifically by avoiding flawed reasoning or through a more general effect of changing the teacher distribution. See Q2 below as a way to address the concern

  4. While the accuracy improvements are encouraging, I think the paper needs to better justify how the proposed training objective addresses the particular failure mechanism of OPSD described in the motivation.

Questions for authors
  1. Based on W1, can the authors show whether OPSD actually reduces reasoning diversity or the ability to identify and correct intermediate errors? It would be particularly useful to compare the base model, OPSD, and NSD on the same problems and examine whether these behavioral changes help explain their performance differences.

  2. Based on W3, this makes me wonder whether explicitly inducing flawed reasoning is necessary for NSD to work. Does the gating mechanism actually assign higher weights to incorrect reasoning steps? An analysis of the selected tokens against annotated reasoning errors, together with a controlled comparison using neutral or random conditioning, would help distinguish learning from reasoning flaws from learning from general distribution shifts.

  3. Section 5.2 studies the effect of removing KL regularization, but I would also like to understand the individual contributions of the gating mechanism and bounded unlikelihood objective. How does downstream accuracy change when the gate is removed or when bounded unlikelihood is replaced with standard unlikelihood, while keeping the other components fixed? This would help establish which parts of the proposed objective are responsible for the improvements.

  4. Could the authors clarify the derivation of Equation (4)? The paper describes the reference-weighted term as a single-sample importance-weighted estimator of forward KL divergence, but the tokens are sampled from the student distribution. As written, the weighting does not appear to yield an unbiased estimator of the stated KL divergence. Is the term intended as an approximation or a different regularization objective?

7 comments
AuthorsOpenReview ↗

Response to Reviewer jBsG [Part 1/5]

We thank you for your very insightful questions on behavioral changes of NSD-ed and OPSD-ed models, the soundness of adaptive gating function, and some confusion about Eq.(4). We also thank you for considering that our overall idea is relatively interesting. We further answer your questions in detail:

Q1: Does OPSD really limit the reasoning diversity? And further comparison with NSD.

In our paper, we show the reflection token frequency of OPSD in Section 4.2. Furthermore, we provide a detailed case study in Table 9, which shows that the base model prefers a symmetric-guess method (which is wrong). However, in the OPSD-ed model’s answer, there’s no self-correction but over-confident behavior that treats this minimum value as the answer with no attempt. In contrast, the NSD-ed model explicitly abandons the approach preferred by the base model and reframes the problem.

Nevertheless, we provide an additional experiment beyond single case study to solve your concerns: We analyze all the answers in AIME 2024, AIME 2025, and HMMT 2025 by prompting Kimi-K3 (96.67% acc on AIME 26) to judge the effective/ineffective reflection times in every reasoning trajectory. Kimi-K3 is also given the gold solution for reference. The result is shown in this table:

ModelTotalEffectiveIneffectiveEff./Ineff.
Base2.3440.9331.4110.661
NSD3.3561.9221.4331.341
GRPO3.8441.8002.0440.880
Intuitor1.7000.7330.9670.759
OPSD2.2780.9561.3220.723

The results show that NSD achieves the highest Eff./Ineff. ratio, indicating the highest reflection efficiency. In contrast, OPSD shows only a small increase in effective reflections over the base model, while its total number of reflections is even lower. This suggests that OPSD primarily replaces incorrect reflections with correct ones guided by privileged oracle information, rather than strengthening the model’s intrinsic ability to reflect. Some recent work (https://openreview.net/forum?id=Az7guis46K, https://openreview.net/forum?id=NmHpi2SCtC) also supports that OPSD brings overconfidence because of the gold solution hint. These evidence together support our motivation.

AuthorsOpenReview ↗

Response to Reviewer jBsG [Part 2/5]

Q2 (1): Does the gating mechanism actually assign higher weights to incorrect reasoning steps?

To prove the effectiveness and explainability of our gating mechanism, we conduct an analysis experiment and illustrate that these tokens are highly relevant to the reasoning errors.

We define error tokens during the student rollout stage as the tokens in a sentence where a reasoning error really happens. We annotate these tokens by Kimi-K3.1-Pro, on all samples in the training dataset. After finishing, we calculate the summary value of adaptive gating on these tokens. We define the enrichment factor of gating as: E(S)=∑t∈Sgt/∑t∈Ωgt∣S∣/∣Ω∣E(S)=\frac{\displaystyle\sum_{t\in S}g_t\big/\sum_{t\in\Omega}g_t}{\displaystyle |S|/|\Omega|} Where the Ω\Omega represents all tokens, and SS represents the error tokens. The result is shown in the following table.

TypeE(S)
Random sentences0.9374
Reasoning errors1.4385

The result shows that gating function tends to allocate more value on real erroneous reasoning tokens, which means the student learns from meaningful distribution shifts. To illustrate the result more visually, we show the density distribution of the gating value of the random and error tokens in Figure 9, Appendix H.

Furthermore, if we drop the gating function, the training process will collapse, and the entropy will increase heavily because of indiscriminatively suppressing all tokens’ probability. The training dynamic is shown in Figure 10.

We also evaluate a random gating function that randomly selects 44% of all tokens as activated tokens in every sample, matching the average gate activation rate in NSD, and assigns them a gate value of 1. This variant yields only marginal improvements on the evaluation benchmarks. For a fairer comparison, we also evaluate a variant that randomly assigns gate values between 0 and the 90th percentile of the gate values in the original NSD. This variant likewise fails to yield meaningful improvements, because these settings make LLMs learn meaningless distribution shifts. The detailed experiment result is shown in the following table:

DatasetQwen3-4BNSDRandom 44% gate = 1gate ∈ [0 , P90 max gate]
AIME 202423.8035.8027.0829.17
AIME 202520.4031.3021.6719.58
AIME 202617.9029.2017.9222.92
HMMT-Feb 202510.8016.3012.5010.83
AMC 202368.8076.3066.5667.81
MATH-50071.2073.1071.8871.80
OlympiadBench47.8051.0048.9848.85
Avg.37.2444.7138.0838.71
AuthorsOpenReview ↗

Response to Reviewer jBsG [Part 3/5]

Q2 (2) Whether explicitly inducing flawed reasoning is necessary for NSD to work?

We have almost resolved this confusion in our answer of Q2 (1). For NSD to improve performance, the negative condition should induce meaningful shifts in the teacher model’s output distribution (e.g., the default negative condition strategy can induce the exact error tailored for the model); otherwise, the gains are limited (e.g., random gating yields no improvement). However, negative conditions need not be carefully crafted: even long, irrelevant Wikipedia passages can yield considerable improvements. Such noise serves a meaningful purpose by simulating conditions that induce errors during long-context reasoning, thereby improving the model’s robustness over long CoTs (it’s not a casual setting). So we explored Wikipedia noise to show that even simple methods can produce meaningful negative conditions. However, since this raised a concern for you, we conducted the following additional experiment:

We further designed an experiment to compare the effectiveness of a random condition, which is a style instruction (e.g., “Please be cute”).

ModelAIME 2024AIME 2025AIME 2026HMMT-Feb 2025AMC 2023MATH-500OlympiadBenchAvg
1.7B9.6010.009.607.1044.1062.5037.1025.71
NSD14.2017.9010.007.1045.9062.6038.7028.06
Random condition12.0810.4210.835.4247.1962.2538.1326.62

The results show that the meaningless condition results in limited improvement. But high quality conditions lead to better results.

Reviewer_uYk9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Negative Self-Distillation (NSD) for LLM reasoning without ground-truth solutions or external teachers. The student generates negative instructions from its own solutions. NSD compares a frozen model’s token probabilities with and without these instructions, using gated, bounded unlikelihood and reference-based regularization. Experiments on three Qwen3 model sizes and seven math benchmarks show gains over base models and baselines.

Critical strengths
  • The idea of using self-generated negative conditions to improve reasoning without ground-truth supervision is interesting. The token-level gating mechanism and bounded unlikelihood objective address the practical difficulty of suppressing undesirable behavior without indiscriminately penalizing ordinary linguistic tokens.
Critical weaknesses
  • Insufficient validation of the student’s ability to construct useful negative supervision. It is unclear whether the student can generate negative instructions that induce meaningful reasoning errors from which it can learn. Examining a solution and devising such errors, beyond simple mistakes or calculation errors, appears to be a challenging task in itself. However, the paper does not sufficiently analyze whether the evaluated models possess this ability. Additional validation or qualitative analysis of actual generated instructions and their effects would help substantiate this central assumption of the proposed approach.
Questions for authors
  • Why are baseline gains quite marginal, particularly for OPSD despite using ground-truth solutions?
3 comments
AuthorsOpenReview ↗

Response to Reviewer uYk9 [Part 1/2]

We thank you for your instructive question about the effectiveness of OPSD and LLM’s negative conditioning ability. And we’re delighted to hear that you think our main idea and gating function are interesting! We further answer your questions and concerns below:

Weakness: Insufficient validation of the student’s ability to construct useful negative supervision.

Thank you for raising this valuable concern. We would like to clarify that NSD places relatively modest demands on the quality of negative conditions: even irrelevant Wikipedia passages can serve as effective negative contexts. Thus, generating sophisticated negative conditions is not a strict prerequisite for NSD to work.

Nevertheless, the student’s ability extends beyond this minimal requirement. It can generate targeted negative conditions that exploit specific reasoning errors already present in its own responses, rather than merely provide generic instructions to reason poorly.

The following case study illustrates this capability: For a three-digit integer ending in 3, Qwen3-4B incorrectly computed the probability of divisibility by 3 as 310\frac{3}{10}. Writing N=100a+10b+3N=100a+10b+3, it assumed that every aa admits exactly three valid values of bb. However, when a∈3,6,9a\in\\{3,6,9\\}, four values—b∈0,3,6,9b\in\\{0,3,6,9\\}—are valid, giving the correct probability 13\frac{1}{3}.

The corresponding negative condition explicitly targets this same wrong assumption:

“You are a student who assumes that the number of valid bb values for each aa is always 3.”

Thus, the condition targets a specific error already present in the unconditioned response, rather than a generic reasoning weakness. This targeted negative signal helps the NSD-trained model break out of its habitual reasoning pattern and solve the problem correctly.

Before presenting the additional quantitative experiments, we want to emphasize the core function of NSD: The key for a negative condition to work is to induce meaningful shifts in the teacher model’s output distribution so that the gating function can identify the erroneous sentences and further punish them.

Therefore, we tested whether the current negative conditioning approach can help the model identify actual errors.

We define error tokens during the student rollout stage as the tokens in a sentence where a reasoning error really happens. We annotate these tokens by Kimi-K3.1-Pro, on all samples in the training dataset. After finishing, we calculate the summary value of adaptive gating on these tokens. We define the enrichment factor of gating as: E(S)=∑t∈Sgt/∑t∈Ωgt∣S∣/∣Ω∣E(S)=\frac{\displaystyle\sum_{t\in S}g_t\big/\sum_{t\in\Omega}g_t}{\displaystyle |S|/|\Omega|} Where the Ω\Omega represents all tokens, and SS represents the error tokens. The result is shown in the following table.

TypeE(S)
Random sentences0.9374
Reasoning errors1.4385

The experimental results show that the current negative conditions enable the model to effectively identify the reasoning errors. You may check Figure 9, Appendix H for the visual distribution!

AuthorsOpenReview ↗

Response to Reviewer uYk9 [Part 2/2]

Question: Why are baseline gains quite marginal, particularly for OPSD despite using ground-truth solutions?

Theoretically, OPSD regards the model with privileged knowledge as the teacher model. But the external hint leads the teacher model to answer directly, lacking reflection. Therefore, it further guides the student to the shortcut reasoning (Similar conclusions could be found in https://openreview.net/forum?id=Az7guis46K and https://openreview.net/forum?id=NmHpi2SCtC).

Empirically, our analysis experiment (Section 4.2) on reflection illustrates the conclusion is right. Furthermore, we provide a detailed case study in Table 9, which specifically shows in the OPSD-ed model’s answer, there’s no self-correction but over-confident behavior that treats this minimum value as the answer with no attempt. In the main experiment table, we thoroughly tuned OPSD’s hyperparameters, evaluated both LoRA and full fine-tuning as described in Appendix D.1, and reported results from the best-performing configuration.

Nevertheless, we provide an additional experiment beyond single case study to solve your concerns: We analyze all the answers in AIME 2024, AIME 2025, and HMMT 2025 by prompting Kimi-K3 (96.67% acc on AIME 26) to judge the effective/ineffective reflection times in every reasoning trajectory. Kimi-K3 is also given the gold solution for reference. The result is shown in this table:

ModelTotalEffectiveIneffectiveEff./Ineff.
Base2.3440.9331.4110.661
NSD3.3561.9221.4331.341
GRPO3.8441.8002.0440.880
Intuitor1.7000.7330.9670.759
OPSD2.2780.9561.3220.723

We can tell that OPSD shows little effective reflection improvement than the base model. Compared with GRPO, which also utilizes the gold answer, OPSD rarely improves reasoning effectiveness.

So in conclusion, though OPSD uses ground-truth, it’s the framework design issue that leads to marginal improvement.

We hope that our clarifications and additional experiments have addressed your concerns. If so, we would be grateful if you could consider revisiting your overall assessment of our work.

Dear Reviewer uYk9,

As the discussion period draws to a close, we would be grateful if you could let us know whether our responses and additional experiments have addressed your main concerns. If so, we would appreciate it if you could consider revisiting your overall assessment. Thank you again for your time and constructive feedback.

Best regards,

The Authors

Reviewer_ximHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Negative Self-Distillation (NSD), a post-training method that uses problem statements without answer labels. The student generates a reasoning trace and a negative instruction intended to induce flawed reasoning. The reference model and negative teacher are frozen copies of the initial model. They assign probabilities to the student's sampled tokens using the same reasoning prefix; only the negative teacher receives the additional instruction. The gate selects tokens whose probabilities increase under the negative instruction. A sigmoid-bounded unlikelihood penalty then reduces the student's probability of generating these tokens. The method also adds a regularization term computed from the student and reference probabilities for each sampled token.

Experiments use three Qwen3 model sizes and seven mathematical reasoning benchmarks, comparing NSD with OPSD, Intuitor, and an adaptation of TTRL that uses training data. Table 1 reports average improvements over the corresponding base models of 2.3, 7.5, and 6.0 percentage points. Additional experiments examine alternative negative instructions, evaluation in thinking mode, and a policy-gradient variant.

Critical strengths
  1. The method addresses a meaningful constraint: improving reasoning without answer labels or a stronger external teacher. Comparing token probabilities from the same frozen model with and without a negative instruction provides a simple training signal. The experiments with alternative negative conditions show that the method can also use irrelevant text that is inexpensive to obtain.

  2. The experiments cover a reasonably broad range of mathematical reasoning tasks. Table 1 includes seven benchmarks and three model sizes; the average gains for the 4B and 8B models are substantial. The pass@8 results and evaluation in thinking mode provide additional evidence beyond the main evaluation in non-thinking mode. These results make the method worth investigating, although they do not resolve the concerns below.

Critical weaknesses

Weakness 1:

Equation (4) does not have forward KL as its expectation under the stated sampling procedure. This changes the interpretation of a central component of NSD.

For a fixed token prefix, let p be the student's probability distribution and r the reference distribution. Equation (4) evaluates r(y) log[r(y)/p(y)] for a token y sampled from p. The two quantities are:

Expected value of Equation (4): sum_y p(y) r(y) log[r(y)/p(y)]. Forward KL: sum_y r(y) log[r(y)/p(y)].

The extra factor p(y) means that the expected value is not the claimed divergence. An individual sample can give a negative contribution even for a valid KL estimate; the problem here is that the expectation itself can be negative. For example, r=(0.8,0.2) and p=(0.9,0.1) give an expectation of approximately -0.07094, while the actual forward KL is +0.04440. Algorithm 1 repeats the same expression. The caption of Figure 6 describes its plotted statistic as forward KL, so the authors should specify exactly how it was computed. Equation (4) may still be useful as a regularizer. The paper should clarify which objective was implemented and assess how its actual properties affect training.

The argument for training stability also needs clarification. Appendix A gives the derivative of standard gated unlikelihood with respect to the sampled token's logit as G*p, where G is the gate value and p is that token's probability. This derivative is bounded by 1, so an unbounded loss does not by itself establish an exploding logit gradient.

The sigmoid-bounded penalty does attenuate this gradient: for a fixed gate, it becomes Gp(1-p)/(2-p)^2, is at most G/8, and tends to zero as p approaches 1. This supports reduced updates for high-confidence tokens. However, it does not establish that standard gated unlikelihood causes the claimed gradient explosion or training collapse in this setting. Bounded logit gradients also do not rule out instability in gradients with respect to model parameters; that requires separate evidence.

Weakness 2:

The experiments do not yet establish that the gate identifies erroneous reasoning steps, or that successful self-correction explains the accuracy gains. Equation (2) detects tokens whose probabilities increase under a negative context. Whether these probability increases identify erroneous tokens remains to be tested.

Table 3 reports the “style-task ratio,” which compares the average weights assigned to style and task tokens. It shows that the gate assigns less weight to style tokens relative to task tokens than the other weighting methods do. This ratio does not measure reasoning correctness: correct calculations also contain task tokens. Table 2 counts “reflection tokens” per response without controlling for response length. Longer responses can contain more such tokens without correcting more errors.

Appendix E.4 and Table 9 provide a qualitative example of successful self-correction, summarized using Sonnet. This example is useful, but does not establish how often self-correction succeeds or whether it explains the accuracy gains across the benchmarks.

Section 5 provides an analysis of token weights, training logs without the regularizer, and an analytical comparison of the penalties. It does not report task accuracy after removing the gate or replacing the sigmoid-bounded penalty with standard gated unlikelihood. Comparisons on a representative model, accounting for changes in the overall loss or gradient scale, would help establish the contribution of these components to accuracy and stability.

Questions for authors

The requested training comparisons can be limited to one representative model and setting.

  1. Which objective was used for the reported experiments? Please specify the sampling distribution and any importance weighting or stop-gradient operations in the implementation. If it matches Equation (4), how does this term compare with a correctly implemented KL regularizer in terms of training stability and task performance? What statistic does Figure 6 plot, and how is it computed? Please also clarify whether the claimed gradient instability concerns logits or model parameters, and provide evidence for that claim.

  2. How do task accuracy and generation stability change when the gate is removed or the sigmoid-bounded penalty is replaced by standard gated unlikelihood? Please account for changes in the overall loss or gradient scale, using rescaling or tuning on validation data where appropriate. Explain how these differences are controlled. On a representative sample of responses, do the selected tokens belong to erroneous reasoning steps? Does reflection lead to successful correction after accounting for response length?

  3. What validation data were used to select the checkpoints in Table 1, and how were these data kept separate from training and all reported test benchmarks? Table 1 reports confidence intervals and one-sided p-values, but how were they computed? Please specify what was sampled and whether the comparison paired results from the same test problems. Also distinguish variation across sampled answers or test problems from variation across independent training runs. This distinction is needed to assess the claim of stable gains, especially for the 1.7B model.

7 comments
AuthorsOpenReview ↗

Response to Reviewer ximH [Part 1/5]

We thank you for your constructive questions on Eq.(4), the design of the adaptive gating function and experimental settings! Meanwhile we’re happy to see you approve the meaningful topic and the reasonable range of experiments.

Below, we address your questions for us in detail:

Q1: Question on our KL objective and gradient instability.

Yes, we do use a biased estimator of KL in our training objective. But we would respectfully clarify that the main role of Eq.(4) is a regularizer rather than the main objective in NSD. Its formulation is tailored to anchoring the student to the reference model and need not coincide exactly with a conventional distillation loss.

For student-sampled tokens, the standard importance-sampled forward-KL term is:

LKL−S(t)=πref(yt∣xi,y<t)πθ(yt∣xi,y<t)log⁡πref(yt∣xi,y<t)πθ(yt∣xi,y<t)\mathcal{L}_{\mathrm{KL-S}}^{(t)}=\frac{\pi_{\mathrm{ref}}(y_t\mid x_i,y_{\lt t})}{\pi_\theta(y_t\mid x_i,y_{\lt t})}\log\frac{\pi_{\mathrm{ref}}(y_t\mid x_i,y_{\lt t})}{\pi_\theta(y_t\mid x_i,y_{\lt t})}

Our Eq. (4) is related to this term by

LKL−ours(t)=πθ(yt∣xi,y<t)⋅LKL−S(t)\mathcal{L}_{\mathrm{KL-ours}}^{(t)}=\pi_\theta(y_t\mid x_i,y_{\lt t})\cdot\mathcal{L}_{\mathrm{KL-S}}^{(t)}

This additional factor relatively downweights rare student tokens, concentrating the regularization signal on tokens likely under both models. This design helps counteract the indiscriminate suppression of linguistic tokens (which usually gain high probability under both models) during unlikelihood training.

To further support this design, we replace the KL regularization term in the paper with the standard forward KL divergence LKL-S\mathcal L_{\mathrm{KL\text{-}S}} and conduct NSD on Qwen3-4B. The experimental results are shown in this table:

MethodAIME 2024AIME 2025AIME 2026HMMT-Feb 2025AMC 2023MATH-500OlympiadBenchAvg
Qwen3-4B Baseline23.820.417.910.868.871.247.837.2
NSD35.831.329.216.376.373.151.044.7
NSD-KL32.927.524.614.671.371.547.941.5

The result shows that the performance of NSD-KL (with standard KL) is slightly lower than our default setting. This supports the empirical effectiveness of our regularizer. Overall, the variant of KL doesn’t change our main conclusion: Even when standard KL is used, NSD still outperforms the other baselines.

Regarding your remaining questions, Figure 6 shows the dynamics of the standard forward KL divergence, which is a different metric from Eq (4), in order to illustrate distribution shifts of the student policy.

Regarding your question about gradient instability, we would clarify that our gradient instability is with regard to logits rather than model parameters, as shown in Appendix A. We would also clarify our wording on “gradient explosion”. Our real intention is to illustrate that, without the sigmoid function, a considerable number of tokens have a relatively high probability and meanwhile every token in them generates a high gradient. These two factors together easily make the training process unstable. We have corrected our description in Section 2.2. Thank you for pointing it out!

The direct evidence for our claim is shown in Figure 11 (left), which illustrates gradient instability under the standard unlikelihood (UL) objective, showing large unstable initial gradient norms and occasional spikes in gradient norms during the middle stages of training. The accompanying forward-KL curve provides complementary evidence of changes in the student distribution.

AuthorsOpenReview ↗

Response to Reviewer ximH [Part 2/5]

Q2 (1): How do task accuracy and generation stability change when the gate is removed or the sigmoid-bounded penalty is replaced by standard gated unlikelihood?

Thank you for pointing out the mechanism analysis to be further proved! We first conduct an ablation study on the adaptive gating function. Our conclusion is that if we drop the gating function, the training process will collapse, and the entropy will increase heavily because of indiscriminatively suppressing all tokens’ probability. The training dynamic is shown in Figure 10.

Then we conduct an ablation study on the sigmoid function. The result is shown in the following table:

MethodAIME 2024AIME 2025AIME 2026HMMT-Feb 2025AMC 2023MATH-500OlympiadBench
NSD35.831.329.216.376.373.151.0
NSD w/o sigmoid34.622.124.615.873.868.950.7

Furthermore, we visualize the KL divergence value and gradient norm changes during training in Figure 11, Appendix H. It shows that the NSD without the sigmoid function causes violent updates and unstable training in the initial 100 steps, which is the expected behavior as we analyzed in Section 2.2, RQ 2.

AuthorsOpenReview ↗

Response to Reviewer ximH [Part 3/5]

Q2 (2): Do the selected tokens (by our gating function) belong to erroneous reasoning steps?

To prove the effectiveness and explainability of our adaptive gating function, we conduct an analysis experiment and illustrate that these tokens are highly relevant to the reasoning errors.

We define error tokens during the student rollout stage as the tokens in a sentence where a reasoning error really happens.

We annotate these tokens by Kimi-K3.1-Pro, on all samples in the training dataset. After finishing, we calculate the summary value of adaptive gating on these tokens. We define the enrichment factor of gating as: E(S)=∑t∈Sgt/∑t∈Ωgt∣S∣/∣Ω∣E(S)=\frac{\displaystyle\sum_{t\in S}g_t\big/\sum_{t\in\Omega}g_t}{\displaystyle |S|/|\Omega|} Where the Ω\Omega represents all tokens, and SS represents the error tokens. The result is shown in the following table.

TypeE(S)
Random sentences0.9374
Reasoning errors1.4385

The result shows that gating function tends to allocate more value to real erroneous reasoning tokens. To illustrate the result more visually, we show the density distribution of the gating value of the random and error tokens in Figure 9, Appendix H.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overall Response to All Reviewers

We sincerely thank all reviewers (ximH, uYk9, jBsG) for their instructive and detailed feedback! We are happy to see almost all reviewers think the label-free topic is important, our simple method is interesting and the main evaluation results are encouraging:

  • Reviewer ximH thinks our NSD method addresses a meaningful constraint, the experiments cover a reasonably broad range of mathematical reasoning tasks, and our additional experiments provide more evidence for our main conclusion.

  • Reviewer uYk9 emphasizes that the core idea of our method is interesting, and acknowledges that the key components of NSD effectively address the practical difficulty.

  • Reviewer jBsG thinks our topic is interesting, the method is simple and efficient, and the experimental results are encouraging. Our analysis experiments are also considered as providing “a better understanding of how the method performs in practice”.

Regarding your concerns about our paper, we have responded to you with our additional explanation or experiments. For your convenience, we have added a section to the appendix (Appendix H) containing the supplementary experimental figures and tables, along with their full descriptions. The added text is shown in blue. We refer to these figures in our comments to help explain the experimental results. Before the end of the paper discussion phase, we will integrate this material into the appropriate sections of the main text.

AuthorsOpenReview ↗

Summary of Responses for the Area Chair and Reviewers

We now summarize our response to reviewers as follows:

  • To Reviewer ximH: We clarified the role of Eq. (4) as a reweighted regularizer and added a comparison with standard forward KL. We strengthened the mechanism analysis through gating and sigmoid ablations, error-token enrichment analysis, and gradient and KL dynamics (Figures 9–11). We also added an LLM-as-a-judge evaluation of reflection effectiveness, clarified the validation and paired-resampling protocols, and conducted two additional Qwen3-1.7B training runs with different seeds to assess robustness.

  • To Reviewer uYk9: To address concerns about student-generated negative supervision, we provided a case study illustrating how the student constructs a negative condition targeting a specific error in its own response. Furthermore, we show quantitative evidence that the resulting gating weights emphasize error reasoning tokens (Figure 9). To investigate the limited OPSD gains, we clarified its hyperparameter tuning and the comparison between LoRA and full fine-tuning (Appendix D.1), and supplemented the behavioral analysis with a case study and an LLM-as-a-judge evaluation of effective and ineffective reflections.

  • To Reviewer jBsG: We expanded the behavioral comparison between NSD and OPSD through case studies and an LLM-as-a-judge analysis, showing that NSD achieves the highest effective-to-ineffective reflection ratio among the evaluated methods. To examine the gating mechanism and negative conditions, we added error-token enrichment analysis, no-gate and random-gate controls, and a style-only conditioning baseline. We also added sigmoid ablations and a standard forward-KL comparison to assess the contribution of the individual training components (Appendix H).

We sincerely thank all reviewers once again for their time and effort; we now have updated the main text to include the new experimental results, and hope that the revised manuscript more clearly supports the design rationale behind each component of NSD.

[Update]:

As the discussion period draws to a close, we provide a brief update summarizing our discussion with the reviewers:

  • Reviewer ximH acknowledged our clarifications on most of the concerns raised. In response to the only remaining concern regarding response length, while we previously used a ratio-based metric to eliminate length bias, we now provide more direct experimental results comparing reflections across closely matched lengths, further supporting our conclusions.

  • Reviewer jBsG expressed a positive assessment of our responses and raised the rating.

  • We have also provided detailed responses to the concerns raised by Reviewer uYk9 and would welcome further feedback and assessment.

We sincerely thank all reviewers for their time, effort, and consideration of our work throughout the review and discussion process. We also want to express our gratitude to the AC for taking the time to review our paper and for considering the rebuttal discussion in the final decision. Thank you again for your time and efforts!

Sincerely,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.