acceptodds
Under review as a conference paper at ICLR 2027

Masked Self-Distillation: Internalizing the Chain-of-Thought in Language Models

10 views

Abstract

Large Reasoning Models (LRMs) produce long, explicit chains of intermediate steps before generating a final answer at inference time. These intermediate traces dominate latency, memory usage, and serving cost, even though the final answer correctness is not causally related to the trace correctness and the trace length is not a reliable indicator of the problem complexity. This raises an obvious question: can the computation expressed in these intermediate tokens be internalized into the parameters of a language model, enabling it to produce answers with much shorter intermediate traces? We propose masked self-distillation, a knowledge-distillation based post-training framework in which copies of the same model are instantiated as teacher and student, and the student model is trained to internalize all or part of the intermediate trace, thus becoming more efficient at inference. We vary the fraction of the intermediate trace the student is trained to internalize, interpolating between full internalization (the student emits only the solution) and no internalization (the student emits the full trace before the answer). We conduct controlled experiments on two reasoning domains: math and graph coloring. We use the masked self-distillation framework to post-train Qwen3-4B and Qwen3-8B models. Our results demonstrate that this method can be used to improve task performance while increasing inference efficiency across various domains and model sizes. We systematically analyze whether the improved efficiency gains in the post-trained models generalize to out-of-distribution (OOD) problems. We find that the masked self-distillation models generalize well for in-domain OOD problems, and the masked self-distillation training does not induce catastrophic forgetting in the student model on out-of-domain problems. Furthermore, our ablation study shows that supervised fine-tuning can train models to produce shorter traces, but at the cost of generalization, highlighting the importance of on-policy training in masked self-distillation. Lastly, we show that this framework can also be used effectively in a dual-model setting where a larger reasoning model teacher supervises a smaller non-reasoning student model, resulting in improved task performance and inference efficiency.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Knowledge distillation, Self distillation, Large Reasoning Models, Efficient inference, Reasoning traces

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BMi1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes masked self-distillation (MSD), an on-policy distillation method for shortening LRM reasoning traces. Two copies of the same model act as teacher (think mode) and student (no-think mode). After rejection-sampling teacher traces, the frozen teacher is conditioned on the prompt plus the first α fraction of its trace, while the student sees only the prompt and is trained to minimize per-token reverse KL to the teacher on its own rollouts. α interpolates between α=0 (plain on-policy think→no-think self-distillation) and α=1 (student emits only the answer).

Critical strengths
  1. A clean, controllable formulation with a careful protocol. The single knob α interpolates between no and full internalization, and results are reported with 3 seeds and std, in-domain and cross-domain OOD sets, token-budget curves (Fig. 5), and length distributions.

  2. The SFT-vs-reverse-KL ablation (Table 1) is informative: off-policy training on truncated traces matches in-distribution accuracy but generalizes worse, produces much longer cross-domain outputs, and collapses at α=1, supporting the claim that on-policy training—not merely shorter targets—drives the gains.

Critical weaknesses
  1. The "internalization" claim is weakly supported, and the paper's own α=0 baseline undercuts the contribution in math. In Tables 2/4, α=0—no masking, i.e., plain on-policy think→no-think distillation (essentially Agarwal et al. 2024 / Shenfeld et al. 2026)—already gives the best GSM8K accuracy (93.5/95.0), the best MATH-500 accuracy for 4B (78.9, above the teacher), and shorter outputs than α=0.3/0.5/0.7. So in math, intermediate α adds nothing over α=0; only α=1 is a distinct operating point. In graph coloring, α=0.3/0.5/0.7 yield nearly identical lengths (3274/3772/4112 tokens), i.e., length does not scale with the masked fraction as the internalization story predicts. The "mode-seeking" explanation is offered without evidence.

  2. No comparison to any external baseline. The paper cites L1, CoT-Valve, O1-Pruner, GFPO, budget prompting, and—most directly—implicit-CoT distillation (Deng et al. 2023/2024) but compares to none. Also missing is a plain RL baseline (e.g., GRPO with correctness reward, ± length penalty) on the same data, which would separate "on-policy training on the domain" from "the masked-teacher signal". For an efficiency paper this makes the method impossible to position. Scope is also narrow: one model family, a near-saturated training set (GSM8K), one synthetic task; Countdown (appendix) shows no token savings.

Questions for authors
  1. How is "no-think mode" handled for the student? Qwen3's no-think template inserts an empty <think></think> block, yet in Appendix E the α=0 and α∈{0.3,0.5,0.7} students emit long think-style content ending in </think>. Does the student prompt open a <think> tag? Does the teacher context for α>0 include <think> followed by the prefix z≤kα? Please give the exact templates.
  2. Please add controls: (a) GRPO/RL in no-think mode with a correctness reward on the same data; (b) at least one published length-reduction method (e.g., L1 or CoT-Valve); (c) implicit CoT (Deng et al. 2024). Even one of these on graph coloring would clarify the contribution.
  3. Can you give direct evidence for the mode-seeking explanation, e.g., rollout length vs. training step per α, or results with forward-KL/JSD?
  4. Is "response length" the full output including the solution? With a 32k inference cap vs. 8k training cap, what fraction of responses hit the cap per method?

5, What fraction of training problems survive rejection sampling, and does the method still work when teacher pass rate is low?

5 comments
AuthorsOpenReview ↗

Disentangling the effect of masking and self-distillation

W1 "The "internalization" claim is weakly supported, and the paper's own α=0 baseline undercuts the contribution in math."

The α=0\alpha=0 results on math were a surprising result for us too. After the initial submission, we did a thorough code audit and found that there was a bug in the implementation affecting the α=0\alpha=0 experiments. We had implemented the masked self-distillation framework by building on SDPO's [1] repository. SDPO's existing implementation adds a student-generated correct response to the teacher's context along with the question prompt. Due to the bug, when we passed α=0\alpha=0, the code used SDPO's original implementation, which caused the correct student rollout to also be appended to the teacher's context. In contrast, in the α=0\alpha=0 setting, the teacher should have been conditioned only on the question. We have fixed the bug and reported the corrected numbers. The earlier α=0\alpha=0 results reported in the paper should be read as a comparison with SDPO-style training. We have updated the Pareto curves and tables in the manuscript with the correct numbers.

The updated results show how much self-distillation, on its own, can drive the performance improvement and compression. We observe that α=0\alpha=0 training modifies the student's behavior to mimic the teacher. This improves the student's performance but does not induce efficient inference. The prefix masking drives the compression gains observed for α∈0.3,0.5,0.7,1\alpha \in \\{0.3, 0.5, 0.7, 1\\}

Qwen3-4B

DatasetSDPO (old)Non-masked(updated)Base think
Acc (%)TokensAcc (%)TokensAcc (%)Tokens
GSM8K94.06 ± 0.391,061 ± 7994.90 ± 0.042,353 ± 4994.74 ± 0.162,332 ± 27
MATH50093.93 ± 0.423,340 ± 14696.07 ± 0.314,959 ± 8995.67 ± 0.425,137 ± 214
AIME2550.00 ± 5.7712,598 ± 2,08162.22 ± 5.0917,666 ± 74063.33 ± 6.6718,190 ± 840
GC-ER χ3-492.3 ± 0.66,369 ± 12089.67 ± 2.166,982 ± 62091.9 ± 0.66,185 ± 237
GC-ER χ646.7 ± 1.612,439 ± 48744.6 ± 1.716,612 ± 26045.4 ± 2.515,489 ± 643
GC-BA χ665.7 ± 1.810,186 ± 36664.10 ± 2.412,832 ± 59862.1 ± 2.612,134 ± 298

Qwen3-8B

DatasetSDPO (old)Non-masked(updated)Base think
Acc (%)TokensAcc (%)TokensAcc (%)Tokens
GSM8K95.32 ± 0.191,109 ± 1295.63 ± 0.092,395 ± 3195.35 ± 0.192,402 ± 6
MATH50095.13 ± 0.313,348 ± 6496.53 ± 0.615,365 ± 3696.73 ± 0.235,355 ± 116
AIME2546.67 ± 12.0210,924 ± 1,27163.33 ± 3.3317,600 ± 8966.67 ± 3.3317,274 ± 752
GC-ER χ3-494.3 ± 1.05,253 ± 6995.2 ± 0.506,052 ± 19895.5 ± 0.35,752 ± 131
GC-ER χ654.4 ± 1.911,855 ± 26662.8 ± 0.5014,795 ± 10964.4 ± 2.514,434 ± 12
GC-BA χ676.5 ± 1.98,470 ± 29078.7 ± 0.710,931 ± 25381.5 ± 0.410,298 ± 288
AuthorsOpenReview ↗

Additional baselines

W2/Q2

(a) As requested, we performed an RL based baseline experiment using GRPO. GRPO trains the model based on the correctness signal. We post-trained the Qwen3-4B model with GRPO baseline on graph coloring dataset. We post-train the model in think mode to study their effects on the reasoning trace length as the aim of our work is to induce efficient reasoning. The table below reports the performance comparison between RL-baselines and our best performing -masked variant. From the table’s result it is evident that our method performs comparable to GRPO in terms of accuracy while being more inference efficient on ID and OOD datasets. For GRPO training, we used the same hyperparameters as our variants, which are reported in Appendix A.

MethodGC-ER χ3-4 Acc (%)GC-ER χ3-4 Avg tokensGC-ER χ6 Acc (%)GC-ER χ6 Avg tokensGC-BA χ6 Acc (%)GC-BA χ6 Avg tokens
GRPO 4B95.07 ± 2.876,502 ± 38975.93 ± 2.2011,351 ± 49690.73 ± 1.857,893 ± 338
MSD 4B (70%)97.5 ± 1.03,274 ± 12979.5 ± 1.56,906 ± 19688.3 ± 1.05,185 ± 135

We have added systematic comparisons with self-distillation-based baselines (SDPO, OPSD, CRISP) including a comparably tuned response-shortening baseline (CRISP) in Appendix G. Our analysis on the graph coloring domain across models shows that the masked self-distilled variants have the highest accuracy on ID and OOD datasets while being inference efficient compared to other distillation-based baselines.

We postulate that MSD outperforms other baselines by explicitly incentivizing the student to generate shorter traces by mimicking the teacher's distribution and acting as if it has already generated various fractions of the reasoning trace. SDPO and OPSD assume that post-hoc rationalizations of the correct answers are easier to generate and equally helpful for the student model to learn from. However, prior work [2] has shown that post-training using post-hoc explanations does not induce performance gains compared to training on the full reasoning trace. On the other hand, CRISP assumes that LRMs' reasoning behavior is instruction following, and thus relies on the teacher following its concise reasoning instruction.

Graph coloring baselines: Qwen3-4B

DatasetMetricOPSDCRISPSDPOMSD (70%)Base (think)Base (no-think)
GC-ER χ3-4Accuracy (%)89.8 ± 1.485.4 ± 0.492.3 ± 0.697.5 ± 1.091.9 ± 0.629.2 ± 1.4
Tokens8,253 ± 2384,984 ± 436,369 ± 1203,274 ± 1296,185 ± 2371,576 ± 85
GC-ER χ6Accuracy (%)37.8 ± 1.027.7 ± 1.146.7 ± 1.679.5 ± 1.545.4 ± 2.513.7 ± 2.1
Tokens14,927 ± 19012,650 ± 10812,439 ± 4876,906 ± 19615,489 ± 6431,254 ± 166
GC-BA χ6Accuracy (%)60.6 ± 1.342.0 ± 1.765.7 ± 1.888.3 ± 1.062.1 ± 2.618.4 ± 1.9
Tokens11,675 ± 15811,075 ± 29510,186 ± 3665,185 ± 13512,134 ± 2981,566 ± 445

Graph coloring baselines: Qwen3-8B

DatasetMetricOPSDCRISPSDPOMSD (50%)Base (think)Base (no-think)
GC-ER χ3-4Accuracy (%)94.9 ± 0.686.8 ± 2.794.3 ± 1.098.5 ± 0.895.5 ± 0.333.2 ± 1.8
Tokens5,914 ± 2074,127 ± 2075,253 ± 693,766 ± 2335,752 ± 1311,149 ± 28
GC-ER χ6Accuracy (%)61.3 ± 0.138.6 ± 3.254.4 ± 1.985.6 ± 0.664.4 ± 2.513.5 ± 0.5
Tokens12,603 ± 1949,839 ± 8711,855 ± 2668,289 ± 18114,434 ± 121,217 ± 23
GC-BA χ6Accuracy (%)76.0 ± 1.957.4 ± 1.376.5 ± 1.993.3 ± 1.081.5 ± 0.418.4 ± 1.9
Tokens9,432 ± 4058,255 ± 2738,470 ± 2905,509 ± 16410,298 ± 2881,145 ± 22

(b)

"Countdown no token savings clarification"

In the case of Countdown, no token savings on average were observed because the base student has degenerate behavior on problems where it produces incorrect responses. The base student keeps looping until it exhausts the maximum token generation limit. This increases the average token count, as shown by the Pareto plots.

We have now provided a detailed analysis of the various model responses based on correctness. Our results demonstrate that the MSD variants ({0.3,0.5,0.7}) have similar accuracy and are more inference efficient compared to the base think mode on ID and OOD test sets when the response is correct. Please refer to Appendix C.1 for the updated results.

Q1 – How is the "no-think mode" handled for the student model?

All the α\alpha-masked students act in no-think mode, so after the chat template is applied, an empty <think></think> block gets appended after the question. In Appendix E, we have added the student prompt with the chat template (highlighted in blue).

Yes, the teacher context for α\alpha >0 includes <think> followed by the prefix z<k. We have added the teacher prompt with the chat template in Appendix E (highlighted in blue).

Even with the no-think chat template, the trained α\alpha-masked students ({0.0,0.3,0.5,0.7}) produce the end-think tag (</think>) in their responses after training. During training, the teacher model acts in think mode and is only conditioned on its α\alpha fraction of the reasoning trace, so its probability distribution has a tendency to produce the </think> tag. Thus, with MSD training, the α\alpha-masked students also learn to generate an additional </think> tag.

Q4 – Is "response length" the full output including the solution?

Yes, the response length includes the full output: the chain-of-thought tokens followed by the solution tokens. We have also added results on the number of responses truncated across various models and datasets. These analyses are in line with the response length distribution reported in Appendix D. We also see that the truncation rate of the base no-think student is highest for Countdown due to the degenerate behavior mentioned above (in response to W2/Q2).

Q5 – What fraction of training problems survive after rejection sampling?

The GSM8K dataset has 7,473 training problems, and we generate 10,000 GC-ER χ3-4 instances for the graph coloring dataset. We generate the responses from the teacher models on these training instances. During teacher response generation, we set the maximum response length to 8K for GSM8K and 16K for graph coloring.

For constructing our MSD training sets, we filter the teacher-generated rollouts and keep only the data points that have a correct answer in the response. Our filter removes the responses that have incorrect answers or incomplete generations.

The table below shows the number of data points used in student training for both datasets and models.

DatasetQwen3-4BQwen3-8B
GSM8K7,406 / 7,4737,081 / 7,473
Graph coloring9,648 / 10,0009,222 / 10,000

Training without rejection sampling. We believe that MSD-style training will provide efficiency gains even when student models are trained on datasets without rejection sampling, as our framework does not require any correctness signal during training. We might not see performance gains over the teacher model, as seen on the current GC OOD test sets, but the α\alpha-trained students can learn to mimic the teacher's behavior while producing shorter reasoning traces.

Thank you again for a detailed review. We hope that the clarifications provided address your concerns. We would be grateful if you would consider raising your score.

References

[1] Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, et al. Reinforcement learning via self-distillation. arXiv preprint arXiv:2601.20802, 2026.

[2] Bhambri, Siddhant, Upasana Biswas, and Subbarao Kambhampati. "Interpretable traces, unexpected outcomes: Investigating the disconnect in trace-based knowledge distillation." Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Reviewer_Xz2kOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes masked self-distillation to reduce the number of intermediate reasoning tokens generated by language models. In the main setting, a frozen teacher and a trainable student originate from the same checkpoint. The teacher first generates solutions, which are filtered for final-answer correctness. During distillation, the teacher receives the question and a prefix of the collected reasoning trace, while the student receives only the question. The student minimizes a reverse-KL objective against the teacher's next-token distributions along student-generated trajectories. Varying the prefix fraction controls the intended degree of reasoning compression, and inference uses only the student. Experiments primarily use Qwen3-4B and Qwen3-8B on mathematics and graph coloring, with additional Countdown and cross-model distillation experiments. Partial masking produces promising accuracy and token-count improvements in several settings, while full masking can cause substantial performance degradation. The authors interpret the results as internalization of intermediate computations and argue that on-policy distillation preserves generalization better than SFT.

Critical strengths
  1. The problem is relevant, and the method is straightforward to deploy. Reducing the cost of long reasoning traces is a useful objective. Giving the teacher additional reasoning context provides a concrete training signal for producing shorter continuations. The main method requires neither a larger teacher nor an architectural change, and the trained student can be used independently at inference.

  2. Some results demonstrate meaningful improvements in both accuracy and output length. For Qwen3-4B on the graph-coloring ID test set, 70% masking improves accuracy from the thinking baseline's 91.9% to 97.5%, while reducing mean response length from 6,185 to 3,274 tokens. On GC-ER with chromatic number 6, the same variant improves accuracy from 45.4% to 79.5%, with mean length falling from 15,489 to 6,906 tokens (Table 3). These are encouraging results beyond merely obtaining shorter outputs. Figure 5 also provides a useful comparison of correctness within different token budgets on the graph-coloring ID task.

  3. The study includes useful controls and exposes important limitations. The experiments vary the masking fraction, include an unmasked distillation baseline and an SFT comparison, and evaluate both more difficult problems within a domain and transfer between mathematics and graph coloring. The main result tables report means and standard deviations over three seeds. The poor performance of full masking on graph coloring is an informative finding, and the additional cross-model experiment shows that the training construction is not restricted to identical teacher and student sizes.

Critical weaknesses

My main concern is whether the shorter outputs preserve the pretrained model's existing abilities. A specialized model with a favorable cost-quality tradeoff can be valuable, but reductions in token count alone do not establish broadly improved reasoning efficiency. The claims about capability preservation and internalization require stronger evidence than the current experiments provide.

  1. The evidence does not establish preservation of general capabilities, and there are already substantial cross-task regressions. Section 5.2 reports that, after GSM8K training, Qwen3-8B achieves 87.0% graph-coloring accuracy with 30% masking and 85.5% with 50% masking, compared with 95.5% for the original thinking model. Mean response length decreases from approximately 5.7K to 4.6K or 4.2K tokens. Losing 8.5-10 percentage points for these savings is a material tradeoff. The paper emphasizes that the distilled models remain above the original no-think baseline, but exceeding that baseline does not establish preservation of abilities already available from the same original checkpoint in thinking mode. The cross-domain evaluation mainly transfers between mathematics and graph coloring; it does not systematically assess capabilities such as coding, knowledge-based question answering, or instruction following. These observations do not prove irreversible forgetting, but they leave the broad claim of no catastrophic forgetting insufficiently supported.

  2. Compression benefits are strongly dependent on the task and masking fraction. For the GSM8K-trained 8B model, full masking reduces GSM8K responses from 2,402 to 302 tokens with accuracy decreasing from 94.6% to 92.8%, an attractive operating point for that task. However, on AIME25, the same model drops from the teacher's 61.1% accuracy to 21.1%. The 30%-masked variant retains approximately the teacher's AIME25 accuracy, but reduces mean length only from 17,274 to 16,322 tokens (Table 4). Thus, the large compression gain on the training domain does not transfer while preserving difficult-problem performance. Figure 5 is helpful, but its ID graph-coloring result does not resolve this broader issue. Comparisons on unseen tasks at matched quality or matched budgets, including a tuned alternative for shortening responses, would better establish the practical advantage of the method. A fixed configuration selected on development data would also clarify its usefulness across tasks without selecting a different masking fraction for each test set.

  3. The experiments support a change in generation behavior more directly than internalization of the omitted computations. Training the student to match a teacher conditioned on a reasoning prefix explicitly encourages the student to produce continuation-like text. Abrupt openings and missing references, as measured in Appendix F, are therefore consistent with the training construction itself. They do not show that the computations associated with the omitted prefix have been reproduced internally. Shorter outputs with retained accuracy could also reflect reduced repetition, a different solution strategy, or changes in output style. Moreover, unmasked distillation already produces substantial shortening: on 8B GSM8K, it reaches 95.0% accuracy at 1,109 tokens, compared with 94.0% at 1,429 tokens for 70% masking (Table 4). The unmasked control is valuable, but this result makes it especially important to distinguish the contribution of prefix masking from distillation more generally. The stronger interpretation of computation being moved into parameters remains unverified.

  4. The SFT comparison does not cleanly isolate the benefit of on-policy training. Appendix A reports 30,000 SFT training steps with a learning rate of 1e-5, versus 200 steps for masked self-distillation with a learning rate of 1e-6. The schedules and weight decay also differ. A distillation step includes multiple student rollouts, so these step counts are not directly equivalent measures of computation. Nevertheless, the substantial differences in optimization and training exposure make it difficult to attribute SFT's worse transfer specifically to its off-policy objective. This is particularly relevant to the fully masked SFT model's collapse on cross-domain GSM8K. Validation-based checkpoint selection, learning curves, and comparably tuned training budgets are needed to separate effects of the objective from effects of prolonged specialization or overtraining.

Questions for authors
  1. Capability retention: Can the authors evaluate the same distilled checkpoint on a broader set of untrained tasks, comparing against both thinking and no-think modes of the original checkpoint? Does allowing the distilled model to use an explicit thinking prompt and a larger generation budget recover the original performance? Please specify the inference templates. This would help distinguish reduced willingness to generate intermediate steps from a loss of capability.

  2. Practical efficiency: Does one masking configuration chosen on development data improve the accuracy-cost tradeoff across unseen tasks? Can the authors extend the budget analysis beyond ID graph coloring and include a comparably tuned response-shortening baseline, such as explicit budget control or training with correctness and length objectives?

  3. Evidence for internalization: What observation would distinguish internalization of the omitted computation from learning the style of a continuation? Could a controlled task with known intermediate states, or interventions on the information supplied in the teacher prefix, provide more direct evidence? Otherwise, can the mechanistic claim be narrowed to the demonstrated behavioral result?

  4. SFT fairness: How were the 30,000-step SFT schedule and checkpoints selected? Do the generalization differences persist when SFT is appropriately tuned and evaluated at earlier checkpoints, with training exposure and compute reported for both methods?

  5. Role of masking: Why does unmasked distillation already shorten responses substantially and sometimes outperform masked variants, as on 8B GSM8K? Across the tested tasks, which gains require the additional reasoning prefix, and which can be obtained from unmasked on-policy distillation alone?

6 comments
AuthorsOpenReview ↗

Capabilities retention

We thank the reviewer for a critical review. We would like to provide clarifications below.

W1/Q1 – Preservation of general capabilities

To systematically analyse whether the existing capabilities are preserved, we conduct an extensive analysis of the gsm8k and graph coloring trained variants on coding (LiveCodeBench v5), instruction following (IFEval), and general knowledge (MMLU-Redux) benchmarks. We report detailed results in Appendix H. Our analysis demonstrates that the base model's think-mode capabilities are largely retained in our masked trained models. In the no-think mode, most of the MSD variants' performance improves over the base student model and is closer to the base think-mode accuracy. We also observe an inference efficiency gain on the MMLU-Redux and LiveCodeBench v5 tasks. Based on this broad analysis, we conclude that MSD-style training, owing to its on-policy nature, does not induce catastrophic forgetting and retains the model's existing capabilities on out-of-domain datasets.

AuthorsOpenReview ↗

Practical efficiency and additional baselines

W2/Q2 – Practical efficiency (accuracy curve under a token budget for OOD)

(a) We propose α\alpha as a tunable parameter that can be chosen by developers depending on their domain and cost/accuracy trade-off. We do not propose any optimal value of α\alpha. While adopting the framework, we would encourage developers to perform a parameter sweep across different α\alpha values and choose the optimal for their use case.

(b) We have extended the generation budget analysis to OOD datasets and across different models. We have added the accuracy curves over the token budget in Appendix I. Our claims regarding the α\alpha variants performing better than the base think mode at various generation budgets remain consistent in OOD settings as well.

Below, we provide the comparison of accuracy under a token budget between our best-performing α\alpha-masked variant and the base model in think mode for Qwen3-4B on the AIME-25 and GC-ER χ6 OOD datasets. It is evident that our MSD variant does better than the base model in think mode. Please refer to Appendix I for other detailed comparisons.

Qwen3-4B accuracy (%) under token budget

Token budgetAIME-25: Base thinkAIME-25: MSD (α=0.3)GC-ER χ6: Base thinkGC-ER χ6: MSD (α=0.7)
4K0.06.72.051.60
8K13.3323.3318.2071.40
12K26.6743.3328.2475.64
16K405033.6877.68
20K53.3356.6739.0479.04

(c) We have added systematic comparisons with self-distillation-based baselines (SDPO, OPSD, CRISP) including a comparably tuned response-shortening baseline (CRISP) in Appendix G. Our analysis on the graph coloring domain shows that the masked self-distilled variants have the highest accuracy on ID and OOD datasets while being inference efficient compared to other distillation-based baselines.

We postulate that MSD outperforms other baselines by explicitly incentivizing the student to generate shorter traces by mimicking the teacher's distribution and acting as if it has already generated various fractions of the reasoning trace. SDPO and OPSD assume that post-hoc rationalizations of the correct answers are easier to generate and equally helpful for the student model to learn from. However, prior work [1] has shown that post-training using post-hoc explanations does not induce performance gains compared to training on the full reasoning trace. On the other hand, CRISP assumes that LRMs' reasoning behavior is instruction following, and thus relies on the teacher following its concise reasoning instruction.

Graph coloring baselines: Qwen3-4B

DatasetMetricOPSDCRISPSDPOMSD (70%)Base (think)Base (no-think)
GC-ER χ3-4Accuracy (%)89.8 ± 1.485.4 ± 0.492.3 ± 0.697.5 ± 1.091.9 ± 0.629.2 ± 1.4
Tokens8,253 ± 2384,984 ± 436,369 ± 1203,274 ± 1296,185 ± 2371,576 ± 85
GC-ER χ6Accuracy (%)37.8 ± 1.027.7 ± 1.146.7 ± 1.679.5 ± 1.545.4 ± 2.513.7 ± 2.1
Tokens14,927 ± 19012,650 ± 10812,439 ± 4876,906 ± 19615,489 ± 6431,254 ± 166
GC-BA χ6Accuracy (%)60.6 ± 1.342.0 ± 1.765.7 ± 1.888.3 ± 1.062.1 ± 2.618.4 ± 1.9
Tokens11,675 ± 15811,075 ± 29510,186 ± 3665,185 ± 13512,134 ± 2981,566 ± 445

Graph coloring baselines: Qwen3-8B

DatasetMetricOPSDCRISPSDPOMSD (50%)Base (think)Base (no-think)
GC-ER χ3-4Accuracy (%)94.9 ± 0.686.8 ± 2.794.3 ± 1.098.5 ± 0.895.5 ± 0.333.2 ± 1.8
Tokens5,914 ± 2074,127 ± 2075,253 ± 693,766 ± 2335,752 ± 1311,149 ± 28
GC-ER χ6Accuracy (%)61.3 ± 0.138.6 ± 3.254.4 ± 1.985.6 ± 0.664.4 ± 2.513.5 ± 0.5
Tokens12,603 ± 1949,839 ± 8711,855 ± 2668,289 ± 18114,434 ± 121,217 ± 23
GC-BA χ6Accuracy (%)76.0 ± 1.957.4 ± 1.376.5 ± 1.993.3 ± 1.081.5 ± 0.418.4 ± 1.9
Tokens9,432 ± 4058,255 ± 2738,470 ± 2905,509 ± 16410,298 ± 2881,145 ± 22
AuthorsOpenReview ↗

Evidence for internalization and SFT fairness

W3/Q3 – Evidence for internalization

We have used the term internalization in accordance with the existing literature. Prior works [3, 4, 5] have used the term internalization of additional information when the teacher is conditioned on additional context such as in-context examples, environment feedback, etc. We believe that MSD training changes the parameters of the student model such that it now requires fewer forward passes to produce the correct answer. We have modified the manuscript and described what we mean by internalization of computations (see the footnote on page 2).

Note that this account deliberately avoids taking any stance on the end-user interpretability of intermediate tokens, or implying that particular task-specific computations are internalized. We believe that our general view is consistent with the mixed evidence regarding the end-user semantics of intermediate tokens [2].

W4/Q4 – SFT fairness

We chose the training hyperparameters after a sweep over learning rate {5e-6, 1e-5, 5e-5} and batch size {16, 32, 64}. We chose the hyperparameters which had the lowest validation loss on held-out validation set. Other hyperparameters were chosen in line with the SFT setup of [4]. Training used a 30k-step cosine schedule decaying to zero; validation loss plateaued by the end of the schedule, and we report the final checkpoint. We did not retain intermediate checkpoints.

Reviewer_62hdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper trains a no-think Qwen3 student, using on-policy per-token reverse KL, to match a frozen think-mode copy of itself that also sees the first α fraction of its own reasoning trace; partial masking improves accuracy and shortens outputs on synthetic graph coloring, while math results are mixed and full masking collapses on graph coloring and AIME25.

Critical strengths
  1. Strong performance on graph coloring: Every partially masked model beats both the teacher and the no-prefix α=0 control on accuracy and length in all 18 graph settings.
Critical weaknesses
  1. The paper builds on SDFT and SDPO but does not cite the closest on-policy self-distillation papers: OPSD (essentially \alpha=1 on Qwen3), OPCD (2602.12275), CRISP, etc. Those are also missing baselines.
  2. On math, partial masking never beats the \alpha=0 control on both accuracy and length, so the math compression comes mainly from self distillation rather than the prefix.
Questions for authors
  1. How does MSD compare with CRISP and with OPSD-style full-trace conditioning on the same models and benchmarks?
  2. Given the \alpha=0 results on math, what evidence shows that the prefix rather than self distillation drives the math results?
  3. Does a teacher given an irrelevant or shuffled prefix produce similar compression?
  4. Which grader and answer extraction method produce the 77.5% MATH-500 baseline? CRISP reported 95.7% on Qwen3-8B thinking while the paper reported 77.5%.
6 comments
AuthorsOpenReview ↗

Additional self-distillation based baselines

We thank the reviewer for a critical review. We would like to provide clarifications below.

Q1/W1 – Comparison with CRISP and OPSD

We have now added the missing citations of the on-policy self-distillation papers and modified the related works section (highlighted in blue). As suggested by the reviewer, we have now included CRISP and OPSD methods as relevant baselines.

We would like to clarify that OPSD and our fully-masked variant (α=1\alpha=1 setting) are not equivalent. In OPSD, the teacher is conditioned on the question, a reference (which can be a reasoning trace and solution; in their experiments, they used math problems that have human-written procedural step-by-step solutions), and the following instruction: "After understanding the reference solution, please try to solve this problem using your own approach below:". So in OPSD, the teacher still generates a reasoning trace followed by the solution tokens, and the student needs to learn this behaviour. In contrast, in our α=1\alpha=1 setting (fully-masked distillation), the teacher model is conditioned on the question followed by its own reasoning trace. Conditioned on this, the teacher's probability distribution will only produce the solution tokens, and this is the distribution which the student needs to match. The implicit assumption in OPSD is that rationalization of a solution is easier than solving the problem itself, and thus the student will learn to generate shorter traces which are explanations of the given solution.

We post-trained Qwen3-4B and Qwen3-8B models on the Graph Coloring dataset using the OPSD and CRISP frameworks. We have provided the hyperparameter details in Appendix A and a detailed comparison on the GC in-distribution and OOD test sets in Appendix G.

Our results show that the masked self-distilled variants have the highest accuracy on ID and OOD datasets while being inference efficient. We postulate that MSD outperforms other baselines by explicitly incentivizing the student to generate shorter traces by mimicking the teacher's distribution and acting as if it has already generated various fractions of the reasoning trace. SDPO and OPSD assume that post-hoc rationalizations of the correct answers are easier to generate and equally helpful for the student model to learn from. However, prior work [1] has shown that post-training using post-hoc explanations does not induce performance gains compared to training on the full reasoning trace. On the other hand, CRISP assumes that LRMs' reasoning behavior is instruction following and thus relies on the teacher following their concise reasoning instruction.

Graph coloring baselines: Qwen3-4B

DatasetMetricOPSDCRISPSDPOMSD (70%)Base (think)Base (no-think)
GC-ER χ3-4Accuracy (%)89.8 ± 1.485.4 ± 0.492.3 ± 0.697.5 ± 1.091.9 ± 0.629.2 ± 1.4
Tokens8,253 ± 2384,984 ± 436,369 ± 1203,274 ± 1296,185 ± 2371,576 ± 85
GC-ER χ6Accuracy (%)37.8 ± 1.027.7 ± 1.146.7 ± 1.679.5 ± 1.545.4 ± 2.513.7 ± 2.1
Tokens14,927 ± 19012,650 ± 10812,439 ± 4876,906 ± 19615,489 ± 6431,254 ± 166
GC-BA χ6Accuracy (%)60.6 ± 1.342.0 ± 1.765.7 ± 1.888.3 ± 1.062.1 ± 2.618.4 ± 1.9
Tokens11,675 ± 15811,075 ± 29510,186 ± 3665,185 ± 13512,134 ± 2981,566 ± 445

Graph coloring baselines: Qwen3-8B

DatasetMetricOPSDCRISPSDPOMSD (50%)Base (think)Base (no-think)
GC-ER χ3-4Accuracy (%)94.9 ± 0.686.8 ± 2.794.3 ± 1.098.5 ± 0.895.5 ± 0.333.2 ± 1.8
Tokens5,914 ± 2074,127 ± 2075,253 ± 693,766 ± 2335,752 ± 1311,149 ± 28
GC-ER χ6Accuracy (%)61.3 ± 0.138.6 ± 3.254.4 ± 1.985.6 ± 0.664.4 ± 2.513.5 ± 0.5
Tokens12,603 ± 1949,839 ± 8711,855 ± 2668,289 ± 18114,434 ± 121,217 ± 23
GC-BA χ6Accuracy (%)76.0 ± 1.957.4 ± 1.376.5 ± 1.993.3 ± 1.081.5 ± 0.418.4 ± 1.9
Tokens9,432 ± 4058,255 ± 2738,470 ± 2905,509 ± 16410,298 ± 2881,145 ± 22
AuthorsOpenReview ↗

Disentangling the effect of masking and self-distillation

Q2/W2 – The role of masking

The α=0\alpha=0 results on math were a surprising result for us too. After the initial submission, we did a thorough code audit and found that there was a bug in the implementation affecting the α=0\alpha=0 experiments. We had implemented the masked self-distillation framework by building on SDPO's [2] repository. SDPO's existing implementation adds a student-generated correct response to the teacher's context along with the question prompt. Due to the bug, when we passed α=0\alpha=0, the code used SDPO's original implementation, which caused the correct student rollout to also be appended to the teacher's context. In contrast, in the α=0\alpha=0 setting, the teacher should have been conditioned only on the question. We have fixed the bug and reported the corrected numbers. The earlier α=0\alpha=0 results reported in the paper should be read as a comparison with SDPO-style training. We have updated the Pareto curves and tables in the manuscript with the correct numbers.

The updated results show how much self-distillation, on its own, can drive the performance improvement and compression. We observe that α=0\alpha=0 training modifies the student's behavior to mimic the teacher. This improves the student's performance but does not induce efficient inference. The prefix masking drives the compression gains observed for α∈0.3,0.5,0.7,1\alpha \in \\{0.3, 0.5, 0.7, 1\\}.

Qwen3-4B

DatasetSDPO (old)Non-masked(updated)Base think
Acc (%)TokensAcc (%)TokensAcc (%)Tokens
GSM8K94.06 ± 0.391,061 ± 7994.90 ± 0.042,353 ± 4994.74 ± 0.162,332 ± 27
MATH50093.93 ± 0.423,340 ± 14696.07 ± 0.314,959 ± 8995.67 ± 0.425,137 ± 214
AIME2550.00 ± 5.7712,598 ± 2,08162.22 ± 5.0917,666 ± 74063.33 ± 6.6718,190 ± 840
GC-ER χ3-492.3 ± 0.66,369 ± 12089.67 ± 2.166,982 ± 62091.9 ± 0.66,185 ± 237
GC-ER χ646.7 ± 1.612,439 ± 48744.6 ± 1.716,612 ± 26045.4 ± 2.515,489 ± 643
GC-BA χ665.7 ± 1.810,186 ± 36664.10 ± 2.412,832 ± 59862.1 ± 2.612,134 ± 298

Qwen3-8B

DatasetSDPO (old)Non-masked(updated)Base think
Acc (%)TokensAcc (%)TokensAcc (%)Tokens
GSM8K95.32 ± 0.191,109 ± 1295.63 ± 0.092,395 ± 3195.35 ± 0.192,402 ± 6
MATH50095.13 ± 0.313,348 ± 6496.53 ± 0.615,365 ± 3696.73 ± 0.235,355 ± 116
AIME2546.67 ± 12.0210,924 ± 1,27163.33 ± 3.3317,600 ± 8966.67 ± 3.3317,274 ± 752
GC-ER χ3-494.3 ± 1.05,253 ± 6995.2 ± 0.506,052 ± 19895.5 ± 0.35,752 ± 131
GC-ER χ654.4 ± 1.911,855 ± 26662.8 ± 0.5014,795 ± 10964.4 ± 2.514,434 ± 12
GC-BA χ676.5 ± 1.98,470 ± 29078.7 ± 0.710,931 ± 25381.5 ± 0.410,298 ± 288

Q3 – Shuffled prefix

We believe that training with a teacher conditioned on the question plus the trace of a different problem, or on a shuffled trace, will degrade performance, since the teacher itself cannot produce the right answer when conditioned on a shuffled trace. To validate this, we first evaluated the teacher (Qwen3-4B in think mode) on 500 problem instances from the graph coloring training dataset for which it originally gave the correct answer. When the teacher is conditioned on a shuffled 50% fraction of its reasoning trace and then allowed to continue, its performance drops to 42.33%, a relative degradation of 58%. Training the student with a shuffled-prefix-conditioned teacher should therefore incentivize the student to match a distribution that leads to incorrect answers. (Note: to obtain the shuffled prefix, we randomly shuffle the tokens of the correct prefix.)

We then train the student model (Qwen3-4B no-think) with our masked self-distillation framework on the graph coloring dataset, where the teacher is conditioned on a shuffled prefix instead of the correct one. The results are reported in the table below. The student trained with the shuffled-prefix-conditioned teacher does not improve over the base no-think model, which is consistent with the degraded teacher performance observed above.

GC-ER χ3-4

ModelAcc (%)Mean tokens
Base think91.9 ± 0.66,185 ± 237
Base no-think29.2 ± 1.41,576 ± 85
Shuffled 50% masked31.53 ± 0.9013,527 ± 950
AuthorsOpenReview ↗

Updated math evaluation script

Q4 – Lower MATH500 performance

For MATH500 evaluation, we used the default evaluation script provided in the verl framework, which only checks the correctness of a response if the answer is in the \boxed{} format. We now use the CRISP evaluation script, which also checks other answer formats in the model's response, such as "This is the Answer:". This change improves the validation accuracy in cases where the model provides a correct answer in a different format. Below are the updated results on the MATH500 test set. We have updated all the math domain results in the paper. While there is an increase in accuracy across all models, we find no change in the relative trend, and thus our conclusions remain unchanged.

MATH500 accuracy (%)

ModelMasked 100%Masked 70%Masked 50%Masked 30%Masked 0%Base (think)Base (no-think)
Qwen3-4B82.00 ± 1.0694.40 ± 1.3994.40 ± 0.3593.33 ± 1.5096.07 ± 0.3195.67 ± 0.4284.93 ± 0.46
Qwen3-8B83.07 ± 0.2391.67 ± 1.4595.33 ± 0.4295.20 ± 0.2096.53 ± 0.6196.73 ± 0.2383.93 ± 1.55

Thank you again for a detailed review. We hope that the additional baselines, role of masking clarification, and the updated math evaluation script addresses your concerns. We would be grateful if you would consider raising your score.

References

  1. Bhambri, Siddhant, Upasana Biswas, and Subbarao Kambhampati. "Interpretable traces, unexpected outcomes: Investigating the disconnect in trace-based knowledge distillation." Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.
  2. Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, et al. Reinforcement learning via self-distillation. arXiv preprint arXiv:2601.20802, 2026.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Common Response to All Reviewers

We thank the reviewers for their thoughtful comments. We have uploaded a revised version of the manuscript along with this rebuttal. All changes in the manuscript are highlighted in blue. Below, we summarize the main updates that are common across reviews. We would like to thank the reviewers in advance for spending the time to read our response and the revised manuscript.

1. Bug fix in the α=0\alpha=0 (non-masked) experiments

After the initial submission, we did a code audit and found a bug affecting the α=0\alpha=0 experiments. Our implementation builds on the SDPO repository, and when α=0\alpha=0 was passed, the code fell back to SDPO's original behavior, which appends a correct student rollout to the teacher's context. In the intended α=0\alpha=0 setting, the teacher should be conditioned only on the question. The α=0\alpha=0 results in the original submission should therefore be read as SDPO-style training. We have fixed the bug and updated the Pareto curves and tables in the manuscript. The updated results demonstrate that the α=0\alpha=0 setting improves the student's performance but does not induce efficient inference. The prefix masking drives the compression gains observed for α∈0.3,0.5,0.7,1\alpha \in \\{0.3, 0.5, 0.7, 1\\}

2. Comparison with additional self-distillation based baselines

We have added systematic comparisons with self-distillation-based baselines (SDPO, OPSD, CRISP) in Appendix G. Our analysis on the graph coloring domain shows that the masked self-distilled variants have the highest accuracy on ID and OOD datasets while being inference efficient compared to other distillation-based baselines. The hyperparameters are provided in Appendix A and the full results are in Appendix G. We have also updated the related work section to discuss these methods.

3. Retention of general capabilities

To check whether the model's existing capabilities are retained after masked self-distillation training, we evaluated the GSM8K and Graph Coloring trained models on coding (LiveCodeBench v5), instruction following (IFEval), and general knowledge (MMLU-Redux) tasks. The base model's think-mode capabilities are largely retained after MSD training. In no-think mode, most MSD variants improve over the base student and come closer to the base think-mode accuracy. We also observe efficiency gains on MMLU-Redux and LiveCodeBench v5. These results suggest that MSD, due to its on-policy nature, does not cause catastrophic forgetting. Detailed results are provided in Appendix H.

4. Math domain evaluation

After Reviewer 62hd pointed out that there was a discrepancy between the Qwen3-4b and 8B model results on Math500, we changed our math response evaluation script. We switched from the default verl evaluation script, which only accepts answers in \boxed{}, to the CRISP evaluation script, which also accepts other answer formats. Accuracy increases for all models, but the relative trends and our conclusions are unchanged.

Other updates

Token budget analysis on OOD datasets: We extended the accuracy vs. generation budget analysis to OOD datasets across all models (Appendix I).

Prompts: We added the full student and teacher prompts with chat templates (Appendix E).

We hope these updates address the reviewers' concerns, and we are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.