- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes masked self-distillation (MSD), an on-policy distillation method for shortening LRM reasoning traces. Two copies of the same model act as teacher (think mode) and student (no-think mode). After rejection-sampling teacher traces, the frozen teacher is conditioned on the prompt plus the first α fraction of its trace, while the student sees only the prompt and is trained to minimize per-token reverse KL to the teacher on its own rollouts. α interpolates between α=0 (plain on-policy think→no-think self-distillation) and α=1 (student emits only the answer).
Critical strengths
-
A clean, controllable formulation with a careful protocol. The single knob α interpolates between no and full internalization, and results are reported with 3 seeds and std, in-domain and cross-domain OOD sets, token-budget curves (Fig. 5), and length distributions.
-
The SFT-vs-reverse-KL ablation (Table 1) is informative: off-policy training on truncated traces matches in-distribution accuracy but generalizes worse, produces much longer cross-domain outputs, and collapses at α=1, supporting the claim that on-policy training—not merely shorter targets—drives the gains.
Critical weaknesses
-
The "internalization" claim is weakly supported, and the paper's own α=0 baseline undercuts the contribution in math. In Tables 2/4, α=0—no masking, i.e., plain on-policy think→no-think distillation (essentially Agarwal et al. 2024 / Shenfeld et al. 2026)—already gives the best GSM8K accuracy (93.5/95.0), the best MATH-500 accuracy for 4B (78.9, above the teacher), and shorter outputs than α=0.3/0.5/0.7. So in math, intermediate α adds nothing over α=0; only α=1 is a distinct operating point. In graph coloring, α=0.3/0.5/0.7 yield nearly identical lengths (3274/3772/4112 tokens), i.e., length does not scale with the masked fraction as the internalization story predicts. The "mode-seeking" explanation is offered without evidence.
-
No comparison to any external baseline. The paper cites L1, CoT-Valve, O1-Pruner, GFPO, budget prompting, and—most directly—implicit-CoT distillation (Deng et al. 2023/2024) but compares to none. Also missing is a plain RL baseline (e.g., GRPO with correctness reward, ± length penalty) on the same data, which would separate "on-policy training on the domain" from "the masked-teacher signal". For an efficiency paper this makes the method impossible to position. Scope is also narrow: one model family, a near-saturated training set (GSM8K), one synthetic task; Countdown (appendix) shows no token savings.
Questions for authors
- How is "no-think mode" handled for the student? Qwen3's no-think template inserts an empty <think></think> block, yet in Appendix E the α=0 and α∈{0.3,0.5,0.7} students emit long think-style content ending in </think>. Does the student prompt open a <think> tag? Does the teacher context for α>0 include <think> followed by the prefix z≤kα? Please give the exact templates.
- Please add controls: (a) GRPO/RL in no-think mode with a correctness reward on the same data; (b) at least one published length-reduction method (e.g., L1 or CoT-Valve); (c) implicit CoT (Deng et al. 2024). Even one of these on graph coloring would clarify the contribution.
- Can you give direct evidence for the mode-seeking explanation, e.g., rollout length vs. training step per α, or results with forward-KL/JSD?
- Is "response length" the full output including the solution? With a 32k inference cap vs. 8k training cap, what fraction of responses hit the cap per method?
5, What fraction of training problems survive rejection sampling, and does the method still work when teacher pass rate is low?
5 comments
Additional baselines
W2/Q2
(a) As requested, we performed an RL based baseline experiment using GRPO. GRPO trains the model based on the correctness signal. We post-trained the Qwen3-4B model with GRPO baseline on graph coloring dataset. We post-train the model in think mode to study their effects on the reasoning trace length as the aim of our work is to induce efficient reasoning. The table below reports the performance comparison between RL-baselines and our best performing -masked variant. From the table’s result it is evident that our method performs comparable to GRPO in terms of accuracy while being more inference efficient on ID and OOD datasets. For GRPO training, we used the same hyperparameters as our variants, which are reported in Appendix A.
| Method | GC-ER χ3-4 Acc (%) | GC-ER χ3-4 Avg tokens | GC-ER χ6 Acc (%) | GC-ER χ6 Avg tokens | GC-BA χ6 Acc (%) | GC-BA χ6 Avg tokens |
|---|---|---|---|---|---|---|
| GRPO 4B | 95.07 ± 2.87 | 6,502 ± 389 | 75.93 ± 2.20 | 11,351 ± 496 | 90.73 ± 1.85 | 7,893 ± 338 |
| MSD 4B (70%) | 97.5 ± 1.0 | 3,274 ± 129 | 79.5 ± 1.5 | 6,906 ± 196 | 88.3 ± 1.0 | 5,185 ± 135 |
We have added systematic comparisons with self-distillation-based baselines (SDPO, OPSD, CRISP) including a comparably tuned response-shortening baseline (CRISP) in Appendix G. Our analysis on the graph coloring domain across models shows that the masked self-distilled variants have the highest accuracy on ID and OOD datasets while being inference efficient compared to other distillation-based baselines.
We postulate that MSD outperforms other baselines by explicitly incentivizing the student to generate shorter traces by mimicking the teacher's distribution and acting as if it has already generated various fractions of the reasoning trace. SDPO and OPSD assume that post-hoc rationalizations of the correct answers are easier to generate and equally helpful for the student model to learn from. However, prior work [2] has shown that post-training using post-hoc explanations does not induce performance gains compared to training on the full reasoning trace. On the other hand, CRISP assumes that LRMs' reasoning behavior is instruction following, and thus relies on the teacher following its concise reasoning instruction.
Graph coloring baselines: Qwen3-4B
| Dataset | Metric | OPSD | CRISP | SDPO | MSD (70%) | Base (think) | Base (no-think) |
|---|---|---|---|---|---|---|---|
| GC-ER χ3-4 | Accuracy (%) | 89.8 ± 1.4 | 85.4 ± 0.4 | 92.3 ± 0.6 | 97.5 ± 1.0 | 91.9 ± 0.6 | 29.2 ± 1.4 |
| Tokens | 8,253 ± 238 | 4,984 ± 43 | 6,369 ± 120 | 3,274 ± 129 | 6,185 ± 237 | 1,576 ± 85 | |
| GC-ER χ6 | Accuracy (%) | 37.8 ± 1.0 | 27.7 ± 1.1 | 46.7 ± 1.6 | 79.5 ± 1.5 | 45.4 ± 2.5 | 13.7 ± 2.1 |
| Tokens | 14,927 ± 190 | 12,650 ± 108 | 12,439 ± 487 | 6,906 ± 196 | 15,489 ± 643 | 1,254 ± 166 | |
| GC-BA χ6 | Accuracy (%) | 60.6 ± 1.3 | 42.0 ± 1.7 | 65.7 ± 1.8 | 88.3 ± 1.0 | 62.1 ± 2.6 | 18.4 ± 1.9 |
| Tokens | 11,675 ± 158 | 11,075 ± 295 | 10,186 ± 366 | 5,185 ± 135 | 12,134 ± 298 | 1,566 ± 445 |
Graph coloring baselines: Qwen3-8B
| Dataset | Metric | OPSD | CRISP | SDPO | MSD (50%) | Base (think) | Base (no-think) |
|---|---|---|---|---|---|---|---|
| GC-ER χ3-4 | Accuracy (%) | 94.9 ± 0.6 | 86.8 ± 2.7 | 94.3 ± 1.0 | 98.5 ± 0.8 | 95.5 ± 0.3 | 33.2 ± 1.8 |
| Tokens | 5,914 ± 207 | 4,127 ± 207 | 5,253 ± 69 | 3,766 ± 233 | 5,752 ± 131 | 1,149 ± 28 | |
| GC-ER χ6 | Accuracy (%) | 61.3 ± 0.1 | 38.6 ± 3.2 | 54.4 ± 1.9 | 85.6 ± 0.6 | 64.4 ± 2.5 | 13.5 ± 0.5 |
| Tokens | 12,603 ± 194 | 9,839 ± 87 | 11,855 ± 266 | 8,289 ± 181 | 14,434 ± 12 | 1,217 ± 23 | |
| GC-BA χ6 | Accuracy (%) | 76.0 ± 1.9 | 57.4 ± 1.3 | 76.5 ± 1.9 | 93.3 ± 1.0 | 81.5 ± 0.4 | 18.4 ± 1.9 |
| Tokens | 9,432 ± 405 | 8,255 ± 273 | 8,470 ± 290 | 5,509 ± 164 | 10,298 ± 288 | 1,145 ± 22 |
(b)
"Countdown no token savings clarification"
In the case of Countdown, no token savings on average were observed because the base student has degenerate behavior on problems where it produces incorrect responses. The base student keeps looping until it exhausts the maximum token generation limit. This increases the average token count, as shown by the Pareto plots.
We have now provided a detailed analysis of the various model responses based on correctness. Our results demonstrate that the MSD variants ({0.3,0.5,0.7}) have similar accuracy and are more inference efficient compared to the base think mode on ID and OOD test sets when the response is correct. Please refer to Appendix C.1 for the updated results.
Q1 – How is the "no-think mode" handled for the student model?
All the -masked students act in no-think mode, so after the chat template is applied, an empty <think></think> block gets appended after the question. In Appendix E, we have added the student prompt with the chat template (highlighted in blue).
Yes, the teacher context for >0 includes <think> followed by the prefix z<k. We have added the teacher prompt with the chat template in Appendix E (highlighted in blue).
Even with the no-think chat template, the trained -masked students ({0.0,0.3,0.5,0.7}) produce the end-think tag (</think>) in their responses after training. During training, the teacher model acts in think mode and is only conditioned on its fraction of the reasoning trace, so its probability distribution has a tendency to produce the </think> tag. Thus, with MSD training, the -masked students also learn to generate an additional </think> tag.
Q4 – Is "response length" the full output including the solution?
Yes, the response length includes the full output: the chain-of-thought tokens followed by the solution tokens. We have also added results on the number of responses truncated across various models and datasets. These analyses are in line with the response length distribution reported in Appendix D. We also see that the truncation rate of the base no-think student is highest for Countdown due to the degenerate behavior mentioned above (in response to W2/Q2).
Q5 – What fraction of training problems survive after rejection sampling?
The GSM8K dataset has 7,473 training problems, and we generate 10,000 GC-ER χ3-4 instances for the graph coloring dataset. We generate the responses from the teacher models on these training instances. During teacher response generation, we set the maximum response length to 8K for GSM8K and 16K for graph coloring.
For constructing our MSD training sets, we filter the teacher-generated rollouts and keep only the data points that have a correct answer in the response. Our filter removes the responses that have incorrect answers or incomplete generations.
The table below shows the number of data points used in student training for both datasets and models.
| Dataset | Qwen3-4B | Qwen3-8B |
|---|---|---|
| GSM8K | 7,406 / 7,473 | 7,081 / 7,473 |
| Graph coloring | 9,648 / 10,000 | 9,222 / 10,000 |
Training without rejection sampling. We believe that MSD-style training will provide efficiency gains even when student models are trained on datasets without rejection sampling, as our framework does not require any correctness signal during training. We might not see performance gains over the teacher model, as seen on the current GC OOD test sets, but the -trained students can learn to mimic the teacher's behavior while producing shorter reasoning traces.
Thank you again for a detailed review. We hope that the clarifications provided address your concerns. We would be grateful if you would consider raising your score.
References
[1] Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, et al. Reinforcement learning via self-distillation. arXiv preprint arXiv:2601.20802, 2026.
[2] Bhambri, Siddhant, Upasana Biswas, and Subbarao Kambhampati. "Interpretable traces, unexpected outcomes: Investigating the disconnect in trace-based knowledge distillation." Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.
Disentangling the effect of masking and self-distillation
W1 "The "internalization" claim is weakly supported, and the paper's own α=0 baseline undercuts the contribution in math."
The α=0 results on math were a surprising result for us too. After the initial submission, we did a thorough code audit and found that there was a bug in the implementation affecting the α=0 experiments. We had implemented the masked self-distillation framework by building on SDPO's [1] repository. SDPO's existing implementation adds a student-generated correct response to the teacher's context along with the question prompt. Due to the bug, when we passed α=0, the code used SDPO's original implementation, which caused the correct student rollout to also be appended to the teacher's context. In contrast, in the α=0 setting, the teacher should have been conditioned only on the question. We have fixed the bug and reported the corrected numbers. The earlier α=0 results reported in the paper should be read as a comparison with SDPO-style training. We have updated the Pareto curves and tables in the manuscript with the correct numbers.
The updated results show how much self-distillation, on its own, can drive the performance improvement and compression. We observe that α=0 training modifies the student's behavior to mimic the teacher. This improves the student's performance but does not induce efficient inference. The prefix masking drives the compression gains observed for α∈0.3,0.5,0.7,1
Qwen3-4B
Qwen3-8B