- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper studies on-policy self-distillation with a teacher conditioned on the question and a reference reasoning trace, while the student is conditioned only on the question. When both models score the same student-generated prefixes, the teacher assigns lower probabilities to epistemic markers such as “Wait” and “perhaps.” The proposed method masks the full-vocabulary distillation loss at positions where the student samples a token from a predefined marker set. Across three checkpoints experimented on AIME24, AIME25, and MATH-500, the authors find that the proposed method prevents the degradation observed with standard uniform distillation and outperforms a matched non-marker masking control.
Critical strengths
- Skipping a small set of loss terms substantially improves accuracy over ordinary self-distillation across all tested model–benchmark combinations (Table 2).
- The matched non-marker control accounts for mask size and several token statistics, providing useful evidence that the choice of positions matters.
- The paper is generally well written and easy to follow.
Critical weaknesses
- The empirical results shown for outperforming the base model and SFT are less convincing than evidence for avoiding degradation. The 7B gains over SFT are only 0.8 to 1.8 points.
- Also, I find that the proposed setup does not establish that marker meaning explains the benefit. They do not match the teacher’s reduction in the sampled token’s probability, and the entropy mask also avoids much of the degradation (Fig. 4). Marker counts and answer lengths alone do not measure reflection.
Questions for authors
- Does masking non-marker positions with similarly large teacher-induced probability drops recover the benefit? Which token IDs account for most masked positions?
- Why are base-model scores below the model card’s results?
4 comments
Response to Reviewer iLxD (2 / 2)
Q2: Why are the base scores lower than the model card's?
Response. Thank you for flagging this discrepancy with the model-card numbers! The gap arises from three concrete differences between our unified evaluation protocol (Appendix C, Table 9) and the official DeepSeek-R1 evaluation setup:
- Generation length cap and sampling budget: To keep evaluation tractable across all models and training ablations, we use a 16,384-token maximum generation length and
avg@12on AIME, whereas the DeepSeek-R1 model card uses a 32,768-token cap and 64 samples. As a result, longer reasoning traces that exceed 16,384 tokens are truncated before producing a final\boxed{}answer. - System prompt and prefix formatting: To maintain a uniform chat template across model families (
DeepSeek-R1-Distill-QwenandOpenMath2-Llama3.1-8B), we include a standard system prompt ("You are a helpful assistant.", Appendix D) without forcing an initial<think>\nprefix, whereas the DeepSeek-R1 evaluation protocol omits system prompts and enforces<think>\nat the start of every response. - Strict rule-based verification: We grade all outputs strictly using symbolic rule-based verification (
Math-Verify) on extracted\boxed{}answers, without fallback heuristics or LLM-assisted grading.
Crucially, all methods in our paper (Base, SFT, GRPO, Naive OPSD, and Masked OPSD) are evaluated under the exact same prompt template, 16,384-token cap, sampling seeds, and strict Math-Verify grader, ensuring an apples-to-apples comparison. We will explicitly note these protocol differences relative to the model cards alongside Table 2 and in Appendix C.
W1: Modest gains over Base and SFT on the 7B model
Response. We appreciate your careful observation! We agree that while Masked OPSD achieves clear gains over Base and SFT on the 1.5B model, its margins over Base and SFT on the 7B model are modest. Our primary, consistent finding across all nine model–benchmark configurations is that masking reverses the severe accuracy collapse of naive OPSD.
Thanks to the authors for providing additional results and analysis, including the justification for the model card score discrepancy. As a result of the authors' response, I have revised the soundness score.
Thank you very much for taking the time to re-evaluate our response and for raising your Soundness score to 3 ("no substantive concerns"). We truly appreciate your thoughtful and constructive feedback, which helped strengthen the paper.
Please let us know if there are any remaining questions or concerns we could clarify further to support a stronger overall assessment during the discussion period.
Response to Reviewer iLxD (1 / 2)
We sincerely appreciate your careful analysis and your encouraging feedback! Your probing question on probability-drop matching directly inspired a new control experiment that makes our mechanistic evidence much tighter. We hope the following responses resolve all of your questions, and we truly appreciate your support of our work.
Q1 (Part 1) / W2 (Part 1): Does masking non-markers with comparable probability drops (Δp) recover accuracy?
Response. Thank you for suggesting this control experiment! In addition to the covariates matched in Table 2, we ran a new Δp-matched non-marker control on
DeepSeek-R1-Distill-Qwen-1.5Bthat also matches the sampled token's probability drop Δpt=ptS(y^t)−ptT(y^t):DeepSeek-R1-Distill-Qwen-1.5B)Masking high-Δp non-markers remains near
Naive OPSDbecause large probability drops on non-markers primarily occur when the teacher corrects student calculation or branching errors (Figure 3). Consequently, masking them leaves epistemic markers unprotected while removing helpful error correction. We will add this control to Table 2 and Section 5.3.Q1 (Part 2): Which token IDs account for most masked positions?
Response. Thanks for asking for this concrete token-level breakdown! Across the masked positions in our 100-update run on
DeepSeek-R1-Distill-Qwen-1.5B, the top five token IDs account for 33.1% of all masked occurrences:14190(Wait, 15.6%),8365(perhaps, 5.9%),7196(maybe, 4.0%),13824(Wait, 4.0%), and1744(think, 3.6%), followed by3520(actually, 2.1%),3783(wait, 2.0%),4977(seems, 1.8%), and88190(Hmm, 1.6%). In particular, variants ofWait(14190,13824,3783) alone account for 21.6% of all masked positions. We will include this token-level breakdown in Appendix A.3.W2 (Part 2): Why does the student-entropy mask in Figure 4 avoid collapse while still trailing the marker mask?
Response. Thank you for spotting that in Figure 4 (Appendix B), the equal-budget highest-student-entropy mask stays near the base checkpoint (avoiding naive OPSD's collapse while trailing the marker mask). This follows directly from the token-level properties in Section 3.1. First, it avoids collapse because mid-sentence hedging and reconsideration tokens naturally carry high student uncertainty, so the top-entropy mask protects part of the model's reflective behavior. Second, it trails the marker mask because much of the entropy mask budget falls on uncertain calculation steps where teacher correction is beneficial rather than on epistemic markers (Figure 3). We will also make the summary of Figure 4 in Section 6 precise so that it clearly distinguishes the entropy mask from the random mask.
W2 (Part 3): Do marker counts measure reflection?
Response. We appreciate this careful methodological distinction! Tracking epistemic markers as indicators of reflection initiation is a common, lightweight practice in the reasoning literature (e.g., DeepSeek-R1 and Kim et al., 2026a, Appendix E.5). At the same time, we agree that lexical marker counts and response lengths are surface-level proxies rather than direct measures of substantive verification or self-correction, as an individual marker can occasionally act as a conversational pause rather than genuine reflection (Appendix G distinguishes these cases qualitatively). Following your helpful point, we will explicitly acknowledge this limitation in Section 6 and tighten our wording throughout the paper to refer precisely to "epistemic marker usage" rather than equating marker counts directly with reflection depth.