acceptodds
Under review as a conference paper at ICLR 2027

Learning When to Continue and When to Stop: Termination-Aware OPSD

Abstract

On-Policy Self-Distillation (OPSD) improves language model reasoning by distilling from a frozen teacher initialized identically to the student and conditioned on privileged information. However, we identify an underexplored termination mismatch in OPSD: the student fails to reliably learn when to continue and when to stop. This mismatch manifests in two opposing failure modes—premature termination, where reasoning stops before sufficient progress is made, and repetition-driven over-generation, where the model continues generating after reasoning has stalled. Our analysis shows that the teacher–student discrepancy on EOS grows during OPSD training, while length-capped trajectories are frequently dominated by unproductive repetition. To address this issue, we propose TA-OPSD (Termination-Aware On-Policy Self-Distillation), which augments the original OPSD objective with two targeted objectives. Stop/Continue Distillation addresses premature termination by explicitly aligning the teacher and student on whether reasoning should continue, while Repetition-Aware Unlikelihood Training addresses repetition-driven over-generation by discouraging repetitive continuations in the student’s failed trajectories. The resulting objective improves termination behavior from complementary directions: continuing when further reasoning is needed and stopping when reasoning becomes unproductive. Experiments on mathematical reasoning tasks with Qwen3 (1.7B/4B/8B) show that TA-OPSD consistently outperforms OPSD and other baselines across all three model scales. Beyond the performance gains, ablation and behavioral analyses show that the two proposed objectives effectively mitigate their corresponding termination failures, reducing both premature termination and repetition-driven over-generation.

Primary area:
reinforcement learning
Keywords:
OPSD, Generation Termination, Unlikelihood Training

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_EpQ1OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates termination mismatch in On Policy Self Distillation (OPSD), where language models may terminate reasoning prematurely or continue generating repetitive content after reasoning has stalled. To address these issues, the authors propose Termination Aware OPSD (TA-OPSD), which introduces two auxiliary training objectives: Stop/Continue Distillation (SC), which aligns the student and teacher's termination probabilities, and Repetition Aware Unlikelihood Training (UL), which penalizes repetitive continuations. The method is evaluated on Qwen3 models of three different sizes (1.7B, 4B, and 8B) in No Thinking mode across three mathematical reasoning benchmarks. The authors report improvements in reasoning accuracy and termination behavior compared with OPSD and other baselines, and conduct ablation studies on the 1.7B model to examine the contributions of the two components.

Critical strengths

The authors provide empirical analyses of teacher and student EOS probability discrepancies and repetitive generation patterns.

Critical weaknesses

Prior studies, such as Demystifying OPD, When EOS Tokens Disagree, and Mismatch Matters, have already investigated length inflation, repetitive generation, and termination related discrepancies in on policy distillation. The proposed SC objective also shares conceptual similarities with existing EOS alignment approaches, while UL builds on established unlikelihood training techniques. The paper would benefit from a clearer differentiation of its methodological contributions from these closely related works.

All experiments are conducted using Qwen3 models in No Thinking mode, despite the paper's focus on reasoning termination. It remains unclear whether the proposed approach generalizes to explicit Thinking mode, where terminating the reasoning phase and terminating the final response may involve different decisions. This limits the generalizability of the findings to reasoning models with explicit thinking stages.

The average accuracy improvement over OPSD decreases from 4.84 percentage points for Qwen3 1.7B to 3.73 for 4B and only 1.32 for 8B. This trend raises concerns about whether the proposed approach remains effective as model capacity increases. The paper provides limited analysis of why the improvements diminish with model scale.

The ablation experiments examining the individual contributions of SC and UL are conducted only on Qwen3 1.7B. Although the full method is evaluated across three model sizes, the experiments do not establish whether both components remain beneficial for larger models. This is particularly relevant given the smaller overall improvements observed on Qwen3 8B.

The paper identifies premature termination partly by allowing a privileged teacher to continue from incorrectly terminated student responses. However, since the teacher has access to reference answers, its ability to recover correct solutions does not necessarily demonstrate that the original student could have benefited from additional reasoning. Consequently, the evidence does not fully establish that premature stopping, rather than insufficient reasoning capability, is the primary cause of these failures.

Questions for authors

Since the original full vocabulary KL objective already includes EOS probabilities, could the authors compare SC against simpler alternatives, such as increasing the EOS token loss weight or explicitly aggregating equivalent EOS tokens? Such comparisons would help establish whether SC provides benefits beyond strengthening existing termination supervision.

Could the authors conduct controlled experiments in which the original student continues generating after its EOS token is removed, or its EOS probability is selectively reduced at the same prefix? Measuring whether these interventions improve answer accuracy would provide more direct evidence that premature termination contributes causally to reasoning failures.

1 comment
AuthorsOpenReview ↗

Response to Reviewer EpQ1

We thank the reviewer for the constructive feedback. We address W1–W5 and Q1–Q2 below.

W1/Q1. Related work and EOS controls

These works address related phenomena but differ in method or goal. Demystifying OPD, which we implement as a baseline, uses reference constraints and rollout mixing to curb length inflation and repetition saturation during training. TA-OPSD uses SC for premature stops and UL for repetitive over-generation. When EOS Tokens Disagree pools equivalent EOS tokens to reconcile different teacher and student stop tokens. Our OPSD teacher and student share a base model and vocabulary. SC distills the full STOP/CONTINUE decision at every valid prefix, including whether to continue reasoning. Mismatch Matters uses probability shaping and teacher top-KK injection to correct general token excess and deficit. TA-OPSD targets termination through stop supervision and penalties on detected repeats. We use the negative-distillation effect of unlikelihood loss to train on repeated spans replayed with their original prefixes, a use distinct from prior UL work. Jointly addressing when to stop and what to suppress after reasoning stalls distinguishes TA-OPSD. (“Demystifying OPD”) (“When EOS Tokens Disagree”) (“Mismatch Matters”) (“Unlikelihood Training”)

We tested the full binary stopping objective against two enhanced EOS controls on Qwen3-1.7B with thinking off. All three shared initialization, the base OPSD protocol, and 100 steps, with UL off. Auxiliary weights used the same initial logit-gradient norm rule. Let EE be the set of two stop tokens and q,pq,p the teacher and student distributions.

  • EOS-WEIGHT adds separate stop-token contributions ∑e∈Eqelog⁡(qe/pe)\sum_{e\in E}q_e\log(q_e/p_e) to the base loss.
  • EOS-POOL pools stop probabilities into qE=∑e∈Eqeq_E=\sum_{e\in E}q_e and pE=∑e∈Epep_E=\sum_{e\in E}p_e and adds the STOP term qElog⁡(qE/pE)q_E\log(q_E/p_E) to the base loss. SC supervises both STOP and CONTINUE.
Method (step 100)Mean accuracy (3 benchmarks)Short errors
EOS-WEIGHT19.86%261/1,200
EOS-POOL20.74%266/1,200
SC27.08%68/1,200

Short errors are incorrect responses that stop naturally within 1,024 tokens with no repetition flagged by UL's mechanical rules. SC improves mean accuracy by 7.22 and 6.34 percentage points over the two controls, respectively, with fewer short errors. Thus, the full STOP/CONTINUE objective gave higher accuracy and a lower short-error rate than either EOS control.

W5/Q2. Can the original student recover from short errors?

On AIME24, AIME25, and AMC23, the natural incorrect-stop rate is the share of responses stopping naturally within 1,024 tokens with an incorrect or unextractable answer. Each method generated 1,200 responses. For Qwen3-1.7B, the equally weighted rate fell from 12.18% (OPSD) to 3.52% (TA-OPSD), down 8.66 percentage points.

To test whether the original student could continue solving, we selected 42 held-out OpenThoughts problems (§3) with correct natural stops from Base and shorter, incorrect responses from OPSD. We kept the original OPSD checkpoint, question, and full pre-EOS prefix, with no reference answer. We masked both EOS tokens only at the original stop and restored the decoding policy thereafter (four random seeds per problem, up to 4,096 new tokens). 33/168 continuations were correct, with at least one correct continuation on 13 questions. With model, derivation, and budget fixed, intervening only at the original stop recovered some errors. This directly shows that some post-OPSD short errors arose from stopping while the student could still answer correctly.

W2. Explicit thinking mode

On Qwen3-1.7B, we enabled thinking for both teacher and student during distillation and at evaluation. Compared with OPSD, TA-OPSD raised three-benchmark mean accuracy from 50.63% to 61.83%. Capped responses fell from 370/1,200 to 188/1,200, and errors that matched the repetition rules and hit the cap fell from 363 to 160. Repetition with budget exhaustion persists with thinking on, and TA-OPSD reduced these failures here.

W3–W4. Model scale and component ablations

On 460 shared held-out diagnostic questions, OPSD gave 114, 53, and 45 short errors at 1.7B, 4B, and 8B. Of the 35 questions 1.7B answered incorrectly without a derivation, 4B and 8B solved 20 and 28, respectively. This is consistent with larger models deriving more on their own, leaving fewer short errors for stopping supervision to address.

The table gives three-benchmark macro accuracy with thinking off.

ModelOPSDOPSD+SCOPSD+ULTA-OPSD
Qwen3-4B35.88%39.10%36.23%39.61%

Adding SC or UL alone improved accuracy on the larger 4B model.

Reviewer_p5KhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work studies termination behavior in on-policy self-distillation (OPSD). It argues that standard OPSD exhibits a “termination mismatch” problem with two opposite failure modes: premature termination and repetition-driven over-generation. To solve such problem, theyf proposed TA-OPSD method adds two auxiliary objectives to the base full-vocabulary OPSD loss. Stop/Continue Distillation (SC) collapses the vocabulary into STOP versus CONTINUE and distills the teacher’s probability mass over these two classes. Repetition-Aware Unlikelihood Training (UL) mines repeated spans from failure trajectories and applies an unlikelihood loss to those continuations under their original prefixes. Experiments show consistent improvements over OPSD. The paper also reports lower capping rates and improved teacher–student EOS alignment.

Critical strengths
  1. The paper isolates two practically important but opposite termination failures and treats them with targeted objectives.

  2. The empirical study is reasonably systematic within the chosen setting. The method is tested at three Qwen3 scales and improves over OPSD on all nine model–benchmark combinations.

  3. The method does not require an auxiliary detector at inference time. Both interventions are training-side modifications, which makes the approach operationally simple.

Critical weaknesses
  1. Lack evidence of research motivation. In section 3, the authors introduced the two opposite failure mode. However the evidence to prove such problem does exist is insufficient, especially in repetition-driven over-generation. Though the appendix has given the repeat case which can be a certain evidence and response capping does not mean repeating tokens, the authors still need to report statistical evidence such average repeat rate or other evaluation scores to prove the repetition problem is existing.

  2. Research focus shift. TA-OPSD is designed to solve the termination problem in OPSD, so the reported data should also include the scores that could represent the extent of termination problem such as early stop rate or repeat rate instead of the accuracy in main result table.

Questions for authors
  1. Following weakness 2, could authors provide more evaluation scores matching the termination problem for more robust result?

  2. We know training models with a larger number of parameters needs higher resource consumption, but the average results in table 1 make me skeptical about whether this algorithm can be effective on models with a larger number of parameters, because as shown in the table, the average score is steadily declining as the number of parameters increases.

1 comment
AuthorsOpenReview ↗

Response to Reviewer p5Kh

We thank the reviewer for the constructive feedback. We address W1–W2 and Q1–Q2 below.

W1–W2/Q1 | Natural incorrect stops and repetition with budget exhaustion. On AIME24, AIME25, and AMC23, we measured the natural incorrect-stop rate within 1,024 tokens, defined as the proportion of responses that stop naturally within 1,024 tokens and either give an incorrect answer or yield no extractable answer. In each run, each method generated a total of 1,200 responses across the three benchmarks. The table below uses the Qwen3-1.7B checkpoints reported in Table 1, with rates averaged equally across the three benchmarks.

OPSDTA-OPSDChange
12.18%3.52%−8.66 percentage points

To test whether these short errors could be recovered through continuation, we selected 42 problems on which Base answered correctly and stopped naturally, while OPSD gave a shorter, incorrect response from the held-out OpenThoughts diagnostic set in §3 of the submission. We used the OPSD checkpoint that produced the original response and preserved the original problem and the complete prefix before EOS. Without providing a reference answer, we masked the two EOS tokens only at the original stopping position and then restored the original decoding policy. We used four random seeds per problem and allowed up to 4,096 additional tokens. Of the 168 continuations, 33 gave the correct answer. Even with the intervention limited to the original stopping position, the student could continue to a correct answer on some of the problems. This directly shows that some short, incorrect responses under OPSD can be recovered by continuing generation past the original stopping position, demonstrating that the student model does sometimes terminate its output prematurely after OPSD training.

Repetition with budget exhaustion. We used the same text-matching and token-periodicity rules as in UL mining. A response counts toward this metric only if it both matches a repetition rule and exhausts its effective generation budget. Each group contains 1,200 responses.

ModelOPSDTA-OPSD
Qwen3-1.7B7745
Qwen3-4B203179
Qwen3-8B9561

In the original evaluations, 77/85, 203/209, and 95/101 of the capped OPSD responses at 1.7B, 4B, and 8B, respectively, met the repetition criteria. Given the high overlap at all three scales, the paper uses the easier-to-measure capping rate to reflect generation failures involving both repetition and capping, capturing budget exhaustion accompanied by repetition. This lets us efficiently track this aspect of the method's performance while reducing the evaluation cost of identifying repeated spans again.

Q2 | Results across model sizes. In Table 1, TA-OPSD's absolute mean accuracy increases from 27.82% to 39.61% and then to 53.87% as model size increases. The gains over OPSD at the same model sizes are +4.84, +3.73, and +1.32 percentage points, respectively.

We evaluated all three OPSD model sizes on the same set of 460 OpenThoughts diagnostic problems, held out from the corresponding training runs. We counted incorrect responses that stopped naturally within 1,024 tokens and did not match any mechanical repetition rule. The proportions of such responses were 114/460 (24.78%), 53/460 (11.52%), and 45/460 (9.78%) for the 1.7B, 4B, and 8B models, respectively. Among the 35 problems on which 1.7B gave an incorrect answer without developing a derivation, 4B and 8B answered 20 and 28 correctly, respectively. This trend is consistent with the explanation that larger models can reason further on more problems and consequently make fewer such short errors under OPSD. Short incorrect stops still occur at 4B and 8B. Because the larger models are more capable, these short errors become relatively less frequent after OPSD training but still occur. TA-OPSD remains more accurate than OPSD at larger scales, so our algorithm also applies to larger models.

Reviewer_pKYhOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper identifies two failure modes of OPSD—premature termination and repetition-driven over-generation—and proposes TA-OPSD, which introduces Stop/Continue Distillation and Repetition-Aware Unlikelihood Training to address them. Experiments on Qwen3-1.7B/4B/8B show improvements in mathematical reasoning accuracy and termination behavior over standard OPSD.

Critical strengths
  • The paper identifies two interesting and complementary failure modes in OPSD, providing a reasonable motivation for improving termination behavior.

  • The proposed objectives are simple and intuitive, directly targeting premature termination and repetitive generation with relatively little additional training overhead.

  • Experiments across three model scales show consistent improvements over OPSD, supported by ablation studies and analyses of generation behavior.

Critical weaknesses

[w1] Superficial treatment of premature termination

The authors identify interesting failure modes, but prior works [1, 2, 3] have already analyzed underlying mechanisms of reasoning degradation, including suppressed uncertainty, leakage, or self-correction. This paper instead takes a rather superficial approach by focusing on controlling termination without examining these mechanisms.

 

[w2] Questionable SFT and GRPO baselines

Overall, most baselines show little improvement over the base models, except for OPSD and TA-OPSD on Qwen3-8B. It is somewhat puzzling that several baselines even underperform the base models, raising concerns about whether they were adequately trained and optimized.

  • SFT: Most SFT results are worse than the base models. In contrast, [4] reports that the same OpenThoughts SFT on the same Qwen3-8B (non-thinking) improves AIME24 accuracy from 26.9% to 65.1%, whereas this paper reports a drop from 29.72% to 14.17%. The Thinking Machines Lab's OPD blog [5] also reports improved SFT performance on Qwen3-8B-Base with OpenThoughts-3. The authors should clarify the SFT setup and explain why SFT was trained for only 100 steps, compared with 500 for GRPO, and ideally reproduce a more reasonable baseline.
  • GRPO: GRPO also shows little improvement over the base models. Prior work [1] reports substantial gains with non-thinking GRPO on DAPO-Math-17k. Larger batch sizes (the reported batch size of 32 seems rather small), longer training, convergence curves, and comparisons on RLVR-focused datasets (e.g., DAPO-Math-17k and DeepScaleR) would be needed.

 

[w3] Limited evaluation across thinking modes and model families

[4] shows that OPSD performance is highly sensitive to the teacher's thinking mode. However, all experiments are conducted on Qwen3-1.7B/4B/8B in thinking-off mode. It remains unclear whether the proposed improvements persist under thinking-enabled settings, where extended reasoning and self-correction are particularly important. Experiments with all thinking-enabled configurations, as well as other model families such as DeepSeek and OLMo, would substantially strengthen the generalizability of the findings.

 

[w4] Insufficient evidence for long-term generation stability

The paper emphasizes improved generation stability, yet OPSD and TA-OPSD are trained for only 100 optimizer steps, compared with 500 for GRPO. Prior work [1] reports reasoning degradation and unstable generation behavior during continued opsd training. Thus, the observed improvements over 100 steps do not necessarily establish long-term stability.

The authors should clarify whether Table 1 reports the best or final checkpoints. More importantly, extending OPSD and TA-OPSD to 1,000 steps, or training with rollout lengths longer than 1,024 tokens, and reporting accuracy and termination statistics every 100 steps would provide stronger evidence for the claimed stability improvements.

 

Note: I understand that this paper largely follows the experimental setup of the original OPSD work. However, simply adopting prior experimental settings does not absolve the authors of the responsibility to validate the baselines, especially when their reported performance appears questionable. I am also aware that [4] appeared after the ICLR submission deadline, so I am not suggesting that it should be used to challenge the novelty or other contributions of this work. Nevertheless, the substantial discrepancy between the two studies' experimental results warrants clarification.

 

References

[1] Kim, Jeonghye, et al. "Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?." arXiv preprint arXiv:2603.24472 (2026).

[2] Kaur, Simran, et al. "Rethinking On-Policy Self-Distillation for Thinking Models." arXiv preprint arXiv:2607.05184 (2026).

[3] Yang, Chenxu, et al. "Self-distilled rlvr." arXiv preprint arXiv:2604.03128 (2026).

[4] Adhinarta, Jason Ken, et al. "Does self-distillation improve reasoning?." arXiv preprint arXiv:2610.18613 (2026).

[5] Thinking Machines Lab, On-Policy Distillation (https://thinkingmachines.ai/blog/on-policy-distillation/)

Questions for authors
  1. Does Table 1 report the best or final checkpoint? Could the authors train TA-OPSD, OPSD, and GRPO for 1,000 steps and report evaluation results every 100 steps to show how performance evolves throughout training?
  2. Why is SFT trained for only 100 steps? In my view, 100 optimizer steps with a batch size of 32 is considerably shorter than typical SFT training, and I would expect longer training to yield better performance. Could the authors also report SFT results over 1,000 steps?
  3. How does TA-OPSD perform with Qwen3 thinking enabled and on other large reasoning models, such as DeepSeek and OLMo?

I would be willing to raise my score if the authors clearly demonstrate TA-OPSD's superiority through evaluation curves extending to 1,000 training steps (or in increased batch size setting), show that the findings generalize to thinking-enabled settings or other model families, and provide a coherent explanation that reconciles their observations with the mechanistic analyses of similar phenomena in prior work.

2 comments
AuthorsOpenReview ↗

Response to Reviewer pKYh (Part 1/2)

Thank you for your comments on the training budget, baselines, and scope of generalization. We address W1–W4 and Q1–Q3 below.

W4/Q1 | Short-run training stability and checkpoints

We study whether adding two termination objectives improves generation behavior and problem-solving accuracy within the original OPSD budget of 100 optimization steps. We examine two failure modes in termination behavior, premature termination and repetition-driven over-generation. In Table 1, OPSD and TA-OPSD use the same checkpoint step at each model size, with steps 75, 50, and 100 for 1.7B, 4B, and 8B, respectively. We will explicitly list these steps in the table note.

The 1,000-step experiment tests the stability of the entire training recipe under a larger budget. Our method focuses on inference stability after short training. Stability over longer training horizons falls outside the scope of our study. In fact, the step counts for the baselines, including SFT, were chosen to ensure a fair comparison with the same total amount of training. This is consistent with the original OPSD paper and is precisely the advantage of methods such as OPSD over SFT. They can use a smaller training budget to achieve the results that SFT achieves with a larger one. The direct test of our contribution is the comparison of OPSD and TA-OPSD at the same model scale and under the same training budget. (“Self-Distilled Reasoner”)

W2/Q2 | SFT and GRPO baselines

The 100-step SFT and 500-step GRPO settings follow the publicly released configurations of the original OPSD paper. This is a commonly used training setup in OPSD. The original paper observed lower mean math accuracy for SFT than for Base at all three Qwen3 scales, including 61.2% → 58.6% at 4B. Air-OPD also reports 61.2% → 58.2% in a separate 4B setting. Kim et al. likewise found that even when all SFT training trajectories reach correct answers, training on trajectories generated with guidance from reference solutions can yield scores below the starting model on all four math benchmarks. Determining why baseline performance drops therefore requires controlling the training data, training process, and evaluation setup. Accuracy drops relative to Base across different baselines are a common problem with complex causes that fall outside the scope of this paper. (“Self-Distilled Reasoner”, “Air-OPD”, “Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?”)

The 65.1% you point out is indeed the AIME24 score reported by Adhinarta et al. for Qwen3-8B with thinking disabled. It comes from a checkpoint selected from a 200-step SFT run. Adhinarta et al. explain in their repository that their SFT actually uses the chains of thought in the OpenThoughts conversations column. Our study uses the corrected implementation from the original OPSD repository, with formatted reference solutions from the solution column as training targets. The substantial differences between the training setups prevent a direct comparison. (“Does Self-Distillation Improve Reasoning?”, Adhinarta et al.'s training configuration and SFT notes, SFT fix in the original OPSD repository)

Thinking Machines' 60% accuracy result used 400,000 prompts and full-parameter fine-tuning. Measured by the number of prompt instances processed during training, this is 400,000 ÷ 3,200 = 125 times the scale of our SFT, and their fine-tuning method differs from our LoRA fine-tuning. This result reflects a different initialization, training data volume, and fine-tuning method. These differences preclude a fair comparison, and it is to be expected that their SFT accuracy is higher than our baseline. (“On-Policy Distillation”)

Kim et al.'s GRPO uses DAPO-Math-17k, with 256 problems per batch, eight sampled responses per problem, and full-parameter fine-tuning. They train on about 25 times as much data as we do, and we use a different fine-tuning method. These differences preclude a direct comparison with our GRPO results, and it is to be expected that their GRPO accuracy is higher than our baseline. In fact, OPSD itself is more efficient than GRPO. This is why we make a fair comparison under similar training budgets rather than simply giving each method enough training. (“Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?”, “Self-Distilled Reasoner”)

AuthorsOpenReview ↗

Response to Reviewer pKYh (Part 2/2)

W1 | In-depth study of stopping decisions

Thank you for your suggestions on our work. Kim et al. study how rich teacher context suppresses the expression of uncertainty, while our work isolates the stopping decision at a specific prefix and its causal impact. Kaur et al. study impaired self-correction at high-entropy branching points in thinking models, while we examine premature termination and repetition-driven over-generation with thinking disabled and verify reductions in failures that involve both repetition and budget exhaustion with thinking enabled. Yang et al. use RLVR environment rewards to address information leakage and update direction, while our work retains a pure OPSD framework, requires no environment rewards, and directly optimizes STOP/CONTINUE and repetitive continuations. These works therefore differ substantially from ours in both research questions and methods. In fact, termination goes beyond superficial length control. It is a policy action that determines whether a derivation can be completed. Our study forms a complete research cycle, from identifying two opposing failure modes through controlled causal interventions and targeted objective design to validation across thinking modes and model families. (“Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?”, “Rethinking On-Policy Self-Distillation for Thinking Models”, “Self-Distilled RLVR”)

W3/Q3 | Thinking mode and cross-model-family evaluation

We added an experiment with Qwen3-1.7B in which thinking is enabled for both teacher and student during training and is also enabled during evaluation, along with a same-protocol comparison on Olmo-3-7B-Instruct. The table below reports equally weighted mean accuracy at step 100 across AIME24, AIME25, and AMC23.

Model and settingOPSDTA-OPSDGain
Qwen3-1.7B, thinking enabled for teacher and student during training and enabled during evaluation50.65%60.49%+9.84 percentage points
Olmo-3-7B-Instruct59.24%62.18%+2.94 percentage points

In the same thinking-enabled comparison, TA-OPSD reduced capped responses from OPSD's 384/1,200 to 199/1,200. The two experiments provide additional evidence with thinking enabled and across model families, respectively.

Reviewer_r87SOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper examines a "termination mismatch" in On-Policy Self-Distillation (OPSD) for mathematical reasoning, where the student model either stops prematurely before completing a derivation or continues generating repetitive text until reaching the token limit. To address these two failure modes, the authors propose TA-OPSD, which adds two auxiliary objectives to the standard OPSD loss: (1) Stop/Continue Distillation (L_{SC}), which groups the vocabulary into binary STOP and CONTINUE classes and minimizes the binary KL divergence between teacher and student, and (2) Repetition-Aware Unlikelihood Training (L_{UL}), which replays offline-mined repetitive prefixes generated by the initial base model and penalizes the repeated continuation spans. The method is evaluated on Qwen3 (1.7B, 4B, and 8B) in "no-thinking" mode across AIME24, AIME25, and AMC23.

Critical strengths

Clear problem motivation and diagnostic analysis: Categorizing OPSD length anomalies into premature termination and repetition-driven over-generation is intuitive. The diagnostic experiment in Section 3—where removing the EOS token from short, incorrect student responses and allowing the privileged teacher to continue recovers the correct answer in 36.6% of cases—provides direct empirical motivation for studying stopping behavior in self-distillation.

Critical weaknesses
  1. Mismatch between training and evaluation length limits (1,024 vs. 32,768 tokens): According to Appendix C, training rollouts are capped at 1,024 tokens (averaging ~779 tokens), whereas evaluation allows up to 32,768 tokens with actual average lengths of 2,412 (Base) to >5,200 tokens (Figure 3b). Because solving these math problems typically exceeds 1,024 tokens, most training rollouts are truncated before natural completion; thus, within the 1,024-token window, the teacher predominantly signals CONTINUE rather than STOP, providing no on-policy supervision where long-horizon stopping actually occurs. This aligns with Figure 5, where adding L_{SC} alone (OPSD+SC) increases the capping rate above standard OPSD at steps 25–50 (to nearly 12%), indicating that L_{SC} mainly suppresses EOS rather than calibrating long-horizon stopping.
  2. Performance degradation across all baselines on Qwen3-4B: Under the chosen setup (disabling Qwen3's thinking mode, 100 steps, temperature 1.1, and the 1,024-token cap), all five post-training baselines on Qwen3-4B (Table 1) perform worse than the untuned Base model (37.11%): SFT (29.10%), GRPO (36.78%), OPSD (35.88%), Stable-OPD (36.30%), and RLCSD (36.46%) all degrade, while only TA-OPSD improves (39.61%). A similar trend holds on 1.7B, where SFT and GRPO fall below the Base model and OPSD/Stable-OPD barely change, raising concerns that the training configuration is suboptimal for the baselines.
  3. Inconsistent numbers across sections and tables: For Qwen3-1.7B, OPSD and TA-OPSD share identical accuracies (22.99% and 27.82%) across Tables 1–3, but their reported lengths and capping rates differ across pages. Section 5.2 and Figure 3 report capping rates of 8.89% (OPSD) and 4.31% (TA-OPSD) with mean lengths of 5,539 and 5,249, whereas Table 3 reports capping rates of 7.55% and 4.77% with lengths of 5,218.87 and 5,197.91. Even if this reflects an undocumented switch between macro- and micro-averaging across benchmarks, Section 5.3 and Figure 5 directly compare OPSD's 8.89% (from Section 5.2) against TA-OPSD's 4.77% (from Table 3) in the same comparison.
  4. Missing reproducibility details in the appendices: The main text defers key procedures to the appendices that are omitted there. Section 4.3 points to Appendix C for the exact negative selection procedure, but Appendix C only mentions that fixed rules detect spans appearing three times, without specifying matching granularity, minimum span length, or contiguity requirements. Similarly, Section 5.3 points to Appendix E for the LLM-assisted selection details, yet Appendix E.2 only notes that "GPT-5" was used offline without providing the prompt, filtering criteria, or retention rate.
  5. Limited novelty, offline replay design, and missing limitations: The method combines a binary EOS KL term with standard Unlikelihood Training (Welleck et al., 2019). Furthermore, although OPSD is an on-policy framework, L_{UL} relies on an offline static replay set generated by the untuned base model before training rather than penalizing repetitions on-the-fly in current on-policy rollouts. The paper also lacks a discussion of limitations.
Questions for authors
  1. Discrepancies in Section 5.2 vs. Section 5.3 / Table 3: Could you clarify why the mean lengths and capping rates for Qwen3-1.7B (OPSD and TA-OPSD, with accuracies of 22.99% and 27.82%) differ between Section 5.2 (OPSD: 8.89% cap, 5,539 length; TA-OPSD: 4.31% cap, 5,249 length) and Table 3 (OPSD: 7.55% cap, 5,218.87 length; TA-OPSD: 4.77% cap, 5,197.91 length)? Why do Section 5.3 and Figure 5 compare 8.89% against 4.77%?
  2. Effect of the 1,024-token training cap and baseline degradation: Given that test-time solutions average 2,400 to 5,000+ tokens, what fraction of training trajectories actually reach a natural completion (high teacher q_{\text{EOS}}) within the 1,024-token training window? Furthermore, why do all five baselines perform worse than the untuned Base model on Qwen3-4B in Table 1, and to what extent is this caused by the 1,024-token training truncation and disabling thinking mode?
  3. Evaluation in thinking mode and on-policy UL: Does the termination mismatch also occur when Qwen3's thinking mode is enabled? Also, since the repetition detection rule is deterministic, have you compared offline negative replay from the base model against applying the UL loss directly on-the-fly to the current on-policy rollouts?
  4. Missing details in Appendices C and E: What are the exact matching rules for constructing negative spans in Appendix C (e.g., minimum token length, contiguous vs. non-contiguous matching), and what prompt and filtering criteria were used for the "GPT-5" selection in Appendix E.2?
2 comments
AuthorsOpenReview ↗

Response to Reviewer r87S (Part 1/2)

We thank the reviewer for the constructive feedback. We address W1–W3 and Q1–Q2 below.

W3/Q1 | Sections 5.2–5.3 and Table 3. For Qwen3-1.7B, paragraph 3 of the original Section 5.2 reports step-100 generation behavior. Table 1's OPSD and TA-OPSD entries and Tables 2–3 use step 75.

StepCapping rate (OPSD→TA-OPSD)Mean tokens
100 (original Section 5.2, paragraph 3)8.89%→4.31%5,539→5,249
75 (Table 3)7.55%→4.77%5,218.87→5,197.91

Figs. 3 and 5 show these values. The draft left the checkpoints unclear. Section 5.3 compared step-100 OPSD (8.89%) with step-75 OPSD+SC (6.46%), then reported step-75 TA-OPSD (4.77%) relative to OPSD. Our final paper will compare at step 75 in both sections and specify it when citing Fig. 3(b,c). Step-75 capping (OPSD/OPSD+SC/OPSD+UL/TA-OPSD) is 7.55%/6.46%/2.92%/4.77%.

W1/Q2 | Stopping supervision within the training window.

Our SC strengthens the teacher signal in stop/continue classes without aiming to teach the student to stop early. This motivates SC over EOS upweighting alone, extending supervision to continuing. Within the 1,024-token window, 1,736 of 3,200 Qwen3-1.7B OPSD training rollouts (54.25%) naturally emitted a stop token. In 1,410 of these rollouts (44.06% of all rollouts), the teacher assigned a combined probability of at least 0.5 to the two stop tokens at that position, showing that the training window provides substantial STOP supervision.

Fig. 5's 12% is a y-axis tick. At steps 25/50, OPSD+SC caps at 1.50%/4.93% vs. OPSD's 2.15%/5.79%. OPSD+SC caps less than OPSD at every plotted step after 0, contrary to “LSCL_{SC} (OPSD+SC) can raise capping above standard OPSD at steps 25–50.”

Why cap at 1,024? In a separate non-thinking 1.7B comparison, changes from each run's step 0 to 100 (4,096 vs. 1,024) differed by −1.33 pp in three-benchmark mean accuracy and +6.25 pp in eval capping. The original 1.7B OPSD comparison also shows gains vary by benchmark and step, so a longer window offers no guaranteed gain. This supports our use of 1,024 tokens, a common OPSD cap (“Self-Distilled Reasoner”, “C-OPSD”, “Privileged Information”).

W2/Q2 | 4B baselines. Table 1's five Qwen3-4B baselines trail Base at the reported checkpoints. Across scales, SFT training sequences and single GRPO rollouts cap at 16,000 tokens, per the OPSD paper and repo. Our experiments found our training method correct and results model-dependent. Qwen3-4B may have some issues under off/off training that put these baselines below Base. Air-OPD shares both caps. Both studies report Qwen3-4B SFT below Base in their settings (“OPSD”, “Air-OPD”). Li et al. also found OPSD below Base with Qwen3-4B teacher and student thinking off. Four-benchmark mean Avg@8 was 19.6% for Base and 17.2%/15.6% for OPSD at 1,024/4,096-token training caps (“Diagnosing OPSD”).

An added same-protocol Olmo-3-7B-Instruct comparison gives step-100 three-benchmark macro accuracy of 59.24%/62.18% (OPSD/TA-OPSD).

AuthorsOpenReview ↗

Response to Reviewer r87S (Part 2/2)

We thank the reviewer for the constructive feedback. We address W4–W5 and Q3–Q4 below.

W5/Q3 | Thinking mode and offline replay. With Qwen3-1.7B thinking on in training (teacher and student) and evaluation, step-100 TA-OPSD vs. OPSD raised three-benchmark mean accuracy 50.65%→60.49% and cut capped responses 384/1,200→199/1,200. Thinking-on OPSD still repeats until the budget runs out. TA-OPSD markedly reduces such cases.

Your online scheme would apply unlikelihood loss to repeats detected in current on-policy rollouts. We expect substantial extra compute, and some batches may lack eligible spans. A related ablation compared offline negatives from base outputs before training and OPSD-trained outputs. The latter gave a small gain. Given the expected cost and small gap, we did not use online UL screening.

W4/Q4 | Appendices C and E.2. Appendix C uses two deterministic rules for mining negatives.

  1. Text matching. Split separately by paragraph, line, and sentence. Collapse whitespace runs to one space and trim both ends. Find the third exact match of normalized spans with ≥40 characters and ≥8 alphanumerics. Matches may be nonadjacent.
  2. Token periodicity. Detect exact repeats in the original sequence at periods of 4, 8, 16, 32, 64, or 128 tokens. Require ≥3 consecutive periods spanning ≥24 tokens. Use the third period as the negative candidate.

Align text candidates inward to whole-token boundaries and merge both sets. Keep at most one per trajectory, the earliest eligible for replay. Replay keeps the original problem and full generation prefix before the span. Negatives have ≤256 tokens, ≥4 alphanumerics, and no stop tokens. Problem prompt + prefix + 256 reserved tokens ≤20,000 tokens.

E.2 actually used GPT-5-assisted case-by-case offline screening. The relevant original instruction was “Before OPSD+SC+UL training, directly select valid negatives from those initially screened with the base model. Please inspect each original response, the three occurrences, and intervening reasoning. Keep only spans whose third occurrence repeats the same local error or a step without progress. Exclude problem restatements, useful formula reuse, valid transformations, and explicit self-correction.” We retained 5/52 Qwen3-1.7B training negative span candidates (9.62%).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response | Research Contributions, Additional Evidence, and Revision Plan

We thank the Area Chair and reviewers for their feedback. We study when a model should continue a derivation and when it should stop, a decision that directly affects problem-solving outcomes in mathematical reasoning. We identified two opposing failures in the same OPSD student. The model can stop before completing a derivation or repeatedly generate after reasoning stalls until it exhausts its budget. In a controlled experiment, we masked EOS only at the original stopping position while holding the student model, generation prefix, and continuation budget fixed. The same student could continue some short, incorrect responses to a correct answer. This diagnosis motivates two complementary objectives in TA-OPSD. SC distills the teacher’s STOP/CONTINUE decisions on student prefixes, and UL penalizes selected repetitive continuations under their original failure prefixes. TA-OPSD uses SC to align the stop and continue decisions with the teacher on student prefixes and UL to suppress selected repetitive continuations under their original failure prefixes. Both interventions take place during training, with no additional detector required at inference.

At matched model sizes and reporting checkpoints, TA-OPSD outperforms OPSD in all nine comparisons across three model sizes and three mathematical benchmarks. Mean accuracy increases by 4.84, 3.73, and 1.32 percentage points, respectively, across the three sizes. The controlled EOS intervention described above also provides direct evidence that stopping decisions affect correctness. On 42 selected problems with short, incorrect responses, we fixed the original student, the existing derivation, and the continuation budget. Masking EOS only at the original stopping position yielded a correct continuation rate of 19.64% (33/168). This shows that the same student can continue some originally incorrect short responses to a correct answer.

We also measured the rate of repetition with budget exhaustion at the checkpoints corresponding to the main table. This is the proportion of responses that both match a repetition rule used in UL and exhaust their effective generation budget. Each method generated 1,200 responses per model size.

ModelOPSDTA-OPSD
Qwen3-1.7B7745
Qwen3-4B203179
Qwen3-8B9561

The table shows the number of responses exhibiting both repetition and budget exhaustion. TA-OPSD reduced these failures on 1.7B and 4B. In the additional comparisons at step 100, we also observed accuracy gains of 9.84 and 2.94 percentage points on thinking-enabled Qwen3-1.7B and the non-thinking Olmo-3-7B-Instruct, respectively.

The reviewers identified shortcomings in how the original submission reported results, measured generation behavior, and documented the experimental procedures. We will make the following revisions in the final version.

  • Align checkpoints and metric definitions. Sections 5.2 and 5.3 mixed generation behavior results from steps 75 and 100. We will use step 75 for the corresponding cross-method comparisons and label the checkpoints in the figures and tables.
  • Provide direct evidence on termination behavior. Alongside accuracy and capping rates, we will report natural incorrect stops and repetition with budget exhaustion. We will add the same-student EOS intervention, the simpler EOS objective controls, and the thinking-enabled and cross-family results. For the behavioral metrics, we will specify the settings in which we observed improvements.
  • Complete the reproducibility details and define the scope of our conclusions. We will disclose the training budget used for each baseline, the rules for constructing UL negatives, and the offline screening procedure. We will also discuss the coverage limits of the training window and fixed negative pool. Our conclusions about stability will be limited to the evaluated short training budgets and subsequent generation behavior.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.