- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates termination mismatch in On Policy Self Distillation (OPSD), where language models may terminate reasoning prematurely or continue generating repetitive content after reasoning has stalled. To address these issues, the authors propose Termination Aware OPSD (TA-OPSD), which introduces two auxiliary training objectives: Stop/Continue Distillation (SC), which aligns the student and teacher's termination probabilities, and Repetition Aware Unlikelihood Training (UL), which penalizes repetitive continuations. The method is evaluated on Qwen3 models of three different sizes (1.7B, 4B, and 8B) in No Thinking mode across three mathematical reasoning benchmarks. The authors report improvements in reasoning accuracy and termination behavior compared with OPSD and other baselines, and conduct ablation studies on the 1.7B model to examine the contributions of the two components.
Critical strengths
The authors provide empirical analyses of teacher and student EOS probability discrepancies and repetitive generation patterns.
Critical weaknesses
Prior studies, such as Demystifying OPD, When EOS Tokens Disagree, and Mismatch Matters, have already investigated length inflation, repetitive generation, and termination related discrepancies in on policy distillation. The proposed SC objective also shares conceptual similarities with existing EOS alignment approaches, while UL builds on established unlikelihood training techniques. The paper would benefit from a clearer differentiation of its methodological contributions from these closely related works.
All experiments are conducted using Qwen3 models in No Thinking mode, despite the paper's focus on reasoning termination. It remains unclear whether the proposed approach generalizes to explicit Thinking mode, where terminating the reasoning phase and terminating the final response may involve different decisions. This limits the generalizability of the findings to reasoning models with explicit thinking stages.
The average accuracy improvement over OPSD decreases from 4.84 percentage points for Qwen3 1.7B to 3.73 for 4B and only 1.32 for 8B. This trend raises concerns about whether the proposed approach remains effective as model capacity increases. The paper provides limited analysis of why the improvements diminish with model scale.
The ablation experiments examining the individual contributions of SC and UL are conducted only on Qwen3 1.7B. Although the full method is evaluated across three model sizes, the experiments do not establish whether both components remain beneficial for larger models. This is particularly relevant given the smaller overall improvements observed on Qwen3 8B.
The paper identifies premature termination partly by allowing a privileged teacher to continue from incorrectly terminated student responses. However, since the teacher has access to reference answers, its ability to recover correct solutions does not necessarily demonstrate that the original student could have benefited from additional reasoning. Consequently, the evidence does not fully establish that premature stopping, rather than insufficient reasoning capability, is the primary cause of these failures.
Questions for authors
Since the original full vocabulary KL objective already includes EOS probabilities, could the authors compare SC against simpler alternatives, such as increasing the EOS token loss weight or explicitly aggregating equivalent EOS tokens? Such comparisons would help establish whether SC provides benefits beyond strengthening existing termination supervision.
Could the authors conduct controlled experiments in which the original student continues generating after its EOS token is removed, or its EOS probability is selectively reduced at the same prefix? Measuring whether these interventions improve answer accuracy would provide more direct evidence that premature termination contributes causally to reasoning failures.
Response to Reviewer EpQ1
We thank the reviewer for the constructive feedback. We address W1–W5 and Q1–Q2 below.
W1/Q1. Related work and EOS controls
These works address related phenomena but differ in method or goal. Demystifying OPD, which we implement as a baseline, uses reference constraints and rollout mixing to curb length inflation and repetition saturation during training. TA-OPSD uses SC for premature stops and UL for repetitive over-generation. When EOS Tokens Disagree pools equivalent EOS tokens to reconcile different teacher and student stop tokens. Our OPSD teacher and student share a base model and vocabulary. SC distills the full STOP/CONTINUE decision at every valid prefix, including whether to continue reasoning. Mismatch Matters uses probability shaping and teacher top-K injection to correct general token excess and deficit. TA-OPSD targets termination through stop supervision and penalties on detected repeats. We use the negative-distillation effect of unlikelihood loss to train on repeated spans replayed with their original prefixes, a use distinct from prior UL work. Jointly addressing when to stop and what to suppress after reasoning stalls distinguishes TA-OPSD. (“Demystifying OPD”) (“When EOS Tokens Disagree”) (“Mismatch Matters”) (“Unlikelihood Training”)
We tested the full binary stopping objective against two enhanced EOS controls on Qwen3-1.7B with thinking off. All three shared initialization, the base OPSD protocol, and 100 steps, with UL off. Auxiliary weights used the same initial logit-gradient norm rule. Let E be the set of two stop tokens and q,p the teacher and student distributions.
Short errors are incorrect responses that stop naturally within 1,024 tokens with no repetition flagged by UL's mechanical rules. SC improves mean accuracy by 7.22 and 6.34 percentage points over the two controls, respectively, with fewer short errors. Thus, the full STOP/CONTINUE objective gave higher accuracy and a lower short-error rate than either EOS control.
W5/Q2. Can the original student recover from short errors?
On AIME24, AIME25, and AMC23, the natural incorrect-stop rate is the share of responses stopping naturally within 1,024 tokens with an incorrect or unextractable answer. Each method generated 1,200 responses. For Qwen3-1.7B, the equally weighted rate fell from 12.18% (OPSD) to 3.52% (TA-OPSD), down 8.66 percentage points.
To test whether the original student could continue solving, we selected 42 held-out OpenThoughts problems (§3) with correct natural stops from Base and shorter, incorrect responses from OPSD. We kept the original OPSD checkpoint, question, and full pre-EOS prefix, with no reference answer. We masked both EOS tokens only at the original stop and restored the decoding policy thereafter (four random seeds per problem, up to 4,096 new tokens). 33/168 continuations were correct, with at least one correct continuation on 13 questions. With model, derivation, and budget fixed, intervening only at the original stop recovered some errors. This directly shows that some post-OPSD short errors arose from stopping while the student could still answer correctly.
W2. Explicit thinking mode
On Qwen3-1.7B, we enabled thinking for both teacher and student during distillation and at evaluation. Compared with OPSD, TA-OPSD raised three-benchmark mean accuracy from 50.63% to 61.83%. Capped responses fell from 370/1,200 to 188/1,200, and errors that matched the repetition rules and hit the cap fell from 363 to 160. Repetition with budget exhaustion persists with thinking on, and TA-OPSD reduced these failures here.
W3–W4. Model scale and component ablations
On 460 shared held-out diagnostic questions, OPSD gave 114, 53, and 45 short errors at 1.7B, 4B, and 8B. Of the 35 questions 1.7B answered incorrectly without a derivation, 4B and 8B solved 20 and 28, respectively. This is consistent with larger models deriving more on their own, leaving fewer short errors for stopping supervision to address.
The table gives three-benchmark macro accuracy with thinking off.
Adding SC or UL alone improved accuracy on the larger 4B model.