- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the long-horizon training behavior of on-policy self-distillation. While prior OPSD studies primarily evaluate relatively short training horizons, this work extends training to 300 steps and observes substantial performance degradation after an initial improvement. The authors conduct experiments with Qwen3 models at 1.7B, 4B, and 8B scales and demonstrate that continued training with a constant learning rate can reduce mathematical reasoning performance below the untrained baseline. They investigate potential explanations through teacher-refresh experiments, exponential moving average (EMA) teachers, learning-rate schedules, and analyses of the clipped forward-KL objective. Their results indicate that periodically refreshing the teacher can cause severe degradation, whereas reducing the cumulative learning-rate dose substantially mitigates long-horizon performance decline. The paper additionally studies difficulty-aware sample weighting, a hybrid OPSD-RLVR objective, and several implementation issues in the original OPSD framework, including numerical precision and training-evaluation prompt mismatches.
Critical strengths
- A valuable empirical investigation of long-horizon OPSD behavior.
The paper examines an important but relatively underexplored question: whether the improvements obtained from OPSD remain stable under extended training. The observation that performance can peak early and subsequently fall below the base model is practically relevant, particularly because training loss alone does not reliably indicate the degradation. The authors reproduce this phenomenon across three model scales and provide multi-seed experiments and statistical analyses for key comparisons. These results identify a meaningful limitation of existing OPSD training recipes.
- Extensive diagnostic experiments and implementation analysis.
The paper goes beyond simply reporting performance degradation. It systematically examines teacher refresh, EMA teachers, learning-rate schedules, clipping thresholds, alternative divergence objectives, and numerical implementation issues. The severe degradation observed with teacher refresh is particularly interesting and highlights potential risks in naively extending moving-teacher approaches to OPSD. The detailed implementation audit and extensive supplementary results also improve the reproducibility and practical usefulness of the work.
Critical weaknesses
- The explanation of the collapse is not sufficiently established.
Although the paper provides convincing empirical evidence that reducing the learning rate mitigates performance degradation, I am less convinced that the experiments establish the proposed causal explanation. In particular, the failure of teacher refresh does not necessarily rule out teacher staleness as a contributing factor, since refreshing the teacher changes the supervision target and can introduce additional optimization effects. Similarly, the observation that reducing the cumulative learning-rate dose improves performance demonstrates an effective intervention but does not fully identify why continued OPSD training degrades reasoning ability.
- Limited practical improvement over early stopping and restricted experimental generality.
The proposed mitigation primarily reduces the amount of optimization applied during extended training. However, the resulting checkpoints do not clearly outperform models obtained through early stopping. For example, at 1.7B scale, the baseline achieves 42.5 at step 75, whereas cosine decay reaches only 39.3 at step 300. The dose-matched constant learning-rate experiment also performs similarly to cosine decay, suggesting that the benefit is largely attributable to reducing the optimization dose rather than the scheduling strategy itself. The practical advantage over carefully tuned learning rates and checkpoint selection is therefore unclear.
Questions for authors
Q1. What practical advantage does the proposed learning-rate scheduling strategy provide over early stopping or conventional learning-rate tuning?
The best early checkpoints generally outperform the final checkpoints obtained with learning-rate decay. Moreover, the dose-matched constant learning-rate experiment achieves performance similar to cosine decay. Could the authors provide a more systematic comparison against validation-based early stopping and tuned constant learning rates, preferably under comparable training budgets? This would help clarify the incremental practical contribution of the proposed mitigation.
Q2. What evidence distinguishes objective drift from alternative explanations for the observed performance degradation?
The learning-rate experiments establish that smaller optimization doses alleviate degradation, but the mechanism connecting optimizer updates to reasoning capability loss remains unclear. In particular, degradation persists with both unclipped forward KL and symmetric clipping. Could the authors provide additional evidence identifying the optimization dynamics responsible for the loss of reasoning performance, beyond the observed correlation between cumulative learning rate, teacher-student divergence, and benchmark accuracy? A more explicit comparison between the gradients of the clipped and unclipped objectives could be informative.
Q3. To what extent does the teacher-refresh experiment rule out teacher staleness?
Refreshing the teacher changes the supervision distribution and, in the periodic-refresh implementation, also resets the adapter and optimizer state. Although the EMA experiments partially address these concerns, the failure of a moving teacher does not necessarily imply that the frozen teacher remains an appropriate supervisor throughout training. Could the authors clarify the causal interpretation of these experiments, or provide an additional control that changes teacher freshness while retaining an explicit anchor to the initial policy?
Summary of Manuscript Revisions and Responses to All Reviewers
We thank Reviewers 82xj, xh9L, and aT2Q for their constructive assessments. They prompted additional controls, an audit of saved results and code, and corrections to our claims and presentation. We have updated the manuscript PDF and supplementary ZIP. This submission-level comment summarizes the revisions and guides readers to the detailed responses in the three reviewer threads.
1. Main changes in the revised manuscript
Completed experiments and controls. We added two independent half-rate training seeds, a frozen-teacher adapter-merge/reinitialization control, batch-size comparisons, fixed-prefix teacher–student KL measurements, and standard-protocol reevaluations of seven checkpoint conditions implicated by the reporting audit. The final EMA-plus-cosine step-200 evaluation is included. No unfinished experiment is reported as a result.
Evaluation provenance and numerical corrections. The revised tables replace affected legacy entries with traceable reevaluations and report counts, evaluation identifiers, denominators, and aggregation rules. The completed EMA-plus-cosine step-200 result is 40.09/47.78/63.33 for macro Avg/Maj/Pass. We explicitly mark unresolved inherited aggregations rather than inferring their denominators or claiming to have reconstructed them.
Narrowed causal and practical conclusions. Cumulative learning-rate dose is now an organizing variable within the tested schedule family, not a unique determinant of terminal performance; the per-step rate also matters. The reset, batch-size, and fixed-prefix KL results constrain explanations but do not complete a causal decomposition. Teacher staleness remains open. The clip analysis is limited to surrogate geometry, and the gradient-spike discussion no longer claims optimizer-state corruption. Our recommendation acknowledges validation-based early stopping and does not present cosine decay as universally superior or sufficient.
Mathematical, statistical, and reporting clarifications. We corrected Proposition 1(iii), including the minority-clipped branch's sign, and state the finite-vocabulary lower bound as negative teacher entropy. Training-seed counts are explicit; mean ± sample SD over training runs is distinguished from problem-conditional bootstrap uncertainty, and checkpoint reevaluation is not an independent training seed. We distinguish Avg@12 declines from Maj@12 and Pass@12 changes, correct the 8B per-benchmark comparisons, qualify the per-problem analysis as descriptive seed-42 stratification, and correct the discussion of prior long-horizon studies.
Code-audit and artifact updates. The manuscript limits the train/evaluation prompt mismatch to the official code path and distinguishes the scalar-clipped gradient proxy from the actual per-vocabulary-clipped training objective. The supplementary ZIP includes the historical artifacts, revision snapshots, final result record, analysis scripts, and these limitations.
2. Guide to the reviewer-specific responses
We posted 38 threaded author comments, including three reviewer-specific overviews:
Each thread gives the direct answer, numerical evidence, limitations, manuscript locations, and revised text.
3. Evidence boundaries retained in the revision
We do not claim a complete mechanism of reasoning-capability loss. A faithful gradient comparison using the actual per-vocabulary-clipped training objective, a moving-teacher condition with an independent fixed initial-policy anchor, and a prospective equal-total-budget comparison of early stopping and learning-rate tuning remain unavailable. We state these limitations explicitly rather than substituting speculative explanations.
We appreciate the reviewers’ careful reading, which has led to corrected reporting, additional evidence, and a more precise account of what the experiments establish and what remains unresolved.