acceptodds
Under review as a conference paper at ICLR 2027

Schedule the Optimizer, Not the Teacher: Diagnosing and Mitigating Long-Horizon Collapse in On-Policy Self-Distillation

Abstract

On-policy self-distillation (OPSD) lets a single language model teach itself. A frozen copy of the initial policy, conditioned on privileged ground-truth solutions, provides dense per-token supervision on the student's own rollouts. This delivers gains competitive with group relative policy optimization (GRPO) at a fraction of the sampling cost. We ask what the short budget hides. Training every frozen-teacher OPSD variant past 100 steps, we find a consistent trajectory: performance peaks, declines monotonically, and by step 300 falls below the untrained base model, at 1.7B, 4B, and 8B scales. The obvious suspect is the frozen teacher going stale, but the obvious remedy backfires. Refreshing the teacher to track the student collapses the model to a three-benchmark mean of 5.7 (8.9% on AIME24, vs. base 54.7%); its outputs degenerate into fluent but empty self-talk. Gentler anchor motion is no escape. An exponential moving average (EMA) teacher matches the best early score of the frozen-teacher family, yet ends the horizon below the frozen teacher's own endpoint at every rate we test (32.5–32.9 vs. 35.1 across EMA 0.99–0.9995). Only dose decay absorbs the penalty. The frozen teacher is not the bottleneck but the anchor. The collapse is instead objective drift under a constant learning rate, controlled primarily by the cumulative learning-rate dose. The one-sided clip that stabilizes early training explains the drift's visible signature, namely a training loss that descends below zero. It does not explain the drift itself. A horizon-aware cosine decay removes most of the collapse at all three scales, ending at or above the untrained base at 1.7B and 8B. Alongside the diagnosis, we make three supporting contributions. A difficulty-aware sample weighting brings small multi-seed gains and does not alter the long-horizon outcome. A numerically hardened hybrid of self-distillation and reinforcement learning with verifiable rewards trains stably where the naive form suffers a 10^14 gradient explosion that poisons the optimizer state. A defect analysis of the official implementation clarifies which reported numbers are affected. Our results reframe frozen-teacher self-distillation as a short-horizon method: schedule the optimizer, not the teacher. Our code is available at the anonymous repository: https://anonymous.4open.science/r/Schedule-the-Optimizer-Not-the-Teacher-ICLR-2027-DCB6.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Large Language Models, Knowledge Inference, Reinforcement Learning with Verifiable Rewards

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_aT2QOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the long-horizon training behavior of on-policy self-distillation. While prior OPSD studies primarily evaluate relatively short training horizons, this work extends training to 300 steps and observes substantial performance degradation after an initial improvement. The authors conduct experiments with Qwen3 models at 1.7B, 4B, and 8B scales and demonstrate that continued training with a constant learning rate can reduce mathematical reasoning performance below the untrained baseline. They investigate potential explanations through teacher-refresh experiments, exponential moving average (EMA) teachers, learning-rate schedules, and analyses of the clipped forward-KL objective. Their results indicate that periodically refreshing the teacher can cause severe degradation, whereas reducing the cumulative learning-rate dose substantially mitigates long-horizon performance decline. The paper additionally studies difficulty-aware sample weighting, a hybrid OPSD-RLVR objective, and several implementation issues in the original OPSD framework, including numerical precision and training-evaluation prompt mismatches.

Critical strengths
  1. A valuable empirical investigation of long-horizon OPSD behavior.

The paper examines an important but relatively underexplored question: whether the improvements obtained from OPSD remain stable under extended training. The observation that performance can peak early and subsequently fall below the base model is practically relevant, particularly because training loss alone does not reliably indicate the degradation. The authors reproduce this phenomenon across three model scales and provide multi-seed experiments and statistical analyses for key comparisons. These results identify a meaningful limitation of existing OPSD training recipes.

  1. Extensive diagnostic experiments and implementation analysis.

The paper goes beyond simply reporting performance degradation. It systematically examines teacher refresh, EMA teachers, learning-rate schedules, clipping thresholds, alternative divergence objectives, and numerical implementation issues. The severe degradation observed with teacher refresh is particularly interesting and highlights potential risks in naively extending moving-teacher approaches to OPSD. The detailed implementation audit and extensive supplementary results also improve the reproducibility and practical usefulness of the work.

Critical weaknesses
  1. The explanation of the collapse is not sufficiently established.

Although the paper provides convincing empirical evidence that reducing the learning rate mitigates performance degradation, I am less convinced that the experiments establish the proposed causal explanation. In particular, the failure of teacher refresh does not necessarily rule out teacher staleness as a contributing factor, since refreshing the teacher changes the supervision target and can introduce additional optimization effects. Similarly, the observation that reducing the cumulative learning-rate dose improves performance demonstrates an effective intervention but does not fully identify why continued OPSD training degrades reasoning ability.

  1. Limited practical improvement over early stopping and restricted experimental generality.

The proposed mitigation primarily reduces the amount of optimization applied during extended training. However, the resulting checkpoints do not clearly outperform models obtained through early stopping. For example, at 1.7B scale, the baseline achieves 42.5 at step 75, whereas cosine decay reaches only 39.3 at step 300. The dose-matched constant learning-rate experiment also performs similarly to cosine decay, suggesting that the benefit is largely attributable to reducing the optimization dose rather than the scheduling strategy itself. The practical advantage over carefully tuned learning rates and checkpoint selection is therefore unclear.

Questions for authors

Q1. What practical advantage does the proposed learning-rate scheduling strategy provide over early stopping or conventional learning-rate tuning?

The best early checkpoints generally outperform the final checkpoints obtained with learning-rate decay. Moreover, the dose-matched constant learning-rate experiment achieves performance similar to cosine decay. Could the authors provide a more systematic comparison against validation-based early stopping and tuned constant learning rates, preferably under comparable training budgets? This would help clarify the incremental practical contribution of the proposed mitigation.

Q2. What evidence distinguishes objective drift from alternative explanations for the observed performance degradation?

The learning-rate experiments establish that smaller optimization doses alleviate degradation, but the mechanism connecting optimizer updates to reasoning capability loss remains unclear. In particular, degradation persists with both unclipped forward KL and symmetric clipping. Could the authors provide additional evidence identifying the optimization dynamics responsible for the loss of reasoning performance, beyond the observed correlation between cumulative learning rate, teacher-student divergence, and benchmark accuracy? A more explicit comparison between the gradients of the clipped and unclipped objectives could be informative.

Q3. To what extent does the teacher-refresh experiment rule out teacher staleness?

Refreshing the teacher changes the supervision distribution and, in the periodic-refresh implementation, also resets the adapter and optimizer state. Although the EMA experiments partially address these concerns, the failure of a moving teacher does not necessarily imply that the frozen teacher remains an appropriate supervisor throughout training. Could the authors clarify the causal interpretation of these experiments, or provide an additional control that changes teacher freshness while retaining an explicit anchor to the initial policy?

Reviewer_xh9LOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies performance degradation when on-policy self-distillation is extended beyond its usual 100-step budget. Across Qwen3 models from 1.7B to 8B, continued training under an approximately constant learning rate reduces reasoning accuracy. Periodic teacher refresh can cause severe collapse, while learning-rate decay mitigates the decline. Comparisons with linear decay and a constant half learning rate suggest that cumulative learning-rate dose explains much of the effect within the tested setup. The paper also studies divergence clipping, EMA teachers, sample weighting, an RLVR hybrid, and implementation defects. Evaluation uses mathematical reasoning benchmarks, with multiple seeds for selected comparisons.

Critical strengths
  1. The central empirical comparison is supported by several useful checks. The decay advantage appears across three model sizes and repeats across three training seeds at 1.7B. Alternative scoring rules and numerical-precision checks make the observed degradation more credible.

  2. The comparison at matched cumulative learning-rate dose is informative. Constant half-rate training and linear decay perform similarly to cosine decay, helping explain which aspect of the intervention matters. The paper also reports the limited gains from its secondary methods openly.

Critical weaknesses
  1. The extensive experiments provide limited additional understanding of why self-distillation degrades reasoning. The study establishes which configurations deteriorate and which slow the decline, but the proposed objective-drift account leaves the harmful supervision largely unidentified. Performance also degrades under unclipped forward KL, so the clipped objective's negative values explain only part of the observed behavior. The paper does not establish what changes in the model's reasoning or why initially useful supervision becomes harmful. This limits the contribution of a paper centered on diagnosis, especially given the overlap with cited work. Kim et al. already report an initial improvement followed by degradation in an OPSD setting and show that reducing the learning rate delays degradation in their self-distillation experiments (arXiv:2603.24472v1, Figures 12 and 16). Ichihara et al. also extend OP2SD from 100 to 400 updates and report declining average accuracy (arXiv:2608.09228v1, Appendix F), contrary to this paper's description that their study does not extend the horizon. The additional insight beyond these observations needs to be made more precise.

  2. The evidence for a transferable account of training horizon is limited by the modest training budget and fixed batch size. With global batch size 32, the main 300-step experiment processes about 9,600 examples, or 0.33 epochs of the 29,434-example dataset. Even 500 steps cover only about 16,000 examples. These experiments establish degradation within a short fine-tuning regime, with little evidence about behavior at substantially larger data budgets. Holding batch size fixed leaves the effects of sample exposure, update count, and gradient noise unresolved. The paper also reports different outcomes at the same cumulative dose: 36.8 for 150 full-rate steps versus 38.8 for 300 half-rate steps. Although the authors acknowledge the role of the per-step rate, the resulting account offers limited guidance for predicting a safe training budget under a different configuration. The mitigation also remains below the best early checkpoint, leaving the benefit of additional training unclear.

Questions for authors
  1. Can the saved checkpoints clarify whether declining accuracy is associated with worse teacher matching or with better matching of harmful supervision? Comparing teacher-student KL on a fixed set of prompts and prefixes before and after degradation, including the unclipped run, would help distinguish these explanations.

  2. Would a controlled change in batch size place the degradation point at a similar number of processed examples, optimizer updates, or cumulative learning-rate dose? One such comparison would help assess whether the proposed account extends beyond batch size 32.

  3. Given the related results in Kim et al. and the extended-training experiment in Ichihara et al., what new conclusion about the cause of degradation is established here? Please also reconcile the description of Ichihara et al. with their Appendix F.

Reviewer_82xjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper revisits privileged context self distillation beyond the short training budget at which it is usually evaluated. Across three Qwen3 sizes, the authors observe deterioration with a frozen teacher, substantially worse failures under teacher refresh, and improved terminal performance when the learning rate is reduced. They investigate several alternative explanations, including asymmetric clipping, numerical precision, prompt mismatch, and implementation defects. The main contribution is an empirical diagnosis and a practical warning about training duration, rather than a new distillation objective.

Critical strengths
  1. Following the entire trajectory changes the assessment of these methods in a useful way. An attractive score at step 100 can conceal a much worse endpoint. Table 1 makes this visible across model sizes and interventions, and the distinction between improving the peak and preserving later performance is worth emphasizing to practitioners.

  2. The optimizer result is supported by more than one successful cosine run. The three seed comparison at 1.7B ends at 39.4 ± 1.0 with cosine decay versus 34.6 ± 0.4 with a constant learning rate. Linear decay and the constant half rate control also retain substantially more performance. The alternative scoring rules in Appendix G further reduce the likelihood that the main ordering is just an answer extraction artifact.

  3. Several negative results are informative. Symmetric clipping does not remove the decline, EMA teachers do not provide a lasting remedy, and the hardened RLVR hybrid does not outperform the baseline over the full horizon. Reporting these outcomes, together with the implementation audit, makes the study more useful than an account built around the best performing variant alone.

Critical weaknesses
  1. Some of the detailed results cannot be reconciled with the stated evaluation protocol. Appendix D, page 17, specifies 30 problems per competition benchmark and 12 samples per problem. Yet Table 20, page 41, reports 53.7 Avg, 67.6 Maj, and 81.5 Pass for the constant half rate run at step 100 on AIME24. Under that single run protocol, majority and any correct accuracy must be rounded multiples of 100/30; neither 67.6 nor 81.5 is attainable. The mean accuracy 53.7 is also not a rounded multiple of 100/360. Appendix R explains missing entries but does not identify alternative denominators or averaging for these rows. Please supply the underlying counts, run identifiers, and aggregation rules, and regenerate any affected comparisons. This is an unresolved reporting inconsistency, not evidence by itself of how the numbers were produced.

  2. Cumulative learning rate dose is a useful organizing variable, but the causal interpretation is stronger than the evidence warrants in places. Section 6 correctly notes that the full rate run stopped at step 150 scores 36.8, compared with 38.8 to 39.3 for the same nominal dose delivered over 300 steps. Thus dose does not identify the endpoint, even within this experimental family. The converse comparison changes both the rate and the number of sampled batches, so it does not isolate a rate correction either. This is hard to reconcile with Appendix B.1 saying that schedule family, shape, and stopping step matter only through dose. I would retain the empirical dose account, qualify that statement, and repeat the particularly informative matched dose comparisons across seeds before treating the residual differences as a general relationship.

  3. The hard refresh experiment does not isolate teacher replacement. Section 5 and Appendix F.2 explain that the operation also merges the adapter, initializes a new adapter, and resets optimizer state. The EMA results are a valuable response to this concern, but their endpoints around 32.5 to 32.9 are very different from the hard refresh endpoint of 5.7. They show that a moving teacher can hurt without a reset; they do not determine how much of the catastrophic hard refresh failure is caused by each intervention. A reset control with the privileged teacher held fixed would make this part of the diagnosis considerably cleaner. Without it, the strongest conclusion concerns the tested refresh procedure, not teacher freshness in isolation.

Questions for authors
  1. Figure 8 states that a gradient spike is clipped at 0.1 but has already poisoned Adam's second moment. Please spell out the operation order and numerical failure responsible. A large finite gradient norm before clipping, by itself, does not explain corruption of moments that are updated from the clipped gradients. Was the failure an overflow, a nonfinite value, or an update performed before the guard?

  2. Proposition 1(iii) in Appendix J, page 27, contains a proof error. When the minority term is clipped, the majority contribution is negative, not nonnegative as stated. With q = 0.25, s = 0.05, and a clip of 0.05, that contribution is approximately negative 0.177 and the total loss is negative 0.127. This refutes the cited proof branch, not the claimed location of the global minimum. Also, the finite vocabulary objective in Equation 2 is bounded below by negative teacher entropy, so Section 5's description of negative terms as unbounded below should be corrected.

  3. Please make the replication status easier to read in Table 1. Some entries explicitly give a standard deviation, some carry a single seed marker, and several other probe rows provide only point estimates. A uniform seed count field would prevent the smaller controls from appearing as well replicated as the central comparison.

  4. Distinguish loss of individual sample reliability from loss under every decoding metric. Appendix R already notes that 8B majority accuracy remains 70.0 while any correct accuracy moves from 77.8 to 78.9 despite a decline in mean accuracy. That qualification belongs closer to the central collapse claim.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Manuscript Revisions and Responses to All Reviewers

We thank Reviewers 82xj, xh9L, and aT2Q for their constructive assessments. They prompted additional controls, an audit of saved results and code, and corrections to our claims and presentation. We have updated the manuscript PDF and supplementary ZIP. This submission-level comment summarizes the revisions and guides readers to the detailed responses in the three reviewer threads.

1. Main changes in the revised manuscript

Completed experiments and controls. We added two independent half-rate training seeds, a frozen-teacher adapter-merge/reinitialization control, batch-size comparisons, fixed-prefix teacher–student KL measurements, and standard-protocol reevaluations of seven checkpoint conditions implicated by the reporting audit. The final EMA-plus-cosine step-200 evaluation is included. No unfinished experiment is reported as a result.

Evaluation provenance and numerical corrections. The revised tables replace affected legacy entries with traceable reevaluations and report counts, evaluation identifiers, denominators, and aggregation rules. The completed EMA-plus-cosine step-200 result is 40.09/47.78/63.33 for macro Avg/Maj/Pass. We explicitly mark unresolved inherited aggregations rather than inferring their denominators or claiming to have reconstructed them.

Narrowed causal and practical conclusions. Cumulative learning-rate dose is now an organizing variable within the tested schedule family, not a unique determinant of terminal performance; the per-step rate also matters. The reset, batch-size, and fixed-prefix KL results constrain explanations but do not complete a causal decomposition. Teacher staleness remains open. The clip analysis is limited to surrogate geometry, and the gradient-spike discussion no longer claims optimizer-state corruption. Our recommendation acknowledges validation-based early stopping and does not present cosine decay as universally superior or sufficient.

Mathematical, statistical, and reporting clarifications. We corrected Proposition 1(iii), including the minority-clipped branch's sign, and state the finite-vocabulary lower bound as negative teacher entropy. Training-seed counts are explicit; mean ± sample SD over training runs is distinguished from problem-conditional bootstrap uncertainty, and checkpoint reevaluation is not an independent training seed. We distinguish Avg@12 declines from Maj@12 and Pass@12 changes, correct the 8B per-benchmark comparisons, qualify the per-problem analysis as descriptive seed-42 stratification, and correct the discussion of prior long-horizon studies.

Code-audit and artifact updates. The manuscript limits the train/evaluation prompt mismatch to the official code path and distinguishes the scalar-clipped gradient proxy from the actual per-vocabulary-clipped training objective. The supplementary ZIP includes the historical artifacts, revision snapshots, final result record, analysis scripts, and these limitations.

2. Guide to the reviewer-specific responses

We posted 38 threaded author comments, including three reviewer-specific overviews:

  • Reviewer 82xj — 14 comments covering 7 W/Q threads: evaluation denominators and replacements; dose interpretation; refresh/reset confounds; the gradient-spike claim; Proposition 1(iii) and the objective lower bound; replication status; and Avg versus alternative decoding metrics.
  • Reviewer xh9L — 12 comments covering 5 W/Q threads: mechanistic scope and prior work; training budget, batch size, and generality; fixed-prefix KL; examples/updates/dose; and the causal contribution beyond existing studies.
  • Reviewer aT2Q — 12 comments covering 5 W/Q threads: mitigation versus mechanism; early stopping and constant-rate tuning; objective drift and clipped-gradient evidence; and teacher staleness, reset confounds, and an initial-policy anchor.

Each thread gives the direct answer, numerical evidence, limitations, manuscript locations, and revised text.

3. Evidence boundaries retained in the revision

We do not claim a complete mechanism of reasoning-capability loss. A faithful gradient comparison using the actual per-vocabulary-clipped training objective, a moving-teacher condition with an independent fixed initial-policy anchor, and a prospective equal-total-budget comparison of early stopping and learning-rate tuning remain unavailable. We state these limitations explicitly rather than substituting speculative explanations.

We appreciate the reviewers’ careful reading, which has led to corrected reporting, additional evidence, and a more precise account of what the experiments establish and what remains unresolved.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.