acceptodds
Under review as a conference paper at ICLR 2027

When Should the Teacher Move? Temporal Coupling and Stability in Self On-Policy Distillation

Abstract

Self on-policy distillation trains a student policy against a teacher derived from its own parameter history, yet the teacher's update schedule—which governs the temporal coupling between teacher and student—has not been systematically studied as a stability variable. Through a controlled schedule sweep on Qwen3-8B, we establish that isolation periods—complete teacher freezing between updates—are the key structural property enabling stable learning, not teacher age, and we introduce a diagnostic framework of temporal KL structure, refresh shock, and length-tail risk to characterize training dynamics. We further identify state-oblivious collapse: the best short-horizon fixed schedule () catastrophically fails under long-horizon training because a clock-driven refresh can copy a transiently drifting student into the teacher in a single, irreversible step—a failure mode invisible under short-horizon evaluation and mechanistically distinct from EMA's chronic contamination. To address this, we propose Consolidation-Gated Teacher Refresh (CGTR), which preserves isolation periods while gating each refresh on joint evidence of reward improvement and length-tail safety, ensuring every teacher movement responds to genuine student consolidation rather than a clock signal. With a single shared parameter set and no per-dataset retuning, CGTR achieves zero collapse and the best final score on all four tasks (Chemistry, Biology, Physics, ToolUse), self-regulating its refresh frequency to each task's learning dynamics.

Primary area:
interpretability and explainable AI
Keywords:
Self Distillation, on-policy learning, training stability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_N6jkOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper focuses on the self on-policy distillation framework, in which they identify a stablility issue from the teachers update scheduling. The paper uses one base model, Qwen3-8B, and tests on limited set of task catagories, where they find some useful observations that can be used for the proposed CGTR algorithm. Moreover, the proposed CGTR is largely heuristics and does not compare with related adaptive update baselines. There are also some overclaims, and the expalnation on the entropy results is not convincing.

Critical strengths

The only strength I can identify is that the paper provides a clear empirical study cases of the teacher update scheduling. However, due to its limited expeiments setup and comparisons, the advantage is not well established.

Critical weaknesses
  1. Similar ideas have already been studied beyond LLMs, particularly in target-network and momentum-teacher settings. CGTR should therefore be compared with related adaptive strategies, not only the fixed schedules and EMA. More importantly, the long-horizon experiments omit larger fixed intervals (e.g., M=100/150), which makes it unclear whether CGTR's gains come from state-aware refreshing or simply fewer teacher updates. The reward gate also becomes increasingly restrictive, leaving the teacher frozen for the last 170 steps on Chemistry. This raises the concern of whether CGTR genuinely adapts to training dynamics or effectively becomes a fixed teacher.
  2. The paper repeatedly uses token entropy dynamics as evidence of teacher contamination, but the connection to CGTR's design is not well established. Section 3.3 acknowledges that entropy alone is an unreliable consolidation signal, yet Section 5.2 uses elevated entropy at step 150 as the main evidence of an unsafe teacher refresh. However, none of CGTR's gates monitors entropy, and the paper does not show whether the reward or length-tail criteria could detect this instability. Moreover, CGTR's isolation gate alone would prevent a refresh at step 150, leaving unclear whether the state-aware gates actually address the identified failure mechanism.
Questions for authors
  1. What theoretical justification supports the connection between token entropy dynamics and the claimed training stability? In particular, how reliably can entropy distinguish stable learning from the proposed failure modes?
  2. Can the authors show the direct evidence that the collapse is caused by an ill-timed teacher refresh?
6 comments
AuthorsOpenReview ↗

Response to Reviewer N6jk (1/4)

We thank the reviewer for the feedback. In response we added larger fixed intervals and predetermined schedules that change only the refresh timing (Tables R1, R2), an offline replay of CGTR's gates on the logs of the collapsing M=50 runs (Table R3), a counterfactual intervention at the suspected harmful refresh (Table R4), and an analysis of why target-network and momentum-teacher mechanisms do not carry over to this setting (Table R5). We also revise the claims the reviewer found overstated. All new runs use the setup of Appendix A and the unchanged CGTR parameters of Appendix C.

W1(a): Comparison with adaptive strategies from target-network / momentum-teacher literature

We agree that the question of when a reference network should move has been studied outside LLMs, and we will expand Related Work to cover target networks in value-based RL (Mnih et al., 2015; Lillicrap et al., 2016; Zhang et al., 2021), momentum teachers (Tarvainen & Valpola, 2017; Grill et al., 2020; Caron et al., 2021), and learned teacher updates (Pham et al., 2021). These mechanisms were designed for settings with a replay buffer or a fixed data distribution. In self on-policy distillation the student trains on its own rollouts, so whatever the teacher absorbs is fed back into the next training data, and failures such as repetition loops appear abruptly. Three of these families are already baselines in the paper: periodic hard updates, EMA targets, and frozen teachers. Table R5 lists for every family why it does not carry over, and how CGTR addresses it.

Family (origin)Baseline in this paperWhen the reference movesWhy it does not fit LLM self on-policy distillationEvidence in this paper
Periodic hard update (DQN; Mnih et al., 2015)Yes: M=1–50 at 100 steps (Table 1); M=50, 100, 150 at 300 steps (Tables 3, R1)every M stepsA clock cannot tell consolidation from drift; with on-policy rollouts a bad copy is reinforced, not diluted by a replay bufferM=50 collapses at step 189, and skipping only its step-150 copy prevents it (Table R4); the best safe M differs by task (Table R1)
Polyak / EMA target (DDPG; Mean Teacher)Yes: α=0.1 and 0.04 (Tables 1, 3)every step, fixed rateAssumes per-step teacher error is small noise that averaging cancels; in closed-loop training drift accumulates insteadLag-aligned EMA: entropy ≈0.7 → 3.0, final 0.713 (Section 5.2, Table 3)
Annealed momentum (BYOL; DINO)Noevery step, rate annealed to zero on a clockDrifts like EMA while the rate is high, then freezes at a time fixed in advance, which needs the horizon and per-task tuningBehaves as EMA until the rate has decayed, so it inherits the drift above (Section 5.2, Table 3)
Frozen targetYes: Fixed (init), Fixed@100 (Table 1, Appendix H)neverThe reference never improves beyond the model it was frozen fromFixed (init) 0.725, Fixed@100 0.765 (Appendix H)
Learned teacher update (Meta Pseudo Labels; Pham et al., 2021)Noby gradient from student feedbackTrains the teacher itself, adding a second 8B model to the backward pass; a self-teacher is a parameter snapshotNot practical at this scale
CGTR (ours)—after isolation, once the reward gain is verified and the length tail is safeUses reward and length statistics already computed for every rollout: no extra inference, no learned component, one parameter set for four tasksBest final score on all four tasks; no collapse in any CGTR run, including nine sensitivity variants (Appendix D)

We will add Table R5 to the paper. If the reviewer has a specific adaptive method in mind, we will add it as a baseline.

AuthorsOpenReview ↗

Response to Reviewer N6jk (2/4)

W1(b): Larger fixed intervals; is the gain just fewer updates?

We added M=100 and M=150 on Chemistry and Biology (300 steps; Table R1). To separate state-aware timing from the number of updates directly, we also ran predetermined schedules that keep CGTR's refresh count and final freeze on Chemistry but move the refresh times, and transplanted each task's CGTR schedule to the other task (Table R2).

Table R1. Larger fixed intervals (single seed).

TaskScheduleRefresh stepsTest finCollapse
ChemM=50every 50 steps0.000189
ChemM=100100, 2000.771—
ChemM=1501500.748—
ChemFixed@100 (App. H)freeze M=50 teacher from 1000.765—
ChemFixed (init) ≡ M≥300none0.725—
ChemCGTR45, 85, 1300.806—
BioM=50every 50 steps0.000130
BioM=100100, 2000.000240
BioM=1501500.458—
BioEMA α=0.04—0.471—
BioCGTR127, 1920.485—

Table R2. Predetermined schedules that change only the refresh timing, then frozen.

TaskRefresh steps#RefFrozen fromTimingTest finEnt finCollapse
Chem45, 85, 1303130CGTR (gated)0.8060.780—
Chem30, 70, 1303130fixed, first two moved earlier0.7790.71—
Chem60, 100, 1303130fixed, first two moved later0.7840.69—
Chem127, 1922192fixed, CGTR's Biology steps0.7760.63—
Bio127, 1922192CGTR (gated)0.4850.204—
Bio45, 85, 1303130fixed, CGTR's Chemistry steps0.4381.87—

M=100 and M=150 on Chemistry do not collapse but finish at 0.771 and 0.748; on Biology M=100 collapses at step 240 and M=150 finishes at 0.458, below EMA. More directly, with three refreshes and a freeze from step 130 on Chemistry, exactly as in CGTR, moving only the first two refreshes lowers the final score to 0.779 and 0.784 (CGTR 0.806; its 3-seed std is 0.007), and CGTR's schedules lose more when transplanted across tasks (0.438 vs 0.485 on Biology; 0.776 vs 0.806 on Chemistry). The gain is therefore not "fewer updates"; it is updates at task-appropriate moments, which CGTR identifies from training statistics without per-task tuning.

W1(c): The reward gate leaves the teacher frozen for the last 170 steps; is CGTR effectively a fixed teacher?

CGTR is adaptive in when it stops moving, and that is the intended behavior: refresh while the student is still improving past its last verified level, then hold. It is not equivalent to a fixed teacher because the result depends on which snapshot is held: Fixed (init) 0.725, Fixed@100 0.765, fixed schedules frozen from step 130 like CGTR 0.779 and 0.784, CGTR's step-130 snapshot 0.806 (Appendix H, Table R2). Across tasks the number and timing of refreshes differ (Chemistry 3, Biology 2, Physics 1, ToolUse 2) with one parameter set, which a fixed teacher or fixed interval cannot do. We will state the "consolidate-then-hold" semantics explicitly in Section 4. Within our 300-step runs, holding the teacher after the last refresh was beneficial; for longer or non-stationary training, resetting r_ref to the current rolling reward after K steps without a refresh is a natural option, which we have not evaluated and leave to future work.

AuthorsOpenReview ↗

Response to Reviewer N6jk (3/4)

W2 / Q1: The role of entropy; theoretical justification

We want to be precise here, since we agree the current text can be read as using entropy as evidence for the gate design.

  1. Entropy is not a gate input because it is not an early warning. CGTR has to decide before a copy is made, so it needs a signal that moves before a failure does. In our runs entropy fails this in both directions. EMA entropy rises from ≈0.7 to 3.0, yet EMA never collapses (Section 5.2, Table 3). M=50 entropy stays at its pre-collapse level through step 187 and drops only after the collapse at step 189 (Figure 4). Its single-step peak at step 150 (raw 0.758) is not distinctive: entropy reaches similar values at steps 160, 170, 183, and 187, and the 7-step average plotted in Figure 4 at step 150 is 0.60. Entropy is also high or low for benign reasons, during productive exploration or under over-regularization (Section 3.3). The length tail moves earlier: it expands before abrupt changes appear in mean entropy (Section 3.4, Appendix F), preceding the M=2 collapse by about 12 steps (Figure 5), and in the logs of the M=50 run the tail ratio rises from 2.4 at step 175 to 4.9 at step 187. This is why the length gate, not an entropy gate, guards each refresh. In the revision we will remove the phrasing in Sections 3.2 and 5.2 that presents entropy as evidence for the gates.

  2. CGTR does not rely on a theoretical link between entropy and stability, and we will say so explicitly. The claim we make, and now test directly, concerns the refresh event: an ill-timed hard copy can trigger collapse (Table R4), and the reward gate would have vetoed that copy (below).

  3. The isolation gate would not have blocked the harmful refresh. The reviewer notes that CGTR's isolation gate alone blocks step 150 in CGTR's own run (its last refresh was at 130). That holds in CGTR's run, but it is not the relevant counterfactual, for two reasons. First, the harmful refresh occurred in the M=50 run, whose previous refresh was at step 100, so on that trajectory the isolation gate (50 ≥ 30) is satisfied and cannot be what blocks the copy. Second, the step-130 refresh that makes the isolation gate active at step 150 in CGTR's own run is itself an output of the state-aware gates: without the reward gate (the w/o-reward-gate ablation in Table 4, which keeps M_min=30 and the length gate) the teacher refreshes six times and collapses at step 218, so the refresh history that the reviewer's argument takes as given exists only because the reward and length gates vetoed the clock-eligible refreshes before step 130. We therefore replayed all three gates offline on the logged statistics of the 300-step M=50 runs in Tables 3 and 5 (Table R3; the M=50 run in Figure 5 is the separate 100-step sweep run without warm-up):

TaskRefresh stepRolling reward r̄ (w=5)r_ref + δ_rReward gateTail ratio max/mean (w=5)Length gate (≤3.2)Isolation (≥30)Verdict
Chem500.6520.412+0.05pass2.6passpassrefresh
Chem1000.7410.652+0.05pass2.5passpassrefresh
Chem1500.7630.741+0.05 = 0.791fail3.0passpassveto
Chem2000.000 (post-collapse)0.763+0.05fail1.0passpassveto
Bio500.4050.380+0.05fail2.6passpassveto
Bio1000.4250.405+0.05fail2.9passpassveto

At Chemistry step 150 the student's reward had plateaued (gain 0.022 < 0.05): the reward gate, which is state-aware, vetoes the copy; the isolation gate does not. On Biology both early M=50 refreshes would have been vetoed for insufficient reward gain, which is consistent with CGTR's actual first Biology refresh occurring at step 127, once r̄ reached 0.438 > 0.430. The w/o-reward-gate ablation (Table 4), which keeps isolation and length gates but collapses at 218, is the trained-run counterpart of this replay.

Reviewer_tRf4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how teacher-update schedules affect stability in self on-policy distillation, where a historical version of the student provides feedback-conditioned supervision. It compares periodic hard refreshes, a frozen teacher, and exponential moving average updates, and proposes Consolidation-Gated Teacher Refresh (CGTR). CGTR updates the teacher only after a minimum isolation period and when both a rolling reward-improvement condition and a response-length-tail condition are satisfied.

The evaluation uses Qwen3-8B on three SciKnowEval subjects and ToolAlpaca, with a 100-step schedule sweep and 300-step comparisons. Additional validation includes gate ablations, threshold sensitivity, frozen-teacher controls, and three-seed experiments on Chemistry and Biology. The paper reports the highest final accuracy among the compared methods on all four tasks, with no observed CGTR collapse, and attributes these benefits to teacher isolation and state-dependent refresh timing.

Critical strengths
  1. Meaningful problem formulation and practical method. The paper studies teacher-update scheduling as an important but relatively underexplored factor affecting self on-policy distillation stability. The proposed CGTR method is conceptually simple, interpretable, and easy to integrate into existing training pipelines, requiring no additional model inference or learned components.

  2. Systematic empirical analysis and useful experimental validation.The paper provides a relatively comprehensive investigation of teacher-update schedules, including fixed-interval refreshes, EMA, and frozen teachers. The experiments further include long-horizon comparisons, gate ablations, hyperparameter sensitivity, and multi-seed verification. These analyses provide useful empirical insights into training instability and demonstrate the practical potential of adaptive teacher updates.

Critical weaknesses

1、The long-horizon comparison has a related confound: M=50M=50 performs six refreshes, whereas CGTR performs three on Chemistry and then freezes the teacher from step 130 to 300. The gate ablations also change refresh counts. The frozen-teacher controls in Appendix H are valuable, but do not isolate state-dependent timing from fewer updates and a longer final freezing period. Similarly, attributing the step-189 collapse specifically to the step-150 refresh remains an inference without a targeted intervention. These limitations matter because the paper presents isolation and state-aware refresh as necessary structural properties, beyond demonstrating that the proposed heuristic works well against the tested schedules. 2、For M=2M=2, Tables 1 and 6 report mean training entropy of 0.900, while the corresponding entropy trajectory in Figures 2 and 5 appears to remain above approximately 3 for much of the latter part of training. These are difficult to reconcile under a straightforward average over training steps. For the long-horizon M=50M=50 run, Section 5.2 reports entropy of approximately 0.98 at step 150, whereas Figure 4 appears closer to 0.6; Table 3 reports final entropy of 0.525, while the plotted endpoint also appears appreciably lower.

Different runs, aggregation conventions, or illustrative rather than empirical curves could explain these differences, but the manuscript does not make such distinctions explicit. This is consequential because the claims about contamination, consolidation, and harmful refreshes rely directly on these diagnostics. The figures and statistics should be reconciled, with their precise definitions and provenance specified.

Questions for authors
  1. Evidence isolating adaptive timing: Does CGTR retain its advantage over state-independent schedules with comparable refresh counts and final freezing periods? A targeted comparison with a longer fixed interval or a predetermined refresh-and-freeze schedule would help determine whether the benefit specifically comes from conditioning updates on the student’s state.

  2. Evidence for the proposed collapse mechanism: Starting from the same checkpoint immediately before the suspected harmful refresh at step 150, does retaining the previous teacher or delaying that refresh prevent the later collapse under otherwise matched conditions? Also, does “irreversible” mean that no spontaneous recovery was observed through step 300, or is a stronger claim intended? This distinction matters because the run reaches its reported peak accuracy at step 187 before collapsing at step 189.

  3. Clarification of diagnostic reporting: Are Figures 2, 4, and 5 empirical traces from the same runs summarized in Tables 1, 3, and 6? Please specify the aggregation, weighting, and smoothing conventions, and reconcile the entropy discrepancies identified above. If any curves are schematic, please distinguish them explicitly from the empirical evidence.

3 comments
AuthorsOpenReview ↗

Response to Reviewer tRf4 (1/3)

We thank the reviewer for the careful and constructive review, and address each point below. New evidence for this review: larger fixed intervals (Table R1); predetermined schedules that match CGTR's refresh count and final freeze but move the refresh times, plus cross-task transplants (Table R2); a counterfactual intervention branching from the checkpoint immediately before the suspected harmful refresh (Table R4); and a re-derivation of every entropy statistic from the logs, which corrects several entropy statistics (W2 / Q3).

W1 / Q1: Confound between state-dependent timing and fewer updates + longer freeze

We ran exactly the two controls the reviewer suggests, on Chemistry and Biology for 300 steps with the setup of Appendix A and the CGTR parameters of Appendix C unchanged.

(i) Longer fixed intervals (Table R1, single seed).

TaskScheduleRefresh stepsTest finCollapse
ChemM=50every 50 steps0.000189
ChemM=100100, 2000.771—
ChemM=1501500.748—
ChemFixed@100 (App. H)freeze M=50 teacher from 1000.765—
ChemFixed (init) ≡ M≥300none0.725—
ChemCGTR45, 85, 1300.806—
BioM=50every 50 steps0.000130
BioM=100100, 2000.000240
BioM=1501500.458—
BioEMA α=0.04—0.471—
BioCGTR127, 1920.485—

Fewer updates and a longer final freeze do not by themselves reproduce CGTR's result. On Chemistry, M=100 (refreshes at steps 100 and 200, frozen from step 200) and M=150 (refresh at step 150, frozen from step 150) finish at 0.771 and 0.748, both below CGTR (0.806; 3 refreshes, frozen from step 130). On Biology, M=100 collapses at step 240 and M=150 survives at 0.458, below EMA (0.471). The safe interval therefore also differs between the two tasks.

(ii) Predetermined schedules with CGTR's refresh count and final freeze (Table R2). A fixed schedule at CGTR's own Chemistry steps {45, 85, 130} needs no separate run: CGTR's gates only decide when to copy, so with the same seed it is CGTR's run. We therefore hold the refresh count (three) and the final freeze (from step 130) fixed and move only the first two refreshes. We also transplant each task's CGTR schedule to the other task.

TaskRefresh steps#RefFrozen fromTimingTest finEnt finCollapse
Chem45, 85, 1303130CGTR (gated)0.8060.780—
Chem30, 70, 1303130fixed, first two moved earlier0.7790.71—
Chem60, 100, 1303130fixed, first two moved later0.7840.69—
Chem127, 1922192fixed, CGTR's Biology steps0.7760.63—
Bio127, 1922192CGTR (gated)0.4850.204—
Bio45, 85, 1303130fixed, CGTR's Chemistry steps0.4381.87—

With the count and the final freeze matched exactly, as the reviewer requested, moving the first two refreshes earlier or later lowers the final score to 0.779 and 0.784 (CGTR: 0.806; 3-seed std 0.007). Transplanting schedules across tasks costs more: the Chemistry schedule on Biology finishes at 0.438 vs 0.485, and the Biology schedule on Chemistry at 0.776 vs 0.806. The benefit is carried by when the refreshes happen, not by how many there are or how long the final freeze lasts, and the right timing is task-specific. CGTR reads it from the student's state with one parameter set.

We agree with the reviewer's broader point about how strongly the paper words "necessary structural properties." We will restate Section 3.5 as two empirically supported design requirements and remove the word "necessary"; Table 2 will be retitled accordingly.

AuthorsOpenReview ↗

Response to Reviewer tRf4 (2/3)

W1 / Q2: Direct evidence for the collapse mechanism; meaning of "irreversible"

Intervention (Table R4). We branch from the saved M=50 Chemistry checkpoint at step 149 (student, teacher = step-100 snapshot, optimizer state) and run six continuations to step 300 under otherwise identical settings. Rollout sampling is re-seeded in each branch.

BranchTeacher handling from step 150Test finTest peakCollapse
C0-aRefresh at 150 (as in M=50), then 200, 2500.0000.747197
C0-bsame, different rollout seed0.0000.751211
C1-aSkip the step-150 refresh only; keep step-100 teacher; refresh at 200, 2500.7720.779—
C1-bsame, different rollout seed0.7660.774—
C2Delay the refresh from 150 to 170; then 220, 2700.7610.770—
C3Apply CGTR gates from step 149 (t_last=100, r_ref=0.741)0.7910.796—
(orig.)M=50 seed 00.0000.744189

Both continuations that perform the step-150 copy collapse (at 197 and 211, within the 189–230 range of Appendix G); both that skip only that copy do not, and neither do the delayed or gated variants. The branches share everything up to step 149 and differ afterwards only in this refresh decision and in the re-seeded rollout sampling, so we now state the mechanism as an intervention result rather than an inference. In C3 the gates block at 150 and 155–180 (reward gate), release at step 185, and the run finishes at 0.791.

Peak at 187 before collapse at 189. The step-150 copy does not lower accuracy right away; it changes the target the student is distilled toward, and the failure surfaces abruptly. Between steps 150 and 187, test mean@16 stays in its pre-refresh range and peaks at 0.744, and token entropy stays at its pre-collapse level (Figure 4). The length tail changes first: in the logged length statistics, the tail ratio rises from 2.4 at step 175 to 4.9 at step 187. From step 188 responses fall into repetitive loops that run to the 8,192-token cap (mean length 7,850 tokens) and no longer emit the answer tag, and by step 189 reward and test accuracy are at zero. A peak followed by an abrupt collapse is therefore what this mechanism predicts. Accuracy is not the signal that exposes a damaged teacher, which is why CGTR checks reward improvement and the length tail at refresh time.

"Irreversible." We mean the weaker, observational claim: no recovery was observed through step 300 in the original run, in the two additional seeds (Appendix G), or in C0-a/C0-b. We will replace "irreversible" with "no recovery observed within the 300-step horizon" throughout, and note in Limitations that we did not test longer horizons or intervention after collapse.

AuthorsOpenReview ↗

Response to Reviewer tRf4 (3/3)

W2 / Q3: Reconciling entropy statistics with the plotted curves

Figures 2, 4, and 5 are all empirical traces from the same runs summarized in Tables 1, 3, and 6; none of them is schematic. The only conceptual diagram in the paper is Figure 3, which sketches the update rules on the refresh timeline of the Chemistry runs. We re-derived every entropy number in the paper from the raw training logs and address the reviewer's two observations in turn.

(1) M=2: the table is wrong; the figure is right. Recomputing from the complete 100-step entropy log of the M=2 run gives

Quantity (M=2, 100 steps)Paper (Tables 1, 6)Recomputed from logsStatus
Entropy mean over steps0.9003.213error, will be corrected
Entropy std over steps (population)1.0631.685error, will be corrected
Final-step entropy5.2265.226correct
Trajectory rising to ≈5.2 (Figs. 2, 5)as plottedas loggedcorrect

The reviewer's reading is therefore accurate: M=2 entropy does remain above ≈3 for much of the latter part of training, and a straightforward step-average cannot be 0.900. We were unable to trace how the two values were produced; they do not correspond to the statistics of the logged series, and we have no record of the computation that generated them. We apologize for the error. To rule out any similar issue, we regenerated every entropy statistic in Tables 1, 3, and 6 directly from the logged series with a single script. Its scope is limited to these two cells: the values are not cited anywhere in the text, and no claim rests on them. The corrected values make M=2 the highest-entropy and highest-variance schedule in the sweep (next highest mean: M=5 at 2.001; next highest std: M=3 at 1.167), which if anything sharpens the contrast in Section 3.2 between tight coupling and sparse refresh. That re-derivation reproduced every other Ent. mean / std / fin entry in Tables 1, 3, and 6 except the M=50 row of Table 3, which we correct in (2) below.

(2) M=50 (300 steps): Section 5.2 and Table 3. Text and tables report raw per-step values; Figure 4 plots a smoothed curve. The smoothing is a centered seven-step moving average whose window is truncated at the boundaries and renormalized,

H̃_t = Σ_{k=max(1,t−3)}^{min(T,t+3)} H_k / (min(T,t+3) − max(1,t−3) + 1),

so interior points average seven raw values and the plotted endpoint is H̃_T = (H_{T−3} + H_{T−2} + H_{T−1} + H_T)/4. At step 150 the raw M=50 entropy is 0.758, and its 7-step average plotted in Figure 4 is 0.60; the ≈0.98 in Section 5.2 is the CGTR run's value at that step, and we will correct it to 0.758. The M=50 row of Table 3 covers steps 1–188, up to the collapse, while the EMA and CGTR rows cover all 300 steps. Over all 300 steps the M=50 mean is 0.455 and the final value is 0.166, which the formula above smooths to the plotted endpoint of 0.28. We will report all three rows over the full run, with the pre-collapse values (0.539 / 0.525) in a separate column. The thin, lighter lines already drawn in Figure 4 are the raw per-step series; we will say so and state the window in the caption. We note that this centered window is non-causal and used only for display; the rolling statistics inside CGTR's gates are trailing (causal) windows of w=5 steps and are unaffected.

Definitions we will add to Appendix A.

  • Per-step token entropy H_t: mean over all generated tokens of the 64 training rollouts at step t of the entropy of the student's next-token distribution (full vocabulary, temperature 1.0), as logged under actor/entropy.
  • Ent. mean / std (Tables 1, 3, 6): unweighted mean and population standard deviation of H_t over all training steps of the run (t = 1…T); not token-weighted, not smoothed. For M=1, over steps 1–16 only (as footnoted).
  • Ent. fin: raw H_T at the final step.
  • Plotted curves: Figure 4 shows a centered 7-step moving average, truncated and renormalized at the boundaries as above (thick), over the raw series (thin). In Figure 2 (both panels, 100-step sweep), entropy is a centered 3-step moving average with edge replication, H̃_t = (H_{max(1,t−1)} + H_t + H_{min(T,t+1)})/3, and KL is a trailing 2-step average, K̃_t = (K_{max(1,t−1)} + K_t)/2. Figure 5 plots raw values with no smoothing. All smoothing is for display only; no table statistic and no CGTR gate uses smoothed values. In the revision we will state the smoothing window in the caption of each figure.
  • We will retitle Figure 2 from "Illustration of two refresh dynamics" to "Empirical KL and entropy traces from the schedule sweep (M=50, M=2, EMA α=0.1)".

We thank the reviewer again for the thoughtful and detailed review, and we would be grateful if the new experiments and revisions led the reviewer to reconsider the rating.

Reviewer_xejgOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when to update a historical self-teacher during self on-policy distillation. The student learns on its own rollouts, while the teacher is an earlier copy of the student and receives feedback context that the student does not see. The update schedule controls how much separation remains between them. The authors compare periodic hard refresh, a teacher frozen from a selected checkpoint, and exponential moving average updates. Frequent hard refresh leaves little historical separation, while EMA can gradually carry student drift into the teacher.

The proposed method is Consolidation-Gated Teacher Refresh (CGTR). Instead of updating the teacher on a fixed clock, CGTR waits until three checks pass. The teacher must have remained fixed for a minimum period, recent reward must exceed the value recorded at the previous refresh, and the response-length tail must stay below a threshold. The goal is to refresh only after the student appears to have consolidated rather than during a temporary unstable phase.

The experiments use Qwen3-8B on Chemistry, Biology, Physics, and ToolUse. The paper begins with a 100-step schedule study, then follows several methods for 300 steps and adds gate ablations, frozen-teacher controls, and sensitivity checks. Chemistry and Biology are also run with three seeds. CGTR has the highest reported final mean@16 score on all four tasks and avoids the late collapse seen with some fixed refresh schedules.

Critical strengths
  1. The long runs reveal a failure that the short schedule study misses.

    This was the most useful part of the paper for me. With the fixed interval M = 50, the model is strongest at the end of the 100-step comparison, so it would be the natural schedule to select from that experiment. It then collapses in the 300-step Chemistry run. The repeated Chemistry and Biology runs show the same general risk rather than a single isolated curve. A short experiment would therefore have led to the wrong practical choice.

    I also liked that the paper does more than report a lower final score. It follows response length, entropy, teacher-student KL, and the timing of individual refreshes. These traces make the proposed failure story easier to inspect. Whether or not CGTR is the final solution, the delayed failure of a seemingly good fixed schedule is worth reporting.

  2. CGTR is a fairly simple response to the problem, and it works consistently in the reported experiments.

    CGTR uses reward and response-length statistics that are already produced during training. It does not require another learned controller, and the same parameter setting is used for all four tasks. The gate ablations are helpful because they show that removing individual conditions changes the refresh pattern and can reintroduce instability. CGTR also finishes ahead of the reported fixed-refresh, EMA, and frozen-teacher controls on the paper's primary mean@16 metric.

Critical weaknesses
  1. I still can't separate adaptive gating from the choice of refresh schedule or freezing point.

    The longer runs and cross-task tables mainly compare CGTR with the fixed interval M = 50 and an EMA coefficient of 0.04. Yet one of the paper's own findings is that the best fixed interval depends on the task and on how long training continues. This makes M = 50 a somewhat uneven reference: it is strong in the short Chemistry study but need not be the best fixed choice for every task or for 300 steps. The frozen-teacher controls are useful, although freezing only at initialization and step 100 leaves many possible stopping points untested.

    I also think the reference-collapse explanation needs narrower wording. In Equation 1, the teacher receives feedback context f and the student does not. Equal teacher and student parameters therefore do not imply equal output distributions. Some distillation signal can remain even immediately after a hard refresh. The experiments support the value of a historical parameter reference, but they do not show that every corrective signal disappears when the parameters are synchronized.

  2. The reward condition appears to make CGTR converge toward a frozen teacher.

    After a refresh, the next refresh requires the rolling reward to exceed the new reference by the fixed reward margin. Since reward is bounded, this test becomes progressively harder to pass. In the Chemistry run, the third refresh occurs at step 130, raises the next threshold to 0.894, and is followed by 170 steps with no further teacher update.

    This may be a feature rather than a bug. My reading of the Chemistry run is that CGTR finds a useful teacher and then benefits from leaving it alone. But that interpretation is different from an adaptive teacher that keeps moving when the student reaches new stable states.

Questions for authors
  1. The long and cross-task experiments mainly use the fixed interval M = 50 and an EMA coefficient of 0.04 as the moving-teacher baselines. Does CGTR still outperform a reasonable fixed interval chosen for each task, or a nonadaptive schedule with roughly the same number and timing of refreshes? This comparison would help separate useful state-dependent gating from a favorable sparse schedule. Also, how should the reference-collapse claim be interpreted when the teacher receives feedback context f but the student does not? Equal parameters do not appear to imply equal output distributions in Equation 1.

  2. Is eventual teacher freezing an intended outcome of CGTR? In the Chemistry run, the refresh at step 130 raises the next reward threshold to 0.894, and the teacher is not updated again during the remaining 170 steps. If freezing is not the intended endpoint, what should happen after a long reward plateau, a later decline, or a new training phase in which the old teacher may no longer be the right reference? Would the reward reference need to decay or reset, or should the teacher remain fixed?

3 comments
AuthorsOpenReview ↗

Response to Reviewer xejg (1/2)

We thank the reviewer for the precise summary and for highlighting the long-horizon finding. In response we ran two new sets of controls on Chemistry and Biology (300 steps, setup of Appendix A, CGTR parameters of Appendix C unchanged): larger fixed intervals (Table R1) and predetermined schedules built from CGTR's refresh steps that change only the timing, within and across tasks (Table R2). We also revise the wording of "reference collapse" and make the intended semantics of the reward ratchet explicit. We address the two weaknesses and two questions below.

W1 / Q1(a): Separating adaptive gating from the choice of refresh schedule or freezing point

We agree that M=50 and EMA α=0.04 alone cannot separate these. We added two sets of controls.

(i) Longer fixed intervals (Table R1, Chemistry and Biology, 300 steps, single seed).

TaskScheduleRefresh stepsTest finCollapse
ChemM=50every 50 steps0.000189
ChemM=100100, 2000.771—
ChemM=1501500.748—
ChemFixed@100 (App. H)freeze M=50 teacher from 1000.765—
ChemFixed (init) ≡ M≥300none0.725—
ChemCGTR45, 85, 1300.806—
BioM=50every 50 steps0.000130
BioM=100100, 2000.000240
BioM=1501500.458—
BioEMA α=0.04—0.471—
BioCGTR127, 1920.485—

On Chemistry the best fixed interval is M=100; on Biology M=100 collapses and the safe choice is M=150, which is nonetheless worse than EMA. Choosing a fixed M that is both safe and strong on both tasks would require a sweep over M at the full 300-step horizon on each task. The 100-step sweep cannot reveal it: M=50 is the best schedule at 100 steps and collapses at 300. CGTR, with one parameter set fixed from the 100-step Chemistry sweep, is best on both.

(ii) Predetermined schedules with matched refresh count (Table R2). To test whether a non-adaptive schedule with roughly the same number and timing of refreshes suffices, we fix the refresh steps in advance and never consult the student. A fixed schedule at CGTR's own Chemistry steps {45, 85, 130} needs no separate run: CGTR's gates only decide when to copy, so with the same seed it is CGTR's run. We therefore change only the timing. On Chemistry we keep three refreshes and the final refresh at step 130 and move the first two; across tasks we transplant each task's CGTR schedule to the other.

TaskRefresh steps#RefFrozen fromTimingTest finEnt finCollapse
Chem45, 85, 1303130CGTR (gated)0.8060.780—
Chem30, 70, 1303130fixed, first two moved earlier0.7790.71—
Chem60, 100, 1303130fixed, first two moved later0.7840.69—
Chem127, 1922192fixed, CGTR's Biology steps0.7760.63—
Bio127, 1922192CGTR (gated)0.4850.204—
Bio45, 85, 1303130fixed, CGTR's Chemistry steps0.4381.87—

With the refresh count and the final freeze held fixed, moving only the first two refreshes lowers the final score by 0.027 and 0.022, three to four times the 3-seed std of 0.007. Transplanting schedules across tasks costs more. The Chemistry schedule on Biology finishes below EMA and 0.047 below CGTR's self-selected Biology timing (Biology 3-seed std: 0.003), with entropy drifting to 1.87. In the CGTR Biology run the rolling reward at steps 45 and 85 is 0.402 and 0.421, still below the first-refresh threshold of 0.430 (initial reward 0.380 plus δ_r = 0.05), so the transplanted refreshes land where CGTR's reward gate finds no verified improvement. Conversely, Biology's later timing on Chemistry loses 0.030. What is adaptive about CGTR is therefore the timing of each refresh, which it reads from the student's state without per-task tuning, not the number of refreshes or the length of the final freeze.

We will add both tables to the main paper (Section 5.3) and move the frozen-teacher controls alongside them.

AuthorsOpenReview ↗

Response to Reviewer xejg (2/2)

W1 / Q1(b): Wording of "reference collapse" given the feedback context f

The reviewer is right. In Eq. 1 the teacher conditions on f and the student does not, so ϕ=θ does not imply π_ϕ(·|x,f,·)=π_θ(·|x,·), and some distillation signal remains even at M=1. We will rewrite the claim as follows: "At M=1 the historical-parameter component of the teacher signal vanishes; the remaining signal is only the context-conditioning effect of f applied to the current parameters. The M=1 run (response length >8,000 at step 10, terminated at step 16) and the M=2 run (0.225) show that this remaining signal is insufficient to anchor training. We call the loss of the historical component anchor loss rather than reference collapse." We will apply this wording wherever the term appears: Sections 1 and 3.2, Table 2, Appendix A.6, and the caption of Figure 5.

W2 / Q2: Does the reward condition make CGTR converge to a frozen teacher? Is that intended?

Yes, and we should have said so explicitly. The ratchet is monotone by design: CGTR implements "refresh while the student is still consolidating, then hold." Three points:

  1. The value is in which snapshot is held, not in holding per se. Freezing at initialization gives 0.725 and freezing the M=50 snapshot at step 100 gives 0.765, versus 0.806 for the snapshot CGTR selected at step 130 (Appendix H). In Table R2, two fixed schedules freeze at exactly the same step as CGTR, step 130, and finish at 0.779 and 0.784: the quality of the held snapshot depends on the refreshes that led to it. The ratchet's job is to ensure the held snapshot encodes a verified improvement.

  2. The held teacher is not inert. During the final 170 frozen steps the student improved from 0.752 to 0.806 against the step-130 teacher, i.e., the frozen target still carried a corrective signal via f (which is exactly the mechanism the reviewer notes in Q1).

  3. Non-stationary training. The scenarios the reviewer raises are a fair point. Within our 300-step runs, holding the teacher after the last refresh was beneficial (points 1 and 2), so we did not need a way to move it again. For longer or non-stationary training, a natural option is to reset the reference: if no refresh has occurred for K steps, r_ref is re-anchored to the current rolling reward, so the gate asks for improvement over the student's recent level rather than over its best past level. We have not evaluated this variant and see it as a natural direction for future work.

We hope these additions address the reviewer's concerns; we would be grateful if the reviewer would consider raising the score if so.

Thanks to the authors for the reply. After reading all the review and rebuttals, I still lean to a weak accept of this work. I will maintain my scores.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.