- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper focuses on the self on-policy distillation framework, in which they identify a stablility issue from the teachers update scheduling. The paper uses one base model, Qwen3-8B, and tests on limited set of task catagories, where they find some useful observations that can be used for the proposed CGTR algorithm. Moreover, the proposed CGTR is largely heuristics and does not compare with related adaptive update baselines. There are also some overclaims, and the expalnation on the entropy results is not convincing.
Critical strengths
The only strength I can identify is that the paper provides a clear empirical study cases of the teacher update scheduling. However, due to its limited expeiments setup and comparisons, the advantage is not well established.
Critical weaknesses
- Similar ideas have already been studied beyond LLMs, particularly in target-network and momentum-teacher settings. CGTR should therefore be compared with related adaptive strategies, not only the fixed schedules and EMA. More importantly, the long-horizon experiments omit larger fixed intervals (e.g., M=100/150), which makes it unclear whether CGTR's gains come from state-aware refreshing or simply fewer teacher updates. The reward gate also becomes increasingly restrictive, leaving the teacher frozen for the last 170 steps on Chemistry. This raises the concern of whether CGTR genuinely adapts to training dynamics or effectively becomes a fixed teacher.
- The paper repeatedly uses token entropy dynamics as evidence of teacher contamination, but the connection to CGTR's design is not well established. Section 3.3 acknowledges that entropy alone is an unreliable consolidation signal, yet Section 5.2 uses elevated entropy at step 150 as the main evidence of an unsafe teacher refresh. However, none of CGTR's gates monitors entropy, and the paper does not show whether the reward or length-tail criteria could detect this instability. Moreover, CGTR's isolation gate alone would prevent a refresh at step 150, leaving unclear whether the state-aware gates actually address the identified failure mechanism.
Questions for authors
- What theoretical justification supports the connection between token entropy dynamics and the claimed training stability? In particular, how reliably can entropy distinguish stable learning from the proposed failure modes?
- Can the authors show the direct evidence that the collapse is caused by an ill-timed teacher refresh?
6 comments
Response to Reviewer N6jk (2/4)
W1(b): Larger fixed intervals; is the gain just fewer updates?
We added M=100 and M=150 on Chemistry and Biology (300 steps; Table R1). To separate state-aware timing from the number of updates directly, we also ran predetermined schedules that keep CGTR's refresh count and final freeze on Chemistry but move the refresh times, and transplanted each task's CGTR schedule to the other task (Table R2).
Table R1. Larger fixed intervals (single seed).
| Task | Schedule | Refresh steps | Test fin | Collapse |
|---|---|---|---|---|
| Chem | M=50 | every 50 steps | 0.000 | 189 |
| Chem | M=100 | 100, 200 | 0.771 | — |
| Chem | M=150 | 150 | 0.748 | — |
| Chem | Fixed@100 (App. H) | freeze M=50 teacher from 100 | 0.765 | — |
| Chem | Fixed (init) ≡ M≥300 | none | 0.725 | — |
| Chem | CGTR | 45, 85, 130 | 0.806 | — |
| Bio | M=50 | every 50 steps | 0.000 | 130 |
| Bio | M=100 | 100, 200 | 0.000 | 240 |
| Bio | M=150 | 150 | 0.458 | — |
| Bio | EMA α=0.04 | — | 0.471 | — |
| Bio | CGTR | 127, 192 | 0.485 | — |
Table R2. Predetermined schedules that change only the refresh timing, then frozen.
| Task | Refresh steps | #Ref | Frozen from | Timing | Test fin | Ent fin | Collapse |
|---|---|---|---|---|---|---|---|
| Chem | 45, 85, 130 | 3 | 130 | CGTR (gated) | 0.806 | 0.780 | — |
| Chem | 30, 70, 130 | 3 | 130 | fixed, first two moved earlier | 0.779 | 0.71 | — |
| Chem | 60, 100, 130 | 3 | 130 | fixed, first two moved later | 0.784 | 0.69 | — |
| Chem | 127, 192 | 2 | 192 | fixed, CGTR's Biology steps | 0.776 | 0.63 | — |
| Bio | 127, 192 | 2 | 192 | CGTR (gated) | 0.485 | 0.204 | — |
| Bio | 45, 85, 130 | 3 | 130 | fixed, CGTR's Chemistry steps | 0.438 | 1.87 | — |
M=100 and M=150 on Chemistry do not collapse but finish at 0.771 and 0.748; on Biology M=100 collapses at step 240 and M=150 finishes at 0.458, below EMA. More directly, with three refreshes and a freeze from step 130 on Chemistry, exactly as in CGTR, moving only the first two refreshes lowers the final score to 0.779 and 0.784 (CGTR 0.806; its 3-seed std is 0.007), and CGTR's schedules lose more when transplanted across tasks (0.438 vs 0.485 on Biology; 0.776 vs 0.806 on Chemistry). The gain is therefore not "fewer updates"; it is updates at task-appropriate moments, which CGTR identifies from training statistics without per-task tuning.
W1(c): The reward gate leaves the teacher frozen for the last 170 steps; is CGTR effectively a fixed teacher?
CGTR is adaptive in when it stops moving, and that is the intended behavior: refresh while the student is still improving past its last verified level, then hold. It is not equivalent to a fixed teacher because the result depends on which snapshot is held: Fixed (init) 0.725, Fixed@100 0.765, fixed schedules frozen from step 130 like CGTR 0.779 and 0.784, CGTR's step-130 snapshot 0.806 (Appendix H, Table R2). Across tasks the number and timing of refreshes differ (Chemistry 3, Biology 2, Physics 1, ToolUse 2) with one parameter set, which a fixed teacher or fixed interval cannot do. We will state the "consolidate-then-hold" semantics explicitly in Section 4. Within our 300-step runs, holding the teacher after the last refresh was beneficial; for longer or non-stationary training, resetting r_ref to the current rolling reward after K steps without a refresh is a natural option, which we have not evaluated and leave to future work.
Response to Reviewer N6jk (3/4)
W2 / Q1: The role of entropy; theoretical justification
We want to be precise here, since we agree the current text can be read as using entropy as evidence for the gate design.
-
Entropy is not a gate input because it is not an early warning. CGTR has to decide before a copy is made, so it needs a signal that moves before a failure does. In our runs entropy fails this in both directions. EMA entropy rises from ≈0.7 to 3.0, yet EMA never collapses (Section 5.2, Table 3). M=50 entropy stays at its pre-collapse level through step 187 and drops only after the collapse at step 189 (Figure 4). Its single-step peak at step 150 (raw 0.758) is not distinctive: entropy reaches similar values at steps 160, 170, 183, and 187, and the 7-step average plotted in Figure 4 at step 150 is 0.60. Entropy is also high or low for benign reasons, during productive exploration or under over-regularization (Section 3.3). The length tail moves earlier: it expands before abrupt changes appear in mean entropy (Section 3.4, Appendix F), preceding the M=2 collapse by about 12 steps (Figure 5), and in the logs of the M=50 run the tail ratio rises from 2.4 at step 175 to 4.9 at step 187. This is why the length gate, not an entropy gate, guards each refresh. In the revision we will remove the phrasing in Sections 3.2 and 5.2 that presents entropy as evidence for the gates.
-
CGTR does not rely on a theoretical link between entropy and stability, and we will say so explicitly. The claim we make, and now test directly, concerns the refresh event: an ill-timed hard copy can trigger collapse (Table R4), and the reward gate would have vetoed that copy (below).
-
The isolation gate would not have blocked the harmful refresh. The reviewer notes that CGTR's isolation gate alone blocks step 150 in CGTR's own run (its last refresh was at 130). That holds in CGTR's run, but it is not the relevant counterfactual, for two reasons. First, the harmful refresh occurred in the M=50 run, whose previous refresh was at step 100, so on that trajectory the isolation gate (50 ≥ 30) is satisfied and cannot be what blocks the copy. Second, the step-130 refresh that makes the isolation gate active at step 150 in CGTR's own run is itself an output of the state-aware gates: without the reward gate (the w/o-reward-gate ablation in Table 4, which keeps M_min=30 and the length gate) the teacher refreshes six times and collapses at step 218, so the refresh history that the reviewer's argument takes as given exists only because the reward and length gates vetoed the clock-eligible refreshes before step 130. We therefore replayed all three gates offline on the logged statistics of the 300-step M=50 runs in Tables 3 and 5 (Table R3; the M=50 run in Figure 5 is the separate 100-step sweep run without warm-up):
| Task | Refresh step | Rolling reward r̄ (w=5) | r_ref + δ_r | Reward gate | Tail ratio max/mean (w=5) | Length gate (≤3.2) | Isolation (≥30) | Verdict |
|---|---|---|---|---|---|---|---|---|
| Chem | 50 | 0.652 | 0.412+0.05 | pass | 2.6 | pass | pass | refresh |
| Chem | 100 | 0.741 | 0.652+0.05 | pass | 2.5 | pass | pass | refresh |
| Chem | 150 | 0.763 | 0.741+0.05 = 0.791 | fail | 3.0 | pass | pass | veto |
| Chem | 200 | 0.000 (post-collapse) | 0.763+0.05 | fail | 1.0 | pass | pass | veto |
| Bio | 50 | 0.405 | 0.380+0.05 | fail | 2.6 | pass | pass | veto |
| Bio | 100 | 0.425 | 0.405+0.05 | fail | 2.9 | pass | pass | veto |
At Chemistry step 150 the student's reward had plateaued (gain 0.022 < 0.05): the reward gate, which is state-aware, vetoes the copy; the isolation gate does not. On Biology both early M=50 refreshes would have been vetoed for insufficient reward gain, which is consistent with CGTR's actual first Biology refresh occurring at step 127, once r̄ reached 0.438 > 0.430. The w/o-reward-gate ablation (Table 4), which keeps isolation and length gates but collapses at 218, is the trained-run counterpart of this replay.
Response to Reviewer N6jk (1/4)
We thank the reviewer for the feedback. In response we added larger fixed intervals and predetermined schedules that change only the refresh timing (Tables R1, R2), an offline replay of CGTR's gates on the logs of the collapsing M=50 runs (Table R3), a counterfactual intervention at the suspected harmful refresh (Table R4), and an analysis of why target-network and momentum-teacher mechanisms do not carry over to this setting (Table R5). We also revise the claims the reviewer found overstated. All new runs use the setup of Appendix A and the unchanged CGTR parameters of Appendix C.
W1(a): Comparison with adaptive strategies from target-network / momentum-teacher literature
We agree that the question of when a reference network should move has been studied outside LLMs, and we will expand Related Work to cover target networks in value-based RL (Mnih et al., 2015; Lillicrap et al., 2016; Zhang et al., 2021), momentum teachers (Tarvainen & Valpola, 2017; Grill et al., 2020; Caron et al., 2021), and learned teacher updates (Pham et al., 2021). These mechanisms were designed for settings with a replay buffer or a fixed data distribution. In self on-policy distillation the student trains on its own rollouts, so whatever the teacher absorbs is fed back into the next training data, and failures such as repetition loops appear abruptly. Three of these families are already baselines in the paper: periodic hard updates, EMA targets, and frozen teachers. Table R5 lists for every family why it does not carry over, and how CGTR addresses it.
We will add Table R5 to the paper. If the reviewer has a specific adaptive method in mind, we will add it as a baseline.