acceptodds
Under review as a conference paper at ICLR 2027

Teaching a Moving Student: Rethinking the Curriculum of On-Policy Distillation

Abstract

In on-policy distillation (OPD), the student not only learns from the teacher but also determines which states receive future supervision. As the student evolves, training moves to new response prefixes while teacher–student disagreement can remain on earlier ones. Under matched trajectory and optimization budgets, neither more training queries nor more frequent rollout resampling is uniformly beneficial. Fresh current-policy rollouts outperform initial-policy rollouts at shorter response budgets, but the ranking reverses at longer budgets. Fixed-prefix measurements show that earlier prefixes become less likely under the student while teacher–student disagreement on them persists. Replaying initial-policy rollouts after current-policy training improves accuracy, whereas replaying fixed recent rollouts does not reproduce the gain. We therefore propose R-OPD, a gradient-triggered curriculum that lets the student learn on current-policy states before adaptively introducing replay, without prescribing a fixed transition iteration. Once mean gradient changes fall within minibatch-level variation, training switches to initial-policy rollouts from the next iteration through the remaining training budget. Across eight mathematics benchmarks, R-OPD improves average accuracy over continued current-policy sampling by 2.25 and 4.44 percentage points at 16K and 32K for a 0.6B student across three training runs, and by 2.81 and 6.15 points for a 1.7B student. With a 30B-A3B teacher, R-OPD also improves 8B accuracy by 4.05 points at 32K. Fixed two-stage replay also improves accuracy. At 32K, R-OPD exceeds a fixed schedule of 40 current-policy updates followed by 20 replay updates by 1.39/1.66/1.85 points for 0.6B/1.7B/8B, averaged over three training-data orders. At the same 32K generation cap, R-OPD also produces longer responses, suggesting that well-timed revisits help the student use more of its reasoning capacity.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, knowledge distillation, large language models, curriculum learning, mathematical reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Pr8AOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies curriculum design in on-policy distillation (OPD), where the evolving student policy determines which states receive teacher supervision. The authors find that query count interacts with rollout resampling frequency, and that the relative advantage of current-policy versus initial-policy rollouts reverses as the response budget increases. Fixed-prefix diagnostics show that earlier prefixes become less likely while teacher–student disagreement persists. Motivated by these findings, R-OPD switches from current-policy training to initial-policy replay when changes in mean gradients fall within an empirical within-iteration variation scale. Across three student sizes and eight mathematics benchmarks, the method improves average accuracy at longer response budgets.

Critical strengths

Important problem and clear conceptual framing. The distinction between decreasing state visitation and incomplete conditional alignment provides a useful perspective on OPD curriculum design.

Informative empirical findings. The reversal of rollout-source rankings across response budgets shows that freshness is not uniformly beneficial and that training curricula should be evaluated in relation to deployment budgets.

Thoughtful experimental controls. Matching trajectory uses and optimizer updates, restoring identical model and optimizer states, and including fixed-recent-bank, order-reversal, and EOS controls strengthen the internal comparisons.

Critical weaknesses

The independent value of adaptive triggering remains unclear. In Table 5b, fixed switching at τ=5 trails R-OPD by only 0.16/0.15 percentage points at 16K/32K. Much of the advantage over the fixed 40-current-update/20-replay-update schedule may therefore come from switching earlier rather than from gradient information. A comparison against a reasonable, pre-specified fixed rule on held-out configurations is needed. It would also be useful to test whether the trigger signal predicts the subsequent benefit of replay relative to continued current-policy training.

The core experiments cover a narrow training regime. R-OPD uses the same 48-query set and 60 optimizer updates. The three repetitions primarily capture data-order variation, not query-set sampling or broader training randomness. Since Section 3 already demonstrates interactions between query count and sampling schedules, transfer to larger query sets or longer training remains uncertain.

The cost–performance advantage is not established. At 32K, R-OPD produces approximately 41%–73% longer responses, while current-policy training is generally stronger at short budgets. Gains under the same generation cap are valid, but do not establish superiority at matched actual inference cost. Accuracy-versus-generated-token curves for current-policy training, a competitive fixed schedule, and R-OPD would clarify this trade-off.

Questions for authors

At matched actual inference-token cost (e.g., accuracy vs.\ realized-token curves, or equal-budget best-of-n), does R-OPD's advantage over current-policy training and fixed \tau=5 persist? If it vanishes or shrinks substantially, should the conclusions be revised accordingly?

Does a fixed switch at τ=5, chosen from the 0.6B scan (Table 5b), remain competitive when transferred to 1.7B/8B students or new query sets?

Were the D ≤ V criterion and the 'two consecutive comparisons' requirement fixed before or after inspecting evaluation results?

Reviewer_HYLsOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how the evolving student policy changes the states receiving supervision during on-policy distillation. It finds that initial-policy prefixes become less likely under the trained student while retaining teacher–student disagreement, and that revisiting these prefixes can improve performance at longer generation budgets. Based on these observations, the paper proposes R-OPD: training first on current-policy rollouts, then switching entirely to a fixed initial-policy rollout bank when a gradient-based criterion is satisfied. Experiments examine rollout freshness, replay source, training order, and transition timing across multiple student sizes.

Critical strengths
  1. An interesting examination of the assumption that fresher rollouts are always preferable. The distinction between how frequently a prefix is visited and how well the student matches the teacher at that prefix is useful. The paper raises a meaningful question about whether current-policy sampling leaves valuable supervision on earlier states underused. I consider this a worthwhile contribution to understanding distillation curricula.

  2. Thoughtful controls support the empirical observations. The paper compares initial-policy replay with recent-policy replay, reverses the order of current-policy and replay stages, and examines fixed versus adaptive transition schedules. The mixed-source and EOS controls further investigate alternative explanations. These experiments provide a substantive basis for the proposed curriculum, beyond a simple comparison of training with and without replay.

Critical weaknesses
  1. The justification for permanently switching to the initial-policy state distribution remains incomplete. Recomputing current student and teacher probabilities avoids stale supervision, but does not remove the mismatch between initial-policy prefixes and the states visited by the current student. Moreover, residual disagreement on an old prefix does not by itself establish that learning this prefix is more useful than learning current-policy states: the student may have learned to avoid it. I appreciate the existing recent-bank and mixed-source controls, but the rationale for allocating all remaining training to a fixed initial bank would be stronger with an analysis of how this trade-off evolves over longer training horizons.

  2. The scope of the curriculum results is relatively narrow. The main R-OPD experiments use 48 queries and 60 optimizer updates. Broader query sets and longer training would help establish whether the benefits persist as the current policy moves further from the initial bank. In addition, the gains at longer generation limits accompany substantially longer responses, while current-policy training remains stronger at shorter limits. This is an informative trade-off, but limits claims of a generally preferable training curriculum.

Questions for authors
  1. Could you extend the replay-stage ablation to compare complete initial-policy replay with mixed current/initial-policy sampling at several replay ratios, under matched trajectory and optimizer-update budgets? Although the paper already includes mixed-source controls, testing these alternatives over a longer post-transition horizon would help determine whether permanently abandoning current-policy sampling is beneficial or whether continued exposure to current states becomes necessary.

  2. Could you compare the fixed initial bank with a periodically refreshed or age-limited replay bank over the same extended horizon? Tracking downstream performance together with teacher–student disagreement on both initial and freshly sampled current-policy prefixes would help distinguish beneficial learning from historical states from increasing specialization to states the student no longer visits.

  3. Do the replay benefits and the preferred replay ratio persist with substantially more than 48 training queries? This would clarify whether the proposed curriculum generalizes beyond the few-query setting.

Reviewer_SmiPOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes R-OPD, a hybrid on-policy distillation method that switches from training on the student’s current-policy rollouts to replaying trajectories generated by the initial policy, using gradient stability as the switching criterion. Experiments across three model sizes show improvements over a fixed rollout schedule although the gains over the best fixed onset are small in the reported sweep and limited training runs leave uncertainty about the robustness and statistical significance of the results.

Critical strengths
  1. The topic of the paper is relevant in the literature, i.e. how the choice of student-generated versus initial-policy trajectories affects on-policy distillation.

  2. The paper's approach is simple and intuitive.

Critical weaknesses
  1. The paper is very hard to follow and there is no explanation why this setup is chosen for this problem or why particular baselines are chosen which makes it hard to understand what exactly the contribution of the paper is.

  2. There is no variance or uncertainty estimation reported in any of the results. Some of the results are very close to each other and not having this confidence intervals makes it impossible to know whether any of the reported differences are meaningful.

  3. There is only one fixed schedule reported which already is doing a good job. No sweep or further study is done to comapre the gradient based proposed approach to fix schedules.

  4. The novelty is a bit under the question. Even though the paper raises a good question, there is no analysis or checking of the insights on what is the contribution of the different components in what authors are proposing as the reason of OPD failing.

  5. Longer responses may contribute to the gains, and the paper does not fully establish whether improvements reflect better reasoning rather than response length or other differences in training cost.

Questions for authors
  1. How does R-OPD compare with the best fixed rollout onset across all three model sizes, and are the differences statistically significant across independent training runs?

  2. How sensitive are the results to generation temperature for pass@n, response-length controls, and matched training compute?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.