- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies curriculum design in on-policy distillation (OPD), where the evolving student policy determines which states receive teacher supervision. The authors find that query count interacts with rollout resampling frequency, and that the relative advantage of current-policy versus initial-policy rollouts reverses as the response budget increases. Fixed-prefix diagnostics show that earlier prefixes become less likely while teacher–student disagreement persists. Motivated by these findings, R-OPD switches from current-policy training to initial-policy replay when changes in mean gradients fall within an empirical within-iteration variation scale. Across three student sizes and eight mathematics benchmarks, the method improves average accuracy at longer response budgets.
Critical strengths
Important problem and clear conceptual framing. The distinction between decreasing state visitation and incomplete conditional alignment provides a useful perspective on OPD curriculum design.
Informative empirical findings. The reversal of rollout-source rankings across response budgets shows that freshness is not uniformly beneficial and that training curricula should be evaluated in relation to deployment budgets.
Thoughtful experimental controls. Matching trajectory uses and optimizer updates, restoring identical model and optimizer states, and including fixed-recent-bank, order-reversal, and EOS controls strengthen the internal comparisons.
Critical weaknesses
The independent value of adaptive triggering remains unclear. In Table 5b, fixed switching at τ=5 trails R-OPD by only 0.16/0.15 percentage points at 16K/32K. Much of the advantage over the fixed 40-current-update/20-replay-update schedule may therefore come from switching earlier rather than from gradient information. A comparison against a reasonable, pre-specified fixed rule on held-out configurations is needed. It would also be useful to test whether the trigger signal predicts the subsequent benefit of replay relative to continued current-policy training.
The core experiments cover a narrow training regime. R-OPD uses the same 48-query set and 60 optimizer updates. The three repetitions primarily capture data-order variation, not query-set sampling or broader training randomness. Since Section 3 already demonstrates interactions between query count and sampling schedules, transfer to larger query sets or longer training remains uncertain.
The cost–performance advantage is not established. At 32K, R-OPD produces approximately 41%–73% longer responses, while current-policy training is generally stronger at short budgets. Gains under the same generation cap are valid, but do not establish superiority at matched actual inference cost. Accuracy-versus-generated-token curves for current-policy training, a competitive fixed schedule, and R-OPD would clarify this trade-off.
Questions for authors
At matched actual inference-token cost (e.g., accuracy vs.\ realized-token curves, or equal-budget best-of-n), does R-OPD's advantage over current-policy training and fixed \tau=5 persist? If it vanishes or shrinks substantially, should the conclusions be revised accordingly?
Does a fixed switch at τ=5, chosen from the 0.6B scan (Table 5b), remain competitive when transferred to 1.7B/8B students or new query sets?
Were the D ≤ V criterion and the 'two consecutive comparisons' requirement fixed before or after inspecting evaluation results?