acceptodds
Under review as a conference paper at ICLR 2027

Optimize More, Sample Less: Efficient On-Policy Distillation

Abstract

On-policy distillation (OPD) improves a student model by repeatedly sampling from its current policy and distilling feedback from a stronger teacher. However, standard OPD tightly couples sampling and optimization: after each round of rollouts, the student is updated only briefly before new trajectories are collected. This can be inefficient when on-policy sampling and teacher inference are expensive, while optimization on already collected trajectories is comparatively cheap. We revisit this design choice and ask a simple question: should we optimize more before sampling again? We introduce a simple modification to OPD that performs multiple optimization steps on each batch of on-policy trajectories before refreshing the rollout distribution. By increasing the number of optimization steps per rollout batch, the student moves closer to the teacher within each rollout round and extracts more learning signal from every sampled trajectory. On DeepSeek-R1-Distill-Qwen-1.5B across a suite of mathematical reasoning benchmarks, additional optimization reaches a given performance level with substantially fewer rollouts and teacher generations. Compared with vanilla OPD, taking 128 optimization steps per rollout batch achieves a wall-clock speedup and uses % fewer FLOPs, while maintaining or improving final performance. Surprisingly, aggressive reuse of on-policy data remains effective well beyond the conventional one-update-per-rollout regime. Our results suggest that frequent resampling is not necessary for effective on-policy distillation: when sampling is expensive, a simple principle works remarkably well—optimize more, sample less.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, OPD

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_V85XOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies whether on-policy distillation benefits from more updates per rollout batch. It caches teacher top-k probabilities and reuses the same student trajectories for up to 128 updates before collecting new data. Experiments cover three teacher and student pairs on mathematical reasoning benchmarks. The results show comparable best-checkpoint accuracy at lower sampling and teacher inference costs.

Critical strengths
  1. The paper addresses a practical bottleneck. Reducing rollout generation and teacher inference could make distillation substantially cheaper. The proposed change is simple to implement.

  2. The controlled comparison keeps the loss and optimizer fixed. Ablations on replay frequency, learning rate, and KL regularization help assess the effect of data reuse.

Critical weaknesses
  1. The contribution is not sufficiently distinguished from related work. The method retains an existing top-k distillation loss. Its main change is to perform more updates on each rollout batch before resampling. Several relevant works are missing from the discussion:
  • DistiLLM-2 [1] (ICML 2025) generates student responses in batches before each training epoch. It therefore already separates generation from subsequent optimization and reduces sampling frequency. This differs from repeatedly optimizing the same complete batch, but it is directly relevant to the paper’s motivation.
  • Proximal Policy Distillation [2] (TMLR 2025), explicitly reuses student trajectories for multiple optimization epochs to improve sample efficiency. It studies control tasks and combines distillation with PPO, importance weighting, and clipping. These differences leave room for the present contribution, but multi-epoch optimization on student trajectories is already established within policy distillation.
  • Blockwise Policy-Drift Gating [3] studies rollout reuse in LLM distillation, including Teacher-TopK supervision. It uses old versus current student probabilities to reweight losses during reuse. This is particularly relevant to the claim that aggressive replay works without additional drift control. These works do not establish the exact result reported here. However, they make it important to identify what is new about the proposed schedule. Lightning OPD is discussed, but no direct experimental comparison is provided. Comparisons with cached offline distillation and drift-aware replay under matched budgets would help establish the value of aggressive consecutive reuse. Without these comparisons, the contribution remains difficult to assess beyond the observation that a large replay count works in the tested settings.
  1. The evidence is too limited to establish when aggressive reuse remains reliable. Experiments use small students, math tasks, and one training seed. The main setting has only 256 updates with K=128, so it covers just two rollout batches. Results are reported at the checkpoint with the highest evaluation AVG5. This leaves uncertainty about robustness across seeds, longer training, and repeated distribution refreshes. Lower loss on a fixed batch does not establish that overfitting is absent.

[1] DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs

[2] Proximal Policy Distillation

[3] Blockwise Policy-Drift Gating for On-Policy Distillation

Questions for authors
  1. What distinguishes the proposed schedule from existing multi-epoch policy distillation and rollout reuse? How would drift gating affect the results under the same loss and replay budget?

  2. How does the method compare with offline distillation using the same top-k targets and generation budget? This comparison would help isolate the value of periodic refresh.

  3. Are the gains stable across training seeds and additional rollout rounds? Were any checkpoints selected using a separate validation set?

  4. Can you conduct experiments using more frontier LLMs, such as Qwen3.5, MiniCPM5?

1 comment

Thank you for the detailed responses and additional experiments. The Block64 comparison, offline control, repeated replay runs, and MiniCPM5 results address several of my requests. The fixed-step results also strengthen the evidence beyond the original best-checkpoint comparisons. I appreciate these additions.

My main remaining concern is the significance and scope of the contribution relative to existing trajectory reuse. The distinctions from DistiLLM-2 and Proximal Policy Distillation clarify differences in training schedules, objectives, and application domains. However, the underlying idea of separating generation from optimization and reusing student trajectories is already established. The contribution therefore rests substantially on demonstrating that unusually aggressive reuse works reliably in the studied setting. The additional results strengthen this empirical finding, but its boundaries remain insufficiently characterized.

In particular, the K=128 setting still covers only two rollout batches. Experiments with smaller K and more refreshes provide useful complementary evidence, but do not establish the behavior of aggressive K=128 reuse over many refresh cycles. Similarly, the three replay runs support repeatability within the tested configuration without resolving this longer-horizon concern. The Qwen3 thinking results also show AVG5 decreasing from 60.77 at step 128 to 56.08 at step 256. This does not by itself establish overfitting, but early improvements on held-out problems are insufficient to settle the concern about degradation during continued training.

The fixed-step comparisons partially address my evaluation concern. Nevertheless, selecting the headline best checkpoints using evaluation AVG5 without a separate validation set leaves those peak results subject to selection effects.

Thus, the remaining concerns about the extent of the contribution and the reliability of aggressive reuse are sufficient for me to maintain my rating.

Reviewer_ja2fOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors present an interesting approach for OPD, and some encouraging experimenting results. They designed a more efficient training approach that can achieve better or similar performance at lower cost. The protocol is clearly laid out, and systematic examination on several factors show that the approach is indeed valuable and can be potentially used to training student models from stronger teacher models.

Critical strengths
  1. Authors showed detailed implementation of this 'more optimization, less sampling' approach for OPD.
  2. Performance gains/retains at lower cost were observed from multiple testing cases, with clear quantitative evaluations.
Critical weaknesses
  1. It is not clear whether this approach have any limitations, compared to Vanilla approach, for larger scale training (say, beyond the testing scale presented in this work).
  2. How to choose the batch size and several other hyper parameters, and whether the choice is very model dependent. These analysis can be included to make the work more complete.
  3. There will be some 'magic' values that balance the optimization and replay the sampled trajectory?
Questions for authors
  1. The performance of the supervised model trained using this approach will encounter any limitations? Compared to the K=1 case?
  2. how to choose the hyper parameters ?
  3. Will weighing on sampled trajectories in some biased ways affect the performance? For example, if the trajectories are replayed more times than others, or optimized longer than others, affect the model performance?

Comments on the paper

1 comment
Area_Chair_zfNzOpenReview ↗

Third Review

Overview. The authors introduce a simple modification to OPD that performs multiple optimization steps on each batch of on-policy trajectories before refreshing the rollout distribution. Their main observation: when sampling is expensive, a simple principle works remarkably well—optimize more, sample less. They obtained good numerical experiments.

The paper is clearly presented. One question that is not addressed in the paper is the purpose of revisiting OPD. Why does one need to improve OPD at all? Can the authors explain the practical needs of such a technology? A reference only means that someone has worked on it before; it does not establish the urgency of developing such a technique in modern times.

Also, the improvement of the OPD sounds incremental; it's a small modification of an existing method. This reviewer fails to see the significance of the contribution. Can the authors elaborate on the significance by identifying which current research community will benefit from this article? Sometimes an active community is demonstrated by an active leaderboard for the same task, like LMAnera for LLM evaluation. Can authors provide information that can point to a similar community?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.