- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies whether on-policy distillation benefits from more updates per rollout batch. It caches teacher top-k probabilities and reuses the same student trajectories for up to 128 updates before collecting new data. Experiments cover three teacher and student pairs on mathematical reasoning benchmarks. The results show comparable best-checkpoint accuracy at lower sampling and teacher inference costs.
Critical strengths
-
The paper addresses a practical bottleneck. Reducing rollout generation and teacher inference could make distillation substantially cheaper. The proposed change is simple to implement.
-
The controlled comparison keeps the loss and optimizer fixed. Ablations on replay frequency, learning rate, and KL regularization help assess the effect of data reuse.
Critical weaknesses
- The contribution is not sufficiently distinguished from related work. The method retains an existing top-k distillation loss. Its main change is to perform more updates on each rollout batch before resampling. Several relevant works are missing from the discussion:
- DistiLLM-2 [1] (ICML 2025) generates student responses in batches before each training epoch. It therefore already separates generation from subsequent optimization and reduces sampling frequency. This differs from repeatedly optimizing the same complete batch, but it is directly relevant to the paper’s motivation.
- Proximal Policy Distillation [2] (TMLR 2025), explicitly reuses student trajectories for multiple optimization epochs to improve sample efficiency. It studies control tasks and combines distillation with PPO, importance weighting, and clipping. These differences leave room for the present contribution, but multi-epoch optimization on student trajectories is already established within policy distillation.
- Blockwise Policy-Drift Gating [3] studies rollout reuse in LLM distillation, including Teacher-TopK supervision. It uses old versus current student probabilities to reweight losses during reuse. This is particularly relevant to the claim that aggressive replay works without additional drift control. These works do not establish the exact result reported here. However, they make it important to identify what is new about the proposed schedule. Lightning OPD is discussed, but no direct experimental comparison is provided. Comparisons with cached offline distillation and drift-aware replay under matched budgets would help establish the value of aggressive consecutive reuse. Without these comparisons, the contribution remains difficult to assess beyond the observation that a large replay count works in the tested settings.
- The evidence is too limited to establish when aggressive reuse remains reliable. Experiments use small students, math tasks, and one training seed. The main setting has only 256 updates with K=128, so it covers just two rollout batches. Results are reported at the checkpoint with the highest evaluation AVG5. This leaves uncertainty about robustness across seeds, longer training, and repeated distribution refreshes. Lower loss on a fixed batch does not establish that overfitting is absent.
[1] DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs
[2] Proximal Policy Distillation
[3] Blockwise Policy-Drift Gating for On-Policy Distillation
Questions for authors
-
What distinguishes the proposed schedule from existing multi-epoch policy distillation and rollout reuse? How would drift gating affect the results under the same loss and replay budget?
-
How does the method compare with offline distillation using the same top-k targets and generation budget? This comparison would help isolate the value of periodic refresh.
-
Are the gains stable across training seeds and additional rollout rounds? Were any checkpoints selected using a separate validation set?
-
Can you conduct experiments using more frontier LLMs, such as Qwen3.5, MiniCPM5?
Thank you for the detailed responses and additional experiments. The Block64 comparison, offline control, repeated replay runs, and MiniCPM5 results address several of my requests. The fixed-step results also strengthen the evidence beyond the original best-checkpoint comparisons. I appreciate these additions.
My main remaining concern is the significance and scope of the contribution relative to existing trajectory reuse. The distinctions from DistiLLM-2 and Proximal Policy Distillation clarify differences in training schedules, objectives, and application domains. However, the underlying idea of separating generation from optimization and reusing student trajectories is already established. The contribution therefore rests substantially on demonstrating that unusually aggressive reuse works reliably in the studied setting. The additional results strengthen this empirical finding, but its boundaries remain insufficiently characterized.
In particular, the K=128 setting still covers only two rollout batches. Experiments with smaller K and more refreshes provide useful complementary evidence, but do not establish the behavior of aggressive K=128 reuse over many refresh cycles. Similarly, the three replay runs support repeatability within the tested configuration without resolving this longer-horizon concern. The Qwen3 thinking results also show AVG5 decreasing from 60.77 at step 128 to 56.08 at step 256. This does not by itself establish overfitting, but early improvements on held-out problems are insufficient to settle the concern about degradation during continued training.
The fixed-step comparisons partially address my evaluation concern. Nevertheless, selecting the headline best checkpoints using evaluation AVG5 without a separate validation set leaves those peak results subject to selection effects.
Thus, the remaining concerns about the extent of the contribution and the reliability of aggressive reuse are sufficient for me to maintain my rating.