acceptodds
Under review as a conference paper at ICLR 2027

Flash On-Policy Distillation

Abstract

Full-vocabulary KL provides on-policy distillation (OPD) with supervision over every next-token alternative, rather than a teacher-top-k truncation or a sampled-token estimate. Making this objective efficient is a framework-level challenge: dense distribution materialization strains memory and communication, while complete-response barriers delay updates and serialize teacher work. We introduce Flash-OPD, an OPD training framework that makes full-vocabulary KL a native training path through three coordinated designs. Partial on-policy rollouts retain unfinished prefixes as context and optimize only fresh suffixes generated by the current student snapshot. Teacher overlap evaluates each ready suffix while other requests continue generating, preserving one synchronized student update per round. A tiled KL loss kernel reconstructs teacher logits from hidden states and a frozen output head, using online normalization and analytic backward recomputation instead of materializing the complete token–vocabulary matrix. Together, these designs preserve full-vocabulary supervision and token-level behavior-policy matching while reducing update latency and loss workspace, with FSDP and vocabulary-parallel Megatron support. In a representative wall-clock comparison, Flash-OPD reaches reverse KL 0.01 in 4.2 hours versus 8.5 hours for full-rollout OPD, a 2.02× time-to-target speedup, with comparable final divergence. Efficient OPD need not obtain its systems benefits by truncating the vocabulary supervision.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Large Language Models, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8Y72OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Flash-OPD, an efficient on-policy distillation framework that separates the update horizon from the full response. Instead of waiting for a complete rollout, Flash-OPD trains each newly generated suffixes, updates and synchronizes the student, and then resumes unfinished trajectories. It further overlaps teacher evaluation with ongoing rollout generation and adopts a memory-efficient full-vocabulary loss. The paper reports substantial wall-clock speedups over vanilla OPD while maintaining similar downstream performance.

Critical strengths
  1. The consume-before-continution design is intuitive and practically motivated. It allows the student model to learn from a newly generated suffix immediately, while still preserving and continuing the unfinished trajectory after the update.
  2. The paper reports promising wall-clock improvements over vanilla OPD. Flash-OPD reduces training time from 8.5 to 4.2 hours on math, 6.3 to 3.4 hours on code, and 12.7 to 7.9 hours on agentic training. The reported downstream score remains largely unchanged relative to vanilla OPD.
Critical weaknesses
  1. Table 1 only compares the proposed Flash-OPD with vanilla OPD. However, several recent approaches are closely related and already target the same rollout-efficiency bottleneck. More specifically, [1] early terminates student rollouts and report large reductions in training FLOPs. [2] explicitly controls the rollout horizon and reports up to 3 times improved training efficiency. [3] decouples rollout generation and learner updates and reports similar improved throughput. This work discusses these baseline work but no matched experiment is provided to show whether the proposed scheme is actually better in wall-clock efficiency or downstream performance. While the proposed scheduling idea is interesting, it is unclear whether it provides a meaningful advantage over existing fast OPD methods. The current experimental evaluation is below the acceptance bar. 2 The paper lacks an end-to-end ablation of its proposed two main scheduling components, i.e. partial updates and teacher overlap. Table 1 only compares vanilla OPD with the whole Flash-OPD system. Appendix D explicitly states that its component measurements do not attribute the end-to-end gains to individual components. Therefore, the source of the reported training speedup remains unclear.

References: [1] Zhang et al. Fast and Effective On-Policy Distillation from Reasoning Prefixes. 2026. [2] Zhang et al. Are Full Rollouts Necessary for On-Policy Distillation? 2026. [3] Kang et al. AsyncOPD: How Stale On-Policy Distillation Be? 2026. [4] Rang et al. Near-Policy: Accelerating On-Policy Distillation via Asynchronous Generation and Selective Packing. 2026

Questions for authors

Questions:

  1. Could the authors provide matched performance comparisons against recent efficient OPD methods (particularly those included in references) under the same compute budget and downstream evaluation? Such comparisons are necessary to demonstrate the practical benefit of the proposed design.
  2. Could the authors provide an end-to-end ablation separating partial updates from teacher overlap? This would help clarify how much of the reported speedup is from the proposed consume-before-continuation design.
Reviewer_8DBoOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a method to accelerate on-policy distillation through partial rollout updates and by overlapping teacher inference with student generation.

Critical strengths
  • The paper provides a detailed description of the fresh suffix alignment and synchronization protocol.
Critical weaknesses

[W1] The claimed speedup lacks reproducible experimental details. The abstract and table 1 claim 2.02x end-to-end speedup. However, in section 4.1, L310, authors state that this result "predates the B200 study and lacks complete run provenance." They have also written: "The latter's online screen completed no optimizer update" and appendix D.6 confirms that integrated training failed during initialization.

This is very concerning. In other words, the authors acknowledge that: 1) they cannot fully document how the headline result was obtained, 2) the only end-to-end training runs whose setup the paper actually specifies (the B200 study) never completed a single optimizer update.

Furthermore, the paper does not specify which models were used as student and teacher for the main end-to-end training experiments in table 1. The Qwen models in appendix D are not explicitly identified as the models used in the main experiments, can the authors clarify this?

[W2] While the proposed method aims to speed up OPD by minimizing GPU idle time, it does not provide a direct comparison with existing asynchronous OPD methods, which are alternative techniques that speed up the training by minimizing the GPU idle time. The lack of comparison makes it unclear if the benefit of Flash-OPD's strict synchronization via partial updates outweighs the additional synchronization and prefill overhead it pays compared to asynchronous alternatives.

[W3] Readiness-driven teacher overlap, which is presented as one of the central design components, has limited novelty. The authors acknowledge this in appendix C, that the existing verl's OPD implementation already supports such overlap. It is thus unclear what new contribution the paper makes regarding per-sample teacher overlap beyond adapting existing techniques to partial-rollout training.

[W4] The paper lacks ablation of the proposed components. Since teacher overlap can be applied independently of partial updates, the current results do not establish how much of the speedup comes from the proposed partial-update algorithm versus existing scheduling techniques. Showing results for 1) full-rollout OPD with teacher overlap and 2) partial-rollout OPD without teacher overlap should be helpful. The benefit of the "consume-before-continuation" contract is also not isolated, so its contribution is left unclear.

Questions for authors

[Q1] How sensitive is Flash-OPD's end-to-end training efficiency to the rollout horizon H? The paper measures teacher-service overhead across different H values, but does not provide end-to-end training results. How was H selected for the experiments in Table 1?

Reviewer_98PROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes Flash-OPD, a training scheme that speeds up on-policy distillation (OPD) by reducing idle time between student generation, teacher evaluation, and learner updates. Rollouts are split into bounded partial segments: each newly generated suffix is trained on once within its round, with earlier tokens kept only as context, before the trajectory resumes. Teacher evaluation is dispatched as soon as each suffix is ready, overlapping with other requests' decoding. Across math, code, and agentic training, the paper reports 2.02×, 1.85×, and 1.61× wall-clock speedups over OPD, with task scores within 0.4 points.

Critical strengths
  • Timely and practical: better GPU utilization in distillation matters given the cost of accelerators, and overlapping teacher evaluation with generation is a promising direction.
Critical weaknesses
  1. Evaluation is not rigorous.
    • No comparison with prior techniques. The only baseline is stage-serialized OPD. Related methods are cited, but citing them is not enough; Flash-OPD should be benchmarked against them, most importantly AsyncOPD, and also Near-Policy Distillation, Prefix OPD, Truncated/Progressive OPD, and APRIL. This is my main concern.
    • Convergence speed rather than quality. The headline speedup is time to reach a reverse-KL target (Fig. 1). A small reverse KL can still leave large quality gaps; task success rate is a more meaningful metric. Task scores over wall-clock time or training steps for both methods would make the claim more convincing.
    • No confidence intervals or statistical significance. Results come from single runs, for both task scores and wall-clock times. Multiple runs are needed to show the variation: without them, neither the ±0.4-point score differences nor the reported speedups in Tab. 1 can be interpreted. Fig. 1 is a historical single-run trace that, per the authors, lacks complete run provenance.
  2. Clarity. The paper is dense and hard to follow.
    • There is no clear storyline: paragraphs with highlighted openings read as detached pieces without a connecting flow.
    • Notation is used without definition, e.g., sg (L93) and HS (L190).
    • Fig. 2, which conveys the core idea, would be more helpful earlier in the paper.
Questions for authors
  1. How does Flash-OPD compare with AsyncOPD at matched compute, in both wall-clock time and task scores? Comparisons with Near-Policy Distillation, Prefix/Truncated OPD, and APRIL-style partial rollouts would further help.
  2. How do task scores evolve over wall-clock time (or training steps) for OPD and Flash-OPD, and what is the variance across seeds?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.