acceptodds
Under review as a conference paper at ICLR 2027

ChunkOPD: Efficient Strict On-Policy Distillation via Chunk-Wise Execution

Abstract

On-policy distillation (OPD) trains student language models with teacher feedback on their own generations, aligning supervision with the states they visit. For long reasoning trajectories, however, conventional strict OPD execution incurs teacher-scoring waits and substantial computation and activation-memory costs during student training. Asynchronous execution improves utilization but introduces policy staleness that may degrade distillation quality, motivating a more efficient schedule that preserves strict on-policy property. We present ChunkOPD, which exploits prefix causality and student parameter consistency between rollout and training to reorganize both teacher scoring and student training around token chunks. Streaming Chunk Scoring scores committed prefixes before trajectories finish, overlapping teacher computation with rollout to shorten the scoring tail. Reverse Chunk Training reuses rollout key–value (KV) states to avoid an additional cache-population pass and recomputes chunks in reverse order to propagate gradients through the cached context while keeping activations local to each chunk. Across the evaluated workloads under matched physical GPU budgets, ChunkOPD achieves 1.11–1.98 speedups over synchronous OPD. In an end-to-end case study, ChunkOPD reduces time costs by 41% and achieves almost identical quality relative to synchronous OPD, whereas asynchronous OPD attains lower final accuracy.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
On-Policy Distillation, LLM, Efficient Training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_r9VLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ChunkOPD is a systems approach to accelerating strict on-policy language-model distillation, where generation and gradient computation use the same student parameter version. It targets teacher-scoring delays and the computation and activation-memory costs of training on long trajectories.

Its main contributions are:

  • Streaming Chunk Scoring: incrementally scores generated prefixes using persistent teacher KV caches, overlapping teacher computation with ongoing generation.
  • Reverse Chunk Training: reuses student rollout KV and recomputes chunks from suffix to prefix, propagating gradients through cached context while retaining only chunk-local activations. This avoids an additional KV-population pass and enables overlapping cache transfers.
  • Implementation and evaluation: integrates these mechanisms into verl/vLLM and evaluates performance, numerical consistency, and training quality.

Experiments across model pairs, GPU budgets, sequence lengths, and batch sizes show 1.11–1.98× speedups over synchronous OPD. Ablations support both components, while numerical checks show highly aligned but nonidentical gradients. Gains over chunked-forward training are smaller under matched GPU placement.

A single-seed, 200-update experiment achieves 41% lower training time with similar accuracy (22.14% vs. 22.48%). Broader quality preservation and the cause of Async’s lower accuracy remain unestablished; performance gains also reflect deployment and implementation differences.

Critical strengths
  • A coherent systems design that exploits strict on-policy consistency. ChunkOPD combines early teacher scoring with rollout-KV reuse and reverse gradient reconstruction, preserving the full-batch update boundary and cross-chunk gradient dependencies. This is consequential because it improves execution efficiency while retaining strict on-policy semantics, making the approach useful when policy staleness is undesirable.
  • Substantial measured efficiency gains with complementary validation. Across the tested configurations, ChunkOPD achieves 1.11–1.98× speedups over synchronous OPD, with component ablations supporting both mechanisms. Numerical diagnostics and a 200-update experiment—showing 41% lower measured training time with similar final accuracy—provide complementary evidence that the efficiency improvements are practically useful without an obvious quality penalty in the evaluated setting.
Critical weaknesses
  • The evaluation does not cleanly isolate the proposed mechanisms’ performance contribution. Comparisons vary GPU placement, operator implementations, sharding, and batching alongside the execution schedule. Under matched dedicated placement, ChunkOPD reduces end-to-end time over CF by only approximately 1.6–6.4%; the reverse-training ablation also removes KV reuse and associated optimizations together. Consequently, the headline speedups establish the effectiveness of the overall implementation more clearly than the incremental value of reverse chunk training, weakening the evidence for the central technical contribution.
  • Evidence for preserving training quality is too limited. The quality evaluation covers one model pair, one mathematical-reasoning workload, and one training seed over 200 updates, despite measurable gradient discrepancies from native training. CF is absent from this evaluation, and Async’s lower final accuracy is not causally linked to staleness. This leaves uncertainty about whether the efficiency gains reliably preserve learning outcomes across larger models, longer training runs, and other tasks.
Questions for authors

Please refer to the weaknesses.

2 comments

W1. Mechanism attribution and the CF comparison

Our system evaluation aims to compare methods under the same target (e.g., model pair, prompt workload, training objective) and total GPU budget, with configurations optimized for each method. Placement is part of this optimization: imposing the same placement can constrain methods differently because their execution schedules expose different opportunities for overlap. Particularly, both Sync and ChunkOPD require complete trajectories and their teacher scores before training can begin. A dedicated trainer therefore has an initial waiting period. By using otherwise idle trainer GPUs for rollout and scoring before reusing them for training, union placement improves execution efficiency for Sync and ChunkOPD, as confirmed by our placement experiments. CF can instead begin differentiable forward computation as rollout chunks arrive, overlapping it with ongoing generation and scoring. This earlier start motivated the dedicated placement used for CF. These scheduling differences support allowing each method to use a placement suited to its execution pattern under the same GPU budget.

The original w/o Reverse Chunk Training ablation evaluates the full training component, including KV reuse and transfer overlap. To isolate their contributions, we added finer-grained ablations within the same reverse-training implementation, holding inputs, operators, sharding, synchronization, and batching fixed. We successively enable rollout KV reuse and transfer overlap; the w/o KV reuse variant disables both. The table reports mean trainer time (s), with batch size 128, chunk size 1024, and microbatch size 2.

SettingStudentTrainer GPUsCapw/o KV reusew/o overlapChunkCombined speedup
Basic settings8B48192124.78110.64104.611.19×
Smaller model4B4819279.7469.2665.471.22×
Shorter context8B4409659.3752.0348.891.21×
More GPUs8B8819268.8061.3056.831.21×

KV reuse reduces trainer time by 10.9–13.1%, and transfer overlap provides a further 5.5–7.3% reduction. These results quantify the benefits of avoiding redundant KV construction and overlapping cache transfers under fixed configurations. The combined 1.19–1.22× speedup applies to trainer execution; replay excludes rollout KV export and phase switching.

W2. Training quality and Async interpretation

Our numerical consistency analysis aims to quantify gradient deviations introduced by KV reuse and reverse training. To assess their practical impact on learning, we expanded the end-to-end quality evaluation with a second training seed for the 1.7B student and a new 4B-student experiment, each covering 200 updates with evaluations every 25 updates. The table reports final macro avg@16 across AIME24, AIME25, and AMC23. CF was an development-time exploration without efficiency advancement, so it is not included in the end-to-end evaluation.

Student / teacherSeedSyncChunkOPDAsync
1.7B-Base / 4B-Base-GRPO122.48%22.14%19.98%
1.7B-Base / 4B-Base-GRPO221.68%21.46%21.37%
4B-Base / 4B-Base-GRPO234.60%34.65%34.97%

ChunkOPD's final scores differ from Sync by at most 0.35 percentage points across all three comparisons, with similar learning curves throughout the evaluated training period. For the larger 4B student, both methods improve performance from 10.45% to approximately 34.6%. These results extend the evidence for comparable distillation quality beyond the original single seed and student size. For Async, the original accuracy gap does not recur in the second seed: its 21.37% score is close to Sync's 21.68% and ChunkOPD's 21.46%. The revised paper therefore remove the causal attribution of the original gap to staleness.

I thank the authors for the clarifications and additional experiments, which substantially address my concerns about performance attribution and training quality; I maintain my weak-accept rating.

Reviewer_emWmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ChunkOPD for efficient strict on-policy distillation. Streaming Chunk Scoring overlaps teacher computation with rollout, while Reverse Chunk Training reuses rollout KV states to reconstruct causal gradients with chunk-local activations. Experiments show speedups over synchronous OPD, with similar distillation quality in an end-to-end case study.

Critical strengths
  1. The paper improves OPD efficiency through systems optimizations while preserving strict on-policy updates.
  2. The efficiency evaluation covers multiple student–teacher pairs and GPU budgets, with useful latency breakdowns and numerical consistency analyses.
Critical weaknesses
  1. My main concern is the contribution of Reverse Chunk Training beyond OOMB [1]. OOMB already uses chunk-wise recomputation and KV-cache gradient propagation to reduce activation memory. It remains unclear how much additional speedup rollout KV reuse and overlapped KV transfer provide over OOMB-style training. Furthermore, the motivation for adopting OOMB-style training in OPD is not fully convincing. To my knowledge, this approach is not widely adopted, and existing OPD implementations such as verl’s native pipeline do not use it.

  2. The evidence for preserving training quality is limited to one student–teacher pair and one training seed per method. The 4B and 8B students used in the main speed benchmarks are not evaluated for training quality.

[1] Out of the Memory Barrier: A Highly Memory Efficient Training System for LLMs with Million-Token Contexts (ICLR 2026)

Questions for authors

Please refer to the Weaknesses section above.

4 comments

W1. Contribution beyond OOMB and motivation in OPD

OOMB is an important inspiration for our work, and we explicitly credit its KV recomputation and KV-gradient propagation techniques. It addresses the activation-memory cost of long-context training by retaining chunk-local activations while propagating gradients across chunk boundaries. Strict OPD shares this motivation: training on long student-generated trajectories also incurs activation-memory costs that grow with sequence length.

Strict OPD additionally produces rollout KV under the same student parameters used for training. Reverse Chunk Training exploits this OPD workflow by integrating rollout KV reuse, causal gradient reconstruction, and KV transfer overlap to improve training efficiency. To quantify these benefits, we supplement finer-grained ablations in Section 4.3 (Component Contributions). With inputs and other settings fixed, we successively enable KV reuse and transfer overlap within the same reverse-training implementation. The w/o KV reuse variant also disables overlap, providing an OOMB-style comparison. The table reports mean trainer time (s) and Chunk's speedup over the OOMB.

SettingStudentGPUsCapw/o KV reuse (OOMB-style)w/o overlapChunkSpeedup
Basic settings8B48192124.78110.64104.611.19×
Smaller model4B4819279.7469.2665.471.22×
Shorter context8B4409659.3752.0348.891.21×
More GPUs8B8819268.8061.3056.831.21×

KV reuse reduces trainer time by 10.9–13.1%, and transfer overlap provides a further 5.5–7.3% reduction, yielding 1.19–1.22× combined speedup beyond the shared reverse-training path. These results demonstrate the complementary benefits of avoiding redundant KV construction and hiding cache-transfer latency.

W2. Limited training-quality evidence

We agree that the original quality evaluation was limited in model and seed coverage. We have added a second training seed for the 1.7B student and a new 4B-student experiment, each covering 200 updates. The quality experiments follow the Base-student/RL-trained-teacher OPD recipe of Li et al.. We use this recipe to evaluate distillation quality, while the efficiency benchmarks cover broader model configurations. The table reports final macro avg@16 across AIME24, AIME25, and AMC23.

Student / teacherSeedSyncChunkOPDSpeedup
1.7B-Base / 4B-Base-GRPO122.48%22.14%1.70×
1.7B-Base / 4B-Base-GRPO221.68%21.46%1.77×
4B-Base / 4B-Base-GRPO234.60%34.65%1.59×

ChunkOPD achieves 1.59–1.77× speedup while maintaining comparable distillation quality across the tested seeds and student sizes. Final scores differ from Sync by at most 0.35 percentage points across all three comparisons, with similar learning curves at 25-update intervals. The added 4B experiment further supports comparable quality at a larger student scale, with both methods improving student model from 10.45% to approximately 34.6%.

Thank you for the additional experiments. May I ask why training quality was not evaluated for the 8B student used in the main efficiency experiments?

Thank you for your valuable question.

For the rebuttal, we prioritized the 4B-Base student because it allowed us to extend the quality evaluation to a larger student while keeping the teacher fixed. Our original experiment followed the training recipe of Rethinking On-Policy Distillation, pairing Qwen3-1.7B-Base with Qwen3-4B-Base-GRPO. Reusing the same RL-trained teacher gave us a controlled extension of that setup with limited need for retuning.

We also considered reusing the post-trained Qwen3-8B and Qwen3-14B checkpoints from the efficiency experiments. Their reported scores are close: 76.0% vs. 79.3% on AIME24 and 67.3% vs. 70.4% on AIME25 (8B vs. 14B). Given these small gaps, we expected modest gains from OPD with this pairing, making it harder to assess whether ChunkOPD preserves the learning gains achieved by Sync. To allow more room for improvement at the 8B scale, we plan to pair Qwen3-8B-Base (13.3% on AIME24, 10.2% on AIME25) with the post-trained Qwen3-14B teacher. The experiment remains pending due to time and compute constraints during the rebuttal.

In the completed experiments, ChunkOPD and Sync show similar learning curves across two 1.7B seeds and one 4B run. Their final macro avg@16 scores differ by at most 0.35 percentage points after 200 updates, supporting comparable training quality across the student sizes and seeds evaluated so far.

Thank you for the clarification. Please let me know if any preliminary results from the 8B experiment become available. I will finalize my recommendation during the reviewer-AC discussion period.

Reviewer_zf2dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ChunkOPD, a system for accelerating strict on-policy distillation through chunk-wise execution. Streaming Chunk Scoring overlaps teacher supervision with ongoing rollout, while Reverse Chunk Training reuses rollout KV states and reconstructs gradients with chunk-local activations. Under matched physical GPU budgets, ChunkOPD achieves 1.11–1.98× speedups over synchronous OPD. A 200-update case study reduces cumulative update time by 41% while obtaining similar final accuracy, although numerical differences and broader scalability warrant further validation.

Critical strengths
  • Useful execution design. The paper combines streaming teacher scoring and rollout-state reuse to reduce overhead while retaining a fixed student parameter version within each update.

  • Detailed systems evaluation. The experiments include complete-update timing, component ablations, memory measurements, and numerical consistency checks across several deployment configurations.

Critical weaknesses
  • Incomplete attribution of training speedups. Although component ablations are provided, removing Reverse Chunk Training jointly disables KV reuse, reverse traversal, and transfer scheduling, while also changing operator and memory-management choices. Differences in Liger usage and gradient synchronization further complicate attribution. More targeted controls are needed to isolate the benefit of KV reuse.

  • Unverified equivalence of KV reuse. Inference-engine KV states are not guaranteed to match trainer-generated states, even with identical parameters. The reported gradient differences of approximately 4.3–4.7% warrant further investigation. Quality validation on a single 1.7B training run over 200 updates provides limited evidence that these discrepancies remain harmless at larger scales.

  • Incomplete asynchronous comparison. The Async baseline uses different resource placement and may generate individual trajectories across multiple parameter versions. Its lower final accuracy therefore does not isolate the effect of policy staleness. Controlled comparisons separating version lag, mixed-version generation, and scheduling are needed.

  • Unclear long-context scalability. Experiments are limited to single-node A100 deployments with token caps up to 8192, and relative speedups decrease at the longer cap in the main matrix. KV states and their adjoints remain sequence-wide, with substantial cache transfers. Longer-context and multi-node evaluations are needed to establish scalability.

Questions for authors

same as weakness

2 comments

W1. Attribution of training speedups

The differing Liger and gradient-synchronization settings aim to compare each method at its best observed performance within the same GPU budget. Sync and Async use optimized native training operators, with limited additional benefit from Liger in our preliminary tests; ChunkOPD and CF use custom chunk-aware training paths, where Liger improves operator efficiency. Reverse Chunk Training also frees activation memory for deferred gradient synchronization (no_sync), reducing communication at the cost of retaining unsharded gradients. In memory-constrained native configurations, enabling this option causes OOM; it is enabled wherever feasible.

Building on the existing component contribution analysis, we added controlled ablation experiments to isolate the contributions of KV reuse and transfer overlap. With all other configurations held fixed, KV reuse reduces trainer time by 10.9–13.1%, and transfer overlap by a further 5.5–7.3%, yielding 1.19–1.22× combined speedup beyond the shared reverse-training path. The table reports mean trainer time (s), with batch size 128, chunk size 1024, and microbatch size 2. Time savings compare adjacent variants, first enabling KV reuse and then transfer overlap. These are trainer-level gains; replay excludes rollout KV export and phase switching.

SettingStudentTrainer GPUsCapw/o KV reusew/o overlapChunkTime saved: KV reuseTime saved: overlap
Basic settings8B48192124.78110.64104.6111.3%5.5%
Smaller model4B4819279.7469.2665.4713.1%5.5%
Shorter context8B4409659.3752.0348.8912.4%6.0%
More GPUs8B8819268.8061.3056.8310.9%7.3%

W2. Numerical consistency of rollout KV reuse

Reverse Chunk Training with rollout KV is mathematically equivalent to native training in exact arithmetic: cross-chunk KV-gradient propagation reconstructs the full causal gradient. In finite precision, however, even native training can produce different gradients when kernels, microbatching, or reduction order change. We therefore conduct numerical consistency experiments to quantify these deviations and distinguish the effects of reverse chunk execution from those of rollout KV reuse.

We acknowledge that rollout KV and recomputed KV differ numerically, with relative L2 differences of 0.785% for K and 2.438% for V. With parameters, trajectories, and teacher targets fixed, the added recomputed-KV control shows that rollout KV reuse increases the discrepancy from native gradients, while the recomputed-KV path also exhibits numerical differences. The table reports relative L2 gradient differences normalized by the native gradient norm.

Reverse chunkRecomputed-KV reverse vs. nativeRollout-KV reverse vs. native
5123.436%4.700%
20483.862%4.312%

A larger gradient discrepancy does not by itself imply worse distillation quality. Relative L2, ∥g−gref∥2∥gref∥2\frac{\lVert g-g_{\mathrm{ref}}\rVert_2}{\lVert g_{\mathrm{ref}}\rVert_2}, can be appreciable even for a small absolute difference when the reference gradient norm is small. We therefore assess its practical impact through end-to-end training, considering both final performance and learning curves. After 200 updates, Sync/ChunkOPD macro avg@16 is 22.48%/22.14% and 21.68%/21.46% for the 1.7B model across two random seeds, and 34.60%/34.65% for the new 4B-Base / 4B-Base-GRPO experiment. Evaluations every 25 updates also show similar learning curves, supporting comparable distillation quality throughout the evaluated training period in these settings.

W3. Interpretation of the Async comparison

We agree that attributing the original accuracy gap to staleness was not justified: the comparison did not control for other execution differences, and the quality evaluation used only one random seed. In the added run with a second random seed, Sync/ChunkOPD/Async achieve similar final scores of 21.68%/21.46%/21.37%, compared with 22.48%/22.14%/19.98% in the original run. The original gap does not recur in the second run, reinforcing the need for caution in attributing it to staleness. In the revised paper, we report both runs, remove the causal attribution to staleness, and focus on efficiency of strict on-policy distillation rather than the impact of staleness.

W4. Long-context and multi-node scalability

We agree that KV storage and transfer costs grow with sequence length, and that the original 8K experiments provided limited evidence of scalability. Due to limited computational resources, we have not evaluated multi-node execution, but have extended the single-node evaluation to longer contexts. We added 16K and 32K long-context experiments with Qwen3-4B/14B on eight A100-80GB GPUs using matched resource placement. The table reports average complete-update times, based on three measured updates.

Total-token capSyncChunkOPDSpeedup
16K340.05 s240.80 s1.41×
32K577.71 s520.47 s1.11×

ChunkOPD achieves 1.41×/1.11× speedup at 16K/32K. The stage measurements show that Streaming Chunk Scoring still reduces the exposed teacher tail from 93.66/53.22 s to 0.32/0.38 s at 16K/32K, while the training window decreases from 33.66/47.00 s to 25.49/41.47 s. Rollout remains the dominant cost at 32K, limiting the total speedup despite the continued teacher-tail and training-time savings. The revised paper includes these results and acknowledges the remaining limitation in multi-node evaluation.

Comments on the paper

1 comment

We thank the reviewers for their careful reading and constructive feedback, and for recognizing ChunkOPD's coherent execution design, practical efficiency benefits, and detailed systems evaluation. The reviews identified the need for clearer attribution of the training speedups, closer examination of numerical differences, and broader validation of training quality and long-context performance. We have revised the paper to address these concerns through additional experiments, numerical analyses, and clarifications, with changes highlighted in blue:

  • Finer-grained ablations demonstrate the additional training efficiency gains from rollout KV reuse and transfer overlap over an OOMB-style trainer (Section 4.3).
  • Paired numerical analyses distinguish the effects of reverse chunk execution from those of rollout KV reuse (Section 4.4).
  • Additional experiments evaluate longer-context performance at 16K and 32K token caps (Appendix F).
  • Expanded training-quality evaluations, including a second 1.7B seed and a new 4B-student experiment, support comparable distillation quality between ChunkOPD and Sync (Section 4.5). Async's original accuracy gap does not recur in the second seed, and we have removed its causal attribution to staleness (Appendix G.4).

Please see the individual reviewer responses below for point-by-point answers to each reviewer's questions and concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.