acceptodds
Under review as a conference paper at ICLR 2027

AOSR: Asynchronous On-Policy Speculative Rollout for Efficient Distillation of Autoregressive Image Generators

Abstract

On-policy distillation (OPD) improves compact autoregressive image generators by training them on image-token sequences generated by the current student. However, producing these sequences autoregressively requires repeated sequential forwards, making student rollout a major training bottleneck. We explore speculative multi-token rollout to reduce this cost while maintaining distillation performance, but find that a batch-synchronous speculative implementation can even reduce efficiency: different sequences obtain different amounts of verified progress, while synchronous execution prevents faster sequences from advancing independently. Based on this observation, we propose AOSR, an Asynchronous On-policy Speculative Rollout framework that preserves sequence-specific verified progress and adaptively removes completed sequences. We further introduce Cross-Step Speculative Refill (CSR), which reuses completed batch positions to pre-generate sequences for the next rollout and treats the prefetched tokens as speculative candidates. After the student update, these tokens are re-verified with the updated student, resampling mismatched tokens when necessary, thereby reusing computation across OPD steps while maintaining comparable generation quality. Experiments on LlamaGen and Janus-Pro with GKD and VarKD show that AOSR substantially reduces rollout and end-to-end training time, while adding CSR provides further acceleration with comparable generation quality.

Primary area:
generative models
Keywords:
On-Policy Distillation, Autoregressive Image Generation, Rollout Acceleration, Speculative Decoding, Cross-Step Speculative Reuse

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_8N2QOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AOSR, an asynchronous speculative rollout framework for accelerating on-policy distillation of autoregressive image generators. The key observation is that batch-synchronous speculative rollout suffers from substantial shared-boundary waiting because different sequences make uneven verified progress. AOSR addresses this through sequence-wise asynchronous advancement, adaptive batch execution, and CSR-based cross-step prefetching, achieving clear rollout and end-to-end training speedups while largely preserving generation quality. The work has strong practical motivation and solid empirical validation, but the main contribution is primarily execution-level optimization with limited algorithmic novelty, and its gains may depend on sequence-progress heterogeneity and speculative acceptance behavior. Overall, the system design is effective and practically useful, though the algorithmic contribution is relatively modest.

Critical strengths

The paper identifies shared-boundary waiting as a major source of inefficiency in speculative OPD rollouts, and the proposed asynchronous execution directly addresses this bottleneck with substantial end-to-end training speedups.

Critical weaknesses
  1. The main contribution is largely an execution-level redesign of OPD rollout, combining asynchronous advancement, dynamic batch adaptation, speculative decoding, and cross-step prefetching. While effective, the teacher, distillation objective, and optimization procedure remain unchanged, making the work more system-oriented than algorithmically novel.

  2. AOSR benefits from heterogeneous sequence-wise progress and idle batch slots; if speculative acceptance rates are low or different sequences progress at similar speeds, the advantage of asynchronous scheduling and CSR may become much smaller. More systematic sensitivity analysis across such regimes would strengthen the paper.

Questions for authors
  1. How much of the speedup comes specifically from the proposed asynchronous scheduling, rather than from existing speculative decoding, dynamic batching, and prefetching techniques?

  2. How sensitive is AOSR to sequence-progress heterogeneity and speculative acceptance rates? In what regimes does the speedup become marginal?

  3. Can the proposed framework generalize beyond autoregressive image OPD, for example to video generation or other rollout-intensive post-training settings?

7 comments
AuthorsOpenReview ↗

For W1. Research Insights and Algorithmic Contributions

We thank the reviewer for recognizing the effectiveness of our method. We would like to clarify that unlike OPD acceleration approaches that reduce or adapt rollout lengths, our work focuses on improving student rollout execution efficiency without changing the distillation procedure. Specifically, the asynchronous advancement, dynamic batch adaptation, and cross-step prefetching in our framework are designed to address the challenges of adapting speculative decoding to OPD rollouts.

As demonstrated in our paper, directly applying batch-synchronous speculative decoding can be slower than standard AR due to heterogeneous verified progress and synchronization overhead. To address this, we develop asynchronous advancement and adaptive batch execution to eliminate synchronization bottlenecks, and further introduce CSR for cross-step computation reuse with post-update re-verification. These problem-driven algorithmic designs are tailored to the characteristics of OPD, achieving substantial rollout and end-to-end training acceleration while maintaining comparable generation quality.

AuthorsOpenReview ↗

For W2. Sensitivity to Sequence-progress Heterogeneity

We thank the reviewer for this concern. We would like to clarify that limited progress heterogeneity does not necessarily imply poor overall acceleration, as speculative decoding itself also contributes to AOSR's speedup. When sequences advance rapidly, AOSR can remain efficient even with limited asynchronous gains (the impact of speculative acceptance is further examined in Q2). For CSR, more synchronized completion reduces opportunities for cross-step prefetching, potentially limiting its additional benefits.

To examine progress heterogeneity in practice, we analyze 50 OPD training batches.

Table 1. Verified Progress Heterogeneity in Real OPD Rollouts.

MetricValue
Average Acceptance Rate0.742
Sequences with less than 30% progress0.162
Sequences with less than 50% progress0.356
Sequences with less than 80% progress0.756

When the first sequence finishes, 35.6% of the remaining sequences have completed less than half of their generation, indicating substantial progress heterogeneity and opportunities for both AOSR and CSR.

We further evaluate synchronization sensitivity by progressively constraining sequences to wait for the slowest sequence in their group, without changing the verification procedure.

Table 2. Impact of Controlled Synchronization on Rollout Latency.

Constrained SequencesRollout Latency (ms) ↓Speedup vs. Fully Sync. ↑
0% (AOSR)3904.181.53×
10%3924.341.52×
50%4765.141.25×
80%5482.631.09×
100% (Fully Synchronized)5972.811.00×

The results show that increasing synchronization constraints reduces the additional benefit of asynchronous advancement. Nevertheless, fully synchronized SJD++ still achieves a 1.34× speedup over AR (5972.81 vs. 8003.57 ms), confirming that reduced asynchronous gains do not necessarily eliminate overall speculative rollout acceleration.

AuthorsOpenReview ↗

For Q1. Contribution of Asynchronous Scheduling

We would like to clarify that asynchronous advancement alone reduces rollout latency by 30.9% (1.45× speedup) compared with synchronous execution using the same SJD++ verifier. This contribution has already been isolated in the component ablation in Table 4 of our original submission, reproduced below for clarity.

VariantRollout Latency (ms) ↓
GKD (AR)8003.57
Batch-Sync. SJD8514.44
+ SJD++ Draft Refinement5972.81
+ Async. Advancement4125.26
+ Batch Adaptation (AOSR)3904.18

As shown above, asynchronous advancement provides the primary improvement among the execution optimizations after SJD++ refinement, while batch adaptation further reduces latency by 5.4%. The contribution of CSR is reported separately in Table 1 of our original submission, where adding CSR to AOSR improves E2E speedup from 1.52× to 1.63× in the LlamaGen-L/GKD setting.

These results clearly isolate the contributions of draft refinement, asynchronous advancement, batch adaptation, and cross-step prefetching.

Reviewer_9FMpOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

For accelerating the on-policy distallation, this paper proposed the AOSR, a speculative roll out method to tackle the speeding up challenge without trade off of the quality. The core contributions/ novelty for this paper are more GPU scheduling algorithm wise, where sequence-wise asynchtonous method is introduced to take more utilize of the GPU or reuse the GPU batch.

Critical strengths

Strengths:

  1. the problem is well-motivated and the solution is engineering pratical, this is a good engineering work on GPU scheduling algorithm.

  2. I really like the CSR abalation (table 7), it shows the harm of reuse token without verification, really informative.

Critical weaknesses

Weakness:

  1. The main corcern here is the whole paper is for GPU scheduling algorithm if I understand correctly, but the main table (table 1) lacks of comprehensive comparison with different GPU scheduling method, there are similar paper doing the GPU scheduling, so here are my suggestions

Method for AOSR comparison

Correctness Forensics for Batch Speculative Decoding: Diagnosing the Ragged Tensor Problem

BASS: Batched Attention-optimized Speculative Sampling

Method for CRS cimparison:

SPEC-RL: Accelerating On-Policy Reinforcement Learning with Speculative Rollouts

Taming the Long-Tail: Efficient Reasoning RL Training with Adaptive Drafter

(optional) Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decoding

  1. Plain SJD preserves the student’s distribution, but the paper adds SJD++ confidence-based reuse (Eq. 4) and CSR re-verification, and gives no argument that these keep the exact rollout distribution.

I am happy to raise my score if AOSR CRS shows a better performance on more baseline comparison, since it will be more convincing for me.

Questions for authors

Besides the weakness part, i have some questions

  1. If there are more harmful/bad token after the verification, so how to make sure the distribution doesn't shift much.

  2. I am also unsure how CSR retains about 15 of 16 prefetched tokens (Table 5) when the updated student must reproduce the old student’s sampled tokens exactly. How are the random draws coupled?

5 comments
AuthorsOpenReview ↗

For W1. Comparison with Existing Rollout Acceleration Methods

We thank the reviewer for the constructive feedback and for recognizing the motivation and practical value of our work, particularly the CSR re-verification ablation.

W1. Comparison with Existing Rollout Acceleration Methods

We thank the reviewer for suggesting these additional baselines. To provide a more comprehensive comparison, we evaluate existing rollout acceleration mechanisms under our OPD setting. For fair comparison, we adapt their relevant mechanisms to unified speculative rollout backends rather than reproducing the complete original systems.

Comparison with batched speculative decoding methods. We compare AOSR with BASS, EQSPEC, and EXSPEC under the same SJD++ verifier, student checkpoint, and rollout workload. We adapt their respective draft-length and batching strategies to a unified execution backend for fair comparison. EXSPEC corresponds to the regrouping-based baseline reported in Table 8.

MethodRollout latency (ms) ↓Speedup vs. Batch-Sync. ↑
Batch-Sync. + SJD++5972.811.000×
BASS [1] + SJD++5014.951.191×
EQSPEC [2] + SJD++4598.001.299×
EXSPEC [2] + SJD++4290.811.392×
AOSR + SJD++3904.181.530×

AOSR achieves the lowest rollout latency, reducing it by 9.0% compared with EXSPEC, the strongest competing baseline.

Comparison with RL-oriented rollout acceleration methods. We further evaluate AOSR and CSR alongside rollout acceleration mechanisms adapted from TLT and NeMo-RL. All speculative variants use the same fixed EAGLE3 drafter without additional drafter training or online updates. We adapt the speculative decoding scheduling strategies of TLT and NeMo-RL to our OPD setting, rather than reproducing their complete RL training systems.

MethodRollout Time (ms) ↓Rollout Speedup ↑E2E Speedup ↑Pass@8 ↑
GKD (AR)64941.00×1.00×50.00%
EAGLE353231.22×1.20×53.33%
EAGLE3 + AOSR39121.66×1.60×53.33%
EAGLE3 + AOSR + CSR38201.70×1.65×53.33%
EAGLE3 + TLT [4]48101.35×1.28×50.00%
EAGLE3 + TLT + AOSR38651.68×1.59×50.00%
EAGLE3 + TLT + AOSR + CSR37971.71×1.62×50.00%
EAGLE3 + NeMo-RL [5]45731.42×1.30×53.33%
EAGLE3 + NeMo-RL + AOSR38421.69×1.53×53.33%
EAGLE3 + NeMo-RL + AOSR + CSR37531.73×1.56×53.33%

Comparison with SPEC-RL. We separately implement the historical trajectory reuse mechanism of SPEC-RL using the same fixed EAGLE3 drafter. We evaluate two epochs with repeated prompts, where the first epoch establishes the trajectory cache and the second benefits from historical reuse. The reported latencies are averaged over both epochs, including the initial cache-building stage.

MethodRollout Time (ms) ↓Rollout Speedup ↑E2E Speedup ↑Pass@8 ↑
GKD (AR)64941.00×1.00×50.00%
EAGLE3 + SPEC-RL [3]45731.42×1.22×53.33%
EAGLE3 + SPEC-RL + AOSR33981.91×1.76×50.00%
EAGLE3 + SPEC-RL + AOSR + CSR33301.95×1.79×53.33%

These results demonstrate that AOSR improves rollout efficiency over the evaluated batching baselines and provides additional acceleration when combined with existing RL-oriented rollout mechanisms. CSR further improves efficiency through cross-step speculative reuse. Together, these comparisons demonstrate the effectiveness and compatibility of our rollout framework across different speculative execution strategies.

References

[1] Qian et al. BASS: Batched Attention-optimized Speculative Sampling. Findings of ACL, 2024. https://aclanthology.org/2024.findings-acl.489/

[2] Zhang et al. Correctness Forensics for Batch Speculative Decoding: Diagnosing the Ragged Tensor Problem. Findings of EMNLP, 2026. https://aclanthology.org/2026.findings-emnlp.498/

[3] Liu et al. SPEC-RL: Accelerating On-Policy Reinforcement Learning via Speculative Rollouts. arXiv:2509.23232, 2025. https://arxiv.org/abs/2509.23232

[4] Hu et al. Taming the Long-Tail: Efficient Reasoning RL Training with Adaptive Drafter. ASPLOS, 2026. https://arxiv.org/abs/2511.16665

[5] Iso et al. Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decoding. arXiv:2604.26779, 2026. https://arxiv.org/abs/2604.26779

AuthorsOpenReview ↗

For W2. Distributional Consistency of SJD++ and CSR

W2. Distributional Consistency of SJD++ and CSR

We thank the reviewer for this question. We would like to clarify that while standard SJD provides distribution-preserving verification, SJD++ does not have a separate strict unbiasedness guarantee. AOSR retains the SJD++ verification rules, while CSR re-verifies prefetched tokens with the updated student and relies on SJD++ for subsequent verification. We therefore evaluate distributional consistency empirically by estimating token distributions from independent samples under fixed prefixes and computing TV distances against AR sampling.

Table 1. Empirical Distributional Consistency of Rollout Methods.

Rollout MethodToken-level TV ↓Mean Log-prob.
AR0.0000-6.4595
SJD0.0454-6.4620
SJD++0.0440-6.4671
AOSR0.0453-6.4543
AR (Updated Student)0.0000-6.5883
AOSR + CSR (Updated Student)0.0389-6.5491

AR serves as the empirical reference (TV = 0 by construction). CSR is compared against AR sampling from the updated student.

The results show that AOSR exhibits comparable empirical TV distance to SJD++ (0.0453 vs. 0.0440), while CSR also maintains a small deviation (0.0389). We further evaluate downstream generation quality on GenEval and DPG-Bench.

Table 2. GenEval Results (Janus-Pro-1B Student).

MethodSingle ObjectTwo ObjectsCountingColorsPositionColor AttributionOverall ↑
GKD (AR)0.980.830.500.900.660.560.74
GKD + AOSR0.970.830.510.910.650.560.74
GKD + AOSR + CSR0.980.830.510.900.630.570.74

Table 3. DPG-Bench Results (Janus-Pro-1B Student).

MethodGlobalEntityAttributeRelationOtherOverall ↑
GKD (AR)87.6988.7288.2889.0188.4282.85
GKD + AOSR87.7088.7388.2488.9888.4182.84
GKD + AOSR + CSR87.7188.7488.2189.0088.4482.85

Both AOSR and AOSR+CSR maintain comparable GenEval (0.74) and DPG-Bench (82.84–82.85) performance to AR. Together with the FID and IS results in our submission, these findings suggest no material degradation in downstream distillation performance under our evaluated settings.

AuthorsOpenReview ↗

For Questions 1 & 2

Q1. Impact of Low-quality Tokens after Verification

We thank the reviewer for this question. We would like to clarify that low-quality tokens do not necessarily imply distributional shifts, as they may also arise under standard AR sampling. The key concern is whether speculative rollout introduces additional deviations from the target AR distribution.

As shown in W2, AOSR exhibits comparable token-level TV distance to SJD++ (0.0453 vs. 0.0440), while CSR also maintains a small deviation (0.0389) against the updated student. Together with comparable FID, IS, GenEval, and DPG-Bench results, these findings suggest no material degradation in downstream distillation performance under our evaluated settings.

Q2. Clarification of CSR Token Retention and Random-Draw Coupling

We apologize for the ambiguity in our original description. We would like to clarify that the 15.01/16 tokens in Table 5 represent the average number of originally prefetched token values retained after verification, rather than exact matches during initial re-verification.

Specifically, CSR replaces the first mismatched token using the updated student, while retaining the remaining prefetched tokens as drafts for subsequent SJD++ verification. Resampling may also reproduce the original token value, which is counted as retained. Therefore, CSR does not require coupling random draws between the old and updated students. We will clarify this metric in the revised paper.

Reviewer_7QxhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AOSR accelerates on-policy distillation of autoregressive image generators by making student rollout asynchronous: the authors find that batch-synchronous speculative rollout is actually slower, so they let each sequence advance by its own verified progress, adaptively drop finished sequences, and use CSR to refill freed batch slots by prefetching the next rollout's prefix, which is re-verified after the student update.

Critical strengths

1 This quantitative observation well motivated the paper. Rather than claiming that speculative decoding helps, the authors show that a batch-synchronous speculative rollout is actually slower

2 The paper cleanly separates the autoregressive constraint from the scheduling constraint.

Critical weaknesses

1 The end-to-end gain is relatively small for CSR. For example, in Table 1, the CSR provides an additional 0.11 speedup in end2end, making its contribution small.

2 This paper provides Janus-pro on COCO but still lacks strong t2i generation experiment. If possible, please provide the results on GenEval and DPG-Bench.

Questions for authors

Please refer to the above

3 comments

We sincerely thank the reviewer for recognizing the value of our work and for the constructive feedback. We address your comments and questions point by point below.

W1. Incremental E2E Speedup of CSR

We thank the reviewer for this comment. We would like to clarify that CSR serves as a complementary optimization, while AOSR provides the primary rollout acceleration. In our original OPD setting, only the student-generated suffix requires rollout, and the remaining training components are unchanged, limiting the additional E2E gains.

To further investigate this effect, we additionally evaluate full-sequence student rollout using LlamaGen-B and report the training-step latency breakdown below.

Training componentGKD (AR)AOSRAOSR + CSR
Student rollout (ms)782444234085
Teacher supervision (ms)106106106
Loss + backward (ms)128128128
Optimizer step (ms)222
Other overhead (ms)507895
E2E step time (ms)811047374416
E2E Speedup (↑)1.00×1.71×1.84×
Non-rollout time fraction (%)3.53%6.63%7.50%

Under full rollout, AOSR achieves 1.71× E2E speedup, which further increases to 1.84× with CSR, compared with 1.52× and 1.63× in the original OPD setting. Notably, CSR provides an additional 6.8% reduction in E2E training time on top of AOSR, consistent with its relative improvement in the original setting.

These results demonstrate that AOSR is responsible for the primary acceleration, while CSR consistently provides additional efficiency gains across different rollout workloads.

W2. Additional Evaluation on GenEval and DPG-Bench

We thank the reviewer for this suggestion. Following the reviewer's recommendation, we additionally evaluate the trained Janus-Pro-1B students on GenEval and DPG-Bench under identical evaluation settings.

GenEval Results (Janus-Pro-1B Student).

MethodSingle ObjectTwo ObjectsCountingColorsPositionColor AttributionOverall ↑
GKD (AR)0.980.830.500.900.660.560.74
GKD + AOSR0.970.830.510.910.650.560.74
GKD + AOSR + CSR0.980.830.510.900.630.570.74

DPG-Bench Results (Janus-Pro-1B Student).

MethodGlobalEntityAttributeRelationOtherOverall ↑
GKD (AR)87.6988.7288.2889.0188.4282.85
GKD + AOSR87.7088.7388.2488.9888.4182.84
GKD + AOSR + CSR87.7188.7488.2189.0088.4482.85

Both AOSR and AOSR+CSR achieve comparable GenEval (0.74) and DPG-Bench (82.84–82.85) performance to AR rollout, further supporting the generation-quality results reported in Table 2 of our submission.

Thank you for the response. It addressed most of my concerns, and I decide to keep my score.

We sincerely thank the reviewer for the positive feedback and for taking the time to evaluate our rebuttal. We appreciate your constructive comments, which have helped us strengthen the experimental evaluation and improve the clarity of our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.