acceptodds
Under review as a conference paper at ICLR 2027

AdaFlash: Adaptive Speculative Decoding via On-Policy Distilled Diffusion Drafters

Abstract

Speculative decoding, in which a lightweight draft model first generates a draft sequence that is then verified by the target model, has become a prevalent paradigm for accelerating large language model inference. Recent work such as DFlash further boosts drafting efficiency by leveraging diffusion drafters, whose parallel denoising mechanism enables draft generation in a single forward pass. In this work, we uncover a *central pitfall* of diffusion drafters: the *global dependency*, arising from bidirectional attention and KV injection in diffusion drafters, is a double-edged sword. On one hand, it endows the model with parallel generation and global contextual modeling capabilities; on the other hand, this global dependency introduces *high variance* at both the domain-level and the token-level: acceptance rates fluctuate substantially across different domains, and the acceptance probability varies across draft positions. To tackle this issue, we propose the AdaFlash framework, comprising two components: (i) an *on-policy distillation* (OPD) algorithm with reverse-KL divergence tailored for diffusion drafters, which delivers stable convergence and continuously adapts the drafter to the deployment distribution; and (ii) an *adaptive length head* that dynamically adjusts the candidate length on the fly, substantially lowering the verification cost of the target model. Experiments demonstrate that AdaFlash consistently improves the speedup during deployment, with especially significant gains under high-concurrency, achieving up to 66% higher average throughput than previous state-of-the-art methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language model, speculative decoding, parallel decoding, efficient inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PJPSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

To deal with the high variance in draft quality in diffusion drafters for speculative decoding, this paper proposes AdaFlash, which contains two components for an online adaptive framework. First, an OPD algorithm continuously updates the drafter during deployment. Second, a lightweight adaptive length head predicts the acceptance ratio of each draft block from the drafter’s input, truncates the candidate sequence, and is trained online on the observed acceptance ratio. The experiments on three models across eight benchmarks show that AdaFlash achieves up to 5.3× speedup over autoregressive decoding and up to 66% higher average throughput than DFlash.

Critical strengths
  1. Clear and meaningful target. How to make diffusion-based speculative decoding robust is a practically important way to improve the current architecture.
  2. Simple method design. Both components are simple and lightweight. Therefore, it is easy to integrate into existing serving engines.
  3. Comprehensive evaluation. The experiments cover eight benchmarks spanning math, code, dialogue, and mixed domains. It also tests the three frontier models with different architectures.
Critical weaknesses
  1. Limited novelty. The ablations do not prove the effectiveness of the proposed designs. In Table 2, there is almost no improvement for reverse-KL, mixture loss, and divergence clipping.
  2. Limited practical gains. The paper emphasizes that AdaFlash’s advantage is especially remarkable at high concurrency. However, although it is better than other speculative decoding methods in this regime, its improvement over standard AR decoding is limited, and in some cases AdaFlash is even slower than standard AR.
  3. The cost of online training is not reported. AdaFlash relies on continuous online training, but there is no report about the resources consumed by training.
Questions for authors
  1. What's the performance if the request distribution changes over time? (Like math queries followed by chat queries)
  2. Were the standard high-concurrency optimizations applied to the baselines? (Like running EAGLE-3 with a smaller draft tree)
  3. How many GPUs are used for online training? Are the results obtained under the same total GPU budget?
2 comments

We appreciate the reviewer's feedback. Below, we address your technical questions and resolve any potential misunderstandings.

Q1. What's the performance if the request distribution changes over time? (Like math queries followed by chat queries)

A1. Thank you for this question. In Table 1, we already evaluate on the Blend dataset, which mixes math, code, and chat requests to simulate a heterogeneous serving workload. AdaFlash maintains its advantage under this mixed distribution: for example, with Qwen3-Coder-30B-A3B at concurrency 128, AdaFlash achieves 1.33× speedup compared to 0.92× for DFlash and 0.99× for OSD. This demonstrates the robustness of AdaFlash to mixed-domain request distributions.


Q2. Were the standard high-concurrency optimizations applied to the baselines? (Like running EAGLE-3 with a smaller draft tree)

A2. Thank you for this suggestion. Our ablation (Table 2) already includes DFlash with a fixed budget. Following your advice, we additionally sweep EAGLE-3's draft-token budget from 32 down to 2 at high concurrency. The results are as follow:

DatasetEAGLE-3 (32)EAGLE-3 (16)EAGLE-3 (8)EAGLE-3 (4)EAGLE-3 (2)DFlash (11)AdaFlash
MathQA3,449.234,935.076,822.847,244.766,279.149,621.7810,800.03
GSM8K3,550.934,995.076,812.077,152.976,360.929,114.1410,625.03
OpenCodeInstruct3,235.464,561.106,481.546,658.295,732.608,832.629,365.62
CodeAlpaca3,215.634,622.666,345.516,740.546,044.908,011.838,365.35

The above table demonstrates that, even at the best fixed budget (budget=4), AdaFlash maintains a 24.1%–49.1% throughput advantage. A fixed budget applies a uniform cap across all requests and decoding rounds, whereas AdaFlash's adaptive length head dynamically shortens verification when draft quality is low to avoid wasted compute, and extends it when quality is high to accept more tokens per round. This per-step adaptivity is what a static budget cannot provide.


Q3. How many GPUs are used for online training? Are the results obtained under the same total GPU budget?

A3. Thank you for this question. AdaFlash uses one H200 for inference and another for asynchronous training; the frozen baseline uses one H200. Both process the identical stream of 30,000 OpenCodeInstruct requests from the same checkpoint.

MethodInference GPU-hTraining GPU-hOutput tokens / effective GPU-h
Frozen drafter10.8580.0001,134,059
AdaFlash8.0861.5321,281,368

Effective GPU-hours combine inference service time and training computation time. Over the entire adaptation process, AdaFlash improves cumulative output per effective GPU-hour by 12.99%, including the training cost.


Q4. Novelty and practical gain.

A4. We thank the reviewer for this comment. We would like to take this opportunity to clarify the novelty and practical gain of our work:

  • Novelty: Our core observation is that diffusion drafters exhibit high variance in draft quality across decoding rounds. Based on this, we design a tailored mixture OPD loss with divergence clipping and a dynamic length head to address this issue.

  • Practical gain: Table 2 demonstrates the contribution of each component, where every module yields consistent improvements. Regarding high-concurrency performance: we remark that scaling to high concurrency is an open challenge for the entire speculative decoding field, as sufficiently large batch sizes will always favor standard AR decoding. AdaFlash pushes this frontier—it maintains speedup at concurrency 64 and partially at concurrency 128, a significant improvement over prior state-of-the-art methods.


We sincerely appreciate your helpful feedback. We have revised the manuscript accordingly. If our responses have addressed your concerns, we would be grateful if you would consider raising the score, and we would be glad to discuss any further questions. Thank you!

Dear Reviewer PJPS, thank you again for your time.

As the discussion phase is closing soon, we would appreciate to hear if our response has addressed your concerns, and if there is anything else to discuss.

We sincerely hope that our clarifications have been helpful and would be grateful if you could take them into account in your final assessment.

Best Regards,

Authors.

Reviewer_XBfHOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AdaFlash extends DFlash with online drafter adaptation and adaptive verification lengths. It combines hard-label supervision with clipped reverse-KL distillation, trains a lightweight length predictor, and implements asynchronous training and variable-length batching in SGLang. Experiments span eight benchmarks and three principal target models, including long reasoning and stochastic sampling. The paper reports up to 5.32× speedup over autoregressive decoding and 66% higher average throughput than DFlash in a high-concurrency setting.

Critical strengths
  • Potentially meaningful serving improvements. The method addresses both draft quality and verification overhead. On Qwen3-Coder-30B-A3B at concurrency 128, Table 1 reports average speedups of 1.69× versus 1.02× for DFlash and 1.14× for OSD. These gains would be practically consequential if established under a fully specified deployment budget.

  • Broad evaluation with useful ablations. The paper examines multiple architectures, domains, concurrency levels, long outputs, and stochastic sampling. The loss and length-head ablations help distinguish improvements in draft acceptance from savings in verification computation.

Critical weaknesses
  • The released implementation does not fully match the proposed training method. Section 3.1 specifies clipping individual vocabulary contributions before summation. However, the released loss implementation, lines 189–205, sums over the vocabulary, normalizes by the valid-token count, and then clips. These are materially different objectives. Furthermore, the released training path appears to compute teacher distributions on completed, verified responses rather than draft-generated prefixes containing rejected proposals. This leaves unclear whether the experiments validate the claimed entry-wise clipping and on-policy supervision.

  • The overall deployment advantage is insufficiently established. Appendix B.1 places training on separate GPUs but does not specify the experimental GPU types/counts, training cost, or elapsed adaptation time. Consequently, serving-pool speedups do not establish improved efficiency under a fixed total resource budget. The closest adaptive-verification comparison is also limited: Table 11 compares offline heads using a fixed threshold on one dataset, with roughly 2% gains and no uncertainty estimates; it does not evaluate DSpark’s calibrated, hardware-aware scheduling. This limits the broader state-of-the-art claim.

Questions for authors
  • Which code version produced the reported results? Please reconcile the clipping discrepancy and specify whether teacher distributions are evaluated on actual draft prefixes, including rejected proposals, or on verified target trajectories.

  • What hardware and GPU allocation were used for inference and training? Are the main results averaged during adaptation or measured after convergence, and what is the cumulative throughput per total GPU-hour, including adaptation?

  • Under matched training data and compute, how does AdaFlash compare with OSD using the same adaptive head and scheduler? Does the advantage over DSpark-style verification persist with calibrated scheduling and repeated measurements?

4 comments

We sincerely appreciate the reviewer's feedback. Below, we address your technical concerns.

Q1. Which code version produced the reported results? Please reconcile the clipping discrepancy and specify whether teacher distributions are evaluated on actual draft prefixes, including rejected proposals, or on verified target trajectories.

A1. Thank you for this question. We address both parts below.

  • Code implementation. We thank the reviewer for carefully examining both our implementation and the paper. The implementation in our anonymous repository (pipeline/training/rkl_model.py), which produced all reported results, performs token-level divergence clipping: the KL divergence at each draft position is first summed over the vocabulary, normalized by the total weight of valid draft positions, clipped with threshold δ=0.01\delta=0.01, and then summed with the position weights. We have revised the manuscript to reconcile the description with the implementation. Thank you again for helping us improve the clarity of the paper!

  • Teacher distributions. We clarify that teacher distributions are evaluated on verified target trajectories, not on rejected draft prefixes. In speculative decoding, rejected draft branches are discarded after each verification round, and the drafter always continues from the target model's verified or corrected prefix. Therefore, the target-corrected trajectory is the actual runtime context that the drafter must adapt to. By "on-policy" in this paper, we refer to training on this runtime context produced by the online speculative decoding system, rather than requiring independent student rollouts. We have clarified this distinction in the revised manuscript. Thank you!


Q2. What hardware and GPU allocation were used for inference and training? Are the main results averaged during adaptation or measured after convergence, and what is the cumulative throughput per total GPU-hour, including adaptation?

A2. Thank you for this question. In our experiments, inference runs on a single NVIDIA H200, and online training runs asynchronously on a separate H200. The main results are averaged after convergence. As you suggested, we report the cumulative throughput per total GPU-hour, including the adaptation (training) cost of AdaFlash:

Methodτ\tauTPSOutput tokensEffective GPU-hTokens / effective GPU-h
Frozen drafter2.677315.0212,313,64210.8581,134,059
AdaFlash (+train time)3.844423.3612,324,3849.6181,281,368

Here, effective GPU-h is the sum of the inference service time and the online training time. Both methods start from the same offline-trained checkpoint. Even when we include the training compute and give no credit for the asynchronous overlap, AdaFlash achieves 12.99% more tokens per effective GPU-hour and 1.34× higher service throughput than the frozen drafter. We have added this table to the revised manuscript.


Q3. Under matched training data and compute, how does AdaFlash compare with OSD using the same adaptive head and scheduler?

A3. Thank you for this question. The upper half of Table 2 (ablation study) already presents this controlled comparison, where the adaptive length head, online update mechanism, and inference scheduler are held fixed. Relative to the standard reverse-KL OSD baseline, adding divergence clipping and mixture OPD loss improves speedup from 4.17× to 4.54× at concurrency 1 (+8.9%) and from 1.18× to 1.27× at concurrency 128 (+7.6%), with mean acceptance length improving accordingly.


Q4. "...adaptive-verification comparison..."

A4. Thank you for this question. Regarding the adaptive-verification comparison, we clarify two points.

  • First, as we mentioned in the paper, "DSpark’s hardware-aware scheduling is complementary to our approach and can be integrated into our AdaFlash framework for further improvement", and we leave this integration as future work.

  • Second, following your suggestion, we add two additional datasets, repeat each configuration three times, and report the mean throughput. The GSM8K results are taken from Table 11:

DatasetMethodτ\tauVerify Len.Speedup
GSM8KDSpark's head5.0746.6731.00×
AdaFlash's length head5.1506.4371.02×
OpenCodeInstructDSpark's head5.9267.8171.00×
AdaFlash's length head5.7987.4791.02×
ShareGPTDSpark's head3.0474.3391.00×
AdaFlash's length head3.0004.2061.05×

The length head consistently achieves a stable 2%–5% throughput improvement across all datasets. As a component-level design that shares the same backbone and differs only in the head, we consider this a meaningful and reliable gain.


We hope the above responses have adequately addressed your technical concerns. If so, we would be grateful if you would consider raising the score, and we would be glad to discuss any remaining questions. Thank you!

Thank you for the clarifications and additional experiments. The response resolves the implementation ambiguity and provides useful evidence that gains persist after accounting for training costs. However, the matched OSD comparison remains unresolved: Section 4.1 uses forward KL, whereas Table 2 compares against reverse KL. The additional length-head results are encouraging, but the small gains without reported variability provide limited evidence of a reliable advantage.

Overall, my assessment has improved, but the evidence remains insufficient to establish the incremental contribution over appropriately matched baselines. I therefore maintain my rating of 2 (weak rejection).

AuthorsOpenReview ↗

Further Response to Reviewer XBfH

We believe the remaining concerns stem from misunderstandings, and we clarify them below.

1. Regarding KL direction. First, we emphasize that Table 2 does give a matched comparison: its first row shares the length head and scheduler with AdaFlash, and differs only in the objective. This row instantiates OSD with reverse KL, one of the distances used in the OSD paper (the OSD paper explicitly allows the distance to be forward KL or reverse KL). Second, Table 5(a) shows that our mixture loss also outperforms forward KL (α=1\alpha=1). Third, for the soft forward KL in Section 4.1, Table 1 at C=1C=1 is also matched, since the length head is disabled at this concurrency (Appendix B.1). Here, AdaFlash outperforms OSD in all 12 settings.

2. Regarding the length head. Both heads share the same DFlash backbone, serving engine, and scheduler, and differ only in the head. Regarding variability, we report the standard deviation over 3 runs:

DatasetMethodτ\tauVerify Len.Speedup
GSM8KAdaFlash's length head5.1506.437+2.21% ± 0.78%
OpenCodeInstructAdaFlash's length head5.7987.479+1.80% ± 0.88%
ShareGPTAdaFlash's length head3.0004.206+5.09% ± 0.79%

Our length head achieves higher mean throughput on all three datasets, and the gain on ShareGPT is well beyond the run-to-run variation. Besides, we respectfully disagree that these gains are too small to be meaningful: as one component of AdaFlash, our length head provides a consistent 2%–5% gain over DSpark's head, and we believe this is a meaningful gain for a single component.

Finally, we remark that both concerns relate to misunderstandings about experimental settings (where, as clarified above, the comparisons are properly matched and the gains are meaningful), and do not affect our core contribution: we identify the high variance of diffusion drafters across decoding rounds, and address it via on-policy distillation and an adaptive length head. We hope these clarifications resolve your remaining concerns.

Dear Reviewer XBfH,

Thank you again for your constructive review. As the discussion period is coming to a close, we would like to kindly follow up on whether our responses have addressed your concerns.

If any questions remain or further clarification would be helpful, please let us know. We would be happy to continue the discussion and improve our submission accordingly.

We sincerely hope that our clarifications have been helpful and would be grateful if you could take them into account in your final assessment.

Best regards, Authors

Reviewer_V2JfOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies diffusion drafters for speculative decoding and argues that their global dependency, which arises from bidirectional attention and KV injection, causes variance in draft quality across domains and draft positions. The authors propose AdaFlash, which combines online on-policy distillation with an adaptive length head that truncates the draft sequence before verification. Experiments across benchmarks and target models show that AdaFlash improves throughput over the baselines, particularly under high concurrency.

Critical strengths
  1. The paper identifies a critical limitation of previous diffusion drafters and proposes methods that address it.

  2. The paper considers high-concurrency serving, a practical setting that much prior work on speculative decoding has overlooked.

  3. The experiments cover multiple target models, benchmarks, and serving settings.

Critical weaknesses
  1. Considerable implementation and deployment complexity. AdaFlash requires substantial modifications to the underlying serving infrastructure, including variable-length verification, KV-cache management, CUDA Graph execution, and architecture-specific kernels. Its public implementation also depends on a patched SGLang version rather than a standard release. Together with the separate training pipeline and dynamic weight updates, these requirements raise concerns about deployment complexity, maintainability, and portability. The paper does not sufficiently discuss these practical costs.

  2. Unanalyzed policy staleness in asynchronous online distillation. AdaFlash decouples training and inference through asynchronous workers and periodically hot-reloads updated drafter weights. Consequently, training samples may have been collected under earlier drafter policies, introducing policy staleness and potentially undermining the claimed on-policy training benefits. However, the paper does not quantify policy lag or investigate its effects on training stability, convergence, and acceptance rates.

Questions for authors
  1. Section 3.2 suggests that γ\gamma can be adjusted according to system load. Is γ\gamma dynamically adjusted during the experiments, or is it held fixed? If dynamically adjusted, what specific system metrics and adjustment rule are used?

  2. How is the ground-truth acceptance ratio at⋆a_t^\star obtained when only the first k^t<k\hat{k}_t<k tokens are verified? If all verified tokens are accepted, the true acceptance length of the full draft remains unknown. Does the method perform additional target verification or explicitly account for such censored observations during online training?

5 comments

We sincerely thank you for your helpful comments and positive rating. Below, we clarify your concerns.

Q1. Section 3.2 suggests that gamma can be adjusted according to system load. Is gamma dynamically adjusted during the experiments, or is it held fixed? If dynamically adjusted, what specific system metrics and adjustment rule are used?

A1. We thank the reviewer for this question. We clarify that γ\gamma is fixed at 1.0. The discussion in Section 3.2 also considers practical deployment scenarios in which the system load varies between peak and off-peak periods (e.g., DSpark). We note that load-aware adjustment of γ\gamma is orthogonal to our length prediction mechanism. We leave dynamic scheduling of γ\gamma to future work, and have clarified this point in the revised manuscript.


Q2. How is the ground-truth acceptance ratio obtained when only the first tokens are verified? If all verified tokens are accepted, the true acceptance length of the full draft remains unknown. Does the method perform additional target verification or explicitly account for such censored observations during online training?

A2. Thank you for this question. We would like to clarify a potential misunderstanding here: the online inference verification and the training label construction are decoupled. During serving, only the first k^t\hat{k}_t draft tokens are verified to reduce overhead. During asynchronous training, we separately recompute target-model logits on verified target trajectories and construct at⋆a_t^\star for the full training-time draft block, so all valid positions, including those beyond k^t\hat{k}_t, are evaluated. Thus, the ground-truth supervision labels are not affected by the truncated verification. We have clarified this in the revised manuscript.


Q3. Implementation and delayed feedback.

A3. Thank you for pointing this out. We answer each question as below:

  • Regarding implementation complexity: our implementation modifies SGLang to support asynchronous training and the dynamic length head. After refactoring, the diff against the SGLang main branch remains manageable, and the design is portable to other serving frameworks (e.g., vLLM). We will add a detailed description of the implementation in the revised manuscript.

  • Regarding policy staleness: we acknowledge that asynchronous training introduces a lag between data collection and weight updates, which is an interesting direction for further investigation. In this paper, we mainly focused on the variance issue of diffusion drafter, and leave the study of policy staleness as an important future work. Thank you again for your valuable comments!


We sincerely appreciate the reviewer's helpful feedback, and have revised the manuscript accordingly. If our responses have addressed your concerns, we would be grateful if you would consider raising the score. Thank you!

Thanks for authors' response, I will keep my positive evaluation.

Dear Reviewer V2Jf,

We sincerely hope that our clarifications have been helpful and would be grateful if you could take them into account in your final assessment. As the discussion period is coming to an end, we would be glad to clarify any remaining questions. Thank you again for your time and feedback.

Best Regards,

Authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.