- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
To deal with the high variance in draft quality in diffusion drafters for speculative decoding, this paper proposes AdaFlash, which contains two components for an online adaptive framework. First, an OPD algorithm continuously updates the drafter during deployment. Second, a lightweight adaptive length head predicts the acceptance ratio of each draft block from the drafter’s input, truncates the candidate sequence, and is trained online on the observed acceptance ratio. The experiments on three models across eight benchmarks show that AdaFlash achieves up to 5.3× speedup over autoregressive decoding and up to 66% higher average throughput than DFlash.
Critical strengths
- Clear and meaningful target. How to make diffusion-based speculative decoding robust is a practically important way to improve the current architecture.
- Simple method design. Both components are simple and lightweight. Therefore, it is easy to integrate into existing serving engines.
- Comprehensive evaluation. The experiments cover eight benchmarks spanning math, code, dialogue, and mixed domains. It also tests the three frontier models with different architectures.
Critical weaknesses
- Limited novelty. The ablations do not prove the effectiveness of the proposed designs. In Table 2, there is almost no improvement for reverse-KL, mixture loss, and divergence clipping.
- Limited practical gains. The paper emphasizes that AdaFlash’s advantage is especially remarkable at high concurrency. However, although it is better than other speculative decoding methods in this regime, its improvement over standard AR decoding is limited, and in some cases AdaFlash is even slower than standard AR.
- The cost of online training is not reported. AdaFlash relies on continuous online training, but there is no report about the resources consumed by training.
Questions for authors
- What's the performance if the request distribution changes over time? (Like math queries followed by chat queries)
- Were the standard high-concurrency optimizations applied to the baselines? (Like running EAGLE-3 with a smaller draft tree)
- How many GPUs are used for online training? Are the results obtained under the same total GPU budget?
2 comments
Dear Reviewer PJPS, thank you again for your time.
As the discussion phase is closing soon, we would appreciate to hear if our response has addressed your concerns, and if there is anything else to discuss.
We sincerely hope that our clarifications have been helpful and would be grateful if you could take them into account in your final assessment.
Best Regards,
Authors.
We appreciate the reviewer's feedback. Below, we address your technical questions and resolve any potential misunderstandings.
Q1. What's the performance if the request distribution changes over time? (Like math queries followed by chat queries)
A1. Thank you for this question. In Table 1, we already evaluate on the Blend dataset, which mixes math, code, and chat requests to simulate a heterogeneous serving workload. AdaFlash maintains its advantage under this mixed distribution: for example, with Qwen3-Coder-30B-A3B at concurrency 128, AdaFlash achieves 1.33× speedup compared to 0.92× for DFlash and 0.99× for OSD. This demonstrates the robustness of AdaFlash to mixed-domain request distributions.
Q2. Were the standard high-concurrency optimizations applied to the baselines? (Like running EAGLE-3 with a smaller draft tree)
A2. Thank you for this suggestion. Our ablation (Table 2) already includes DFlash with a fixed budget. Following your advice, we additionally sweep EAGLE-3's draft-token budget from 32 down to 2 at high concurrency. The results are as follow:
The above table demonstrates that, even at the best fixed budget (budget=4), AdaFlash maintains a 24.1%–49.1% throughput advantage. A fixed budget applies a uniform cap across all requests and decoding rounds, whereas AdaFlash's adaptive length head dynamically shortens verification when draft quality is low to avoid wasted compute, and extends it when quality is high to accept more tokens per round. This per-step adaptivity is what a static budget cannot provide.
Q3. How many GPUs are used for online training? Are the results obtained under the same total GPU budget?
A3. Thank you for this question. AdaFlash uses one H200 for inference and another for asynchronous training; the frozen baseline uses one H200. Both process the identical stream of 30,000 OpenCodeInstruct requests from the same checkpoint.
Effective GPU-hours combine inference service time and training computation time. Over the entire adaptation process, AdaFlash improves cumulative output per effective GPU-hour by 12.99%, including the training cost.
Q4. Novelty and practical gain.
A4. We thank the reviewer for this comment. We would like to take this opportunity to clarify the novelty and practical gain of our work:
Novelty: Our core observation is that diffusion drafters exhibit high variance in draft quality across decoding rounds. Based on this, we design a tailored mixture OPD loss with divergence clipping and a dynamic length head to address this issue.
Practical gain: Table 2 demonstrates the contribution of each component, where every module yields consistent improvements. Regarding high-concurrency performance: we remark that scaling to high concurrency is an open challenge for the entire speculative decoding field, as sufficiently large batch sizes will always favor standard AR decoding. AdaFlash pushes this frontier—it maintains speedup at concurrency 64 and partially at concurrency 128, a significant improvement over prior state-of-the-art methods.
We sincerely appreciate your helpful feedback. We have revised the manuscript accordingly. If our responses have addressed your concerns, we would be grateful if you would consider raising the score, and we would be glad to discuss any further questions. Thank you!