- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes a training free diffusion drafter based speculative decoding method named "Fail Fast, Win Big". The core idea is to couple an autoregressive LLM as the target model with a separate diffusion LLM (dLLM) as the draft model to do the speculative decoding. The major intuition is that the dLLM drafter’s token confidence can serve as a proxy for whether a generation segment is easy or hard to speculate. The drafter always uses a minimal denoising budget, so for hard segments, where a low-confidence token appears early, the draft is cut short and little drafting compute is wasted on tokens likely to be rejected (“fail fast”); while for easy segments, where all tokens remain high-confidence, the draft is aggressively extended in chunks to increase the number of draft tokens to increase the acceptance length ("win big"). This design is also supported by the dLLM's fast decoding capability of generating multiple draft tokens with minimal forward passes. For the evaluation, this paper compares FailFast with an AR drafter (Qwen2.5-1.5B), a Fast-dLLM drafter, and EAGLE-3 on Qwen2.5-7B/14B/32B targets across MATH, AIME, GSM8K, GPQA, and HumanEval, and shows that FailFast achieves up to 4.9× speedup over vanilla decoding.
Critical strengths
Overall, I think the idea of fail fast, win big introduced in this work for the diffusion drafter based speculative decoding is well motivated and makes sense. The preliminary analysis and observations in Section 3 are interesting; and the corresponding designed method in Section 4 is straightforward and easy to implement. I think this idea may help the adaptive draft length design of the diffusion based SD.
Critical weaknesses
(1) Even though the idea is neat, I feel the evaluation is quite unfair and unconvincing. The speedups in Table 1 can be derived from a simple cost model rather than measured one. The reported speedups can be closely reproduced from the per-round data in Table 4 (accepted length and drafter passes) by assuming a fixed cost per drafter pass and a fixed cost per verification process, regardless of draft length. Meanwhile, EAGLE-3 is measured in SGLang. Therefore, comparing modeled numbers against measured ones is actually unfair and not correct. This work is about speculative decoding, where the real measured end-to-end throughput speedup is a must.
(2) The generalization ability of this idea is also quite worrying. The paper only shows the proof of concept (also the evaluation setting is un-fair) for the Qwen2.5-{32B,14B,7B}-Instruct target models with the corresponding Fast-dLLM-v2-1.5B diffusion drafter which is trained from the Qwen2.5-1.5B model (Also LLaMA2-13B with DiffuLLaMA-7B). Therefore, if we wanna generalize this idea to other model families like Gemma, DeepSeek, or KIMI, we must train a new diffusion model, which is actually non-trivial and very computational costly. Therefore, I do not appreciate the generalization ability of this idea as well.
Questions for authors
(1) Are the FailFast, Fast-dLLM and AR-drafter speedups in Table 1, Table 6 and Fig. 7 measured end to end results? If not, please do provide end-to-end measurements for all methods in the same serving framework.
(2) Are the proposed FailFast lossless speculative decoding for both T=0 and T>1 sampling scenarios?
I do not see any rebuttal and response from author. I think it is not enough to be accepted. I highly do not recommend to accept this paper.