acceptodds
Under review as a conference paper at ICLR 2027

Fail Fast, Win Big: Rethinking the Drafting Strategy in Speculative Decoding via Diffusion LLMs

Abstract

Speculative decoding efforts have largely optimized draft quality for enhanced performance, treating policy as a secondary focus. While effective, this approach can be costly and brittle, as speedups depend on continually improving drafter quality. We argue that drafting policy should be a first-class driver of performance. Our key insight is that intrinsic properties of emerging diffusion LLMs (dLLMs) make this shift both possible and effective. Their accuracy improves sublinearly with compute, yielding diminishing returns to quality, and unlike with autoregressive models, drafting cost is nearly independent of length, enabling long drafts at low marginal cost. Together, these properties favor policies that hedge against draft quality – dynamically allocating compute across the sequence based on heterogeneous decoding difficulties across the sequence – rather than attempting to maximize it. We present FailFast, a speculative decoding framework that operationalizes this perspective. FailFast allocates minimal compute to hard regions ("fail fast") while aggressively extending drafts in easy regions ("win big"), often accepting tens of tokens per step. Without any fine-tuning, FailFast provides lossless acceleration of autoregressive LLMs, achieving up to 4.9× speedup over standard decoding, 1.7× over a naive dLLM drafter baseline, and 1.7× over EAGLE-3 across diverse models and workloads.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Inference, Speculative Decoding, Diffusion LLMs

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_F7RDOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a training free diffusion drafter based speculative decoding method named "Fail Fast, Win Big". The core idea is to couple an autoregressive LLM as the target model with a separate diffusion LLM (dLLM) as the draft model to do the speculative decoding. The major intuition is that the dLLM drafter’s token confidence can serve as a proxy for whether a generation segment is easy or hard to speculate. The drafter always uses a minimal denoising budget, so for hard segments, where a low-confidence token appears early, the draft is cut short and little drafting compute is wasted on tokens likely to be rejected (“fail fast”); while for easy segments, where all tokens remain high-confidence, the draft is aggressively extended in chunks to increase the number of draft tokens to increase the acceptance length ("win big"). This design is also supported by the dLLM's fast decoding capability of generating multiple draft tokens with minimal forward passes. For the evaluation, this paper compares FailFast with an AR drafter (Qwen2.5-1.5B), a Fast-dLLM drafter, and EAGLE-3 on Qwen2.5-7B/14B/32B targets across MATH, AIME, GSM8K, GPQA, and HumanEval, and shows that FailFast achieves up to 4.9× speedup over vanilla decoding.

Critical strengths

Overall, I think the idea of fail fast, win big introduced in this work for the diffusion drafter based speculative decoding is well motivated and makes sense. The preliminary analysis and observations in Section 3 are interesting; and the corresponding designed method in Section 4 is straightforward and easy to implement. I think this idea may help the adaptive draft length design of the diffusion based SD.

Critical weaknesses

(1) Even though the idea is neat, I feel the evaluation is quite unfair and unconvincing. The speedups in Table 1 can be derived from a simple cost model rather than measured one. The reported speedups can be closely reproduced from the per-round data in Table 4 (accepted length and drafter passes) by assuming a fixed cost per drafter pass and a fixed cost per verification process, regardless of draft length. Meanwhile, EAGLE-3 is measured in SGLang. Therefore, comparing modeled numbers against measured ones is actually unfair and not correct. This work is about speculative decoding, where the real measured end-to-end throughput speedup is a must.

(2) The generalization ability of this idea is also quite worrying. The paper only shows the proof of concept (also the evaluation setting is un-fair) for the Qwen2.5-{32B,14B,7B}-Instruct target models with the corresponding Fast-dLLM-v2-1.5B diffusion drafter which is trained from the Qwen2.5-1.5B model (Also LLaMA2-13B with DiffuLLaMA-7B). Therefore, if we wanna generalize this idea to other model families like Gemma, DeepSeek, or KIMI, we must train a new diffusion model, which is actually non-trivial and very computational costly. Therefore, I do not appreciate the generalization ability of this idea as well.

Questions for authors

(1) Are the FailFast, Fast-dLLM and AR-drafter speedups in Table 1, Table 6 and Fig. 7 measured end to end results? If not, please do provide end-to-end measurements for all methods in the same serving framework.

(2) Are the proposed FailFast lossless speculative decoding for both T=0 and T>1 sampling scenarios?

1 comment

I do not see any rebuttal and response from author. I think it is not enough to be accepted. I highly do not recommend to accept this paper.

Reviewer_ninXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  1. Prior work on speculative decoding has mostly improved draft quality, so speedups depend on continued investment in drafters. This paper instead treats the drafting policy as a first-class driver of performance, using a diffusion LLM (dLLM) as the drafter for an autoregressive target model.
  2. The approach builds on two dLLM properties. First, drafting cost is nearly independent of draft length. Second, accuracy improves sublinearly with compute, so returns diminish: in Fig. 2, 2.6× more drafter forward passes raise the acceptance rate by only 7.3%. Since decoding difficulty also varies within a sequence, the authors generate long drafts in easy regions and spend minimal compute in hard regions. This hedges against draft quality instead of trying to maximize it.
  3. The proposed framework, FailFast, needs no training and has two parts. "Fail fast" generates drafts with a single denoising pass, typically only one denoising step (model forward pass), which avoids spending extra compute on hard-to-predict tokens. "Win big" uses the drafter's token confidence to find easy regions and aggressively extends the speculation length there, up to 70 tokens in a single round. This reduces verification overhead by amortizing repeated drafter–verifier interactions.
  4. The experiments use Qwen2.5-7B/14B/32B-Instruct as targets and an off-the-shelf Fast-dLLM v2 1.5B as the drafter, without fine-tuning. On MATH, AIME, GSM8K, GPQA and HumanEval, FailFast achieves lossless speedups of 1.9–4.9× over vanilla decoding. It is 1.3–2.1× faster than an AR drafter and 1.2–1.7× faster than a naive dLLM drafter. Against EAGLE-3 it is 1.0–1.4× faster on average and up to 1.7× faster on individual datasets.
Critical strengths
  1. The paper is clearly written and easy to follow. Each design choice is motivated by a simple empirical observation in Section 3 (Figs. 2–5), and Fig. 1 and Algorithm 1 make the method easy to understand. The "fail fast / win big" framing captures the core idea well, and the method is simple to implement on top of an off-the-shelf dLLM.
  2. Combining the observation that generated tokens fall into easy and hard regions with intrinsic dLLM properties is a novel angle. The paper shows that many tokens lie in long easy regions (Fig. 5: 50–70% of tokens belong to runs of more than 10 easy tokens). Because dLLM drafting cost is nearly independent of length, very long drafts in these regions are cheap, which autoregressive drafters cannot afford. This shifts the focus of speculative decoding from improving draft quality to designing the drafting policy.
  3. The speedups are substantial and lossless: 1.9–4.9× over vanilla decoding, 1.3–2.1× over an AR drafter, and 1.2–1.7× over a naive dLLM drafter. FailFast also beats a trained EAGLE-3 baseline on average for all three target models (by up to 1.7× on individual datasets), even though it uses an off-the-shelf dLLM without fine-tuning. The gains carry over to a LLaMA-based setup (4.24× vs. 1.61× with an AR drafter), with the same hyperparameters used across datasets.
Critical weaknesses
  1. The proposed method is evaluated on Qwen 2.5 series, which has limited capacity. I understand that this is because Fastdllm v2 is trained on Qwen 2.5, but it also limits the study to merely one model series.
  2. Broader evaluations are needed as Fastdllm v2 is know to overfit on math and coding benchmarks like gsm8k, math500. I suggest add more evaluations on GSM-Plus, LiveCodeBench (late 2026 questions), AIME 2026, HMMT 2026, etc.
Questions for authors
  1. It would be better to provide the baseline performance of the draft dLLM on its own.
  2. Can the proposed method be combined with more advanced dLLM drafters such as DSpark (though already listed as concurrent work)?
Reviewer_PG12OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that speculative decoding should treat drafting policy, rather than drafter quality, as the main lever, and that diffusion LLM (dLLM) drafters make this possible because their drafting cost is nearly independent of draft length and their accuracy improves sublinearly with denoising steps. It also observes that generation contains long runs of easy tokens interleaved with hard regions. FailFast uses an off-the-shelf dLLM drafter with a single denoising step per block ("fail fast") and keeps extending the draft while every token's confidence exceeds a threshold ("win big"). Validation uses Qwen2.5 targets of three sizes on math, science, and code benchmarks at batch size one. FailFast reports speedups over vanilla decoding of up to 4.9x and consistent gains over an autoregressive drafter, a Fast-dLLM drafter, and EAGLE-3, with further results on a LLaMA2 target, sampling at temperature 1, and a data-matched EAGLE-3.

Critical strengths

A simple, training-free rule gives consistent wall-clock speedups over strong, carefully handled baselines across target sizes and benchmarks (Table 1). The autoregressive drafter's length is tuned per dataset, EAGLE-3 is trained to convergence and run with its draft tree, and the data-matched EAGLE-3 (Table 8) addresses the concern that the dLLM drafter saw more math data. This is the main reason for acceptance: the gains survive the comparisons most likely to remove them, and the method is easy to adopt.

The motivating measurements are informative and explain the result. Additional denoising steps raise acceptance only slightly (Figure 2), a large share of the output lies in long easy runs (Figures 4 and 5), and the speedup comes from fewer verification rounds and cheaper drafting (Table 4, Figure 7). This matters because it shows why long, cheap drafts pay off with dLLM drafters, which is useful beyond this particular rule.

Critical weaknesses

The two components are not ablated separately, so the source of the gain is unclear. No configuration reports one-step drafting at a fixed length or multi-step drafting with the extension rule, and Appendix B indicates that the draft is extended in only a minority of rounds. This is consequential because the paper's framing centers on the adaptive policy, while the evidence is equally consistent with most of the speedup coming from one-step drafting alone. Confidence-gated adaptive draft length is itself established for autoregressive drafters (SpecDec++, EAGLE-2), so the paper's contribution depends on this split.

The claim that the gains come from "drafting policy alone, without fine-tuning", and the experimental scope, are broader than the evidence. The drafter used is adapted from the target's own model family and post-trained, and the LLaMA2 experiment likewise uses a dLLM adapted from LLaMA2, so draft quality comes from that adaptation rather than being avoided. Every experiment also runs at batch size one, while long drafts become costly to verify at larger batches, as Appendix C acknowledges. This weighs on the recommendation as a framing problem that narrower claims would fix, not as a defect in the method.

Questions for authors
  • What is the end-to-end speedup of one-step drafting at a fixed draft length with no extension, and of multi-step drafting with the extension rule? These two settings would isolate the two components.
  • How does FailFast behave at moderate batch sizes, and how quickly does the best maximum draft length shrink as verification becomes compute-bound?
  • How were the per-model confidence thresholds chosen, and on which data? The sensitivity analysis covers only one model and one dataset.
  • All benchmarks are math, science QA, or code. A chat workload such as MT-Bench, which matches EAGLE-3's training data, would show whether the advantage holds there.
  • Scoping the abstract and conclusion to batch-1 decoding with a dLLM drafter from the same model family would match the evidence. The block-diffusion drafting work discussed in Appendix H would also be easier to find in the main related-work section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.