- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes D3D, a standalone block-diffusion drafter for asynchronous speculative decoding. Unlike target-conditioned drafters such as DFlash, D3D does not use target hidden states. This lets it draft successor blocks during verification, following the SSD framework. The drafter is initialized from a pretrained AR model and trained on target-generated responses with a position-weighted denoising loss and an auxiliary AR loss. It is then progressively pruned according to acceptance length, with distillation-based recovery. During verification, one packed forward pass prepares blocks for likely rejection outcomes and for full acceptance, and a matching block is reused on a cache hit. With Qwen3-8B and Qwen3-32B targets, D3D reports average speedups of 5.00× and 5.71× across seven benchmarks, compared with 4.03× and 3.67× for DFlash.
Critical strengths
- Strong empirical results. D3D achieves the highest acceptance length on all seven benchmarks for both targets and the highest average speedup among the evaluated methods.
- Sensible design. Removing the dependency on target hidden states is a clean way to enable asynchronous drafting. One-pass diffusion drafting also fits naturally with SSD, whose AR drafters are limited by sequential drafting steps.
- Thorough ablations and reproducibility. The paper ablates the AR loss, drafter scale, initialization, depth, draft budget, block size, and anchor count, and provides code.
Critical weaknesses
- Fairness of comparison and attribution of gains. D3D uses two GPUs while synchronous baselines use one, and its ~7B drafter is roughly 3× the size of DFlash; there is no comparison against 2-GPU tensor parallelism for the target. Without a synchronous D3D result or a same-size AR drafter under SSD, it is unclear whether the gains come from asynchrony, diffusion drafting, or scale. D3D-1.7B is also missing from Table 1. In addition, the baselines are retrained without reported budgets, and the SSD configuration is unspecified.
- Contribution and novelty. Each claimed contribution is incremental. The drafter training mostly follows prior work, with the auxiliary AR loss as the main new element, and its gain over DFlash is confounded with drafter scale. Pruning yields only a ~5.6% speedup gain on Qwen3-8B, with no comparison to a smaller drafter or simpler pruning. The asynchronous decoding is essentially a simplified SSD, which already anticipates diffusion drafters.
- Incomplete evaluation. t_d, t_v, and cache-miss overhead are not reported, so the claim that drafting is hidden by verification is unverified. Only greedy decoding is evaluated, leaving cache-hit rates and losslessness under sampling untested. Under batching, cache misses occur almost every round, and the paper offers no fallback strategy for this. Evaluation is also limited to the Transformers backend.
Questions for authors
- How does D3D compare with running the target alone using 2-GPU tensor parallelism?
- To disentangle the sources of the gains: what speedup does D3D achieve under synchronous decoding, and how does D3D-1.7B compare with SSD using a same-size AR drafter?
- What are t_d, t_v, and the cache-miss overhead? Does pruning help with the Qwen3-32B target?
7 comments
Response to Reviewer c1Bv
W2: Contribution and novelty
Contribution and novelty. Each claimed contribution is incremental. The drafter training mostly follows prior work, with the auxiliary AR loss as the main new element, and its gain over DFlash is confounded with drafter scale. Pruning yields only a ~5.6% speedup gain on Qwen3-8B, with no comparison to a smaller drafter or simpler pruning. The asynchronous decoding is essentially a simplified SSD, which already anticipates diffusion drafters.
D3D's contribution is to make asynchronous speculative decoding effective with a standalone diffusion drafter. SSD uses an autoregressive drafter to prepare drafts for likely verification outcomes while the target verifies the current block. It mentions diffusion LLMs only in its related work, as one way to speed up drafting, and neither builds nor evaluates a diffusion drafter. The asynchronous scheme alone does not explain D3D's results. With a same-size Qwen3-1.7B AR drafter, SSD is 1.94$\times$ to 2.62$\times$ slower than D3D-1.7B, as shown in our response to W1, and in Table 1 D3D achieves the highest speedup on both targets, exceeding the strongest baseline by 12.6% on Qwen3-8B and 45.7% on Qwen3-32B.
Pairing SSD-style asynchronous drafting with a diffusion drafter is not straightforward. Existing diffusion drafters such as DFlash need the target's hidden states from the current round, so they cannot draft during verification at all. A drafter for this setting must reach long acceptance without any target features, which is what our target-aligned training achieves. Asynchronous drafting also relaxes the latency constraint of synchronous decoding. Since drafting time is hidden behind verification, the drafter need not be as fast as possible and can focus on acceptance length, as long as all successor blocks are prepared within verification. Our acceptance-guided pruning keeps this preparation fast while retaining over 99% of the acceptance length.
Our training is what makes a standalone drafter accurate. Under fully matched conditions, D3D is 11.7% faster than PARD-2 and accepts 5.8% more tokens per round. Its gain over DFlash is not confounded with scale, since D3D-1.7B, which is smaller than DFlash, still achieves longer acceptance on every benchmark in Table 3(b), as discussed in our response to W1. The auxiliary AR loss also improves at a fixed architecture on both Qwen3 and Llama-3.1-8B-Instruct.
Pruning targets the cost that asynchronous execution cannot hide. On a cache miss, which occurs in about 30% of rounds on Qwen3-32B, the redraft returns to the critical path, and smaller targets with shorter verification leave less room for drafting.
The ~5.6% end-to-end gain comes with almost no loss in acceptance, since pruning removes 13.9% of the layers while keeping over 99% of the acceptance length. A smaller drafter cannot offer the same trade-off. Moving from the 8B to the 4B backbone lowers by 6.2%, and on Qwen3-32B the D3D-1.7B drafter has a lower cache-hit rate than the pruned 8B drafter, 65.4% compared with 69.7%, and a higher amortized recovery cost, 8.1 ms compared with 7.2 ms per round.
To compare with simpler pruning, we used each method to remove five layers from the same frozen 36-layer drafter, without recovery training. We then measured the acceptance length on 512 prompts that were not used for layer selection.
| Pruning method | Acceptance length |
|---|---|
| Remove the last five layers | 2.28 |
| Uniform deletion | 4.12 |
| One-shot ranking | 4.94 |
| Iterative selection (ours) | 7.31 |
Iterative selection preserves far more of the drafter's accuracy, and in end-to-end decoding it reaches 1.93$\times$ the throughput of uniform deletion and 3.04$\times$ that of removing the last five layers.
Response to Reviewer c1Bv
W3: Completeness of the evaluation
Incomplete evaluation. t_d, t_v, and cache-miss overhead are not reported, so the claim that drafting is hidden by verification is unverified. Only greedy decoding is evaluated, leaving cache-hit rates and losslessness under sampling untested. Under batching, cache misses occur almost every round, and the paper offers no fallback strategy for this. Evaluation is also limited to the Transformers backend.
Drafting, verification, and cache-miss costs. Drafting is hidden behind verification. With the Qwen3-32B target, packed drafting takes 27.0 ms and verification 46.2 ms on average, and the remaining overhead comes from cache misses. We profile the pruned 31-layer D3D-8B and the D3D-1.7B drafters on the 222 prompts, measuring GPU stage times with CUDA events.
| Drafter | (ms) | (ms) | Cache hit rate | Recovery per miss (ms) | Amortized per round (ms) |
|---|---|---|---|---|---|
| D3D-8B, 31 layers | 27.0 | 46.2 | 69.7% | 24.0 | 7.2 |
| D3D-1.7B | 24.0 | 46.1 | 65.4% | 23.4 | 8.1 |
Each miss costs 24.0 ms, which amortizes to 7.2 ms per round at a 69.7% cache hit rate.
Losslessness and speedup under sampling. Under sampling, D3D follows the standard speculative sampling rule of Leviathan et al. (2023). The verifier accepts each drafted token according to the ratio of the target and drafter probabilities, and upon the first rejection resamples the token from the normalized residual distribution. This procedure provably preserves the target distribution. To verify this empirically, we ran D3D with Qwen3-8B and the 31-layer drafter on all 30 AIME25 problems, generating up to 1,024 tokens, and compared the distribution that D3D samples from with the target distribution recomputed independently on the same prefix. The empirical total variation (TV) and Jensen–Shannon (JS) discrepancies are summarized below.
| Precision | Temperature / top-p | Mean TV | Max TV | Mean JS (nats) |
|---|---|---|---|---|
| BF16 | 1 / 1 | |||
| BF16 | 0.6 / 0.95 | |||
| FP32 | 1 / 1 | |||
| FP32 | 0.6 / 0.95 |
In FP32, the maximum TV distance between the two distributions is below . The larger BF16 deviations are consistent with rounding error in the logits and probability computations, and the much smaller FP32 discrepancies support the correctness of the sampling implementation. With Qwen3-8B and the 31-layer drafter, running asynchronously on two H800 GPUs, D3D reaches an end-to-end speedup, including prefill, of 2.92$\times$ at T=1 and 3.71$\times$ at T=0.6 with top-p 0.95, with cache hit rates of 59.7% and 68.6%, and every benchmark improves in both settings.
Batched serving and backend. Cache misses do not occur in almost every round under batching. At concurrency 4, the cache-hit rate is 79.2% on the 222 prompts. We implemented D3D on an SSD-derived PyTorch serving backend with paged KV caching, in addition to the Transformers backend used in the paper. Cache hits and misses are handled per request, so a miss triggers a redraft only for that request, while the other requests keep their cached drafts. Against native AR decoding on the same backend, D3D is 3.73$\times$ faster at concurrency 1 and 2.03$\times$ faster at concurrency 4. At higher concurrency, decoding becomes compute-bound and the benefit of speculative decoding shrinks for any drafter. As in vLLM, which disables speculative decoding when the number of queued requests exceeds a threshold, a simple dispatcher then switches to AR decoding, so D3D matches native AR at concurrency 8 and 16.
Reference
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274–19286. PMLR, 2023.
Response to Reviewer c1Bv
We thank the reviewer for the detailed comments. Below we address each of the raised concerns:
W1: Fairness of comparison and attribution of gains
Two-GPU budget. On the same two H800 GPUs, D3D is 6.25$\times$ faster than the target alone with TP=2 on Qwen3-8B and 7.87$\times$ faster on Qwen3-32B, and it is faster on every benchmark. Table 1 reports the speedup of a single request at batch size 1, where TP=2 is slower than TP=1 because of communication overhead, so the 5.00$\times$ and 5.71$\times$ in Table 1, measured against TP=1, underestimate D3D's speedup over a two-GPU baseline. The synchronous baselines cannot use the second GPU for drafting either, because their drafters need the target's hidden states from the current round. This comparison uses the 750 prompts of Table 1, which consist of 128 prompts sampled from each of five benchmarks, together with all 80 MT-Bench and all 30 AIME25 prompts.
Drafter scale. D3D's advantage over DFlash does not come from drafter scale. DFlash draws its language prior from the target's hidden states, whereas D3D learns it within the drafter through auxiliary AR supervision, which preserves the pretrained model's next-token prediction on the unmasked prefix and improves τ at a fixed architecture in Table 3(a). With Qwen3-32B as the target, D3D-1.7B has 2.03B parameters including its embedding and LM head, fewer than DFlash's 2.57B even without the embedding and LM head it shares with the target, yet it reaches 6.1$\times$, 5.3$\times$, and 2.9$\times$ on MATH-500, HumanEval, and MT-Bench in Table 2, compared with 4.37$\times$, 3.75$\times$, and 3.00$\times$ for DFlash in Table 1.
Synchronous execution and SSD with a same-size drafter. Asynchronous execution and single-pass diffusion drafting each contribute to the speedup. Running the same pruned 31-layer D3D-8B checkpoint synchronously and asynchronously on the same two H800 GPUs, asynchronous execution is faster on every benchmark. increases average throughput by 33.3% on Qwen3-8B and 22.5% on Qwen3-32B, and reduces mean end-to-end latency by 24.7% and 18.4%.
To compare diffusion drafting with AR drafting at the same scale, we run SSD with a Qwen3-1.7B AR drafter against D3D-1.7B, with the Qwen3-8B target on two H800 GPUs and 222 prompts consisting of the first 32 prompts of six benchmarks and all 30 prompts of AIME25. In SSD, K is the number of drafted tokens per round, giving a block size of K+1, and F is the number of alternative tokens prepared per position. The three configurations below span block sizes from 4, the SSD setting for Qwen3-8B in Table 1, to 16, the block size of D3D, with F=2 matching D3D's fan-out at the larger block sizes.
D3D is faster than SSD in all three configurations, and the gap widens as K grows because SSD drafts K tokens sequentially for each branch, whereas D3D drafts a whole block in one forward pass. The effect of scale is covered by the drafter-scale results above, where a drafter smaller than DFlash still runs faster asynchronously. We will add these D3D-1.7B results to the revised paper.
Training budgets and SSD configuration. D3D and all retrained baselines use the same training data generated by the target model and are trained for three epochs on two H800 GPUs. SSD in Table 1 uses a Qwen3-0.6B AR drafter with F=4 for both targets, K=3 for Qwen3-8B, and K=5 for Qwen3-32B, matching the block sizes 4 and 6 in Table 1. Since Qwen3-32B has a higher verification latency than Qwen3-8B, the drafter can generate more tokens within each verification, which supports a larger K. The target and drafter each run on one H800 with CUDA graphs enabled. These details were omitted from the paper, and we will add them to Section 4.1.