acceptodds
Under review as a conference paper at ICLR 2027

D3D: Decoupled Diffusion Drafting for Asynchronous Speculative Decoding

10 views

Abstract

Target-conditioned diffusion drafters predict candidate tokens in parallel, yet each new block must wait for updated target-model hidden states. This forces drafting and verification to run sequentially, limiting speedup even though each block is generated in one pass. We introduce D3D, a standalone diffusion drafter that removes this inter-round dependency. D3D combines blockwise denoising with auxiliary autoregressive supervision to achieve long accepted prefixes without target hidden states. This independence lets D3D draft successor blocks during verification in one packed pass, enabling asynchronous speculative decoding. We progressively prune layers according to their effect on acceptance length to reduce the cost of packed drafting. Removing five of 36 layers retains over 99% of mean acceptance length across three benchmarks with Qwen3-8B as the target. Across seven benchmarks, D3D averages 5.00× and 5.71× decoding speedup with Qwen3-8B and Qwen3-32B targets, compared with 4.03× and 3.67× for DFlash. On GSM8K with Qwen3-32B, speedup reaches 7.48×.

Primary area:
foundation or frontier models, including LLMs
Keywords:
diffusion, asynchronous, speculative decoding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_c1BvOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes D3D, a standalone block-diffusion drafter for asynchronous speculative decoding. Unlike target-conditioned drafters such as DFlash, D3D does not use target hidden states. This lets it draft successor blocks during verification, following the SSD framework. The drafter is initialized from a pretrained AR model and trained on target-generated responses with a position-weighted denoising loss and an auxiliary AR loss. It is then progressively pruned according to acceptance length, with distillation-based recovery. During verification, one packed forward pass prepares blocks for likely rejection outcomes and for full acceptance, and a matching block is reused on a cache hit. With Qwen3-8B and Qwen3-32B targets, D3D reports average speedups of 5.00× and 5.71× across seven benchmarks, compared with 4.03× and 3.67× for DFlash.

Critical strengths
  1. Strong empirical results. D3D achieves the highest acceptance length on all seven benchmarks for both targets and the highest average speedup among the evaluated methods.
  2. Sensible design. Removing the dependency on target hidden states is a clean way to enable asynchronous drafting. One-pass diffusion drafting also fits naturally with SSD, whose AR drafters are limited by sequential drafting steps.
  3. Thorough ablations and reproducibility. The paper ablates the AR loss, drafter scale, initialization, depth, draft budget, block size, and anchor count, and provides code.
Critical weaknesses
  1. Fairness of comparison and attribution of gains. D3D uses two GPUs while synchronous baselines use one, and its ~7B drafter is roughly 3× the size of DFlash; there is no comparison against 2-GPU tensor parallelism for the target. Without a synchronous D3D result or a same-size AR drafter under SSD, it is unclear whether the gains come from asynchrony, diffusion drafting, or scale. D3D-1.7B is also missing from Table 1. In addition, the baselines are retrained without reported budgets, and the SSD configuration is unspecified.
  2. Contribution and novelty. Each claimed contribution is incremental. The drafter training mostly follows prior work, with the auxiliary AR loss as the main new element, and its gain over DFlash is confounded with drafter scale. Pruning yields only a ~5.6% speedup gain on Qwen3-8B, with no comparison to a smaller drafter or simpler pruning. The asynchronous decoding is essentially a simplified SSD, which already anticipates diffusion drafters.
  3. Incomplete evaluation. t_d, t_v, and cache-miss overhead are not reported, so the claim that drafting is hidden by verification is unverified. Only greedy decoding is evaluated, leaving cache-hit rates and losslessness under sampling untested. Under batching, cache misses occur almost every round, and the paper offers no fallback strategy for this. Evaluation is also limited to the Transformers backend.
Questions for authors
  1. How does D3D compare with running the target alone using 2-GPU tensor parallelism?
  2. To disentangle the sources of the gains: what speedup does D3D achieve under synchronous decoding, and how does D3D-1.7B compare with SSD using a same-size AR drafter?
  3. What are t_d, t_v, and the cache-miss overhead? Does pruning help with the Qwen3-32B target?
7 comments
AuthorsOpenReview ↗

Response to Reviewer c1Bv

We thank the reviewer for the detailed comments. Below we address each of the raised concerns:

W1: Fairness of comparison and attribution of gains

Fairness of comparison and attribution of gains. D3D uses two GPUs while synchronous baselines use one, and its ~7B drafter is roughly 3× the size of DFlash; there is no comparison against 2-GPU tensor parallelism for the target. Without a synchronous D3D result or a same-size AR drafter under SSD, it is unclear whether the gains come from asynchrony, diffusion drafting, or scale. D3D-1.7B is also missing from Table 1. In addition, the baselines are retrained without reported budgets, and the SSD configuration is unspecified.

Two-GPU budget. On the same two H800 GPUs, D3D is 6.25$\times$ faster than the target alone with TP=2 on Qwen3-8B and 7.87$\times$ faster on Qwen3-32B, and it is faster on every benchmark. Table 1 reports the speedup of a single request at batch size 1, where TP=2 is slower than TP=1 because of communication overhead, so the 5.00$\times$ and 5.71$\times$ in Table 1, measured against TP=1, underestimate D3D's speedup over a two-GPU baseline. The synchronous baselines cannot use the second GPU for drafting either, because their drafters need the target's hidden states from the current round. This comparison uses the 750 prompts of Table 1, which consist of 128 prompts sampled from each of five benchmarks, together with all 80 MT-Bench and all 30 AIME25 prompts.

Drafter scale. D3D's advantage over DFlash does not come from drafter scale. DFlash draws its language prior from the target's hidden states, whereas D3D learns it within the drafter through auxiliary AR supervision, which preserves the pretrained model's next-token prediction on the unmasked prefix and improves τ\tau at a fixed architecture in Table 3(a). With Qwen3-32B as the target, D3D-1.7B has 2.03B parameters including its embedding and LM head, fewer than DFlash's 2.57B even without the embedding and LM head it shares with the target, yet it reaches 6.1$\times$, 5.3$\times$, and 2.9$\times$ on MATH-500, HumanEval, and MT-Bench in Table 2, compared with 4.37$\times$, 3.75$\times$, and 3.00$\times$ for DFlash in Table 1.

Synchronous execution and SSD with a same-size drafter. Asynchronous execution and single-pass diffusion drafting each contribute to the speedup. Running the same pruned 31-layer D3D-8B checkpoint synchronously and asynchronously on the same two H800 GPUs, asynchronous execution is faster on every benchmark. increases average throughput by 33.3% on Qwen3-8B and 22.5% on Qwen3-32B, and reduces mean end-to-end latency by 24.7% and 18.4%.

To compare diffusion drafting with AR drafting at the same scale, we run SSD with a Qwen3-1.7B AR drafter against D3D-1.7B, with the Qwen3-8B target on two H800 GPUs and 222 prompts consisting of the first 32 prompts of six benchmarks and all 30 prompts of AIME25. In SSD, K is the number of drafted tokens per round, giving a block size of K+1, and F is the number of alternative tokens prepared per position. The three configurations below span block sizes from 4, the SSD setting for Qwen3-8B in Table 1, to 16, the block size of D3D, with F=2 matching D3D's fan-out at the larger block sizes.

SSD configurationBlock sizeThroughput ratio (D3D / SSD)
K3/F441.94$\times$
K7/F282.12$\times$
K15/F2162.62$\times$

D3D is faster than SSD in all three configurations, and the gap widens as K grows because SSD drafts K tokens sequentially for each branch, whereas D3D drafts a whole block in one forward pass. The effect of scale is covered by the drafter-scale results above, where a drafter smaller than DFlash still runs faster asynchronously. We will add these D3D-1.7B results to the revised paper.

Training budgets and SSD configuration. D3D and all retrained baselines use the same training data generated by the target model and are trained for three epochs on two H800 GPUs. SSD in Table 1 uses a Qwen3-0.6B AR drafter with F=4 for both targets, K=3 for Qwen3-8B, and K=5 for Qwen3-32B, matching the block sizes 4 and 6 in Table 1. Since Qwen3-32B has a higher verification latency than Qwen3-8B, the drafter can generate more tokens within each verification, which supports a larger K. The target and drafter each run on one H800 with CUDA graphs enabled. These details were omitted from the paper, and we will add them to Section 4.1.

AuthorsOpenReview ↗

Response to Reviewer c1Bv

W2: Contribution and novelty

Contribution and novelty. Each claimed contribution is incremental. The drafter training mostly follows prior work, with the auxiliary AR loss as the main new element, and its gain over DFlash is confounded with drafter scale. Pruning yields only a ~5.6% speedup gain on Qwen3-8B, with no comparison to a smaller drafter or simpler pruning. The asynchronous decoding is essentially a simplified SSD, which already anticipates diffusion drafters.

D3D's contribution is to make asynchronous speculative decoding effective with a standalone diffusion drafter. SSD uses an autoregressive drafter to prepare drafts for likely verification outcomes while the target verifies the current block. It mentions diffusion LLMs only in its related work, as one way to speed up drafting, and neither builds nor evaluates a diffusion drafter. The asynchronous scheme alone does not explain D3D's results. With a same-size Qwen3-1.7B AR drafter, SSD is 1.94$\times$ to 2.62$\times$ slower than D3D-1.7B, as shown in our response to W1, and in Table 1 D3D achieves the highest speedup on both targets, exceeding the strongest baseline by 12.6% on Qwen3-8B and 45.7% on Qwen3-32B.

Pairing SSD-style asynchronous drafting with a diffusion drafter is not straightforward. Existing diffusion drafters such as DFlash need the target's hidden states from the current round, so they cannot draft during verification at all. A drafter for this setting must reach long acceptance without any target features, which is what our target-aligned training achieves. Asynchronous drafting also relaxes the latency constraint of synchronous decoding. Since drafting time is hidden behind verification, the drafter need not be as fast as possible and can focus on acceptance length, as long as all successor blocks are prepared within verification. Our acceptance-guided pruning keeps this preparation fast while retaining over 99% of the acceptance length.

Our training is what makes a standalone drafter accurate. Under fully matched conditions, D3D is 11.7% faster than PARD-2 and accepts 5.8% more tokens per round. Its gain over DFlash is not confounded with scale, since D3D-1.7B, which is smaller than DFlash, still achieves longer acceptance on every benchmark in Table 3(b), as discussed in our response to W1. The auxiliary AR loss also improves τ\tau at a fixed architecture on both Qwen3 and Llama-3.1-8B-Instruct.

Pruning targets the cost that asynchronous execution cannot hide. On a cache miss, which occurs in about 30% of rounds on Qwen3-32B, the redraft returns to the critical path, and smaller targets with shorter verification leave less room for drafting.

The ~5.6% end-to-end gain comes with almost no loss in acceptance, since pruning removes 13.9% of the layers while keeping over 99% of the acceptance length. A smaller drafter cannot offer the same trade-off. Moving from the 8B to the 4B backbone lowers τ\tau by 6.2%, and on Qwen3-32B the D3D-1.7B drafter has a lower cache-hit rate than the pruned 8B drafter, 65.4% compared with 69.7%, and a higher amortized recovery cost, 8.1 ms compared with 7.2 ms per round.

To compare with simpler pruning, we used each method to remove five layers from the same frozen 36-layer drafter, without recovery training. We then measured the acceptance length on 512 prompts that were not used for layer selection.

Pruning methodAcceptance length τ\tau
Remove the last five layers2.28
Uniform deletion4.12
One-shot ranking4.94
Iterative selection (ours)7.31

Iterative selection preserves far more of the drafter's accuracy, and in end-to-end decoding it reaches 1.93$\times$ the throughput of uniform deletion and 3.04$\times$ that of removing the last five layers.

AuthorsOpenReview ↗

Response to Reviewer c1Bv

W3: Completeness of the evaluation

Incomplete evaluation. t_d, t_v, and cache-miss overhead are not reported, so the claim that drafting is hidden by verification is unverified. Only greedy decoding is evaluated, leaving cache-hit rates and losslessness under sampling untested. Under batching, cache misses occur almost every round, and the paper offers no fallback strategy for this. Evaluation is also limited to the Transformers backend.

Drafting, verification, and cache-miss costs. Drafting is hidden behind verification. With the Qwen3-32B target, packed drafting takes 27.0 ms and verification 46.2 ms on average, and the remaining overhead comes from cache misses. We profile the pruned 31-layer D3D-8B and the D3D-1.7B drafters on the 222 prompts, measuring GPU stage times with CUDA events.

Draftertdt_d (ms)tvt_v (ms)Cache hit rateRecovery per miss (ms)Amortized per round (ms)
D3D-8B, 31 layers27.046.269.7%24.07.2
D3D-1.7B24.046.165.4%23.48.1

Each miss costs 24.0 ms, which amortizes to 7.2 ms per round at a 69.7% cache hit rate.

Losslessness and speedup under sampling. Under sampling, D3D follows the standard speculative sampling rule of Leviathan et al. (2023). The verifier accepts each drafted token according to the ratio of the target and drafter probabilities, and upon the first rejection resamples the token from the normalized residual distribution. This procedure provably preserves the target distribution. To verify this empirically, we ran D3D with Qwen3-8B and the 31-layer drafter on all 30 AIME25 problems, generating up to 1,024 tokens, and compared the distribution that D3D samples from with the target distribution recomputed independently on the same prefix. The empirical total variation (TV) and Jensen–Shannon (JS) discrepancies are summarized below.

PrecisionTemperature / top-pMean TVMax TVMean JS (nats)
BF161 / 14.9×10−44.9\times10^{-4}1.9×10−21.9\times10^{-2}1.49×10−51.49\times10^{-5}
BF160.6 / 0.958.1×10−48.1\times10^{-4}1.3×10−21.3\times10^{-2}4.84×10−54.84\times10^{-5}
FP321 / 13.06×10−63.06\times10^{-6}2.08×10−52.08\times10^{-5}1.89×10−111.89\times10^{-11}
FP320.6 / 0.952.97×10−62.97\times10^{-6}5.17×10−55.17\times10^{-5}3.46×10−113.46\times10^{-11}

In FP32, the maximum TV distance between the two distributions is below 5.2×10−55.2\times10^{-5}. The larger BF16 deviations are consistent with rounding error in the logits and probability computations, and the much smaller FP32 discrepancies support the correctness of the sampling implementation. With Qwen3-8B and the 31-layer drafter, running asynchronously on two H800 GPUs, D3D reaches an end-to-end speedup, including prefill, of 2.92$\times$ at T=1 and 3.71$\times$ at T=0.6 with top-p 0.95, with cache hit rates of 59.7% and 68.6%, and every benchmark improves in both settings.

Batched serving and backend. Cache misses do not occur in almost every round under batching. At concurrency 4, the cache-hit rate is 79.2% on the 222 prompts. We implemented D3D on an SSD-derived PyTorch serving backend with paged KV caching, in addition to the Transformers backend used in the paper. Cache hits and misses are handled per request, so a miss triggers a redraft only for that request, while the other requests keep their cached drafts. Against native AR decoding on the same backend, D3D is 3.73$\times$ faster at concurrency 1 and 2.03$\times$ faster at concurrency 4. At higher concurrency, decoding becomes compute-bound and the benefit of speculative decoding shrinks for any drafter. As in vLLM, which disables speculative decoding when the number of queued requests exceeds a threshold, a simple dispatcher then switches to AR decoding, so D3D matches native AR at concurrency 8 and 16.

Reference

Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274–19286. PMLR, 2023.

Reviewer_D6wqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes D3D, which accelerate speculative decoding via async drafting and verification. The key idea is to train a diffusion drafter that is NOT dependent on target hidden states. Further, authors use acceptance-guided layer pruning to further make the draft faster. Overall the promising results are demonstrated on 2 Qwen-3 models with rich ablation studies provided.

Critical strengths

The overall system demonstrates promising improvements against strong baselines.

Critical weaknesses
  1. Related works are not discussed/compared sufficiently.
  2. Main claim might be questionable due to unfair hardware usage: D3D uses 2 GPUs while some key baselines use only 1.
Questions for authors
  1. Can you provide further ablation on the contribution of async by running the method in a synced manner?
  2. Can you provide discussion and comparison to the related work DPara [1]? I think it share the similar async idea and should be at least discussed, if not empirically compared. Making your work positioning properly is important.

[1] Liu et al., When Parallel Drafter Meets Parallel Speculative Decoding, arXiv:2609.27396

4 comments
AuthorsOpenReview ↗

Response to Reviewer D6wq

We sincerely thank the reviewer for the careful review and for bringing DPara to our attention. Below we address each of the raised concerns:

W1: Related work and comparison with DPara

Related works are not discussed/compared sufficiently.

DPara appeared on arXiv in late September 2026, close to the submission deadline, so we regard it as concurrent work, and we are glad to discuss it in the related-work section.

DPara and D3D both overlap drafting with verification, but differ in how the drafter works. DPara conditions its diffusion backbone on cached target features, precomputes backbone outputs for every possible acceptance length during verification, and then completes the draft with a lightweight AR head once the bonus token is known. D3D uses no target hidden states and prepares complete drafts for the most likely outcomes in one packed pass. A matching draft is used directly, and a miss requires a new draft. DPara thus avoids misses but depends on target features, whereas D3D removes this dependency.


W2: Fairness of the two-GPU comparison

Main claim might be questionable due to unfair hardware usage: D3D uses 2 GPUs while some key baselines use only 1.

Under the same two-GPU budget, D3D remains the fastest method. Table 1 in the main paper reports the speedup of a single request at batch size 1. DFlash, DSpark, and EAGLE-3 cannot use a second GPU to reduce this latency, because their drafters need the target's hidden states from the current round and therefore cannot draft during verification. The other way to use a second GPU is tensor parallelism for the target, but at batch size 1 it makes the target slower because of communication overhead, so it is unlikely to speed up these baselines either. On the same two H800 GPUs, D3D is 6.25$\times$ faster than target-only TP=2 on Qwen3-8B and 7.87$\times$ faster on Qwen3-32B. Running one replica per GPU raises aggregate throughput but leaves per-request latency unchanged.

PARD-2 and SSD use the same two GPUs as D3D, drafting asynchronously on the second GPU, and D3D outperforms both.

MethodQwen3-8BQwen3-32B
SSD1.84$\times$3.54$\times$
PARD-24.08$\times$3.88$\times$
D3D5.00$\times$5.71$\times$

Q1: Contribution of asynchronous execution

Can you provide further ablation on the contribution of async by running the method in a synced manner?

Asynchronous execution is faster on every benchmark. We run the same pruned D3D-8B checkpoint synchronously and asynchronously on the same two H800 GPUs. using the 750 prompts of Table 1.

TargetAsync / sync throughputMean E2E latency, sync → async (s)
Qwen3-8B+33.3%3.93 → 2.96
Qwen3-32B+22.5%5.55 → 4.53

Asynchronous execution increases average throught by 33.3% on Qwen3-8B and 22.5% on Qwen3-32B, and reduces mean end-to-end latency by 24.7% and 18.4%.


Q2: Discussion and comparison with DPara

Can you provide discussion and comparison to the related work DPara [1]? I think it share the similar async idea and should be at least discussed, if not empirically compared. Making your work positioning properly is important.

[1] Liu et al., When Parallel Drafter Meets Parallel Speculative Decoding, arXiv:2609.27396

We have added this discussion in our response to W1. In brief, DPara is concurrent work, and the two methods differ in what the drafter depends on during verification: DPara relies on cached target features, whereas D3D drafts without any target features.

Following the reviewer's suggestions, we have added the DPara discussion and the new hardware and synchronous ablations to the revised manuscript. They have helped us position D3D more clearly, and we look forward to the reviewer's further comments.

AuthorsOpenReview ↗

Looking forward to your feedback

We hope this message finds you well. In our response, we compared D3D with target-only tensor parallelism on the same two GPUs, added an ablation that runs D3D synchronously, and discussed DPara as concurrent work. The revised manuscript reflects these changes. As the discussion period closes in less than three days, we would appreciate knowing whether these results resolve your concerns about hardware fairness and positioning. If they do, we would be grateful if you could take them into account in your final evaluation.

AuthorsOpenReview ↗

Your feedback is very important to us

Dear Reviewer,

As the discussion period ends in less than a day, we would like to check whether any questions remain about the hardware comparison, the synchronous ablation, or the positioning relative to DPara. We would be glad to provide any clarification that would help. If our responses have addressed your concerns, we would sincerely appreciate it if you would consider updating your score.

Thank you very much for your time and consideration.

Best regards,

The Authors

AuthorsOpenReview ↗

Follow-up on our response to your review

Dear Reviewer D6wq,

As the discussion period draws to a close, we wanted to follow up on our point-by-point response and the additional experimental results we shared. We would appreciate your thoughts on whether the new evidence addresses the concerns you raised. If any point still needs clarification, we would be happy to discuss it while there is still time.

Thank you again for your thoughtful review and for taking the time to consider our response.

Best regards,

The Authors

Reviewer_XNysOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

D3D introduces a standalone diffusion drafter that eliminates its reliance on fresh hidden states from the target model, enabling asynchronous drafting and verification. The method adapts a pretrained autoregressive model via blockwise denoising and auxiliary autoregressive supervision, compresses the resulting drafter through acceptance-guided progressive layer pruning, and employs an SSD-style execution scheme to prepare drafts for possible verification outcomes. Across seven benchmarks, it reports average decoding speedups of 5.00x and 5.71x for Qwen3-8B and Qwen3-32B targets, respectively, under a two-GPU setup.

Critical strengths
  1. The main evaluation spans two target sizes, seven benchmarks, and five baselines, including asynchronously executed PARD-2 and SSD. D3D achieves consistently strong acceptance lengths and average decoding speedups.

  2. The ablations cover auxiliary AR supervision, pretrained initialization, drafter size, pruning depth, asynchronous computation budget, block size, and the number of training anchors. Table 3(a) supports the benefit of AR supervision, while Table 3(b) shows that a smaller D3D drafter can outperform DFlash in acceptance length.

  3. The experiments also reveal useful practical tradeoffs. The reported pruning experiment removes five layers while retaining over 99% of the average acceptance length, and the asynchronous budget study shows that higher cache-hit rates do not necessarily yield greater speedups. Concurrency experiments and the long-context extension provide additional evidence of practical applicability.

Critical weaknesses
  1. Limited model-family coverage. Both targets, Qwen3-8B and Qwen3-32B, as well as all drafter initializations, belong to the Qwen3 family. These experiments demonstrate effectiveness across sizes within a single family, but they do not establish whether the benefits transfer to other model families. Because pretrained initialization and target-aligned training are central to the method, this limitation matters when assessing the generality of the proposed design.

  2. Drafter-target configurations and attribution of gains require stronger evaluation. The main experiments use a drafter initialized from Qwen3-8B and subsequently pruned, with Qwen3-8B itself serving as one target. Although the paper does include a larger-target setting and smaller-drafter experiments, the size ablation primarily reports acceptance length rather than systematically comparing actual speedups and resource costs. Furthermore, the reported comparisons do not sufficiently control for backbone capacity, pretrained initialization, and training budget across standalone parallel drafters. It therefore remains difficult to isolate the additional benefit of the proposed adaptation from the advantages of a large, closely related pretrained backbone.

Questions for authors
  1. To address model-family coverage, I suggest evaluating the same training procedure on another model family using a representative subset of tasks. Reporting acceptance length, decoding speedup, and an auxiliary AR-loss ablation would help establish whether the main benefits transfer beyond Qwen3.

  2. To clarify configuration tradeoffs and the attribution of gains, I suggest reporting acceptance length, end-to-end latency, and GPU memory usage for several drafter sizes with a fixed target. A representative comparison with a standalone parallel drafter such as PARD-2—using the same pretrained initialization, comparable model capacity and training budget, and, where practical, a matched asynchronous execution setup—would also help isolate the additional benefit of D3D's adaptation.

6 comments
AuthorsOpenReview ↗

Response to Reviewer XNys

We really appreciate the reviewer's constructive feedback. Below we address each of the raised concerns:

W1: Generalization beyond the Qwen3 family

Limited model-family coverage. Both targets, Qwen3-8B and Qwen3-32B, as well as all drafter initializations, belong to the Qwen3 family. These experiments demonstrate effectiveness across sizes within a single family, but they do not establish whether the benefits transfer to other model families. Because pretrained initialization and target-aligned training are central to the method, this limitation matters when assessing the generality of the proposed design.

D3D transfers to Llama. With Llama-3.1-8B-Instruct as the target and a drafter initialized from Llama-3.2-1B-Instruct, asynchronous D3D reaches an average speedup of 3.69$\times$, and the auxiliary AR loss improves acceptance length τ\tau and decoding speedup on every benchmark in both modes.

We apply the same training procedure as for Qwen3. To test the auxiliary AR loss, we train two drafters for three epochs each, with AR loss weights of 0 and 1, and evaluate them on MATH-500, HumanEval, and MT-Bench in synchronous and asynchronous modes on the same two H800 GPUs.

DatasetAR loss weightτ\tau (sync.)τ\tau (async.)Sync. speedupAsync. speedup
MATH-50004.494.493.92$\times$4.27$\times$
MATH-50014.654.644.01$\times$4.43$\times$
HumanEval04.174.163.68$\times$3.98$\times$
HumanEval14.354.353.74$\times$4.14$\times$
MT-Bench02.672.662.24$\times$2.44$\times$
MT-Bench12.752.742.28$\times$2.51$\times$

With the same checkpoints and GPUs, asynchronous execution is 10.4% faster than synchronous execution with the AR loss and 8.7% faster without it.

AuthorsOpenReview ↗

Response to Reviewer XNys

W2: Attribution of gains to the drafter backbone

Drafter-target configurations and attribution of gains require stronger evaluation. The main experiments use a drafter initialized from Qwen3-8B and subsequently pruned, with Qwen3-8B itself serving as one target. Although the paper does include a larger-target setting and smaller-drafter experiments, the size ablation primarily reports acceptance length rather than systematically comparing actual speedups and resource costs. Furthermore, the reported comparisons do not sufficiently control for backbone capacity, pretrained initialization, and training budget across standalone parallel drafters. It therefore remains difficult to isolate the additional benefit of the proposed adaptation from the advantages of a large, closely related pretrained backbone.

We would like to clarify that initializing the drafter from Qwen3-8B is not the main source of D3D's gains. The initialization provides pretrained language priors, but the drafter is retrained into a block diffusion model that predicts B−1B-1 masked tokens in a single forward pass, where BB is the block size, without any access to the target's hidden states. Moreover, the same drafter is used for Qwen3-32B, whose weights it does not share, and the speedup there is even higher, 5.71$\times$ compared with 5.00$\times$.

To isolate our training from the backbone, we compare D3D with PARD-2 under fully matched conditions. Both drafters are initialized from Qwen3-0.6B and trained with their own objectives on the same 32,768 samples for three epochs. Both are evaluated asynchronously on two H800 GPUs with Qwen3-8B as the target model, on 222 prompts consisting of the first 32 prompts of six benchmarks and all 30 prompts of AIME25.

MethodDrafter sizeE2E throughput (tokens/s)Acceptance length τ\tauMean E2E latency (s)Drafter peak memory (GiB)
PARD-20.6B107.123.976.121.79
D3D0.6B119.634.205.511.78

D3D achieves 11.7% higher decoding throughput and 5.8% longer acceptance length. As all training and evaluation conditions are matched, the gain comes from the training objective: PARD-2 is trained to run both with and without target hidden states, whereas D3D is trained only for the standalone setting of asynchronous drafting, combining block denoising LdiffL_{diff} with auxiliary AR supervision LARL_{AR}.

To measure the actual speed and cost of different drafter sizes, we fix Qwen3-8B as the target model. All three D3D drafters are trained for one epoch and run synchronously on the 750 prompts of Table 1 in the main paper, which consist of 128 prompts sampled from each of five benchmarks, together with all 80 MT-Bench and all 30 AIME25 prompts.

Drafter sizeDecoding speedupMean E2E latency (s)Acceptance length τ\tauTarget + drafter memory (GiB)
1.7B3.22$\times$4.45.8420.04
4B3.47$\times$4.16.9524.63
8B3.66$\times$3.87.4131.62

Scaling the drafter from 1.7B to 8B raises τ\tau by 27%, but slower drafting limits the improvement in decoding speedup to 14%. This suggests that D3D remains effective with smaller drafters.

AuthorsOpenReview ↗

Response to Reviewer XNys

Q1: Llama experiment with the AR-loss ablation

To address model-family coverage, I suggest evaluating the same training procedure on another model family using a representative subset of tasks. Reporting acceptance length, decoding speedup, and an auxiliary AR-loss ablation would help establish whether the main benefits transfer beyond Qwen3.

As detailed in our response to W1 above, we conducted the Llama experiment the reviewer suggested. We trained D3D for Llama-3.1-8B-Instruct and report acceptance length, decoding speedup, and the AR-loss ablation on MATH-500, HumanEval, and MT-Bench. D3D reaches an average speedup of 3.69$\times$, and the AR loss improves both acceptance length and speedup on every benchmark.


Q2: Drafter-size study and matched PARD-2 comparison

To clarify configuration tradeoffs and the attribution of gains, I suggest reporting acceptance length, end-to-end latency, and GPU memory usage for several drafter sizes with a fixed target. A representative comparison with a standalone parallel drafter such as PARD-2, using the same pretrained initialization, comparable model capacity and training budget, and, where practical, a matched asynchronous execution setup, would also help isolate the additional benefit of D3D's adaptation.

Both suggested analyses are included in our response to W2 above. It reports acceptance length, end-to-end latency, and memory for drafters from 1.7B to 8B with Qwen3-8B fixed, together with a PARD-2 comparison that uses the same initialization, model size, training data, and asynchronous runtime. Under these matched conditions, D3D achieves 11.7% higher throughput.

We have incorporated the Llama experiments, the PARD-2 comparison, and the drafter-size analysis into the revised manuscript. We hope these results resolve the reviewer's concerns about generality and gain attribution, and we would warmly welcome any further discussion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Discussion Phase

We sincerely thank the Area Chair for organizing the discussion and all three reviewers for their careful reading and constructive suggestions. Their feedback led to a substantially more complete set of experiments, all of which are included in the revised manuscript. Below we summarize the strengths the reviewers recognized and how each of their concerns is resolved.

Strengths recognized by the reviewers

All three reviewers recognize the strength of D3D's results, and all three rate their soundness concerns as "minor or fixable," which we address below.

  • Best acceptance length and speedup on every benchmark (all reviewers). Reviewer c1Bv notes that D3D "achieves the highest acceptance length on all seven benchmarks for both targets and the highest average speedup among the evaluated methods." Reviewer XNys notes "consistently strong acceptance lengths and average decoding speedups," and Reviewer D6wq notes "promising improvements against strong baselines."
  • Comprehensive evaluation and rich ablations (all reviewers). Reviewer XNys notes that the evaluation "spans two target sizes, seven benchmarks, and five baselines, including asynchronously executed PARD-2 and SSD," that "Table 3(a) supports the benefit of AR supervision," and that a smaller D3D drafter "can outperform DFlash in acceptance length." Reviewer D6wq notes "rich ablation studies," and Reviewer c1Bv lists "thorough ablations and reproducibility" among the paper's strengths.
  • A clean and natural design (Reviewer c1Bv). Removing the dependency on target hidden states "is a clean way to enable asynchronous drafting," and one-pass diffusion drafting "fits naturally with SSD, whose AR drafters are limited by sequential drafting steps."
  • Practical insight and applicability (Reviewer XNys). The experiments "reveal useful practical tradeoffs": pruning retains "over 99% of the average acceptance length," the budget study shows that "higher cache-hit rates do not necessarily yield greater speedups," and the concurrency and long-context experiments "provide additional evidence of practical applicability."

Every reviewer concern has been addressed

  1. Fairness of the two-GPU comparison (Reviewers D6wq and c1Bv). On the same two H800 GPUs, D3D is 6.25$\times$ and 7.87$\times$ faster than the target model with tensor parallelism on Qwen3-8B and Qwen3-32B, and it outperforms PARD-2 and SSD, which also use two GPUs. Synchronous baselines such as DFlash cannot use the second GPU for drafting, as they need the target's current hidden states.

  2. Sources of the gains (all reviewers). To separate asynchronous execution, training, and drafter scale: with the same checkpoint and GPUs, asynchronous execution increases throughput by 33.3% on Qwen3-8B and 21.0% on Qwen3-32B. In the fully matched PARD-2 comparison proposed by Reviewer XNys, D3D is 11.7% faster. In the same-size comparison proposed by Reviewer c1Bv, SSD with a Qwen3-1.7B AR drafter is 1.94$\times$ to 2.62$\times$ slower than D3D-1.7B, and D3D-1.7B, which is smaller than DFlash, also achieves longer acceptance.

  3. Generality beyond Qwen3 (Reviewer XNys). On Llama-3.1-8B-Instruct with a 1B drafter, D3D reaches an average speedup of 3.69$\times$, and the auxiliary AR loss improves acceptance length and speedup on every benchmark.

  4. Novelty and positioning (Reviewers D6wq and c1Bv). We thank Reviewer D6wq for pointing us to DPara, which relies on target features, whereas D3D drafts without them. D3D makes SSD-style asynchronous drafting effective with a standalone diffusion drafter, while SSD mentions diffusion drafters only in its related work and does not build one. Against simpler pruning, our iterative layer selection retains an acceptance length of 7.31, compared with 4.94 for one-shot ranking and 4.12 for uniform deletion.

  5. Completeness of the evaluation (Reviewer c1Bv). On Qwen3-32B, drafting takes 27.0 ms and is hidden behind 46.2 ms of verification, and cache misses add 7.2 ms per round on average. Under temperature sampling, D3D preserves the target distribution, with a maximum TV distance below 5.2×10−55.2\times10^{-5} in FP32, and reaches 2.92$\times$ to 3.71$\times$ end-to-end speedup. In batched serving, the cache-hit rate is 79.2% at concurrency 4, where D3D is 2.03$\times$ faster than native AR.

We are grateful to all three reviewers, whose suggestions have made the paper considerably stronger. We believe the revised manuscript now offers a clear and well-supported case for asynchronous speculative decoding with a standalone diffusion drafter, and we thank the Area Chair for considering our work.

Best Regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.