acceptodds
Under review as a conference paper at ICLR 2027

D2-SD: Accelerating Speculative Decoding with Dual Diffusion Draft Models

11 views

Abstract

Speculative decoding accelerates autoregressive large language model inference by drafting multiple tokens and verifying them in a single target-model forward pass. Recent diffusion-based drafters produce an entire block in parallel but usually commit to a single draft sequence per verification: once the first mismatch occurs, all subsequent draft tokens are discarded, resulting in a limited acceptance rate. Naively batching more draft candidate sequences only introduces a marginal improvement, as redundant or poorly placed branches increase the cost of drafting and verification without proportionally increasing the number of accepted tokens. We propose -SD, a dual diffusion draft speculative decoding framework that organizes candidates into a confidence-guided prefix tree, where the first diffusion drafter generates a block together with per-position confidence scores that are used to identify the most likely rejection boundary and select the top- prefix ranges for recovery; the second variable-prefix diffusion drafter re-anchors at each selected prefix and proposes alternative continuations in one batched pass; the resulting shared-prefix candidates are jointly verified via cascade attention. Empirically, -SD shows clear improvements over both the underlying diffusion approach and strong autoregressive speculative decoding baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Speculative Decoding, Machine Learning, ML System

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_HrZcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

D2-SD uses one diffusion drafter to predict likely rejection points. A second drafter proposes new continuations from those points. The target verifies the branches together. Tests show speedups over DFlash.

Critical strengths
  • Drafting again near likely rejection points makes sense. The ablations test both branch placement and second-drafter training.
  • The paper measures runtime and stage costs, including where extra branches stop paying off.
Critical weaknesses
  • Compared with DFlash plus four extra samples, the gain is only about 6% (5.35x versus 5.06x in Table 3). This needs a second trained drafter and extra memory.
  • The concurrency test uses independent streams, not continuous batching. Throughput falls sharply with more requests. It is unclear whether the gains carry over to a batched serving engine.
Questions for authors
  1. How much extra memory does the second drafter need compared with the resampling baseline?
  2. How much of the concurrency slowdown comes from using independent streams? What changes with batched verification?
3 comments

Thank you for the positive assessment and the focused questions. All new numbers below were measured on B300 with Qwen3-8B on MATH-500 (128 requests, γ=16, T=0), using the Table 4 protocol.

W1. Gain over DFlash + 4 samples.

The core contribution shows up in acceptance, and there the gap is large:

DFlash+4 samplesD²SD K=4
α (Table 3 average)5.756.087.62
α (B300, MATH-500)6.487.138.48

Under the same branching budget, D²SD recovers 3–6× more accepted tokens per cycle than resampling. Resampled branches share a single per-position distribution and collapse onto the same tokens, whereas the VP-Drafter re-anchors at likely failure points with a different inductive bias. Ablation 2 (DF→DF) confirms that the gain comes from the variable-prefix training rather than from simply adding a second forward pass.

This acceptance advantage carries over to wall-clock time and widens as the fixed per-cycle costs are amortized. Under batched verification (Table R2), D²SD reaches 1.19–1.20× over DFlash for B ≤ 4, whereas resampling gives 1.05–1.15×.

Q1. Extra memory of the second drafter relative to the resampling baseline.

ComponentExtra memory
VP-Drafter weights1.95 GiB
VP-Drafter KV cache20 KiB per token per request (5 layers × 8 KV heads × 128 dims), about one seventh of the target's KV
Measured peak, B=1+1.99 GiB (19.49 vs. 17.50 GiB)
Measured peak, B=16+2.52 GiB (25.70 vs. 23.18 GiB)

On a 275 GiB device this is below 1% of HBM. The resampling baseline has the same verification activations at equal K.

Q2. Independent streams vs. batched verification.

Most of the slowdown comes from the independent-stream protocol itself, and it affects DFlash equally.

  • DFlash's own aggregate per-GPU throughput falls from 178 to 66 tok/s from B=1 to B=16 (Table 4).
  • The GPU's SM-active fraction drops from 44–56% at B=1 to 18–22% at B=8/16, because concurrent Python threads contend to launch per-request kernels. The GPU sits idle, so the measured decline does not reflect the method's compute cost.

We re-ran the same kernels with all B requests batched into a single draft and verification forward, keeping a shared prefix per request via ragged cascade attention (Table R2). For 8 requests, the DFlash cycle drops from 573 ms (independent streams) to 19.5 ms (batched). With batched verification, D²SD delivers 1.20× / 1.20× / 1.19× / 1.12× over DFlash at B = 1 / 2 / 4 / 8, with K chosen per load by Eq. (8).

The mechanism is explained in Table R3. As long as the verification width B·(K+1)·γ stays below the GPU's ridge point (≈284 tokens on B300), extra branches share the target's weight reads and are nearly free. The load-adaptive K keeps D²SD in that regime as concurrency grows. We will replace the independent-stream sweep in Section 4.5 with these batched results.


Tables referenced above

Table R2. Independent streams vs. batched verification (same eager kernels as our released implementation; context 224; per-cycle cost for the whole batch of B requests; D²SD speedup over DFlash = α/cycle ratio with load-adaptive K).

B=1B=2B=4B=8
DFlash cycle, independent streams (ms)19.266.0242.8572.6
DFlash cycle, batched (ms)19.319.319.419.5
D²SD batched, ×DFlash (selected K)1.20 (K=8)1.20 (K=8)1.19 (K=4)1.12 (K=2)

Table R3. Roofline of the verification GEMMs (one Qwen3-8B layer: QKV+O+gate/up+down on B300; measured HBM 6.53 TB/s, dense BF16 1.85 PFLOP/s, ridge ≈ 284 FLOP/B, i.e., ≈ 284 verified tokens per forward).

Tokens per target forward16801603206401280
GEMM time per layer (ms)0.0740.0800.0920.1060.1680.302
Relative to 16 tokens1.001.091.241.442.284.09
Achieved HBM (TB/s)5.224.814.223.642.291.28

Thanks for the memory breakdown and batching experiments. These address much of my concern, though performance in a full serving setup remains an open question. I’ll keep my weak accept recommendation.

Thank you for raising this point. Evaluating performance in a full serving setup remains an important direction for future work.

Reviewer_6dyhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

D²-SD proposes a dual-diffusion-draft speculative decoding framework. The first DFlash-style diffusion drafter generates an entire block of tokens in parallel with per-position confidence scores. These confidences identify the most likely rejection boundaries and select the top-K prefix lengths. A second, specially trained Variable-Prefix Drafter (VP-Drafter) then re-anchors at each selected prefix and proposes alternative continuations in one batched pass. The resulting K+1 shared-prefix candidates are jointly verified via cascade attention. Compared to single-draft DFlash, this method significantly improves both acceptance length and end-to-end decoding speed

Critical strengths
  1. Unlike naive resampling or blindly increasing block size, the authors observe that resampled branches suffer from severe "error homogeneity"—since they share the same per-position categorical distribution, they collapse onto the same high-confidence paths and collectively fail at the same difficult decision boundaries. The solution introduces an independently trained VP-Drafter to provide a distinct inductive bias. Meanwhile, the first drafter's own confidence scores are used to predict rejection boundaries without requiring an additional model. This design is well-validated: with the same K=4 branching budget, naive resampling improves α by only 0.37, whereas D² SD improves it by 2.25.

  2. The paper evaluates on 8 benchmarks (math, code, chat), two target models (Qwen3-8B, GPT-OSS-20B), both decoding modes (greedy T=0, sampling T=1), and includes concurrency throughput experiments. The sampling-mode acceptance length comparison is indeed presented: on Qwen3-8B, average α improves from 4.33 to 5.45; on GPT-OSS-20B, from 1.99 to 2.35. Ablation studies are progressive—resampling baseline → DFlash-reuse baseline → third-level cascade baseline—carefully isolating each component's contribution.

Critical weaknesses
  1. Doubled training cost with no discussion of cross-target-model reusability D²-SD requires independently training two diffusion draft models, doubling the training cost. This cost is not trivial in an industrial setting with a large-scale MoE target model and long-context samples.

  2. The largest model tested is GPT-OSS-20B; whether the paper's mechanism transfers directly to large-scale MoE remains an open question. Additionally, for trillion-parameter-scale models, the draft model scale, training cost, and integration with pipeline/tensor parallelism are unaddressed.

  3. Although a concurrency experiment is provided (Table 4, MATH-500), it is tested on a single task, and the advantage decays rapidly with increasing concurrency (1.28× → 1.14×). The authors attribute this to "rising verification time share" (57.6% → 67.8%), but no hardware-level modeling (HBM bandwidth, compute intensity, SM occupancy, warp scheduler behavior) is established to explain this trend.

Questions for authors

See Weakness.

1 comment

Thank you for the positive and detailed assessment.

W1. Training cost and cross-target reusability.

  • Cost. The VP-Drafter has the same size and recipe as DFlash: ≈439 EFLOPs of dense-equivalent drafter compute, or ≈603 EFLOPs counting the context (Appendix A).
    • Both drafters are conditioned on the same frozen target features, so they can be trained in a single data pass that shares one teacher forward (2N_t·T ≈ 428 EFLOPs). The total is then ≈1.63 ZFLOPs versus ≈1.03 ZFLOPs for DFlash, i.e., about 1.6×, not 2×.
    • The cost is set by the drafter size. For MoE targets, the teacher forward scales only with the active parameters, so the overhead does not grow with the target's total parameter count.
  • Reusability. The VP-Drafter does not depend on the first drafter's weights, so one VP-Drafter serves any first drafter for the same target. For GPT-OSS-20B we paired our VP-Drafter directly with the public third-party DFlash checkpoint, with no joint training.
  • Across targets. The architecture is identical to DFlash's, so an existing DFlash checkpoint of a new target is a ready initialization for its VP-Drafter.

W2. Large-scale MoE targets.

We applied D²SD unchanged to the large-scale open-source MoE model minimax m3.1. Both drafters have 5 layers, block size γ=7, and are trained on the full open-PerfectBlend data. The table reports the mean acceptance length α.

BenchmarkTFirst drafter onlyD²SDΔα
BBH05.716.00+0.29 (+5.1%)
MGSM04.334.70+0.37 (+8.5%)
MMLU03.693.89+0.20 (+5.4%)
MMLU-Pro03.914.11+0.20 (+5.1%)
Winogrande03.363.51+0.15 (+4.5%)
GSM8K05.375.57+0.20 (+3.7%)
AIME 24/2514.995.23+0.24 (+4.8%)
MBPP15.195.44+0.25 (+4.8%)
LiveCodeBench (test6, Python, 2-shot)14.574.82+0.25 (+5.5%)
Average4.574.81+0.24 (+5.3%)

The mechanism transfers directly. α improves on all nine benchmarks, spanning reasoning, knowledge, math and code, under both greedy and sampled decoding. With the short γ=7 block, at most 7 − α ≈ 1.3–3.6 tokens per cycle are recoverable, and D²SD recovers up to 22% of that headroom without any change to the method or its hyperparameters.

W3. Hardware-level modeling of the concurrency trend.

Roofline. We measured the B300 directly (Table R3): HBM 6.53 TB/s, dense BF16 1.85 PFLOP/s, ridge ≈ 284 FLOP/B. A Qwen3-8B verification forward therefore stays memory-bound up to ≈284 tokens. Its layer GEMMs take only 9% longer at 80 tokens than at 16, because the extra branches reuse the same HBM weight reads. The cycle-level model is

speedup(D²SD)/speedup(DFlash) = (α_D² / α_DF) · [T_d1 + T_v(γB)] / [T_d1 + T_d2 + T_v((K+1)γB)],

where T_v(N) is flat for N below the ridge. In that regime D²SD's only overhead is the small second-draft pass T_d2, and the α gain dominates. Beyond the ridge, the model directly gives the K that maximizes throughput at a given load. We use this as a load-adaptive K rule.

The trend in Table 4. The rising verification share in Table 4 reflects host-side contention in the independent-stream protocol, not HBM saturation:

  • DFlash's own cycle time grows 57× from B=1 to B=16, far more than the 16× a saturated GPU would imply.
  • The GPU's SM-active fraction drops from 44–56% at B=1 to 18–22% at B=8/16.
  • Batching the same kernels removes the growth (Table R2: 19.3 ms for 1 request, 19.5 ms for 8).

With batched verification and the adaptive K, D²SD delivers 1.20× / 1.20× / 1.19× / 1.12× over DFlash at B = 1 / 2 / 4 / 8. We will add this roofline-based analysis to Section 4.5.


Tables referenced above

Table R2. Independent streams vs. batched verification (same eager kernels as our released implementation; context 224; per-cycle cost for the whole batch of B requests; D²SD speedup over DFlash = α/cycle ratio with load-adaptive K).

B=1B=2B=4B=8
DFlash cycle, independent streams (ms)19.266.0242.8572.6
DFlash cycle, batched (ms)19.319.319.419.5
D²SD batched, ×DFlash (selected K)1.20 (K=8)1.20 (K=8)1.19 (K=4)1.12 (K=2)

Table R3. Roofline of the verification GEMMs (one Qwen3-8B layer: QKV+O+gate/up+down on B300; measured HBM 6.53 TB/s, dense BF16 1.85 PFLOP/s, ridge ≈ 284 FLOP/B, i.e., ≈ 284 verified tokens per forward).

Tokens per target forward16801603206401280
GEMM time per layer (ms)0.0740.0800.0920.1060.1680.302
Relative to 16 tokens1.001.091.241.442.284.09
Achieved HBM (TB/s)5.224.814.223.642.291.28
Reviewer_KoKuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a dual-diffusion drafting framework for accelerating speculative decoding. The method addresses the limited acceptance length of single-sequence diffusion drafting by constructing confidence-guided, shared-prefix candidate branches. Specifically, a first drafter generates a token block and estimates likely rejection boundaries, while a separately trained variable-prefix drafter generates alternative continuations from selected prefixes. These candidates are then jointly verified using cascade attention. Experiments on Qwen3-8B and GPT-OSS-20B across eight benchmarks demonstrate improvements in acceptance length and end-to-end decoding speedup over DFlash and EAGLE-3.

Critical strengths
  1. Clear motivation and practical relevance. The paper addresses an important limitation of diffusion-based speculative decoding, where early token rejection causes subsequent draft tokens to be discarded. Confidence-guided branching offers an intuitive way to allocate additional computation to potentially valuable recovery positions.

  2. Effective integration of parallel drafting and tree-based verification. D²SD combines parallel diffusion drafting, variable-prefix continuation generation, and cascade attention into a coherent framework. The separately trained VP-Drafter enables batched generation of alternative suffixes without introducing sequential autoregressive drafting.

  3. Consistent empirical improvements and useful ablations. The method improves both acceptance length and wall-clock speedup across two target models and multiple benchmarks.

Critical weaknesses
  1. The second drafter is not explicitly trained to complement the first drafter. Its objective is still standard cross-entropy, without encouraging alternative continuations that cover the first drafter's failures. Why not directly optimize the additional acceptance gained by the second drafter?

  2. The rejection-boundary estimation is not sufficiently reliable. The method uses drafter confidence as a proxy for target acceptance, but these are not necessarily correlated. The reported top-1 prediction accuracy is only 36.1% on GSM8K and 37.8% on MATH-500. How does inaccurate boundary prediction affect the effectiveness of the method?

  3. The additional computational cost is substantial. The method requires two separately trained diffusion drafters and an additional drafting pass. Moreover, the throughput improvement over DFlash drops from 28% to 14% as concurrency increases from 1 to 16. It is unclear whether maintaining two models is worthwhile under realistic serving workloads.

Questions for authors

See weaknessnes

4 comments

Thank you for the thoughtful review. New measurements below were taken on B300 with Qwen3-8B, γ=16, K=4 and greedy decoding, using 128 prompts each from MATH-500 and GSM8K.

W1. Why not train the VP-Drafter to directly maximize the acceptance added on top of the first drafter?

The quantity to optimize, namely where the first drafter actually fails, is not present in the training data. It would have to be generated online by putting the first drafter, and for the true objective also the target, inside the VP-Drafter's training loop. We decided against this because of the overhead:

  1. First drafter in the loop. Locating DFlash's real rejection boundary for every training anchor block requires a frozen DFlash forward pass. Using the accounting of Appendix A, this adds ≈146 EFLOPs (≈201 EFLOPs with context) to the VP-Drafter's own 439 (603) EFLOPs, i.e., +33% drafter compute, plus 1.95 GiB of extra weights and activations per rank.
  2. Target in the loop for the actual objective. "Additional acceptance" is only defined after the VP-Drafter's own proposals are verified. Optimizing it faithfully requires on-policy VP rollouts verified by the target. Re-running the 8B target over the drafted blocks costs ≈1,142 EFLOPs per epoch, about 1.9× the VP-Drafter's training compute. The total VP training pipeline would grow by roughly 2.3×.
  3. Non-differentiable, serial, and coupled. The objective E[max_b L_b] − L_DFlash is a discrete longest-prefix match. Optimizing it needs RL or a surrogate, which gives up the dense, fully parallel cross-entropy training over 512 anchors per sample. Training would also become serial, since DFlash must finish before VP training starts, and each VP-Drafter would be tied to a specific first-drafter checkpoint. With independent training, the two runs proceed in parallel, and our VP-Drafter could be paired directly with the public third-party DFlash checkpoint for GPT-OSS-20B.

Instead, D²SD obtains complementarity at inference time. The VP-Drafter is invoked only at the prefixes where DFlash's own confidence predicts failure, it is trained for every prefix length, and its loss emphasizes the positions right after the re-anchor point (Eq. 7). This works well in practice. At the same confidence-selected boundaries, the VP-Drafter rescues 57.6% / 52.7% of the rejection-hit cycles on MATH-500 / GSM8K. Reusing DFlash itself as the second drafter rescues only 28.1% / 29.9%, and the gain per rescued cycle on MATH-500 is +2.10 vs. +0.76 tokens.

Outlook. Failure-aware training of the second drafter is a natural next step, and we will discuss it in the paper. Cheaper routes than full on-policy optimization, in increasing order of cost:

  • replace the geometric prior in Eq. (6) with DFlash's empirical first-mismatch distribution, computed once offline;
  • re-weight the VP loss toward positions where a frozen DFlash is wrong, which needs one extra forward and no target in the loop;
  • use a differentiable expected-acceptance surrogate;
  • fine-tune with acceptance signals from logged serving traces, where rejection boundaries are observed for free.

W2. How does inaccurate boundary prediction affect the method?

Because D²SD launches K=4 branches, the relevant figure is the top-4 hit rate: 77.5% on GSM8K and 80.5% on MATH-500 (Table 11). Misses also cost little. We measured this in two ways.

(a) Gain on boundary hits vs. misses.

Cycles (share of all cycles)MATH-500: share / gainGSM8K: share / gain
Boundary hit72.9% / +2.10 tokens66.9% / +2.06 tokens
Boundary miss18.5% / +1.66 tokens19.5% / +1.48 tokens

A branch anchored at an earlier prefix regenerates through the true boundary, so missed cycles still keep 72–79% of the hit-case gain. This robustness is the reason we keep top-K prefixes rather than only the arg-max.

(b) Oracle upper bound. We replaced the top-1 choice with the true rejection boundary.

Confidence-based selectionOracle boundaryChange
α, MATH-5008.488.61+1.5%
α, GSM8K9.139.22+1.0%

Our training-free, confidence-based selection therefore achieves about 94% of the gain obtainable with perfect boundary knowledge.

W3. Computational cost and behavior under concurrency.

  • Cost. The VP-Drafter adds one small forward per cycle (4.7 ms) and 1.95 GiB of weights. This is offset in verification: D²SD's cascade verifier checks K+1=5 branches in 19.5 ms, versus 28.4 ms for DFlash's single-branch verification, and it stays memory-bound below the GPU's ridge point (≈284 tokens on B300). A D²SD cycle thus costs within a few percent of a DFlash cycle (33.5 vs. 33.9 ms), while α is 31% higher. The result is 1.33× throughput over DFlash at B=1 and 1.14× at B=4 (1.18× with K=8).
  • Concurrency. The decline in Table 4 is a property of the independent-stream evaluation protocol, not of the method:
    • DFlash's own aggregate per-GPU throughput falls from 178 to 66 tok/s from B=1 to B=16 (Table 4), and SM-active time drops from 44–56% to 18–22%.
    • Batching the same kernels keeps the DFlash cycle at ≈19 ms for 8 requests, versus 573 ms with independent streams.
    • With batched verification and K chosen per load through Eq. (8), D²SD delivers 1.20× / 1.20× / 1.19× / 1.12× over DFlash at B = 1 / 2 / 4 / 8.
  • Is a second model worth maintaining? It adds under 1% of HBM, and it is trained independently and can be reused with any first drafter for the same target. We will add these batched serving results to Section 4.5.

Thank you for the detailed clarifications and additional experiments. The new results, particularly the boundary prediction analysis and practical efficiency measurements, have addressed my main concerns. I appreciate the authors' efforts and have updated my rating to 3.

Thank you. We will incorporate the additional boundary-prediction analysis and efficiency measurements later.

Reviewer_fih5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes D²SD, a speculative decoding framework that uses a second diffusion drafter to regenerate suffixes from confidence-selected prefixes of an initial draft. The resulting branches are jointly verified by the target model. Experiments on Qwen3-8B and GPT-OSS-20B show speedups over DFlash and EAGLE-3.

Critical strengths

The paper addresses a practical limitation of single-branch diffusion drafting: an early mismatch discards the remaining draft. Confidence-guided branching and a variable-prefix drafter provide a useful, if incremental, approach. The experiments report wall-clock speed and targeted ablations, showing that the method improves speed in many evaluated settings.

Critical weaknesses

The contribution is incremental, and the benefit is not uniform across settings. For example, on GPT-OSS-20B at T=1, D²SD is slower than DFlash on LiveCodeBench (1.62× versus 1.68× speedup over autoregressive decoding) and MT-Bench (1.59× versus 1.60×). The second drafter adds computation and memory. The paper reports an analytical training-compute estimate and some per-stage timing, but a clear matched breakdown of inference compute, peak memory, and latency would make the deployment trade-off easier to assess. These concerns temper the significance of the otherwise useful gains.

Questions for authors
  1. Could the authors report peak GPU memory and a matched breakdown of per-cycle drafting and verification time, including the overhead of the second drafter, relative to DFlash? How do these costs change with the number of branches and concurrency?
  2. What explains the regressions relative to DFlash on GPT-OSS-20B at T=1 for LiveCodeBench and MT-Bench? Are these differences stable across repeated runs, and which stage accounts for them?
1 comment

Thank you for the positive assessment and the concrete requests. We ran all requested measurements on a B300 SXM6 GPU, using Qwen3-8B on MATH-500 (128 requests, γ=16, T=0) and the Table 4 protocol.

Q1. Peak memory and a matched per-cycle breakdown, including how costs change with K and concurrency.

Table R1. Per-cycle breakdown and peak memory (one B300 SXM6, Qwen3-8B, MATH-500: 128 requests, ≤256 tokens, γ=16, T=0, Table 4 protocol; CUDA-event ms per request-cycle).

MethodαDraft-1Draft-2VerifyOtherCycletok/s B=1 (×DFlash)tok/s B=4 (×DFlash)Weights (GiB)Peak B=1Peak B=16
DFlash6.485.11–28.400.3533.86171.4 (1.00)78.3 (1.00)17.2117.4421.98
D²SD K=28.084.964.5119.534.1233.13219.2 (1.28)85.8 (1.10)19.1619.4624.58
D²SD K=48.484.964.6619.534.3433.50227.2 (1.33)89.3 (1.14)19.1619.4925.70
D²SD K=88.625.365.3021.464.5736.68211.4 (1.23)92.7 (1.18)19.1619.6026.96
  • Where the time goes. The second drafter costs 4.5–5.3 ms per cycle, the same order as the first drafter. This cost is offset in verification. D²SD's cascade verifier checks K+1=5 branches in 19.5 ms, less than DFlash's single-branch verification (28.4 ms), because the shared prefix KV is attended once and never expanded or copied per branch. Verification also barely grows with K (21.5 ms for 9 branches): 80–144 verified tokens are well below the B300 ridge point of ≈284 tokens (Table R3), so the target forward stays memory-bound and extra branches reuse the same weight reads. A D²SD cycle therefore costs within a few percent of a DFlash cycle (33.5 vs. 33.9 ms here; 34.0 vs. 31.9 ms on GSM8K in Table 14), while α is 31% higher. Consistent with Table 4, this gives 1.33× throughput over DFlash at B=1 and 1.14× at B=4, with K=8 reaching 1.18× at B=4.
  • Memory.
    • VP-Drafter weights: 1.95 GiB.
    • VP KV cache: 20 KiB per token per request (5 layers × 8 KV heads × 128 dims), about one seventh of the target's 144 KiB.
    • Measured peak, relative to DFlash: +2.0 GiB at B=1 and +3.7 GiB at B=16. On a 275 GiB device this is below 1.5% of HBM.
  • Concurrency. Under the paper's independent-stream protocol, the throughput decline with B is shared by all methods; DFlash's own aggregate per-GPU throughput falls from 178 to 66 tok/s (Table 4). The protocol is host-bound: batching the same kernels keeps the DFlash cycle at ≈19 ms for up to 8 requests, versus 573 ms with independent streams (Table R2). With batched verification and K selected per load through Eq. (8), D²SD delivers 1.20× / 1.20× / 1.19× / 1.12× over DFlash at B = 1 / 2 / 4 / 8.

Table R2. Independent streams vs. batched verification (same eager kernels as our released implementation; context 224; per-cycle cost for the whole batch of B requests; D²SD speedup over DFlash = α/cycle ratio with load-adaptive K).

B=1B=2B=4B=8
DFlash cycle, independent streams (ms)19.266.0242.8572.6
DFlash cycle, batched (ms)19.319.319.419.5
D²SD batched, ×DFlash (selected K)1.20 (K=8)1.20 (K=8)1.19 (K=4)1.12 (K=2)

Table R3. Roofline of the verification GEMMs (one Qwen3-8B layer: QKV+O+gate/up+down on B300; measured HBM 6.53 TB/s, dense BF16 1.85 PFLOP/s, ridge ≈ 284 FLOP/B, i.e., ≈ 284 verified tokens per forward).

Tokens per target forward16801603206401280
GEMM time per layer (ms)0.0740.0800.0920.1060.1680.302
Relative to 16 tokens1.001.091.241.442.284.09
Achieved HBM (TB/s)5.224.814.223.642.291.28

Q2. GPT-OSS-20B at T=1 on LiveCodeBench and MT-Bench.

  • Which stage, and why. These two entries follow directly from the break-even condition of Eq. (8), α_D²/α_DF > C_D²/C_DF. At T=1 the public GPT-OSS DFlash checkpoint accepts only α≈2 tokens per cycle. The α gain is therefore +10.2% on LiveCodeBench and +13.3% on MT-Bench, roughly equal to the extra cost of the second-draft forward. In our per-stage measurement on GPT-OSS-20B the verification time is unchanged, so this extra forward is the only additional cost. Whenever the acceptance headroom is larger, the α gain dominates: every T=0 task improves, with α ratios of 1.14–3.04 and an average speedup of 6.15× vs. 3.53×.
  • Stability. Both entries are within run-to-run noise. Repeating DFlash on GPT-OSS-20B at T=1 with three sampling seeds gives speedups of 1.334 / 1.438 / 1.530 on LiveCodeBench (std 0.098) and 1.404 / 1.430 / 1.391 on MT-Bench (std 0.020). These spreads exceed the reported gaps of 0.06× and 0.01×. By contrast, D²SD's gains are stable across seeds where the headroom is large, e.g., Qwen3-8B on LiveCodeBench at T=1: 4.91 ± 0.02× vs. 3.98 ± 0.05×.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.