- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
D2-SD uses one diffusion drafter to predict likely rejection points. A second drafter proposes new continuations from those points. The target verifies the branches together. Tests show speedups over DFlash.
Critical strengths
- Drafting again near likely rejection points makes sense. The ablations test both branch placement and second-drafter training.
- The paper measures runtime and stage costs, including where extra branches stop paying off.
Critical weaknesses
- Compared with DFlash plus four extra samples, the gain is only about 6% (5.35x versus 5.06x in Table 3). This needs a second trained drafter and extra memory.
- The concurrency test uses independent streams, not continuous batching. Throughput falls sharply with more requests. It is unclear whether the gains carry over to a batched serving engine.
Questions for authors
- How much extra memory does the second drafter need compared with the resampling baseline?
- How much of the concurrency slowdown comes from using independent streams? What changes with batched verification?
3 comments
Thanks for the memory breakdown and batching experiments. These address much of my concern, though performance in a full serving setup remains an open question. I’ll keep my weak accept recommendation.
Thank you
Thank you for raising this point. Evaluating performance in a full serving setup remains an important direction for future work.
Thank you for the positive assessment and the focused questions. All new numbers below were measured on B300 with Qwen3-8B on MATH-500 (128 requests, γ=16, T=0), using the Table 4 protocol.
W1. Gain over DFlash + 4 samples.
The core contribution shows up in acceptance, and there the gap is large:
Under the same branching budget, D²SD recovers 3–6× more accepted tokens per cycle than resampling. Resampled branches share a single per-position distribution and collapse onto the same tokens, whereas the VP-Drafter re-anchors at likely failure points with a different inductive bias. Ablation 2 (DF→DF) confirms that the gain comes from the variable-prefix training rather than from simply adding a second forward pass.
This acceptance advantage carries over to wall-clock time and widens as the fixed per-cycle costs are amortized. Under batched verification (Table R2), D²SD reaches 1.19–1.20× over DFlash for B ≤ 4, whereas resampling gives 1.05–1.15×.
Q1. Extra memory of the second drafter relative to the resampling baseline.
On a 275 GiB device this is below 1% of HBM. The resampling baseline has the same verification activations at equal K.
Q2. Independent streams vs. batched verification.
Most of the slowdown comes from the independent-stream protocol itself, and it affects DFlash equally.
We re-ran the same kernels with all B requests batched into a single draft and verification forward, keeping a shared prefix per request via ragged cascade attention (Table R2). For 8 requests, the DFlash cycle drops from 573 ms (independent streams) to 19.5 ms (batched). With batched verification, D²SD delivers 1.20× / 1.20× / 1.19× / 1.12× over DFlash at B = 1 / 2 / 4 / 8, with K chosen per load by Eq. (8).
The mechanism is explained in Table R3. As long as the verification width B·(K+1)·γ stays below the GPU's ridge point (≈284 tokens on B300), extra branches share the target's weight reads and are nearly free. The load-adaptive K keeps D²SD in that regime as concurrency grows. We will replace the independent-stream sweep in Section 4.5 with these batched results.
Tables referenced above
Table R2. Independent streams vs. batched verification (same eager kernels as our released implementation; context 224; per-cycle cost for the whole batch of B requests; D²SD speedup over DFlash = α/cycle ratio with load-adaptive K).
Table R3. Roofline of the verification GEMMs (one Qwen3-8B layer: QKV+O+gate/up+down on B300; measured HBM 6.53 TB/s, dense BF16 1.85 PFLOP/s, ridge ≈ 284 FLOP/B, i.e., ≈ 284 verified tokens per forward).