acceptodds
Under review as a conference paper at ICLR 2027

When Parallel Drafter Meets Parallel Speculative Decoding

12 views

Abstract

DSpark-style parallel drafters have made speculative decoding highly effective, yet their draft phase remains serialized on the critical path of every round. Parallel speculative decoding (PSD) overlaps drafting with verification, yet existing methods must guess the accepted prefix and bonus token in advance: a wrong guess reverts the whole batch to serial drafting. We present DPara, a PSD framework that reuses effective parallel drafters yet guarantees backbone–verification overlap in every round, thereby eliminating this probabilistic fallback altogether. While the target verifies, DPara's diffusion backbone precomputes draft representations for *every* acceptance boundary with the bonus left unspecified; a lightweight autoregressive head then combines the revealed verification outcome with the matching precomputed representation to emit the next round's draft tokens almost instantly—fully parallelizing the dominant backbone forward with verification and leaving only the negligible head cost serial. Experiments on Qwen3-8B and Qwen3-14B across seven math, coding, and chat benchmarks show that DPara achieves average speedups of and over autoregressive decoding, surpassing the strongest serial and parallel speculative decoding baselines alike.

Primary area:
optimization
Keywords:
Parallel Speculative Decoding, LLM inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_USUgOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents DPara, a parallel speculative decoding framework built on a DSpark-style drafter. During verification, the backbone precomputes draft representations for every possible acceptance boundary. After verification, the corresponding branch is selected and DSpark’s lightweight autoregressive head incorporates the actual bonus token.

Critical strengths

The implementation demonstrates measurable latency improvements over DSpark, with average speedups increasing from 2.89× to 3.21× on Qwen3-8B and from 3.35× to 3.52× on Qwen3-14B.

The evaluation includes batching, stochastic sampling, heterogeneous drafting hardware, and single-GPU colocation. These experiments provide useful evidence about deployment trade-offs.

Critical weaknesses

The technical contribution is limited. DPara combines established verification-outcome speculation with DSpark’s existing, frozen autoregressive correction head. The main changes are execution scheduling and backbone finetuning for multiple feature-less anchors, without a clearly established new insight.

Scalability and resource efficiency remain unclear. DPara computes d(d+1) prediction positions per request but uses only one branch. Single-GPU DPara already underperforms DSpark at batches 8 and 16; dedicated-GPU results stop at batch 16 with fixed draft length 7, leaving the compute crossover and benefits under equal hardware budgets unresolved.

Questions for authors
  • What conceptual contribution distinguishes DPara from acceptance-boundary enumeration combined with an existing bonus-conditioned correction mechanism?
  • How do backbone latency and end-to-end throughput scale jointly with batch size and draft length?
  • Under an equal total hardware budget, how does DPara compare with allocating the additional drafting resources to serving more requests?
Reviewer_HG97OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DPara, a parallel speculative decoding (PSD) framework that overlaps drafting and target verification without suffering from "serial fallback". Existing PSD methods (like SSD) guess the accepted prefix and bonus token in advance; when a guess misses, the entire batch reverts to slow serial drafting. DPara avoids guessing by adapting a DSpark parallel drafter into M-DFlash, which uses a tree-structured attention mask to precompute draft logits for all d+1 possible acceptance lengths while verification is running. Once verification reveals the actual bonus token, a tiny, frozen autoregressive (AR) head incorporates it in ~1.3 ms. On Qwen3-8B and Qwen3-14 B, DPara achieves average speedups of 3.21X and 3.52X over autoregressive decoding, outperforming both serial and parallel baselines.

This is a clear, well-engineered paper based on a smart observation: separating the heavy, bonus-independent diffusion backbone from the tiny, bonus-dependent AR head enables 100% verification overlap and 0% fallback. The authors also thoroughly test practical setups, including batching up to 16, temperature sampling, using a cheap A10 GPU for drafting, and single-GPU execution. My main concerns are the drop in average acceptance length compared to serial DSpark, compute contention on a single GPU at larger batch sizes, and the need to normalize throughput per GPU in the main two-GPU experiments.

Critical strengths
  1. Precomputing all d+1 acceptance boundaries eliminates the batch-fallback problem that causes SSD to revert to serial execution on 57%–96% of steps at a batch size of 16.

  2. M-DFlash adds no new parameters and only fine-tunes the diffusion backbone for 1 epoch while keeping the AR head frozen, improving acceptance length by 30.7% over the base checkpoint.

  3. DPara consistently beats baselines across 7 math, coding, and chat benchmarks. Pairing an H800 with a cheap A10 drafting GPU (transferring only hidden states over Ethernet) retains 98% of the speedup while cutting drafting hardware cost by ~90%.

Critical weaknesses
  1. Because the drafter runs before verification finishes, its anchor tokens lack fresh target features. As a result, DPara's acceptance length drops by nearly 1 token compared to serial DSpark (4.41 vs. 5.36 on Qwen3-14B).

  2. Precomputing all d+1 branches expands the drafter's input sequence from 8 tokens to 64 tokens per request (for d=7). Consequently, on a single GPU (DPara-single), DPara becomes compute-bound and is slower than serial DSpark at batch sizes 8 and 16.

  3. The main results (Table 1 and Table 2) use 2 H800 GPUs for DPara against 1 H800 GPU for serial DSpark, without reporting throughput normalized per GPU

  4. Evaluation is limited to dense models up to 14B (Qwen3-8B/14B); testing on 70B+ or MoE architectures would better show how the overlap scales.

Questions for authors

Appendix E notes that Qwen3-8B outputs were used to train both the 8B and 14B drafters. Does this off-policy data explain why Qwen3-14B shows a smaller speedup gain over DSpark (+5.1%) than Qwen3-8B (+11.1%)?

Reviewer_u3FvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

In this paper, authors propose a parallel speculative decoding (PSD) framework to guarantee overlapping of the expensive backbone calculation with the verification step, by precomputing draft predictions for every possible acceptance boundary.

The technique is sound and the results demonstrate the effectiveness. Overall, the paper is well written and shows meaningful improvement over recent baselines at the cost of additional computing.

Critical strengths

The approach to hide the time consuiming backbone calculation within the verification time window is a valid idea even if not entirely new.

Critical weaknesses

My main concern is that precomputing for every possible acceptance boundary increases the drafting cost significantly, as mentioned in Section 4.3.2 where DPara-single loses its advantage over DSpark for batch sizes 8 and 16. However, for multi-GPU setups, the authors have not commented on this overhead or the ideal limit for draft length d in their approach. At what point does this overhead completely overshadow the benefits of their approach?

Questions for authors

Can you comment on the point made in the weakness section?

Comments on the paper

5 comments
~Arvid_Frydenlund1OpenReview ↗

Relation to prior work.

Can you explain the relationship between your work and Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potential (Samragh et al, 2025) and TiDAR: Think in Diffusion, Talk in Autoregression (Liu et al, 2025)? I believe your DPara algorithm is very similar to the former's quadratic decoding algorithm, and the latter applies it in a diffusion setting.

~Xinze_Feng1OpenReview ↗

Question about DPara's overlap and speedup

Thank you for sharing this interesting work. I particularly liked how DPara handles the unknown bonus token: masking it lets the drafter backbone run ahead for each possible acceptance point, while the final autoregressive walk waits for verification. I was curious about how much drafting latency this removes from the critical path.

I profiled the official DSpark checkpoints (deepseek-ai/dspark_qwen3_8b_block7 and deepseek-ai/dspark_qwen3_14b_block7) using SGLang's DSpark worker (commit 95d0e57e83, CUDA graphs, FlashAttention 3) on a single H200. Settings follow Table 1: batch size 1, greedy decoding, thinking disabled, up to 512 new tokens, draft length 7, and 80 prompts per dataset.

The median step breakdown below comes from the built-in step profiler and kernel-level PyTorch traces on GSM8K (172 steps for 8B and 155 for 14B):

Component8B (ms)14B (ms)
Target verification, including LM head, acceptance check, and input preparation5.588.87
Draft feature fusion and planning0.170.20
Drafter backbone (5 transformer layers)0.691.05
Draft LM head0.290.36
Autoregressive walk0.290.29
Total step7.0110.77

Feature fusion and the drafter backbone account for approximately 12.3% and 11.6% of the step for 8B and 14B. The walk timing is close to Appendix A's 0.3 ms, and my DSpark acceptance lengths match Table 1 within 0.06 across all eight settings.

Roofline estimate. If DPara fully hides feature fusion and the backbone—a fraction hh of the DSpark step—while other costs remain unchanged, the idealized speedup is

SDParaideal=SDSparkτDParaτDSpark11−h.S_{\mathrm{DPara}}^{\mathrm{ideal}} = S_{\mathrm{DSpark}} \frac{\tau_{\mathrm{DPara}}}{\tau_{\mathrm{DSpark}}} \frac{1}{1-h}.

Using the paper's DSpark speedups and acceptance lengths, together with our measured overlap fractions, gives the following estimates. These assume complete overlap with no added overhead; the draft LM head and autoregressive walk remain on the critical path.

SettingPaper τ (DSpark / DPara)Paper DSpark speedupEstimated DPara speedupPaper DPara speedup
8B GSM8K6.51 / 5.823.48×3.55×4.17×
8B HumanEval5.82 / 4.783.15×2.95×3.50×
8B Alpaca3.72 / 2.922.00×1.79×2.10×
8B MT-Bench4.01 / 3.122.13×1.89×2.25×
14B GSM8K6.53 / 5.734.07×4.04×4.53×
14B HumanEval5.79 / 4.763.66×3.40×3.84×
14B Alpaca3.71 / 2.912.32×2.06×2.32×
14B MT-Bench4.02 / 3.162.50×2.22×2.53×

Since our profiles use different hardware and potentially a different implementation, these estimates are illustrative rather than bounds on your reported results. They also use the GSM8K timing fractions across datasets. I would appreciate clarification on two points:

  1. Does DPara overlap any additional work, such as the draft LM head, autoregressive walk, or context bookkeeping, beyond feature fusion and the backbone?
  2. Could you share some implementation details or the framework DPara is built on so that I can reproduce the speedups? I understand that your experiments use H800 GPUs, whereas my profiles use an H200. It would also be helpful to understand how the overlap benefits and reported speedups might change across hardware.

I'm happy to share my logs, traces, and commands and rerun with your exact settings and prompt lists. Thanks again for the interesting work!

  1. In our default configuration, the draft LM head is also overlapped. An exception is when using a low-cost draft GPU (A10): we transfer hidden states rather than logits.
  2. Our inference implementation differs substantially from the SGLang setup you profiled, beyond the H800/H200 hardware difference. As you appropriately noted, combining our reported results with intermediate timings measured under a different hardware/framework configuration can therefore yield inconsistent or counterintuitive estimates. The overlap benefits should ideally be evaluated within the same implementation and hardware setup.
  3. DPara’s implementation is similar to SSD.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.