acceptodds
Under review as a conference paper at ICLR 2027

Speculative Pipeline Decoding: Higher-Accuracy Drafting with Hidden Latency via Pipeline Parallelism

Abstract

Speculative Decoding (SD) accelerates low-concurrency LLM inference with a draft-then-verify paradigm. Mainstream methods, however, rely on multi-token prediction, which incurs compounding prediction difficulty and exposed draft latency. We propose Speculative Pipeline Decoding (SPD), which partitions the target LLM into pipeline stages so that tokens of a single sequence advance in parallel. To keep the pipeline saturated, a Pipeline Draft Module (PDM) aggregates multi-depth target features to predict the next token and runs concurrently with each pipeline step, yielding bounded prediction difficulty, higher acceptance, and hidden draft latency. Experiments show that SPD achieves higher theoretical and wall-clock speedup than EAGLE-3 at moderate pipeline width, while more aggressive widths still leave room for further gains.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
pipeline parallel, speculative decoding, LLM efficiency, distributed inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YtuFOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SPD partitions the target model into pipeline stages and uses a separate draft module to keep tokens flowing through them. The drafter consumes target features from different depths and runs alongside each pipeline step to hide its latency.

Critical strengths
  1. Using the target features already available in the pipeline is an interesting idea. It gives the drafter more information without waiting for the full target pass.

  2. I like the pre-step scheduling choice. Giving up some feature depth so drafting can overlap with target execution is a sensible tradeoff to explore.

  3. The paper measures actual distributed decoding speed and breaks down the running time. This helps show where communication and synchronization eat into the theoretical gains.

Critical weaknesses
  1. The discussion of parallel and decoupled speculative decoding is too limited. SSD is briefly cited, but PEARL [1], SpecBranch [2], DOUBLE [3], DSI [4], SpecActor [5], and SPECTRE [6] are missing. These works provide relevant precedents for overlap, resource allocation, and rollback, even though their execution settings differ. I would like a clearer explanation of what splitting the target into stages adds over these approaches, together with comparisons against suitable parallel baselines.

  2. How expensive is gathering target hidden states on the dedicated draft GPU? Figure 2 provides a useful breakdown, but I still cannot tell how much data is transferred per cycle, which transfers remain on the critical path, or how costs change across interconnects. This matters especially because the nine- and seventeen-GPU configurations cross node boundaries. Transfer volume and exposed communication time should be reported, including during pipeline refill after rejection, to establish when drafting latency can actually be hidden.

  3. The hardware comparison is not well matched. The eight-stage SPD configuration uses nine GPUs, while EAGLE-3 and AR use one. On Qwen3.5-9B, their reported greedy speedups are 2.67x and 2.54x for SPD and EAGLE-3, respectively, so the extra resources buy only a small latency improvement. I would like comparisons against tensor-parallel AR and speculative decoding under the same GPU budget. Larger models that already require multiple GPUs would also be a better test of the proposed deployment setting than the current 4B/9B targets.

[1] Tianyu Liu, Yun Li, Qitan Lv, Kai Liu, Jianchen Zhu, Winston Hu, and Xiao Sun. PEARL: Parallel Speculative Decoding with Adaptive Draft Length. ICLR, 2025.

[2] Yuhao Shen, Junyi Shen, Quan Kong, Tianyu Liu, Yao Lu, and Cong Wang. SpecBranch: Speculative Decoding via Hybrid Drafting and Rollback-Aware Branch Parallelism. ICLR, 2026.

[3] Yuhao Shen, Tianyu Liu, Junyi Shen, Jinyang Wu, Quan Kong, Li Huan, and Cong Wang. Double: Breaking the Acceleration Limit via Double Retrieval Speculative Parallelism. arXiv preprint, 2026.

[4] Nadav Timor, Jonathan Mamou, Daniel Korat, Moshe Berchansky, Oren Pereg, Moshe Wasserblat, Tomer Galanti, Michal Gordon-Kiwkowitz, and David Harel. Distributed Speculative Inference (DSI): Speculation Parallelism for Provably Faster Lossless Language Model Inference. ICLR, 2025.

[5] Rongxin Cheng, Kai Zhou, Xingda Wei, Siyuan Liu, Mingcong Han, Mingjing Ai, Yeju Zhou, Baoquan Zhong, Wencong Xiao, Xin Liu, Rong Chen, and Haibo Chen. Fast LLM Post-training via Decoupled and Best-of-N Speculation. arXiv preprint, 2025. (SpecActor.)

[6] Jincheng Xie, Yawen Ling, Qi Xiao, Feiyu Zhang, Zhongyi Huang, Wen Hu, and Yu Zheng. SPECTRE: Hybrid Ordinary-Parallel Speculative Serving for Resource-Efficient LLM Inference. arXiv preprint, 2026.

Questions for authors

Same as the weaknesses above.

3 comments

Thank you for the detailed and constructive review.

W1. Parallel speculative decoding. Thank you for the references. We will add a dedicated paragraph in related works. These works share the same motivation with ours: in standard SD the drafter and the target wait for each other. They reduce this waiting by running more work side by side on top of the usual draft-then-verify loop:

WorkSource of parallelism
PEARLA separate small draft model keeps drafting while the target verifies the first draft token (pre-/post-verify).
DSIThe target keeps running while the drafter drafts, using multiple target instances.
SpecBranchDrafts several branches at once, forking only where the next token is uncertain (similar to tree drafting).
DOUBLEDraft and target both run n-gram retrieval and verification in parallel.
SpecActorPlaces drafter and target on different GPUs to accelerate RL rollouts.
SPECTRELends underused small-model services as drafters to busy large-model services: a serving and scheduling problem.

However, our method, splitting the target into stages, has essential advantages:

  1. No mutual waiting by construction. With a phase shift of n, the token being drafted never depends on a verification still in flight. The methods above actually still keep the draft-to-verify dependency within a step and overlap work around it, while SPD removes that dependency from the algorithm itself.
  2. Target-feature drafting. Most of these works (e.g., PEARL, DSI, SPECTRE) use a separate small LM as the drafter, whose acceptance is generally below EAGLE-style drafters that read the target's own features. In contrast, SPD's drafter stably reads the target's in-flight features from several depths, achieving higher acceptance rate.
  3. Balanced load, easy to schedule. Above methods usually run a large target next to a small drafter on separate devices, giving unequal loads. SPD splits one target into equal stages, and the drafter's cost is hidden behind a stage forward.

W2. Communication cost.

  • Volume. Each cycle moves only O(n) hidden vectors: one per stage-to-stage hop (hidden states of n*B*1*H), the anchor features (num_anchors*B*1*H) sent to the PDM rank, and the last stage's output for verification (B*1*H). A vector is h × 2 bytes when batchsize=1 (5 KB for 4B, 8 KB for 9B), so a cycle transfers tens of KB, which is much lower than TP (TP need twice all-reduce per layer, whose volume is almost 60 times more than PP). Communication is latency-bound, not bandwidth-bound.
  • Exposed time of communication. Since the communication volume is small and we have makd the sending and receiving asynchronous, as we can see in Figure 2, the communication time exposed in the critical path contribute is very little. On the contrary, scheduling and verifying have significantly extended the critical path, which leaves optimization room.
  • Across interconnects. Crossing one node boundary is nearly free because PP's communication volume is much less than TP. The n = 16 degradation comes from drafter latency in the critical path.
  • Refill after rejection. We will add the time breakdown of refilling to our new version. It is similar to the time breakdown when the pipeline is full. The only difference is: (1) not all stages are working (2) no verifying. So the total time is less.

In summary, the communication cost is very low. We will report these in our new version.

W3. Hardware-matched comparison. We added same-GPU baselines on one 9×V100 NVSwitch node (greedy, batch 1). Each method's speedup is relative to single-GPU AR (standard autoregressive) in its own stack (SGLang for AR/EAGLE-3, our runtime for SPD):

MethodGPUs4B9B
Standard, TP-441.60×1.92×
Standard, TP-881.68×2.20×
EAGLE-3 (k=7), TP-111.64×1.90×
EAGLE-3 (k=7), TP-442.35×3.17×
EAGLE-3 (k=7), TP-882.34×3.37×
SPD, n=452.09×2.45×
SPD, n=892.40×2.86×
  • SPD outperforms TP at a similar GPU budget, and outperforms single-GPU EAGLE-3.

  • EAGLE-3 + TP is faster than our method for 9B. However, our method still has advantages and potential:

    1. Our prototype realizes only about 75% of SPD's theoretical speedup (e.g. 3.39 / 2.53), so future engineering on the inference engine (e.g. hidden the latency of scheduling) can narrow this gap.
    2. SPD composes with TP (TP within a stage, PP across stages) and needs only small point-to-point messages, so it applies to situations where we have to deploy a large model across nodes using PP (within a node still TP), where pure TP + standard SD doesn't apply.
  • We will add this comparison in our new version of paper.

Thank you for the response, which addresses some of my concerns. However, I could not find these updates in the PDF. I would like to see them incorporated into the revised manuscript and will raise my score once they are included.

Thank you for your follow-up. We have uploaded a revised PDF that incorporates the content of our previous response. All new or modified text is highlighted in blue:

  • W1 (parallel SD): a new paragraph in Section 2 (Related Work).
  • W2 (communication cost): a new "Communication Cost" paragraph in Section 4.3, and Appendix D. Appendix D covers transfer volume vs. TP, exposed communication time, and a new time breakdown of the pipeline-refill phase.
  • W3 (hardware-matched comparison): a new Section 4.4 with the same-GPU-budget comparison against TP and EAGLE-3+TP. The Limitations section is updated accordingly.

We hope the revision addresses your concerns. Thank you again for your constructive feedback.

Reviewer_8gufOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a pipeline architecture for speculative decoding that breaks the target model into n stages and runs each stage on a dedicated GPU. It also proposes a pipeline draft module to keep the stages of the pipeline completely filled.

Critical strengths

The idea is technically sound, even if not entirely novel. Results demonstrate the effectiveness of this approach. Overall, the paper is well written and shows meaningful improvement at the cost of additional hardware resources.

Critical weaknesses

My main concern with the paper is that while pipelining itself is not a new concept in Computer Architecture, it comes at the cost of additional hardware resources. As such, the authors should note and mention the return on investment (ROI) for this, maybe in terms of speedup achieved per additional GPU as part of their ablation study.

Minor comment: figure 1 says PSM, which I assume meant PDM

Questions for authors

Discuss ROI per additional GPU tradeoff.

2 comments

Thank you for the positive assessment.

ROI per additional GPU. We measured all methods on the same V100 NVSwitch node (fp16, greedy, batch 1), against AR (autoregressive), TP-n AR and EAGLE-3 + TP-n, comparing their speedup and ROI.

MethodGPUs4B9BPer GPU (4B / 9B)
Standard, TP-441.60×1.92×0.40 / 0.48
Standard, TP-881.68×2.20×0.21 / 0.28
EAGLE-3 (k=7), TP-111.64×1.90×1.64 / 1.90
EAGLE-3 (k=7), TP-442.35×3.17×0.59 / 0.79
EAGLE-3 (k=7), TP-882.34×3.37×0.29 / 0.42
SPD, n=452.09×2.45×0.42 / 0.49
SPD, n=892.40×2.86×0.27 / 0.32

We will add this table to the ablation section, with three observations:

  1. As the number of GPUs increases, the average return per GPU declines Like TP, SPD spends more GPUs to cut latency, and the marginal effect inevitably emerges. However, SPD's per-GPU efficiency is on par with TP-4 AR and higher than TP-8 AR.
  2. SPD scales where TP saturates. Going from 4 to 8 GPUs adds only +0.08× / +0.28× to TP AR and −0.01× / +0.20× to EAGLE-3 + TP. Going from n = 4 to n = 8 adds +0.31× / +0.41× to SPD. So considering the incremental speedup from n = 4 to n = 8, SPD's return per additional GPU is higher than other methods, showing a better scalability.
  3. SPD composes with TP, and its marginal cost is small when the model is already sharded. PP does not replicate weights, so a model that already needs n GPUs needs only one extra GPU for the PDM. SPD runs across nodes with PP and can use TP within each node, which suits large models that TP alone cannot scale.

Figure 1 "PSM". Thank you, this should be "PDM". It has been fixed.

Thank you for the response. I will maintain my score.

Reviewer_hhZZOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Speculative decoding is the main lever for low-concurrency decoding, but multi-token prediction has two limits: acceptance decays with draft length, and draft latency stays exposed before verification. The paper takes a different route: it partitions the target LLM into n stages and advances n tokens of the same sequence at different depths, so that one new token enters per step. Three techniques support this: a drafter (PDM) that predicts a single token from target hidden states aggregated over the in-flight staircase, a pre-step schedule that overlaps drafting with the stage forward and hides its latency, and streaming verification with rollback. Training aligns the drafter to the inference layouts through sequence replication, simulated fill, and a structural mask.

Critical strengths
  • The problem is real and important. Pipeline bubbles at batch 1 are waste, and the paper connects them to the two limits of multi-token drafting: compounding feature drift and exposed draft latency.
  • The analysis holds up. The observation that a phase shift of n removes the intra-step draft-to-verify dependency is the key step, and the three designs follow from it.
  • The implementation is substantial. A from-scratch multi-node pipeline runtime, per-cycle profiling, and a report of where the method degrades.
Critical weaknesses
  1. The gain comes from hardware, and it is small. SPD uses n+1 GPUs while EAGLE-3 and the AR baseline use one, and on the larger model the best configuration gives only a small gain. The comparison that matters, tensor parallelism at the same GPU count, is missing, and EAGLE-3's strongest setting (tree drafting) is evaluated only theoretically. The paper explains the cost by noting that large models are already deployed on multiple GPUs, but the experiments use 4B/9B models that fit on one device, so the case that needs multiple GPUs is not measured.

  2. Accuracy and losslessness are not verified. The paper reports only efficiency metrics (speedup and step acceptance): no benchmark accuracy, no token-for-token comparison with AR, and no distributional check under sampling. Rollback and cross-stage cache truncation are where silent errors would appear, and the reported results come from these runs.

  3. Hyperparameter sensitivity is thin, and there is no rule for choosing n and Ls. The sweeps over n and Ls exist, but the aggregation scheme itself is not ablated: the number and positions of anchors rest on a single remark, the drafter width and vocabulary are not studied, and non-uniform partitioning is untested. The choice of width comes from one hardware setting, so a reader with a given model and interconnect cannot tell which configuration to use.

  4. Generality is untested. Both models come from one family (Qwen3.5-4B/9B), thinking mode is disabled, and generation is short, so the evaluation covers chat-style outputs rather than the long reasoning traces where decode speed matters most. Architecture matters too: the design assumes a stage forward costs 1/n of a full forward and splits layers uniformly, while the paper notes that Qwen3.5 "mixes attention types". If linear-attention layers are cheaper than full-attention ones, a uniform split leaves the stages unbalanced and the 1/n cost model no longer holds. Whether the results carry over to another family, or to a model with MoE or MLA, is not shown.

Questions for authors
  1. At the same GPU count, what are the wall-clock speedups of TP-n AR and of EAGLE-3 under TP-n? The paper says all speedups are against "the same single-GPU autoregressive baseline": which implementation is that, SGLang or your own framework? How does the 9B comparison look once the extra GPUs are priced in? What is the wall-clock result for tree drafting?
  2. Please report benchmark accuracy for AR, EAGLE-3, and SPD, a greedy token-for-token match rate against AR, and a distributional check under sampling. How is the smaller drafter vocabulary handled when the target places mass outside its support?
  3. Given a model size and an interconnect, how should n and Ls be chosen? How sensitive is acceptance to the number and position of anchors, to drafter width, and to non-uniform partitioning?
  4. With thinking enabled and longer generations, does the speedup hold or grow? On a second family (Llama, DeepSeek with MLA, an MoE model), how would the stage split and the anchor choice change? Are the stages actually balanced on Qwen3.5 given its mixed attention types?
2 comments

Thank you for the careful review.

W1. Hardware-matched comparison. We added same-GPU baselines on one 9×V100 NVSwitch node (greedy, batch 1). Each method's speedup is relative to single-GPU AR (standard autoregressive) in its own stack (SGLang for AR/EAGLE-3, our runtime for SPD):

MethodGPUs4B9B
Standard, TP-441.60×1.92×
Standard, TP-881.68×2.20×
EAGLE-3 (k=7), TP-111.64×1.90×
EAGLE-3 (k=7), TP-442.35×3.17×
EAGLE-3 (k=7), TP-882.34×3.37×
SPD, n=452.09×2.45×
SPD, n=892.40×2.86×
  • SPD outperforms TP at a similar GPU budget, and outperforms single-GPU EAGLE-3.

  • EAGLE-3 + TP is faster than our method for 9B. However, our method still has advantages and potential:

    1. Our prototype realizes only about 75% of SPD's theoretical speedup (e.g. 3.39 / 2.53), so future engineering on the inference engine (e.g. hidden the latency of scheduling) can narrow this gap.
    2. More importantly, SPD composes with TP (TP within a stage, PP across stages) and needs only small point-to-point messages, so it applies to situations where we have to deploy a large model across nodes using PP (within a node still TP), where pure TP + standard SD doesn't apply.
  • We will add this comparison in our new version of paper.

W2 / Q2. Accuracy and losslessness. SPD uses standard speculative verification: argmax match for greedy decoding, and rejection sampling with residual max(0, p − q) for sampling. Its output distribution therefore equals the target's, and task accuracy equals AR's, in theory. To verify the implementation, including rollback and cache truncation, we compared SPD's greedy outputs with AR(autoregressive) token by token, and proved the losslessness:

PrecisionIdentical outputs
fp16240 / 240
fp32240 / 240

W3 / Q3. Hyperparameters. SPD is not very sensitive to these choices, and we use EAGLE-3-style defaults.

  • Anchors are taken from shallow, middle and deep layers. As shown in our experiment, changing their positions, or increasing the number above 5 barely changes acceptance (Qwen3.5-4B, n = 8, L_s = 4):
Anchor layersAcceptance rate
0,16,310.345
0, 8, 16, 24, 31 (default)0.380
0, 4, 13, 22, 310.376
0, 4, 8, 12, 16, 20, 24, 28, 310.381
  • Drafter width equals the target's hidden size, and the draft vocabulary is 50k, as in EAGLE-3. Neither choice affects losslessness.
  • Choosing n and L_s. We choose the L_s by this empirical formula: L_s ≈ L/n − 1 (section 4.3), which balance the drafter's accuracy and time to make it not exposed to the critical path. For a balanced partition, n is 4,8,16 by default, and the choice depends on hardware resources.
  • Non-uniform partitioning is our on-going work. It involves more complex scheduling and may be the V2 of our method.

W4 / Q4. Generality.

  • Stage balance. Qwen3.5 repeats a pattern of 3 GDN layers + 1 full-attention layer. At n = 4 and n = 8 every stage has the same composition (6 + 2 and 3 + 1), so the stages are balanced. At n = 16 the stages alternate, which partly explains its lower speedup.
  • Long generation or thinking mode belongs to a new task type, so adjusting our training data is needed. However, training on different data is beyond our compute budget. Mainstreaming methods like EAGLE-3 neither test thinking mode or different training datasets, so we just follow their settings.
  • Other families. The runtime already supports Llama-3.1 and Qwen3. Pure-attention models split uniformly, MoE layers have roughly constant per-layer cost, and MLA changes only the attention kernel, so our method should generalize well. But training drafters for more families is beyond our compute budget.

Thanks for the added experiments and clarifications, which partially address several of my concerns. However, some concerns, e.g., Q4, remain largely unresolved. My current assessment is borderline. I would not object to acceptance if one champions the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.