- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SPD partitions the target model into pipeline stages and uses a separate draft module to keep tokens flowing through them. The drafter consumes target features from different depths and runs alongside each pipeline step to hide its latency.
Critical strengths
-
Using the target features already available in the pipeline is an interesting idea. It gives the drafter more information without waiting for the full target pass.
-
I like the pre-step scheduling choice. Giving up some feature depth so drafting can overlap with target execution is a sensible tradeoff to explore.
-
The paper measures actual distributed decoding speed and breaks down the running time. This helps show where communication and synchronization eat into the theoretical gains.
Critical weaknesses
-
The discussion of parallel and decoupled speculative decoding is too limited. SSD is briefly cited, but PEARL [1], SpecBranch [2], DOUBLE [3], DSI [4], SpecActor [5], and SPECTRE [6] are missing. These works provide relevant precedents for overlap, resource allocation, and rollback, even though their execution settings differ. I would like a clearer explanation of what splitting the target into stages adds over these approaches, together with comparisons against suitable parallel baselines.
-
How expensive is gathering target hidden states on the dedicated draft GPU? Figure 2 provides a useful breakdown, but I still cannot tell how much data is transferred per cycle, which transfers remain on the critical path, or how costs change across interconnects. This matters especially because the nine- and seventeen-GPU configurations cross node boundaries. Transfer volume and exposed communication time should be reported, including during pipeline refill after rejection, to establish when drafting latency can actually be hidden.
-
The hardware comparison is not well matched. The eight-stage SPD configuration uses nine GPUs, while EAGLE-3 and AR use one. On Qwen3.5-9B, their reported greedy speedups are 2.67x and 2.54x for SPD and EAGLE-3, respectively, so the extra resources buy only a small latency improvement. I would like comparisons against tensor-parallel AR and speculative decoding under the same GPU budget. Larger models that already require multiple GPUs would also be a better test of the proposed deployment setting than the current 4B/9B targets.
[1] Tianyu Liu, Yun Li, Qitan Lv, Kai Liu, Jianchen Zhu, Winston Hu, and Xiao Sun. PEARL: Parallel Speculative Decoding with Adaptive Draft Length. ICLR, 2025.
[2] Yuhao Shen, Junyi Shen, Quan Kong, Tianyu Liu, Yao Lu, and Cong Wang. SpecBranch: Speculative Decoding via Hybrid Drafting and Rollback-Aware Branch Parallelism. ICLR, 2026.
[3] Yuhao Shen, Tianyu Liu, Junyi Shen, Jinyang Wu, Quan Kong, Li Huan, and Cong Wang. Double: Breaking the Acceleration Limit via Double Retrieval Speculative Parallelism. arXiv preprint, 2026.
[4] Nadav Timor, Jonathan Mamou, Daniel Korat, Moshe Berchansky, Oren Pereg, Moshe Wasserblat, Tomer Galanti, Michal Gordon-Kiwkowitz, and David Harel. Distributed Speculative Inference (DSI): Speculation Parallelism for Provably Faster Lossless Language Model Inference. ICLR, 2025.
[5] Rongxin Cheng, Kai Zhou, Xingda Wei, Siyuan Liu, Mingcong Han, Mingjing Ai, Yeju Zhou, Baoquan Zhong, Wencong Xiao, Xin Liu, Rong Chen, and Haibo Chen. Fast LLM Post-training via Decoupled and Best-of-N Speculation. arXiv preprint, 2025. (SpecActor.)
[6] Jincheng Xie, Yawen Ling, Qi Xiao, Feiyu Zhang, Zhongyi Huang, Wen Hu, and Yu Zheng. SPECTRE: Hybrid Ordinary-Parallel Speculative Serving for Resource-Efficient LLM Inference. arXiv preprint, 2026.
Questions for authors
Same as the weaknesses above.
3 comments
Thank you for the response, which addresses some of my concerns. However, I could not find these updates in the PDF. I would like to see them incorporated into the revised manuscript and will raise my score once they are included.
Thank you for your follow-up. We have uploaded a revised PDF that incorporates the content of our previous response. All new or modified text is highlighted in blue:
- W1 (parallel SD): a new paragraph in Section 2 (Related Work).
- W2 (communication cost): a new "Communication Cost" paragraph in Section 4.3, and Appendix D. Appendix D covers transfer volume vs. TP, exposed communication time, and a new time breakdown of the pipeline-refill phase.
- W3 (hardware-matched comparison): a new Section 4.4 with the same-GPU-budget comparison against TP and EAGLE-3+TP. The Limitations section is updated accordingly.
We hope the revision addresses your concerns. Thank you again for your constructive feedback.
Thank you for the detailed and constructive review.
W1. Parallel speculative decoding. Thank you for the references. We will add a dedicated paragraph in related works. These works share the same motivation with ours: in standard SD the drafter and the target wait for each other. They reduce this waiting by running more work side by side on top of the usual draft-then-verify loop:
However, our method, splitting the target into stages, has essential advantages:
W2. Communication cost.
In summary, the communication cost is very low. We will report these in our new version.
W3. Hardware-matched comparison. We added same-GPU baselines on one 9×V100 NVSwitch node (greedy, batch 1). Each method's speedup is relative to single-GPU AR (standard autoregressive) in its own stack (SGLang for AR/EAGLE-3, our runtime for SPD):
SPD outperforms TP at a similar GPU budget, and outperforms single-GPU EAGLE-3.
EAGLE-3 + TP is faster than our method for 9B. However, our method still has advantages and potential:
We will add this comparison in our new version of paper.