- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes PaPaSpec, a speculative decoding framework that parellel both verification/drafting and drafting itself. The key idea is overlapping the forward process of verification with drafting with a novel idea, depth-causal conditioning. The work further introduces looping of the drafter's intermediate layer to generate a refined representation and fully utilize the unused latency budget of the verification process. Experiments shows improvements over different setups and baselines.
Critical strengths
- The paper categorized the parellel in speculative decoding into macro-level, overlapping the verification and drafting, micro-level, parellel drafting. The attempt of combining both idea is well-motivated.
- The proposed depth-causal conditioning provides a reasonable way to implement macro-level parellel for the speculative decoding method that ultilize the infomation from target model itself.
- Looping for the intermediate layer of the drafter naturally provides a way to fully utilize the unused latency budget and improve acceptence length.
Critical weaknesses
- Although a promising idea is provided, the evaluation is limited using a Huggingface Transformer + torch.compile. Appendix A.1 shows that vllm + Dflash exceeds the performance of papaspec on Hugging face Transformer. In the limitation section, the authors also admit that their scope is limited to single-batch serving. It's unclear that if these idea could be integrated into a practical serving framework.
- Another major concern would be the proposed method, PaPaSpec uses two GPUs, while baselines use only a single GPU. A fair comparison is expected.
- A important mismatch is that, different branches correspond to different prefixes, while the available hidden states are generated from the same verification sequence.
- Minor weakness: the paper also contains several speculative hardware discussions that are not strongly supported by the evaluation, a more concise discussion would improve clarity.
Questions for authors
Q1: Could the authors provide more discussion or experiments on how this work can be intergrated into modern serving engines?
Q2: Could the authors compare to other baselines under the same hardware budget?
Q3: Could the author discuss about how the mismatch between target model's hidden and the different prefix by the drafter affects the improvement?
6 comments
Part II: CPU drafting, no draft and verification mismatch, and future hardware
W2 & Q2 Comparisons under same hardware budget
We thank the reviewer for pushing us on this. Indeed, we initially followed the established practice for research on SSD and used a separate H100 GPU for the drafter (Kumar et al., 2026). This is actually quite well-motivated in latency-sensitive settings where one would rather use two GPUs to serve a single user 20% faster than serving two users at standard speed (e.g., in high-frequency financial analysis/trading, real-time voice-to-voice conversation assistants, or autonomous robotics with expensive VLAs). Nevertheless, in our updated paper (Sec. 4.3, App. A.4, or table below), we also report on PaPaSpec with the verifier on an H100 GPU and the drafter on the available node CPUs (two 48-core Intel Xeon Platinum 8468). Given the substantial compute asymmetry, we implemented the CPU-based drafting with a tailored C++ implementation using W8A8 quantization for the linear layers, reduced the cache to , and ran the last drafting layer and the LM head concurrently on the host GPU. As can be seen in the table below, PaPaSpec still improves on the SoTA by 3.1%, even without an additional GPU available.
| Method | Cache hit (%) | Acceptance length | Speedup | Gain over D-PACE |
|---|---|---|---|---|
| D-PACE | – | 7.00 | 3.39× | – |
| PaPaSpec (B=32, 2 GPUs) | 85.3 | 7.20 | 3.81× | +12.5% |
| PaPaSpec (B=4, CPU drafter) | 53.7 | 7.08 | 3.49× | +3.1% |
Drafting on the host CPU for Gemma-4-26B-A4B-it (greedy decoding, =16, 2048 new tokens, compiled Hugging Face Transformers). Averaged over the same seven benchmarks as in Table 1.
Moreover, we recognize that a faithful comparison against standard SSD was missing. SSD decoding (Kumar et al. 2026) is also macro-parallel and uses two H100 GPUs, but in contrast to our work, it is micro-sequential (no DFlash architecture and lacking our depth-causal conditioning). After extensive tuning of their hyperparameters, their best configuration only achieves a speedup of compared to the speedup of that PaPaSpec achieves. The reported numbers for SSD are consistent with reproductions in prior art.
W3 & Q3: No mismatch between drafting and verification
We politely want to push back against this and clarify our implementation. It is correct that different branches correspond to different prefixes, but the available hidden states are taken from matching prefixes of the verifier. As such, the system remains consistent. To improve the presentation of our paper, we have now added the following explanation to Section 3.1 of the main text.
Branch $(\hat{\tau}, b)$ is masked to condition only on positions $t \leq n+\hat{\tau}-1$; a cache hit thus implies zero mismatch between drafting and verification.
W4: Discussion of future hardware
Thank you for this suggestion. We now run PaPaSpec's macro-parallel drafting on the host CPUs instead of a second GPU, and it still improves over D-PACE by 3.1% without any additional accelerator (see W2 and App. A.4). This shows that PaPaSpec can turn idle compute into decoding speed, and we expect dedicated drafting accelerators to increase this benefit further. We clarify this in Sec. 3.3.
With the additional clarification and results, we hope to have resolved the reviewer's concerns about the presentation, informativeness, and soundness of our method, and to have convinced them to recommend acceptance.
Thank you for the great effort you put into the rebuttal and revised manuscript! I appreciate the vLLM integration and the new batched serving results. I have two follow-up questions about the [P] design.
- Appendix A.3 describes preparing
[P]branches per request for a block of draft tokens, one for each possible acceptance length. If each branch predicts positions, the number of candidate positions grows quadratically with , while only one branch is used. How does this affect the practical block size at higher concurrency? Could you report how throughput scales with block size in the Table 8 setup? - Could you report mean acceptance length under matched settings for
[P]alone,[P]with the correction head, and drafting conditioned on the actual bonus token? This would quantify the acceptance cost of replacing the bonus token with[P]and how much of that loss the correction head recovers.
Overall, I appreciate how directly the revision addresses my earlier concerns. If time is limited, a qualitative discussion of these trade-offs would also be helpful!
Part I: Single- and multi-batch vLLM serving
We thank reviewer
NBs8for the insightful feedback, for agreeing that combining the established practices of micro- and macro-parallel drafting is well-motivated, for appreciating the introduction of depth-causal conditioning, and for recognizing the effectiveness of layer looping in the drafter. In the following, we address all raised questions and concerns.W1-a & Q1: Single-batch vLLM serving
Thank you for pushing us on this. We have now implemented PaPaSpec into vLLM, which has significantly lifted the throughput of our system reported in Sec. 4.3, Table 2, and Table 7 (see results on Qwen3-8B below). The faster verification shrinks the window for hiding drafting: we reduce the cache budget from 32 to 16, and the gains decrease from +30% to +24% over DFlash and from +18% to +10% over D-PACE on Qwen3-8B. Nonetheless, PaPaSpec remains the fastest method on the optimized backend:
Decoding throughput and acceptance length τ on vLLM for Qwen3-8B (greedy decoding, 128 prompts per benchmark, 2048 maximum new tokens, H100). The gains also translate to Qwen3-4B and Gemma-4-26B-A4B.
W1-b & Q1: Batched serving on vLLM
While speculative decoding is most effective at low batch sizes, we have now adapted PaPaSpec to batched serving (Section 4.3 and App. A.3 in the updated paper). Since a single cache miss stalls the entire batch, the probability of a fully macro-parallel step decays exponentially with the number of concurrent requests c. We therefore fine-tune PaPaSpec to also draft without an anchor token, using a trainable placeholder [P], which guarantees a 100% cache hit rate. An optional lightweight correction head connects the eventual bonus token with the anchor-free draft. The same drafter still supports exact anchors, which we use to augment the cache with B′ exact-anchor drafts.
Output throughput (tok/s) for concurrent serving with vLLM on Gemma-4-26B-A4B-it (SPEED-Bench throughput splits, 1536 requests each, greedy decoding, H100).
PaPaSpec improves throughput over AR and D-PACE up to c=16 concurrent requests (up to +11% speedup improvement), with larger gains for longer inputs, as the long-context drafting attention is hidden behind verification. Even without the correction head, it outperforms D-PACE up to c=4 (1k) and c=16 (2k). From c=32, the H100 becomes compute-bound, and PaPaSpec performs on par with D-PACE (see point 6 in the general response).