acceptodds
Under review as a conference paper at ICLR 2027

Parallel-Parallel Speculative Decoding

21 views

Abstract

Speculative decoding accelerates memory-bandwidth-bound inference of large autoregressive target models using a small, fast drafter model. Target-latent-conditioned parallel drafters, such as DFlash, currently offer the best accuracy-latency tradeoff, leading to the highest reported speedups. However, latent conditioning still requires executing drafting and verification sequentially, placing drafting latency on the critical path. This limits achievable speedups and prevents effective use of emerging hardware that can support concurrent drafting and verification. We introduce parallel-parallel speculative (PaPaSpec) decoding, which runs a multi-layer latent-conditioned parallel drafter concurrently with verification, hiding drafting latency entirely. The key enabler is a prefix and depth-causal latent conditioning scheme that restricts each drafter layer to only access the target latents computed so far. Hiding drafting behind verification not only removes it from the critical latency path, but also buys the drafter additional time to refine its draft. We exploit this by looping over intermediate layers until new conditioning information arrives, which increases the acceptance length while keeping the drafter size constant. We evaluate PaPaSpec in single-batch serving across three different target models on NVIDIA H100 GPUs. Across seven benchmarks covering math, coding, and chat, PaPaSpec accelerates autoregressive generation by 3.26–5.50x on average, a 13–22% improvement over DFlash.

Primary area:
foundation or frontier models, including LLMs
Keywords:
speculative decoding, masked diffusion models, autoregressive inference acceleration, looped transformers

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_NBs8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PaPaSpec, a speculative decoding framework that parellel both verification/drafting and drafting itself. The key idea is overlapping the forward process of verification with drafting with a novel idea, depth-causal conditioning. The work further introduces looping of the drafter's intermediate layer to generate a refined representation and fully utilize the unused latency budget of the verification process. Experiments shows improvements over different setups and baselines.

Critical strengths
  1. The paper categorized the parellel in speculative decoding into macro-level, overlapping the verification and drafting, micro-level, parellel drafting. The attempt of combining both idea is well-motivated.
  2. The proposed depth-causal conditioning provides a reasonable way to implement macro-level parellel for the speculative decoding method that ultilize the infomation from target model itself.
  3. Looping for the intermediate layer of the drafter naturally provides a way to fully utilize the unused latency budget and improve acceptence length.
Critical weaknesses
  1. Although a promising idea is provided, the evaluation is limited using a Huggingface Transformer + torch.compile. Appendix A.1 shows that vllm + Dflash exceeds the performance of papaspec on Hugging face Transformer. In the limitation section, the authors also admit that their scope is limited to single-batch serving. It's unclear that if these idea could be integrated into a practical serving framework.
  2. Another major concern would be the proposed method, PaPaSpec uses two GPUs, while baselines use only a single GPU. A fair comparison is expected.
  3. A important mismatch is that, different branches correspond to different prefixes, while the available hidden states are generated from the same verification sequence.
  4. Minor weakness: the paper also contains several speculative hardware discussions that are not strongly supported by the evaluation, a more concise discussion would improve clarity.
Questions for authors

Q1: Could the authors provide more discussion or experiments on how this work can be intergrated into modern serving engines?

Q2: Could the authors compare to other baselines under the same hardware budget?

Q3: Could the author discuss about how the mismatch between target model's hidden and the different prefix by the drafter affects the improvement?

6 comments
AuthorsOpenReview ↗

Part I: Single- and multi-batch vLLM serving

We thank reviewer NBs8 for the insightful feedback, for agreeing that combining the established practices of micro- and macro-parallel drafting is well-motivated, for appreciating the introduction of depth-causal conditioning, and for recognizing the effectiveness of layer looping in the drafter. In the following, we address all raised questions and concerns.


W1-a & Q1: Single-batch vLLM serving

Thank you for pushing us on this. We have now implemented PaPaSpec into vLLM, which has significantly lifted the throughput of our system reported in Sec. 4.3, Table 2, and Table 7 (see results on Qwen3-8B below). The faster verification shrinks the window for hiding drafting: we reduce the cache budget from 32 to 16, and the gains decrease from +30% to +24% over DFlash and from +18% to +10% over D-PACE on Qwen3-8B. Nonetheless, PaPaSpec remains the fastest method on the optimized backend:

MethodMATH-500 tok/sMATH-500 τ\tauHumanEval tok/sHumanEval τ\tauMT-Bench tok/sMT-Bench τ\tauAvg. tok/sAvg. τ\tauGain of PaPaSpec
AR1481.001491.001481.001481.00–
DFlash9208.027586.494004.256936.25+24%
D-PACE10468.738517.054424.567806.78+10%
PaPaSpec11538.879447.224764.718586.94–

Decoding throughput and acceptance length τ\tau on vLLM for Qwen3-8B (greedy decoding, 128 prompts per benchmark, 2048 maximum new tokens, H100). The gains also translate to Qwen3-4B and Gemma-4-26B-A4B.


W1-b & Q1: Batched serving on vLLM

While speculative decoding is most effective at low batch sizes, we have now adapted PaPaSpec to batched serving (Section 4.3 and App. A.3 in the updated paper). Since a single cache miss stalls the entire batch, the probability of a fully macro-parallel step decays exponentially with the number of concurrent requests cc. We therefore fine-tune PaPaSpec to also draft without an anchor token, using a trainable placeholder [P][P], which guarantees a 100% cache hit rate. An optional lightweight correction head connects the eventual bonus token with the anchor-free draft. The same drafter still supports exact anchors, which we use to augment the cache with B′B' exact-anchor drafts.

InputMethodc=1c=2c=4c=8c=16c=32c=64
1kAR174280441702108315972365
1kD-PACE (16)2704427161156187828713037
1kD-PACE (8)2844597391198192031134132
1kPaPaSpec ([P])2984757471183189929433957
1kPaPaSpec ([P]+head)3084877691233195330664175
1kGain of PaPaSpec+8.2%+6.2%+4.1%+2.9%+1.7%−1.5%+1.0%
2kAR164278426694103315622315
2kD-PACE (16)2634276861104180226862580
2kD-PACE (8)2724457041130186828263653
2kPaPaSpec+[P] (8)2914677381189187426813604
2kPaPaSpec+[P]+head (8)3024807631219191827613750
2kGain of PaPaSpec+11.1%+8.0%+8.3%+7.9%+2.7%−2.3%+2.7%

Output throughput (tok/s) for concurrent serving with vLLM on Gemma-4-26B-A4B-it (SPEED-Bench throughput splits, 1536 requests each, greedy decoding, H100).

PaPaSpec improves throughput over AR and D-PACE up to c=16c=16 concurrent requests (up to +11% speedup improvement), with larger gains for longer inputs, as the long-context drafting attention is hidden behind verification. Even without the correction head, it outperforms D-PACE up to c=4c=4 (1k) and c=16c=16 (2k). From c=32c=32, the H100 becomes compute-bound, and PaPaSpec performs on par with D-PACE (see point 6 in the general response).

AuthorsOpenReview ↗

Part II: CPU drafting, no draft and verification mismatch, and future hardware

W2 & Q2 Comparisons under same hardware budget

We thank the reviewer for pushing us on this. Indeed, we initially followed the established practice for research on SSD and used a separate H100 GPU for the drafter (Kumar et al., 2026). This is actually quite well-motivated in latency-sensitive settings where one would rather use two GPUs to serve a single user 20% faster than serving two users at standard speed (e.g., in high-frequency financial analysis/trading, real-time voice-to-voice conversation assistants, or autonomous robotics with expensive VLAs). Nevertheless, in our updated paper (Sec. 4.3, App. A.4, or table below), we also report on PaPaSpec with the verifier on an H100 GPU and the drafter on the available node CPUs (two 48-core Intel Xeon Platinum 8468). Given the substantial compute asymmetry, we implemented the CPU-based drafting with a tailored C++ implementation using W8A8 quantization for the linear layers, reduced the cache to B=4B=4, and ran the last drafting layer and the LM head concurrently on the host GPU. As can be seen in the table below, PaPaSpec still improves on the SoTA by 3.1%, even without an additional GPU available.

MethodCache hit (%)Acceptance lengthSpeedupGain over D-PACE
D-PACE–7.003.39×–
PaPaSpec (B=32, 2 GPUs)85.37.203.81×+12.5%
PaPaSpec (B=4, CPU drafter)53.77.083.49×+3.1%

Drafting on the host CPU for Gemma-4-26B-A4B-it (greedy decoding, γ\gamma=16, 2048 new tokens, compiled Hugging Face Transformers). Averaged over the same seven benchmarks as in Table 1.

Moreover, we recognize that a faithful comparison against standard SSD was missing. SSD decoding (Kumar et al. 2026) is also macro-parallel and uses two H100 GPUs, but in contrast to our work, it is micro-sequential (no DFlash architecture and lacking our depth-causal conditioning). After extensive tuning of their hyperparameters, their best configuration only achieves a speedup of 2.29×2.29\times compared to the speedup of 5.98×5.98\times that PaPaSpec achieves. The reported numbers for SSD are consistent with reproductions in prior art.


W3 & Q3: No mismatch between drafting and verification

We politely want to push back against this and clarify our implementation. It is correct that different branches correspond to different prefixes, but the available hidden states are taken from matching prefixes of the verifier. As such, the system remains consistent. To improve the presentation of our paper, we have now added the following explanation to Section 3.1 of the main text.

Branch $(\hat{\tau}, b)$ is masked to condition only on positions $t \leq n+\hat{\tau}-1$; a cache hit thus implies zero mismatch between drafting and verification.

W4: Discussion of future hardware

Thank you for this suggestion. We now run PaPaSpec's macro-parallel drafting on the host CPUs instead of a second GPU, and it still improves over D-PACE by 3.1% without any additional accelerator (see W2 and App. A.4). This shows that PaPaSpec can turn idle compute into decoding speed, and we expect dedicated drafting accelerators to increase this benefit further. We clarify this in Sec. 3.3.


With the additional clarification and results, we hope to have resolved the reviewer's concerns about the presentation, informativeness, and soundness of our method, and to have convinced them to recommend acceptance.

Thank you for the great effort you put into the rebuttal and revised manuscript! I appreciate the vLLM integration and the new batched serving results. I have two follow-up questions about the [P] design.

  1. Appendix A.3 describes preparing K+1K+1 [P] branches per request for a block of KK draft tokens, one for each possible acceptance length. If each branch predicts KK positions, the number of candidate positions grows quadratically with KK, while only one branch is used. How does this affect the practical block size at higher concurrency? Could you report how throughput scales with block size in the Table 8 setup?
  2. Could you report mean acceptance length under matched settings for [P] alone, [P] with the correction head, and drafting conditioned on the actual bonus token? This would quantify the acceptance cost of replacing the bonus token with [P] and how much of that loss the correction head recovers.

Overall, I appreciate how directly the revision addresses my earlier concerns. If time is limited, a qualitative discussion of these trade-offs would also be helpful!

Reviewer_QKNuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes PaPaSpec, a speculative decoding framework that aims to eliminate two forms of sequential dependency: micro-level sequentiality within drafting and macro-level sequentiality between drafting and verification. The key idea is to enable concurrent execution of target-conditioned parallel drafting and verification by leveraging intermediate hidden states from the target model through prefix- and depth-causal conditioning. The paper further introduces layer-wise looping to improve acceptance length and confidence-based dynamic budget allocation to reduce speculative prefix cache misses. Experiments across three target models and seven benchmarks report average decoding speedup improvements of 13–22% over DFlash.

Critical strengths
  • Meaningful technical approach

    • Prefix- and depth-causal conditioning addresses a key limitation of existing target-conditioned parallel drafters, which must wait for the target model's verification pass to complete before initiating the next drafting step. By enabling drafting to proceed using intermediate target-model states, PaPaSpec allows target-conditioned drafting and verification to execute concurrently. This provides a technically meaningful approach to reducing the sequential dependencies in existing speculative decoding frameworks.
  • Thorough analysis of execution characteristics

    • The paper provides component-wise ablations of the proposed conditioning mechanism, layer-wise looping, and confidence-based budget allocation. The latency breakdowns further illustrate the effectiveness and execution overhead of individual components, helping clarify their respective contributions and performance trade-offs.
Critical weaknesses
  • Limited performance improvements relative to the additional hardware cost

    • PaPaSpec requires dedicated GPUs for the target model and drafter, whereas DFlash and D-PACE operate on a single GPU. Despite this additional hardware requirement, the reported decoding speedup improvements are relatively modest, ranging from 13–22% over DFlash and approximately 8% over the stronger D-PACE baseline. Even when the primary objective is to minimize single-request latency, dedicating an additional full-sized GPU to drafting represents a substantial practical constraint. The paper does not sufficiently establish whether the achieved latency reductions justify this additional resource cost.

    • The paper emphasizes that concurrent execution creates opportunities to exploit otherwise idle drafting capacity through layer-wise looping and speculative branching. However, this additional computational capacity is not inherently free, as it is enabled by dedicating a separate GPU to the drafter. Although the additional computations can be overlapped with verification and thus hidden from the critical path, this does not eliminate the opportunity cost of the dedicated GPU, which could otherwise serve additional requests or independent speculative decoding sessions. While the authors suggest the possibility of using a dedicated drafting accelerator, the current evaluation relies exclusively on full-sized H100 GPUs. Consequently, the practical cost-effectiveness of the proposed architecture remains insufficiently demonstrated.

  • Lack of comparisons with existing macro-parallel approaches

    • A central contribution of PaPaSpec is the integration of target-conditioned parallel drafting into a macro-parallel execution framework. However, the main baselines are limited to macro-sequential approaches, such as DFlash and D-PACE. Prior works, including SSD, SwiftSpec, and Mirror-SD, have already explored concurrent drafting and verification. In particular, PaPaSpec directly builds upon the speculative prefix caching framework introduced by SSD. Therefore, comparisons against existing macro-parallel approaches under matched two-GPU configurations are essential for assessing the actual benefits of the proposed method.

    • Without such comparisons, it is difficult to determine whether PaPaSpec provides meaningful advantages over existing macro-parallel frameworks, or whether the reported improvements primarily arise from overlapping drafting and verification rather than the proposed conditioning mechanism. Given the additional architectural complexity and training requirements, the paper should demonstrate that its design offers substantial benefits beyond those achievable with existing macro-parallel approaches.

  • Incremental technical novelty relative to existing approaches

    • Several key design elements of PaPaSpec have already been explored in prior work. DFlash and P-EAGLE introduce target-conditioned parallel drafting, SSD and Mirror-SD investigate macro-parallel execution, and DFlare explores layer-specific latent conditioning. Against this background, the primary methodological novelty of PaPaSpec appears to lie in its prefix- and depth-causal conditioning mechanism, which enables these previously explored ideas to be combined into a concurrent execution framework.

    • Although this conditioning mechanism meaningfully addresses the dependency that prevents conventional target-conditioned drafters from operating concurrently with verification, its algorithmic novelty appears somewhat limited when considered alongside existing layer-specific conditioning and macro-parallel execution techniques. Similarly, layer-wise looping and confidence-based budget allocation appear to be adaptations or extensions of existing ideas rather than fundamentally new mechanisms. Overall, the paper would benefit from a clearer articulation of the distinct technical insights that go beyond combining and adapting established approaches.

References

[1] Bhendawade, Nikhil, et al. "Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inference." arXiv preprint arXiv:2510.13161 (2025).

[2] Chen, J., Liang, Y. & Liu, Z.. (2026). DFlash: Block Diffusion for Flash Speculative Decoding. <i>Proceedings of the 43rd International Conference on Machine Learning</i>, in <i>Proceedings of Machine Learning Research</i> 306:15343-15355 Available from https://proceedings.mlr.press/v306/chen26bv.html.

[3] Hui, Mude, et al. "P-EAGLE: Parallel-Drafting EAGLE with Scalable Training." arXiv preprint arXiv:2602.01469 (2026).

[4] Kumar, T., Dao, T., & May, A. (2026). Speculative speculative decoding. In Proceedings of the 14th International Conference on Learning Representations (ICLR). https://openreview.net/forum?id=aL1Wnml9Ef

[5] Wu, T., Yao, Y., Qi, Z., Zheng, H., Wang, Z., Ma, H., Liao, L., Lakkaraju, H., Li, J., & Du, Y. (2026). D-PACE: Dynamic position-aware cross-entropy for parallel speculative drafting. arXiv. https://doi.org/10.48550/arXiv.2605.18810

[6] Zhang, J., Yu, Z., Liu, S., Yu, E. J., Li, Z., Zhu, D., Duo, J., Xiong, W., Song, Y., Yu, G., Zhu, J., & Li, S. (2026). DFlare: Scaling up draft capacity for block diffusion speculative decoding. arXiv. https://doi.org/10.48550/arXiv.2606.02091

[7] Zhang, Z., Jiang, Z., Jiang, C., Yu, M., Zheng, S., Lin, H., Liu, X., & Hoffmann, H. (2026). SwiftSpec: Disaggregated speculative decoding and fused kernels for low-latency LLM inference. In Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (pp. 2197–2211). ACM. https://doi.org/10.1145/3779212.3790246

Questions for authors
  • Hardware requirements for drafting. Can PaPaSpec achieve comparable latency improvements when the drafter is allocated a smaller GPU or more limited computational resources, rather than a dedicated H100? What is the minimum hardware capacity required for the drafter to maintain meaningful performance gains?

  • Effectiveness of learned layer-specific mixing. How important are the learned layer-specific mixing parameters to the effectiveness of prefix- and depth-causal conditioning? Could comparable acceptance performance be achieved using fixed mixing weights or simpler depth-causal masking strategies?

  • Efficiency of intermediate-layer looping. Under a matched drafting compute budget, what advantages does intermediate-layer looping offer over simply using a deeper drafter without looping? Is the improvement attributable to the looping mechanism itself, rather than the additional computation allocated to drafting?

  • Sensitivity to inference backend optimization. Table 2 shows substantial performance differences between the two inference backends (HF and vLLM). Can PaPaSpec maintain comparable improvements over DFlash when both methods are evaluated using an optimized backend such as vLLM? In particular, reducing target verification latency through backend optimization also reduces the available window for overlapping drafting computation. To what extent do PaPaSpec's performance advantages persist under these conditions?

6 comments
AuthorsOpenReview ↗

Part I: Performance improvements with new conditioning, equalized hardware through CPU drafting

We thank reviewer QKNu for the insightful feedback, for recognizing that PaPaSpec addresses a ''key limitation of existing target-conditioned parallel drafters'' with a ''technically meaningful approach'', and for highlighting our extensive ablations. In the following, we address all raised questions and concerns.


W1-a & Q2: Improving performance gains with new conditioning

We thank the reviewer for pushing us to further investigate our prefix and depth-causal conditioning. Upon additional investigation, we made three improvements, ordered by impact level from high to low:

  • Block-wise looping. Looping the middle layers as a block, rather than each layer individually, further improves performance.
  • Full-block acceptance. We resolved the missing cache entry for full block acceptance by extending the drafted block by one token.
  • Tied mixing weights. Sharing the mixing weights across the drafter layers slightly improves the performance compared to our initial layer-specific mixing (see the ablation below).

Together, these changes yield an updated PaPaSpec with notably larger gains:

ModelDFlashD-PACEInitial PaPaSpecUpdated PaPaSpec
Qwen3-8B4.55×5.08×5.50× (+21% / +8%)5.98× (+31% / +18%)
Qwen3-4B4.31×4.79×5.23× (+21% / +9%)5.64× (+31% / +18%)
Gemma-4-26B-A4B-it3.04×3.42×3.57× (+17% / +4%)3.81× (+25% / +11%)

Average speedups over AR generation across all seven benchmarks in greedy decoding in compiled HF Transformers. Values in brackets show the improvement over DFlash / our own D-PACE baseline, which we now report for all target models.

The updated PaPaSpec roughly doubles its advantage over the strong D-PACE baseline.

Ablation on the mixing weights As mentioned above, we found that tying the mixing weights across drafter layers is slightly better than training separate mixing weights α\alpha and β\beta per-layer. Having no trainable mixing weights is slightly worse. While the differences are small, given this evidence, we now use the tied mixing weights as conditioning.

Mixing weights strategy# Mixing weightsAcceptance length
Per-layer α\alpha, β\beta (paper)505.20
Layer-tied α\alpha, β\beta105.23
No mixing weights05.18

As in our other ablations, we trained from scratch on the smaller 100k dataset.

Simpler depth-causal masking strategies We kindly ask the reviewer to elaborate on what they mean by ''simpler depth-causal masking strategies''. Having a down-projection is a central piece of the DFlash strategy and required to align the verifier representations with those of the drafter. Moreover, dropping the depth-causal conditioning and instead only conditioning on the token positions with fully available verifier hidden states leads to worse performance, as our ablation on the conditioning mechanism in the initial submission already showed (Figure 5, blue line vs green line). Similarly, using depth-causal conditioning also in the prefix drops the performance too (blue line vs orange line). We are happy to clarify this point further.


W1-b & Q1: Equalized hardware through CPU drafting

Thank you for raising this point. Indeed, we initially followed the established practice for research on SSD and used the symmetric setting of separate H100 GPUs for both drafting and inference (Kumar et al., 2026). However, in our updated paper (Sec. 4.3 and App. A.4), we also report on PaPaSpec with the verifier on an H100 GPU and the drafter on the available node CPUs (two 48-core Intel Xeon Platinum 8468). Given the substantial compute asymmetry, we ran the CPU-based drafting with a tailored C++ implementation using W8A8 quantization for the linear layers, reduced the cache to B=4B=4, and ran the last drafting layer and the LM head concurrently on the host GPU. As can be seen in the table below, PaPaSpec still improves on D-PACE by 3.1%, even without an additional GPU available.

MethodCache hit (%)Acceptance lengthSpeedupGain over D-PACE
D-PACE–7.003.39×–
PaPaSpec (B=32, 2 GPUs)85.37.203.81×+12.5%
PaPaSpec (B=4, CPU drafter)53.77.083.49×+3.1%

Drafting on the host CPU for Gemma-4-26B-A4B-it (greedy decoding, γ\gamma=16, 2048 new tokens, compiled HF Transformers). Averaged over the same seven benchmarks as in Table 1.

AuthorsOpenReview ↗

Part II: Comparison against SSD, efficiency of looping, and vLLM integration

W2: Comparisons against existing macro-parallel approaches

We agree with the reviewers' assessment and thank them for bringing this up. We followed their suggestion and now compare against standard SSD in a fully matched 2-GPU setup (with one NVIDIA H100 for the drafter and one NVIDIA H100 for the verifier) under the same compiled inference backend. After extensive tuning of their hyperparameters, SSD achieves a speedup of 2.29×2.29\times compared to the speedup of 5.98×5.98\times that PaPaSpec achieves. The reported numbers for SSD are consistent with reproductions in prior art. These results confirm the step improvement reported in the literature that target-conditioned micro-parallel drafters achieve over micro-sequential drafters. We thank the reviewer for pointing out the missing baseline and are glad to fill in the gap.

MethodGSM8KMATH-500AIME26HumanEvalMBPPLCBMT-BenchAvg.PaPaSpec gain
SSD (0.6B)2.452.452.362.302.252.301.872.29×+161%
SSD (1.7B)2.002.011.941.891.881.881.631.89×+216%
DFlash4.665.655.244.644.294.852.484.55×+31%
D-PACE5.226.416.015.054.745.352.755.08×+18%
PaPaSpec6.367.576.826.155.656.273.085.98×–

Speedup over AR generation on Qwen3-8B with greedy decoding.


Q3: Efficiency of looping

Great question! We can indeed make the drafter deeper to improve acceptance length while remaining hidden behind the verifier, as shown in our additional experiments in App. B.3. However, first, this increases the number of parameters of the drafter. Secondly, as we show in the table below, looping actually achieves slightly higher acceptance lengths than a deeper model (see Seq. τ\tau column). Thirdly, in case of a cache miss, we fall back to macro-sequential decoding. A looped model allows us to dynamically decide on the depth of the model, depending on whether the drafter is on the critical path (e.g., after a cache miss) or not, yielding the overall highest speedup in macro-parallel drafting.

DrafterDraft layersParams (B)Passes/draftSeq. τ\tauSeq. draft (ms/step)Seq. speedupPar. τ\tauPar. miss draft (ms)Par. speedup
PaPaSpec (1 loop)51.0555.452.233.53×5.441.884.20×
PaPaSpec (2 loops)51.0585.513.003.32×5.521.884.24×
8-layer81.6385.532.953.39×5.522.614.23×
PaPaSpec (3 loops)51.05115.703.773.24×5.661.884.36×
11-layer112.21115.623.753.22×5.623.364.26×

Experiments on Qwen3-8B. All models are trained on the 100k ablation dataset. Seq.: macro-sequential drafting; Par.: macro-parallel drafting.


Q4: vLLM integration

Thank you for pushing us on this. We have now implemented PaPaSpec into vLLM, which has significantly lifted the throughput of our system reported in Sec. 4.3, Table 2, and Table 6 (see results on Qwen3-8B below). As the reviewer anticipated, the faster verification shrinks the window for hiding drafting: we reduce the cache budget from 32 to 16, and the gains decrease from +30% to +24% over DFlash and from +18% to +10% over D-PACE on Qwen3-8B. Nonetheless, PaPaSpec remains the fastest method on the optimized backend:

MethodMATH-500 tok/sMATH-500 τ\tauHumanEval tok/sHumanEval τ\tauMT-Bench tok/sMT-Bench τ\tauAvg. tok/sAvg. τ\tauGain of PaPaSpec
AR1481.001491.001481.001481.00–
DFlash9208.027586.494004.256936.25+24%
D-PACE10468.738517.054424.567806.78+10%
PaPaSpec11538.879447.224764.718586.94–

Decoding throughput and acceptance length τ\tau on vLLM for Qwen3-8B (greedy decoding, 128 prompts per benchmark, 2048 maximum new tokens, H100). The gains also translate to Qwen3-4B and Gemma-4-26B-A4B.

AuthorsOpenReview ↗

Part III: Novelty

W3: Technical novelty

The key insight of PaPaSpec is that target latents become available progressively during verification. Prior latent-conditioned drafters (such as DFlash, P-EAGLE, or DFlare) all require the target's full forward pass before drafting can begin, while prior macro-parallel methods (SSD, Mirror-SD) use micro-sequential drafters without target-latent conditioning. Building on this insight, we contribute:

  • Prefix and depth-causal conditioning, which enables tight target conditioning for drafting during verification.
  • Looping to spend the freed compute budget, raising the acceptance length at no additional parameters or critical-path latency; to our knowledge, the first use of looping in speculative drafters.
  • Confidence-based cache allocation, which reduces cache misses in general SSD frameworks.
  • NEW: Anchor-free drafting, which removes cache misses entirely and enables batched serving (see responses to jHiu and NBs8).

We evaluate these on compiled Hugging Face Transformers, with the drafter on a GPU or (NEW) a CPU, and (NEW) on vLLM, for both single and (NEW) concurrent requests, with SoTA performance in all settings.


With the additional clarification and results, we hope to have resolved the reviewer's concerns about the presentation, informativeness, and soundness of our method, and to have convinced them to recommend acceptance.

Reviewer_jHiuOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PaPaSpec, a speculative decoding framework that aims to eliminate both micro-level and macro-level sequentiality in speculative decoding. The key idea is to execute a target-latent-conditioned parallel drafter concurrently with target-model verification. Since conventional DFlash requires hidden states from multiple target layers that are only available after verification completes, the authors introduce prefix and depth-causal conditioning, allowing each drafter layer to consume only target latents that have already been computed. The paper further exploits the resulting slack by looping intermediate drafter layers to improve draft quality without increasing the number of parameters. PaPaSpec is integrated with SSD, together with a confidence-based dynamic allocation of the drafting cache budget.

Critical strengths

The paper addresses an important systems bottleneck in speculative decoding. Existing latent-conditioned parallel drafters such as DFlash improve micro-level parallelism but still place drafting on the critical path. The solution of prefix and depth-causal conditioning mechanism is technically well motivated. The ablations provide useful evidence for the proposed components, clearly demonstrating the effectiveness of intermediate-state usage and layer looping.

Critical weaknesses
  1. The experiments were conducted on single-batch serving, while the conditioning and looping strategies could differ under larger batch sizes. For example, when batch size is large, the effectiveness of looping may degrade (similar to the case in Fig.6 (a)). Therefore, it is unclear whether the reported gains persist under larger batch sizes and continuous/concurrent serving.
  2. The method requires a separated device or computation resource to execute the draft model, which may limit its applicability to distributed inference systems with symmetric device utilization. In many distributed inference systems, GPUs are symmetrically utilized for tensor, pipeline, data, or expert parallelism rather than reserving one device for a separate drafter.
  3. The effectiveness of SSD lies largely in sufficient token budget, as the verification has not been obtained. Therefore, whether the effectiveness remains when batch size increases is unclear.
Questions for authors
  1. How does PaPaSpec perform with larger batch sizes and concurrent requests?
  2. Can PaPaSpec be deployed efficiently when all GPUs are already occupied by symmetric distributed inference?
  3. Is the number of looping iterations dynamically configurable at inference time? Does it require training another drafter?
4 comments
AuthorsOpenReview ↗

Part I: Batched serving, CPU drafting

We thank reviewer jHiu for the insightful feedback, for recognizing the importance of our contribution, the technical appropriateness of our solution, and the extensive ablations on our system. In the following, we address all raised questions and concerns.


W1 & W3 & Q1: Batched serving

Thank you for pointing out this missing piece. While speculative decoding is most effective at low batch sizes, we have now adapted PaPaSpec to batched serving (Sec. 4.3 and App. A.3 in the updated paper). Since a single cache miss stalls the entire batch, the probability of a fully macro-parallel step decays exponentially with the number of concurrent requests cc. In our revision, we therefore fine-tune PaPaSpec to also draft without an anchor token, using a trainable placeholder [P][P] instead, which guarantees a 100% cache hit rate. An optional lightweight correction head connects the eventual bonus token with the anchor-free draft. The same drafter still supports exact anchors, which we use to augment the cache with B′B' exact-anchor drafts.

InputMethodc=1c=2c=4c=8c=16c=32c=64
1kAR174280441702108315972365
1kD-PACE (16)2704427161156187828713037
1kD-PACE (8)2844597391198192031134132
1kPaPaSpec ([P])2984757471183189929433957
1kPaPaSpec ([P]+head)3084877691233195330664175
1kGain of PaPaSpec+8.2%+6.2%+4.1%+2.9%+1.7%−1.5%+1.0%
2kAR164278426694103315622315
2kD-PACE (16)2634276861104180226862580
2kD-PACE (8)2724457041130186828263653
2kPaPaSpec+[P] (8)2914677381189187426813604
2kPaPaSpec+[P]+head (8)3024807631219191827613750
2kGain of PaPaSpec+11.1%+8.0%+8.3%+7.9%+2.7%−2.3%+2.7%

Output throughput (tok/s) for concurrent serving with vLLM on Gemma-4-26B-A4B-it (SPEED-Bench throughput splits, 1536 requests each, greedy decoding, H100).

PaPaSpec improves throughput over AR and D-PACE up to c=16c=16 concurrent requests (up to +11% speedup improvement), with larger gains for longer inputs, as the long-context drafting attention is hidden behind verification. Even without the autoregressive correction head, it outperforms D-PACE up to c=4c=4 (1k) and c=16c=16 (2k). From c=32c=32, the H100 becomes compute-bound, and PaPaSpec performs on par with D-PACE (see point 6 in the general response).

Regarding looping (Fig. 6a): looping indeed makes the drafter compute-bound earlier. Loops can be reduced at inference time, but this lowers the acceptance length, particularly with the bonus token placeholder. At high concurrency, we instead reduce drafting compute by applying the LM head only after verification, computing only cc instead of c(γ+1+B′)c(\gamma+1+B') blocks. Pruning drafts even during verification, e.g., by estimating the acceptance length from the target latents, is a promising direction for future work.


W2 & Q2: Asymmetric verification and drafting resources

We thank the reviewer for pushing us on this. Indeed, we initially followed the established practice for research on SSD and used the symmetric setting of separate H100 GPUs for both drafting and inference (Kumar et al., 2026). However, in our updated paper (Sec. 4.3 and App. A.4), we also report on PaPaSpec with the verifier on an H100 GPU and the drafter on the available CPU node (two 48-core Intel Xeon Platinum 8468). Given the substantial compute asymmetry, we ran the CPU-based drafting with a tailored C++ implementation using W8A8 quantization for the linear layers, reduced the cache to B=4B=4, and ran the last drafting layer and the LM head concurrently on the host GPU. As can be seen in the table below, PaPaSpec still improves on the SoTA by 3.1%, even without an additional GPU available.

MethodCache hit (%)Acceptance lengthSpeedupGain over D-PACE
D-PACE–7.003.39×–
PaPaSpec (B=32, 2 GPUs)85.37.203.81×+12.5%
PaPaSpec (B=4, CPU drafter)53.77.083.49×+3.1%

Drafting on the host CPU for Gemma-4-26B-A4B-it (greedy decoding, γ\gamma=16, 2048 new tokens, compiled Hugging Face Transformers). Averaged over the same seven benchmarks as in Table 1.

AuthorsOpenReview ↗

Part II: Loop configurations at inference time

Q3 Configuring the number of loops at inference time

Yes, the number of loops is configurable at inference time without training another drafter. We already leverage this to reduce the latency on the critical path: PaPaSpec runs 3 loops in macro-parallel mode and does not loop in macro-sequential mode (e.g., after a cache miss). Our new ablation in App. B.3 shows that the drafter also generalizes to 2 loops, which it never saw during training (it was trained with 1 and 3 loops only): the acceptance length increases monotonically from 5.45 (1 loop) to 5.51 (2 loops) and 5.70 (3 loops). Moreover, at the same number of layer passes, looping matches or exceeds a deeper drafter without looping (5.51 vs. 5.53 at 8 passes, 5.70 vs. 5.62 at 11 passes) while using fewer parameters and offering more flexibility.


With the additional clarification and results, we hope to have resolved the reviewer's concerns about the soundness of our method and to have reaffirmed their decision to recommend acceptance.

Thank you for the detailed response, which solved my concerns about larger batch size serving, device asymmetric occupation and looping configurations. I think the CPU drafting design could indeed be a bottleneck as model scales or serving pressure increases, making it the bottleneck in so-called 'parallel-parallel' paradigm. I will maintain my original score as acceptance, but I suggest this point be examined by other reviewers and the AC.

AuthorsOpenReview ↗

Deployment scenarios for PaPaSpec

We thank the reviewer for acknowledging our new results on larger batch sizes, asymmetric device occupation, and looping configurations. We are glad that they maintain their positive assessment, and we thank them for the thoughtful remark on CPU drafting.

We agree that the host CPU is not the drafting platform for every deployment scale, and we see our CPU experiment as a proof of concept: even modest, otherwise idle compute next to the verifier suffices for PaPaSpec to improve upon D-PACE, particularly in low-concurrency, latency-sensitive settings.

PaPaSpec itself is agnostic to the drafting device, and we see three regimes:

  • Limited resources: The host CPU, as we demonstrate in the rebuttal. The cache budget adapts at runtime, so the drafter degrades gracefully by reducing B at increasing demand.
  • Higher serving pressure: A GPU drafter, as in our main experiments. As discussed in point 6 of the general response, compute continues to grow faster than memory bandwidth on future devices. This matches the compute-heavy demand of PaPaSpec, e.g., for large cache budgets or in batched serving.
  • Dedicated drafting accelerators: Finally, we want to emphasize dedicated drafting accelerators, such as the Groq 3 LPX deployed alongside NVIDIA Vera Rubin GPUs, now in production. Macro-sequential drafters cannot benefit from such accelerators, as drafting remains on the critical path, whereas PaPaSpec hides it behind verification.

We also want to remark that the drafter size grows much more slowly than the target size, or even remains constant. For example, DFlash for Qwen3.5-397B-A17B has 1B parameters, about the same size as for Qwen3-8B. Hence, the drafting compute scales with the drafter rather than the target, and larger targets even give the drafter more time, as verification takes longer.

We thank the reviewer again for the positive assessment and all the valuable feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global response and summary of revision

We thank all reviewers for their constructive feedback and for recognizing the strong motivation behind combining micro- and macro-parallel drafting, our prefix and depth-causal conditioning mechanism, and our extensive ablations. Below is a summary of the new experiments and revisions included in our updated paper and individual responses:


1. vLLM integration (QKNU): We integrated PaPaSpec into vLLM, significantly lifting the overall throughput. For example, Qwen3-8B's average throughput increased from 718 to 858 tokens/s. Importantly, PaPaSpec's gain remains at 16-24% against DFlash, and 5-10% against our own strong D-PACE baseline.

2. Batched serving (jHiu, NBs8): We now also cover batched serving on vLLM. To prevent cache misses, we extended PaPaSpec to be able to draft without a bonus token. By fine-tuning our checkpoint to operate without bonus tokens, we eliminate speculative cache misses and sequential fallbacks, achieving 1.7-11.1% speedup gains against D-PACE up to a batch size of 16 on H100 GPUs (before entering the compute-bound regime).

3. Heterogeneous Single-GPU verification + CPU drafting (jHiu, NBs8, QKNU): While previous symmetric 2-GPU setups suit extreme latency-critical applications, we now also evaluate an asymmetric setup with the verifier on a single H100 GPU and the drafter offloaded to two 48-core CPUs. PaPaSpec continues to outperform existing SoTA methods by exploiting spare compute heterogeneously, without requiring an additional second GPU.

4. Matched 2-GPU SSD baseline & related work (NBs8, QKNU): Under a matched 2-H100 GPU setup and compiled inference backend, extensively tuned Speculative Speculative Decoding (SSD; Kumar et al., 2026) achieves a 2.29x speedup compared to PaPaSpec's 5.98x, demonstrating the necessity of target-conditioned micro-parallel drafting for SoTA performance.

5. Updated architecture, ablations & clarifications (jHiu, NBs8, QKNU):

  • Updated architecture: Tying the mixing weights across drafter layers slightly improves acceptance length. Together with block-wise looping and extended block drafting that covers full-block acceptance, the updated PaPaSpec raises the speedup gains to 19–31% over DFlash (previously 13-22%) and to 9–18% over D-PACE (previously 4-9%), a baseline we now report for all target models.
  • Looping vs. deeper drafters: A new ablation shows that, at the same number of layer passes, looping matches or slightly outperforms static deeper drafters with up to 2.1×2.1\times more parameters in acceptance length. Moreover, looping allows dynamic inference-time depth adjustment (e.g., 3 loops during macro-parallel execution vs. 1 loop on the critical path), which yields the highest overall macro-parallel drafting speedups. Beyond PaPaSpec, this suggests looping as a parameter-efficient way to scale the acceptance length of general micro-parallel drafters.
  • Prefix consistency: We clarified in Section 3.1 that branch (τ,b)(\tau, b) conditions strictly on causal target latents up to n+τ−1n+\tau-1 via attention masking, ensuring zero mismatch between drafting and verification.

6. Hardware scaling context (jHiu, NBs8, QKNU): Finally, we highlight an important hardware trend contextualizing our results. All empirical evaluations in our paper are conducted on NVIDIA H100 SXM GPUs (3.35 TB/s3.35\text{ TB/s} memory bandwidth; ∼2,000 TFLOPS\sim 2,000\text{ TFLOPS} FP8 compute), the GPUs available to us. Newer architectures continue the historical scaling laws introduced by Wulf & McKee (1995) in their seminal ''Memory Wall'' paper, where compute outpaces memory bandwidth: on its direct 8-GPU server successor, NVIDIA HGX Rubin NVL8, per-GPU HBM4 bandwidth scales by 4.85×4.85\times (16.25 TB/s16.25\text{ TB/s}), whereas dense peak compute scales by 8.1×8.1\times in FP8 (16,250 TFLOPS16,250\text{ TFLOPS}) and 16.9×16.9\times in NVFP4 (33,750 TFLOPS33,750\text{ TFLOPS}). Because standard autoregressive decoding is memory-bandwidth-bound, PaPaSpec is explicitly designed to trade abundant, otherwise-idle compute for lower latency without incurring additional memory-bus traffic. As newer hardware further widens the gap between available FLOPs and memory bandwidth, PaPaSpec's extra compute from speculative speculative drafting becomes increasingly inexpensive relative to memory transfers, making our approach even more effective on next-generation accelerators.


We thank the reviewers for the extensive time and effort put into their reviews and look forward to a fruitful discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.