- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
H-Spec proposes a parallel speculative decoder that eliminates the separate drafter-side KV cache while leaving the target model unchanged. It initializes Mamba-2 modules using the target’s last-token hidden states and directly reuses target KVs through attention.
The main contribution is hybrid context injection. By combining a compact prefix summary with position-wise context to retain draft quality without maintaining another KV cache. The paper identifies the memory and cache management costs of existing drafters and shows that direct KV reuse alone reduces acceptance at later draft positions. The accompanying Mamba attention architecture implements this combination.
Critical strengths
1.H-Spec eliminates the separate drafter-side KV cache while preserving parallel drafting and improving draft acceptance. This offers a useful way to reduce serving memory overhead without modifying the target model.
2.Controlled ablations show that last-token hidden states and target KVs provide complementary benefits. Together with throughput improvements across three targets and multiple workloads, these results support the effectiveness and practical relevance of hybrid context injection.
Critical weaknesses
1.The paper motivates removing the drafter KV cache as a way to improve concurrency, but does not demonstrate that baselines reach a memory limited concurrency ceiling or that H-Spec raises it. This leaves the practical capacity benefit underlying the motivation unclear.
2.The authors provide a reasonable intuition for choosing Mamba, but do not compare it with simpler fusion architectures given the same contextual information. Consequently, the evidence does not sufficiently establish whether the benefits justify the added complexity of Mamba.
Questions for authors
1.How does H-Spec compare with a simpler attention-based drafter that receives the same last-token hidden states and target KVs through residual or gated fusion? Such a comparison would clarify the benefit of Mamba beyond the additional contextual information.
3 comments
Thanks to the authors for their timely response and thorough experiments. The attention + gated fusion experiments further demonstrate that last-token hidden states provide useful context for the drafter, but weaken the role of Mamba. The authors could further elaborate on the throughput & latency of attention + gated fusion versus attention + Mamba. I tend to keep the Weak Accept score. For AC: I believe the paper's quality is higher than weak acceptance, but not to the extent of clear acceptance.
Throughput and drafting latency for hybrid vs. attention+gated fusion
Dear reviewer 4RuW,
Thank you for the constructive follow-up. We compare throughput and drafting latency of Attention + gated fusion and Hybrid (Mamba+attention), using the same ablation checkpoints above, without the Markov head. We evaluate throughput on the full augmented Spec-Bench, with 80 requests per task and 640 requests total. We report average throughput over three repeats on a single A100 80G.
Throughput in tok/s over concurrency (columns):
| Target | Method | 16 | 64 | 128 |
|---|---|---|---|---|
| Llama-3.1-8B-IT | Attn + gated | 1540 | 2052 | 2057 |
| Hybrid (Mamba+attention) | 1568 (+1.8%) | 2198 (+7.1%) | 2153 (+4.7%) | |
| Qwen3-8B | Attn + gated | 1874 | 2435 | 2371 |
| Hybrid (Mamba+attention) | 1957 (+4.4%) | 2518 (+3.4%) | 2513 (+6.0%) |
Hybrid achieves higher throughput across all evaluated settings, improving throughput by +1.8% to +7.1% on Llama-3.1-8B-IT and +3.4% to +6.0% on Qwen3-8B.
We also profile average drafting latency using the approach in Fig. 7: synthetic requests with 16 input tokens and 2,048 output tokens, with three waves per concurrency level.
Drafting latency in ms over concurrency (columns):
| Target | Method | 16 | 64 | 128 |
|---|---|---|---|---|
| Llama-3.1-8B-IT | Attn + gated | 4.22 | 9.65 | 18.49 |
| Hybrid (Mamba+attention) | 3.99 (−5.5%) | 8.84 (−8.4%) | 16.38 (−11.4%) | |
| Qwen3-8B | Attn + gated | 3.74 | 8.93 | 17.52 |
| Hybrid (Mamba+attention) | 3.48 (−7.0%) | 7.83 (−12.3%) | 14.92 (−14.8%) |
Hybrid's advantage in drafting latency grows with concurrency, consistent with Fig. 7a’s comparison of Hybrid and Attention-only at matched parameter counts, which shows that Hybrid’s runtime advantage increases with concurrency.
Together with the previous MAL results, these results show that Hybrid achieves higher acceptance with fewer trainable parameters, lower drafting latency, and higher end-to-end throughput across concurrency levels than Attention + gated fusion. Additionally, H-Spec removes the drafter KV cache to address a memory bottleneck in concurrent serving. Mamba’s favorable runtime scaling with concurrency further supports its use in serving settings.
We thank the reviewer again for their time and engagement, and we hope this follow-up helps address their remaining concerns regarding the use of Mamba.
Authors' rebuttal
Dear reviewer 4RuW,
Thank you for your positive review. We appreciate your recognition of the effectiveness and practical relevance of hybrid context injection. We also appreciate your suggestions to evaluate memory-limited concurrency and compare Mamba with a gated-fusion variant. We have added both experiments and will incorporate them into the revised version.
W1. Concurrency/throughput when memory-bound.
We add a case study where long context saturates KV cache memory. We use the
longQAtask from Fig. 8, where requests are grouped into 2K/4K/8K/16K/32K bins by prompt length. For each bin, we submit all 900 requests at once, so maximum concurrency is limited by how many requests fit on the A100 80G GPU. We report the max achieved concurrency and throughput for DSpark and H-Spec drafters on Qwen3 4B and 8B targets:Max achieved concurrency, all requests submitted at once:
Throughput (tok/s):
For each prompt-length bin, H-Spec can hold more concurrent requests (+8.3% to +20.0%) than DSpark, as eliminating the drafter KV cache leaves more serving headroom. Throughput increases by +3.3% to +11.5%. This case study directly demonstrates the benefit of eliminating the drafter KV cache when GPU memory limits serving capacity.
W2/Q1. Mamba ablation.
We evaluate the ablation variant suggested by the reviewer: a drafter with five attention-MLP layers, each reusing one target layer’s KV cache; a gated fusion combines last-token hidden states from selected target layers with each draft layer’s input. We train this new ablation variant under a matched recipe and compare it with the Attention-only and Hybrid (Mamba+attention) variants shown in Fig.6b of the paper. The Markov head is excluded from all three variants. Note that Attention-only and Hybrid have approximately matched parameter counts, and adding gated fusion increases the trainable parameter count by 5.8% and 7.8% over hybrid across two target models.
Avg. MAL over augmented Spec-Bench tasks:
Compared to the Attention-only variant, the gated fusion improves MAL by 9.4% and 12.9% on the two targets, suggesting that last-token hidden states provide useful contextual information independent of Mamba. Hybrid achieves slightly better MAL (+3.4% and +2.3%) than the gated-fusion variant despite having fewer trainable parameters, supporting the use of Mamba.