acceptodds
Under review as a conference paper at ICLR 2027

H-Spec: Parallel Speculative Decoding Without a Drafter-Side KV Cache

37 views

Abstract

Block diffusion drafters such as DFlash and DSpark reduce drafting latency in speculative decoding by predicting multiple future tokens in parallel. These drafters condition their prediction on the target model by projecting target hidden states at every input position into a separate drafter-side KV cache. This drafter-side cache consumes extra GPU memory per request, reducing maximum concurrency. Maintaining separate caches can also lead to fragmentation or padding in paged serving systems, increasing memory waste. Direct target KV reuse alone eliminates this extra cache but reduces draft quality at later block positions. We propose a hybrid target-context injection method that complements direct target KV reuse with target hidden states only at the last input position, requiring no separate drafter-side KV cache. Based on this design, we propose H-Spec, a hybrid Mamba-attention parallel drafter. Mamba modules are initialized with projected last-token target hidden states, while attention modules reuse target KVs in place. Despite its recurrent formulation, Mamba's parallel scan allows H-Spec to preserve block-parallel drafting. Across three targets and three workloads, H-Spec achieves 5.1-17.3% higher peak throughput across the evaluated concurrency levels and 4.3-24.6% lower average KV utilization than the best baseline for each metric. Despite having no drafter-side KV cache, H-Spec preserves draft quality, improving mean accepted length by 5.0-13.3% over the best baseline.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Speculative Decoding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_4RuWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

H-Spec proposes a parallel speculative decoder that eliminates the separate drafter-side KV cache while leaving the target model unchanged. It initializes Mamba-2 modules using the target’s last-token hidden states and directly reuses target KVs through attention.

The main contribution is hybrid context injection. By combining a compact prefix summary with position-wise context to retain draft quality without maintaining another KV cache. The paper identifies the memory and cache management costs of existing drafters and shows that direct KV reuse alone reduces acceptance at later draft positions. The accompanying Mamba attention architecture implements this combination.

Critical strengths

1.H-Spec eliminates the separate drafter-side KV cache while preserving parallel drafting and improving draft acceptance. This offers a useful way to reduce serving memory overhead without modifying the target model.

2.Controlled ablations show that last-token hidden states and target KVs provide complementary benefits. Together with throughput improvements across three targets and multiple workloads, these results support the effectiveness and practical relevance of hybrid context injection.

Critical weaknesses

1.The paper motivates removing the drafter KV cache as a way to improve concurrency, but does not demonstrate that baselines reach a memory limited concurrency ceiling or that H-Spec raises it. This leaves the practical capacity benefit underlying the motivation unclear.

2.The authors provide a reasonable intuition for choosing Mamba, but do not compare it with simpler fusion architectures given the same contextual information. Consequently, the evidence does not sufficiently establish whether the benefits justify the added complexity of Mamba.

Questions for authors

1.How does H-Spec compare with a simpler attention-based drafter that receives the same last-token hidden states and target KVs through residual or gated fusion? Such a comparison would clarify the benefit of Mamba beyond the additional contextual information.

3 comments
AuthorsOpenReview ↗

Authors' rebuttal

Dear reviewer 4RuW,

Thank you for your positive review. We appreciate your recognition of the effectiveness and practical relevance of hybrid context injection. We also appreciate your suggestions to evaluate memory-limited concurrency and compare Mamba with a gated-fusion variant. We have added both experiments and will incorporate them into the revised version.

W1. Concurrency/throughput when memory-bound.

We add a case study where long context saturates KV cache memory. We use the longQA task from Fig. 8, where requests are grouped into 2K/4K/8K/16K/32K bins by prompt length. For each bin, we submit all 900 requests at once, so maximum concurrency is limited by how many requests fit on the A100 80G GPU. We report the max achieved concurrency and throughput for DSpark and H-Spec drafters on Qwen3 4B and 8B targets:

Max achieved concurrency, all requests submitted at once:

TargetMethod2K4K8K16K32K
Qwen3-4BDSpark15789472412
H-Spec176103532713
Qwen3-8BDSpark13776412110
H-Spec16388462312

Throughput (tok/s):

TargetMethod2K4K8K16K32K
Qwen3-4BDSpark1761.61092.8651.6330.1139.8
H-Spec1886.11162.0678.3345.1145.0
Qwen3-8BDSpark1284.6839.7521.1268.9110.6
H-Spec1432.3909.2545.3278.1114.2

For each prompt-length bin, H-Spec can hold more concurrent requests (+8.3% to +20.0%) than DSpark, as eliminating the drafter KV cache leaves more serving headroom. Throughput increases by +3.3% to +11.5%. This case study directly demonstrates the benefit of eliminating the drafter KV cache when GPU memory limits serving capacity.

W2/Q1. Mamba ablation.

We evaluate the ablation variant suggested by the reviewer: a drafter with five attention-MLP layers, each reusing one target layer’s KV cache; a gated fusion combines last-token hidden states from selected target layers with each draft layer’s input. We train this new ablation variant under a matched recipe and compare it with the Attention-only and Hybrid (Mamba+attention) variants shown in Fig.6b of the paper. The Markov head is excluded from all three variants. Note that Attention-only and Hybrid have approximately matched parameter counts, and adding gated fusion increases the trainable parameter count by 5.8% and 7.8% over hybrid across two target models.

Avg. MAL over augmented Spec-Bench tasks:

TargetAttention-onlyAttention + gated fusionHybrid (Mamba+attention)
Llama3.1-8B-IT2.652.903.00
Qwen3-8B2.642.983.05

Compared to the Attention-only variant, the gated fusion improves MAL by 9.4% and 12.9% on the two targets, suggesting that last-token hidden states provide useful contextual information independent of Mamba. Hybrid achieves slightly better MAL (+3.4% and +2.3%) than the gated-fusion variant despite having fewer trainable parameters, supporting the use of Mamba.

Thanks to the authors for their timely response and thorough experiments. The attention + gated fusion experiments further demonstrate that last-token hidden states provide useful context for the drafter, but weaken the role of Mamba. The authors could further elaborate on the throughput & latency of attention + gated fusion versus attention + Mamba. I tend to keep the Weak Accept score. For AC: I believe the paper's quality is higher than weak acceptance, but not to the extent of clear acceptance.

AuthorsOpenReview ↗

Throughput and drafting latency for hybrid vs. attention+gated fusion

Dear reviewer 4RuW,

Thank you for the constructive follow-up. We compare throughput and drafting latency of Attention + gated fusion and Hybrid (Mamba+attention), using the same ablation checkpoints above, without the Markov head. We evaluate throughput on the full augmented Spec-Bench, with 80 requests per task and 640 requests total. We report average throughput over three repeats on a single A100 80G.

Throughput in tok/s over concurrency (columns):

TargetMethod1664128
Llama-3.1-8B-ITAttn + gated154020522057
Hybrid (Mamba+attention)1568 (+1.8%)2198 (+7.1%)2153 (+4.7%)
Qwen3-8BAttn + gated187424352371
Hybrid (Mamba+attention)1957 (+4.4%)2518 (+3.4%)2513 (+6.0%)

Hybrid achieves higher throughput across all evaluated settings, improving throughput by +1.8% to +7.1% on Llama-3.1-8B-IT and +3.4% to +6.0% on Qwen3-8B.

We also profile average drafting latency using the approach in Fig. 7: synthetic requests with 16 input tokens and 2,048 output tokens, with three waves per concurrency level.

Drafting latency in ms over concurrency (columns):

TargetMethod1664128
Llama-3.1-8B-ITAttn + gated4.229.6518.49
Hybrid (Mamba+attention)3.99 (−5.5%)8.84 (−8.4%)16.38 (−11.4%)
Qwen3-8BAttn + gated3.748.9317.52
Hybrid (Mamba+attention)3.48 (−7.0%)7.83 (−12.3%)14.92 (−14.8%)

Hybrid's advantage in drafting latency grows with concurrency, consistent with Fig. 7a’s comparison of Hybrid and Attention-only at matched parameter counts, which shows that Hybrid’s runtime advantage increases with concurrency.

Together with the previous MAL results, these results show that Hybrid achieves higher acceptance with fewer trainable parameters, lower drafting latency, and higher end-to-end throughput across concurrency levels than Attention + gated fusion. Additionally, H-Spec removes the drafter KV cache to address a memory bottleneck in concurrent serving. Mamba’s favorable runtime scaling with concurrency further supports its use in serving settings.

We thank the reviewer again for their time and engagement, and we hope this follow-up helps address their remaining concerns regarding the use of Mamba.

Reviewer_z6VVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

H-Spec is a parallel drafter that removes the persistent drafter-side KV cache. Attention reads target KVs in place, while Mamba-2 is initialized from last-position target hidden states. A DSpark Markov head supplies causal correction. Under matched training on Llama3.1-8B-Instruct and Qwen3-4B/8B, it reports higher acceptance and 5.1–17.3% higher peak vLLM throughput over the main baselines.

Critical strengths
  • Effective cache-free conditioning. Target KV reuse and the last-token summary remove the persistent drafter KV cache while improving acceptance, which is a practical improvement in industrial settings.
  • Controlled comparisons. Baselines share training data, optimizer steps, attention windows and vocabulary pruning, with closely matched parameter counts.
  • Broad evaluation. Results cover concurrent vLLM serving, sampling, long contexts and transfer to fine-tuned targets.
Critical weaknesses
  • The benefit under distributed KV storage is unclear. How much GPU memory, maximum concurrency and throughput does removing drafter KV save when KV is distributed or offloaded, accounting for active decode state and transfer costs?
  • Related work is incomplete. Please cite and distinguish GliDe [1], LongSpec [2] and KVShot [3], particularly their target-KV reuse, bounded drafter memory and hybrid hidden/KV conditioning.
  • Compatibility with prefill–decode disaggregation needs clarification. How are last-position hidden states transferred from prefill to decode, and how is in-place KV reuse implemented with paging and sharding?
  • Generalization beyond 4B/8B targets is unclear. Do the gains persist on newer, larger models trained with large-scale RL? The reported fine-tuned-target transfers do not resolve this question.

[1] Cunxiao Du et al. GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding. ICML, PMLR 235:11704–11720, 2024.

[2] Penghui Yang et al. LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verification. ACL, pp. 1826–1844, 2026.

[3] Tianyu Liu et al. When Hidden States Drift: Can KV Caches Rescue Long-Range Speculative Decoding? arXiv:2604.26412, 2026.

Questions for authors

See weaknesses above.

5 comments
AuthorsOpenReview ↗

Authors' rebuttal

Dear reviewer z6VV,

Thank you for your positive review. We are glad that you find eliminating drafter-side KVs a practical improvement in industrial settings. We also appreciate you finding our comparisons controlled and our evaluations broad. We have clarified H-Spec’s compatibility with KV offloading and PD disaggregation, relationships to previous work (GliDe, LongSpec, KVShot), and implementation details on target KV reuse under paging and sharding. We also evaluated the larger, more recent Qwen3.8-27B target. We will incorporate all your feedback into the revised paper and add the three suggested papers to the bibliography.

W1. Compatibility with distributed or offloaded KV store.

Systems for offloaded or distributed KV cache, e.g., Mooncake [1], LMCache [2], and vLLM’s OffloadingConnector [3], only evict idle KVs from the GPU; active KVs for requests in the running batch stay on the GPU. In one decoding step, H-Spec drafts k=7 tokens for each request in the running batch by reading those requests' target KVs in place. Therefore, prefix-cache offloading does not eliminate H-Spec’s memory benefit of removing the active drafter KVs.

GPU memory and concurrency: Active drafter and target KVs share the same GPU memory pool. Therefore, drafters with their own KV cache reduce the resident token capacity relative to no-speculative-decoding, while H-Spec avoids this extra memory consumption. We report the maximum token capacity from vLLM start-up logs, on an A100 80G GPU:

TargetMethodKV budget (GiB)Token capacity
Llama3.1-8B-ITDSpark54.58406,035
H-Spec53.51438,320
Qwen3-4BDSpark63.54408,759
H-Spec63.46462,112
Qwen3-8BDSpark54.63351,404
H-Spec54.54397,120

H-Spec increases resident token capacity by 8.0-13.1% over DSpark.

Throughput: We add a case study to measure serving performance when the running batch’s active KVs saturate GPU memory. We use the longQA task from Fig. 8, where requests are grouped into 2K/4K/8K/16K/32K bins by their prompt length. For each bin, we submit all 900 requests at once, so maximum concurrency is limited by how many requests fit on the A100 80G GPU. We report the max achieved concurrency and throughput for DSpark and H-Spec drafters on Qwen3 4B and 8B targets:

Max achieved concurrency, all requests submitted at once:

TargetMethod2K4K8K16K32K
Qwen3-4BDSpark15789472412
H-Spec176103532713
Qwen3-8BDSpark13776412110
H-Spec16388462312

Throughput (tok/s):

TargetMethod2K4K8K16K32K
Qwen3-4BDSpark1761.61092.8651.6330.1139.8
H-Spec1886.11162.0678.3345.1145.0
Qwen3-8BDSpark1284.6839.7521.1268.9110.6
H-Spec1432.3909.2545.3278.1114.2

For each prompt-length bin, H-Spec can hold more concurrent requests (+8.3% to +20.0%) than DSpark, as eliminating the drafter KV cache leaves more serving headroom. Throughput increases by +3.3% to +11.5%.

[1] MOONCAKE: Trading More Storage for Less Computation – A KVCache-centric Architecture for Serving LLM Chatbot. FAST’25.

[2] LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference. arXiv 2510.09665.

[3] vLLM KV offloading. https://docs.vllm.ai/en/stable/features/kv_offloading_usage/

W2. Related work suggestions.

We will add all suggested papers to the related work and bibliography of the revised paper.

GliDe uses cross-attention in the drafter to read target KVs while retaining drafter self-attention and its own KV cache. H-Spec focuses on eliminating the serving overhead of drafter KVs in block-parallel drafters. Section 3.1 explores target KV reuse as a replacement for traditional hidden-state-based KV injection. We find that this replacement hurts acceptance at later block positions, motivating the hybrid context injection.

LongSpec uses sliding-window attention (SWA) to bound the drafter KV cache to a 512-token window, but the serving system must still manage this separate cache. Our vLLM case study shows allocation padding for SWA DFlash and DSpark alongside a full-attention target (Appendix B, Table 5). H-Spec eliminates this cache entirely, avoiding its GPU memory cost and associated cache management complexities.

KVShot reports that target-KV-only conditioning lowers overall draft acceptance for EAGLE-3. Its hybrid variant combines target KVs and hidden states to improve acceptance, but retains the drafter-side prefix KV cache. H-Spec instead recovers acceptance without the drafter-side cache by combining in-place target-KV reuse with target hidden states only at the last input position through a Mamba-attention architecture.

AuthorsOpenReview ↗

Authors' rebuttal (part 2)

W3. Compatibility with PD disaggregation, paging, and sharding.

PD disaggregation: In vLLM’s disaggregated prefilling [1], the decode instance receives the prompt’s KV cache and runs one target-model forward pass over the last prompt token to sample the first output token. This first pass on the decoding instance generates the hidden states for the last prompt token. H-Spec directly reuses these hidden states along with target KVs to draft. In contrast, drafters with a KV cache require computing the drafter prompt KVs on the prefill instance and transferring them to the decoding instance [2, 3], adding network overhead.

Assuming one typical 200Gb/s link (25 GB/s) between prefill and decode nodes, and a 2,048-token prompt, the added drafter-KV transfer time for the Qwen3-8B target is:

MethodTarget KVDrafter KVEstimated added transfer time
DFlash / DSpark288 MiB40 MiB1.7 ms (+13.9%)
H-Spec288 MiB0 MiB0 ms (+0.0%)

Because active KVs are always resident on the decoder GPU, eliminating the drafter KV still saves decoder-side GPU memory under PD disaggregation.

Paging: Each H-Spec attention module reuses the KV cache of one target layer. vLLM supports a layer that declares another layer as its KV-sharing target [4]. Such a layer allocates no cache and directly reads the sharing target’s paged cache through the same block table as the target layer. H-Spec’s draft layers declare corresponding target-model layers as their sharing target through this mechanism. H-Spec writes K and V only for its draft block into the lookahead slots that vLLM reserves for speculative tokens, and the next verification pass overwrites them with the target’s actual KV.

Sharding: H-Spec matches the target’s attention head layout in order to reuse its KV.

  • Under tensor parallelism (TP), H-Spec’s attention modules are sharded the same way as the targets; each H-Spec shard therefore reads target KV on the same rank, and performs column-parallel Q/K/V projections and row-parallel output projection the same as the target. Mamba modules are not sharded: the last-token hidden states are already duplicated on every rank after target all-reduce, so each rank computes the full Mamba initial state locally.
  • Under pipeline parallelism (PP), co-locating H-Spec with selected target layers supplying KVs and hidden states on the same stage avoids cross-stage data transfer. These layers are selected before training, so their selection must account for the intended PP partition. Other drafters that condition on target hidden states also need to consider the PP partition when selecting input layers.

[1] vLLM Disaggregated Prefilling. https://docs.vllm.ai/en/latest/features/disagg_prefill/

[2] vLLM PR #35158. https://github.com/vllm-project/vllm/pull/35158

[3] vLLM NixlConnector Compatibility Matrix. https://docs.vllm.ai/en/latest/features/nixl_connector_compatibility/

[4] vLLM Hybrid KV Cache Manager. https://docs.vllm.ai/en/latest/design/hybrid_kv_cache_manager/#case-5-kv-sharing

W4. Generalization to larger targets.

We evaluate the larger and more recent Qwen3.8-27B target model. We train DSpark and H-Spec for 5 epochs over 100K randomly selected PerfectBlend samples, using target-generated responses with reasoning enabled.

Qwen3.8-27B has a smaller per-token KV footprint (64 KiB) than the full-attention targets evaluated in the paper (128–144 KiB), while DSpark’s per-token KV size remains 20 KiB across all targets. Therefore, the relative memory overhead introduced by the drafter KV cache becomes more significant.

Max token capacity reported by vLLM start-up logs for DSpark and H-Spec. The 4B/8B targets are served on one A100 80G. Qwen3.8-27B is served on two A100 80Gs under tensor parallelism, with the drafter and target attentions partitioned the same way.

TargetDSparkH-Spec
Llama3.1-8B-IT406,035438,320 (+8.0%)
Qwen3-4B408,759462,112 (+13.1%)
Qwen3-8B351,404397,120 (+13.0%)
Qwen3.8-27B1,046,9691,276,340 (+21.9%)

On Qwen3.8-27B, H-Spec increases maximum token capacity by 21.9% over DSpark, compared with 8.0%-13.1% on the full-attention targets.

MAL on augmented Spec-Bench:

MathQAChatRAGSumm.Transl.CodeToolAvg.
DSpark3.8432.4852.5522.9352.6392.6162.9322.5192.815
H-Spec3.9562.5702.6103.0902.7562.7393.0672.6102.925

H-Spec improves MAL over DSpark by +2.3-5.3% across subsets, and +3.9% on average, while increasing max token capacity.

Thank you for the detailed response, the additional experiments, and the revised manuscript, which address most of my questions well.

On Qwen3.8-27B (W4), achieving +21.9% higher token capacity while still improving MAL without a drafter KV cache is encouraging; reporting end-to-end serving throughput under TP=2 would make this result even more complete. Overall, I view the work positively for now while following the remaining discussion.

Reviewer_fGQgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

H-Spec is a parallel drafter for spec dec, which removes the drafter's KV. It init states from last token hidden states of the target model and make attn layers resue the KV. It also has a DSpark head. Compared to DFLash, DSpark and etc., H-spec has better tps and lower memory usage for KV cache.

Critical strengths
  1. Drafter KV cache is a well-known issue for spec dec serving. The problem is important, but there are little related works about it.
  2. The KV resue study is good, it demonstrates that the hidden states is necessary for the design
  3. The authors made a some ablation studies, which is interesting.
Critical weaknesses
  1. The idea of H-spec is to save kv mem, but I think in the paper's experiments setting, the KV is not the bottleneck here. I think the gain of tps is based on the higher MAL instead of the del of KV.
  2. In this setting, the KV overhead is not that much for the dense model which is evaluated. The author can find a more suitable model to eval, like Qwen-3.5 or something?
  3. The author said that the baselines are reproduced by their recipe. But it is better to compare with official DFlash or DSpark release ckpt. It can demonstrates better what the gap is.
Questions for authors
  1. What is the tps gain for long context with high user concurency?
  2. Is the reproduced baseline performs better or worse than the official ones?
4 comments
AuthorsOpenReview ↗

Authors’ rebuttal

Dear reviewer fGQg,

Thank you for your positive review. We appreciate your acknowledgment of our discussion on the complexities of heterogeneous KV cache management (page-size and access-pattern mismatches), as well as your positive assessment of our motivation and ablation experiments. Based on your suggestions, we have added a memory-bound case study, evaluations on the hybrid Qwen3.8-27B target, and comparisons of a stronger H-Spec checkpoint with official baselines. We will incorporate all your feedback into the revised paper.

W1/Q1. Throughput when memory-bound.

We add a case study where long context saturates KV cache memory. We use the longQA task from Fig. 8, where requests are grouped into 2K/4K/8K/16K/32K bins by prompt length. For each bin, we submit all 900 requests at once, so maximum concurrency is limited by how many requests fit on the A100 80G GPU. We report the max achieved concurrency and throughput for DSpark and H-Spec drafters on Qwen3 4B and 8B targets:

Max achieved concurrency, all requests submitted at once:

TargetMethod2K4K8K16K32K
Qwen3-4BDSpark15789472412
H-Spec176103532713
Qwen3-8BDSpark13776412110
H-Spec16388462312

Throughput (tok/s):

TargetMethod2K4K8K16K32K
Qwen3-4BDSpark1761.61092.8651.6330.1139.8
H-Spec1886.11162.0678.3345.1145.0
Qwen3-8BDSpark1284.6839.7521.1268.9110.6
H-Spec1432.3909.2545.3278.1114.2

For each prompt-length bin, H-Spec can hold more concurrent requests (+8.3% to +20.0%) than DSpark, as eliminating the drafter KV cache leaves more serving headroom. Throughput increases by +3.3% to +11.5%. This case study directly demonstrates the benefit of eliminating the drafter KV cache when GPU memory limits serving capacity.

W2. Evaluation on hybrid targets.

We evaluate the Qwen3.8-27B target model, which uses a memory-efficient hybrid architecture. We train DSpark and H-Spec for 5 epochs over 100K randomly selected PerfectBlend samples, using target-generated responses with reasoning enabled.

Qwen3.8-27B has a smaller per-token KV footprint (64 KiB) than the full-attention targets evaluated in the paper (128–144 KiB), while DSpark’s per-token KV size remains 20 KiB across all targets. Therefore, the relative memory overhead introduced by the drafter KV cache becomes more significant.

Max token capacity reported by vLLM start-up logs, for DSpark and H-Spec. The 4B/8B targets are served on one A100 80G. Qwen3.8-27B is served on two A100 80Gs under tensor parallelism, with the drafter and target attentions partitioned the same way.

TargetDSparkH-Spec
Llama3.1-8B-IT406,035438,320 (+8.0%)
Qwen3-4B408,759462,112 (+13.1%)
Qwen3-8B351,404397,120 (+13.0%)
Qwen3.8-27B1,046,9691,276,340 (+21.9%)

On Qwen3.8-27B, H-Spec increases maximum token capacity by 21.9% over DSpark, compared with 8.0%-13.1% on the full-attention targets.

MAL on augmented Spec-Bench:

MathQAChatRAGSumm.Transl.CodeToolAvg.
DSpark3.8432.4852.5522.9352.6392.6162.9322.5192.815
H-Spec3.9562.5702.6103.0902.7562.7393.0672.6102.925

H-Spec improves MAL over DSpark by +2.3-5.3% across subsets, and +3.9% on average, while increasing max token capacity.

AuthorsOpenReview ↗

Authors' rebuttal (part 2)

W3. Comparing H-Spec to official baseline checkpoints.

We trained H-Spec for one epoch over the full PerfectBlend dataset for Qwen3-4B and evaluated it alongside the official DFlash and DSpark checkpoints [1]. We report each checkpoint’s training data and epochs below.

H-Spec was trained using target-generated responses with reasoning on, while the official DFlash and DSpark checkpoints were trained with reasoning off. We evaluate all three checkpoints with reasoning both on and off at test time. All drafters predict 7 tokens per step. MAL is averaged over the augmented Spec-Bench tasks:

DrafterTraining dataEpochsMAL (reasoning off)MAL (reasoning on)
H-SpecPerfectBlend (1.3M)13.473.72
DFlashNemotron V2 + CodeAlpaca (800K)63.252.67
DSparkPerfectBlend (1.3M)104.153.39

H-Spec exceeds official DFlash in both test-time reasoning settings and official DSpark with reasoning on. Its MAL trails DSpark’s with reasoning off, with only 10% as many training epochs and a reasoning-mode mismatch between training and test time. Matching DSpark’s training conditions would require regenerating PerfectBlend responses with reasoning off and a 10-epoch training run. We estimate this training would cost around 3,300 H100 hours, or 17 days on an 8-GPU node, which we cannot complete during the rebuttal. We plan to continue this effort and release stronger H-Spec checkpoints as future steps.

[1] Hugging Face: z-lab/Qwen3-{4,8}B-DFlash-b16, deepseek-ai/dspark_qwen3_{4,8}b_block7

Q2. Comparing reproduced and official baselines.

The reproduced checkpoints were trained on reasoning-enabled responses, while official DFlash and DSpark were trained on larger datasets of reasoning-disabled responses. We verify that our reproduced DFlash and DSpark checkpoints preserve the official checkpoints’ relative MAL ranking. We evaluate official checkpoints with reasoning off and ours with reasoning on, matching their respective training-time reasoning settings.

Averaged MAL over augmented Spec-Bench:

TargetOfficial (reasoning off)Ours (reasoning on)
Qwen3-4BDSpark 4.15 > DFlash 3.25DSpark 3.09 > DFlash 2.86
Qwen3-8BDSpark 4.08 > DFlash 3.29DSpark 2.95 > DFlash 2.77

P-Eagle and DFlash-2 do not have official checkpoints for targets evaluated in the original submission. For Llama3.1-8B-IT, only DFlash has an official checkpoint. We therefore cannot include these cases in the check above.

Additionally, our reproduced DFlash-2 exceeds DSpark in MAL on Qwen3-8B (3.11 > 2.95), consistent with the trends reported by the DFlash-2 authors on other target models [1]. The original DFlash paper [2] also uses a 100K-sample training corpus to compare its ablation variants, matching the scale of our unified recipe.

[1] DFlash-2: Keep drafting parallel. https://inco.ai/blog/dflash2/

[2] DFlash: Block Diffusion for Flash Speculative Decoding. ICML'26.

Thanks authors. The 27B results is good. Actually most of my concerns are resolved. However, the weakness #3 is still open. I will maintain my score.

AuthorsOpenReview ↗

Authors' follow-up evaluations on W3

Dear reviewer fGQg,

We sincerely thank you for your time and for acknowledging our rebuttal results. Regarding W3, we added end-to-end throughput comparisons to a third-party DSpark checkpoint for the Qwen3.8-27B target model, released by RadixArk [1]. This baseline checkpoint has over 100K downloads in the last month (reported by Hugging Face), supporting its relevance in the community. As DeepSeek has never released DSpark for Qwen3.8-27B, we use RadixArk's release.

We trained a stronger H-Spec: two epochs over full PerfectBlend data (1.3M). We keep H-Spec’s architectural configurations the same as in the paper: four layers (including one partial Mamba-MLP layer), SWA, attention and MLP dimensions matched to the target model, and a pruned 64K-vocab LM head with about 25% of the target vocab size.

The RadixArk DSpark is larger in several components (+24.9% active parameters from a larger Markov head and a full-vocab LM head), uses full attention, and has slightly higher MALs (up to 2.4% across code/chat tasks and concurrency levels) than the H-Spec. Since MALs under different drafting costs do not establish serving gains, we report end-to-end throughput under concurrency. Both drafters propose 7 tokens per step and are served with thinking enabled on the target model, consistent with the DSpark checkpoint’s documented evaluation setting.

Throughput in tok/s over concurrency (columns) on Qwen3.8-27B target, TP=2 on two A100 80Gs, thinking-enabled:

DatasetDrafter48163264128
Chat (MT-Bench)RadixArk/Qwen3.8-27B-DSpark390.5704.51,022.31,216.91,254.71,251.1
H-Spec (PerfectBlend, 2 epochs)401.3 (+2.8%)728.1 (+3.3%)1,049.4 (+2.7%)1,244.8 (+2.3%)1,339.7 (+6.8%)1,340.1 (+7.1%)
Code (HumanEval)RadixArk/Qwen3.8-27B-DSpark410.4763.51,130.91,377.41,459.81,451.4
H-Spec (PerfectBlend, 2 epochs)435.7 (+6.2%)791.5 (+3.7%)1,161.7 (+2.7%)1,409.7 (+2.3%)1,514.2 (+3.7%)1,513.5 (+4.3%)

H-Spec improves end-to-end throughput across concurrency levels by +2.3% to +7.1% over RadixArk’s DSpark; peak throughput improved by +6.8% and +3.7% on Chat and Code, respectively. These end-to-end serving gains suggest that H-Spec achieves a more favorable tradeoff in drafting cost, MAL, and verification cost, all without needing a drafter-side KV cache.

For a large target model at 27B, verification cost contributes the most to the runtime (around 87% to 91% in our runs). Increasing H-Spec capacity for a potentially higher MAL (and thus fewer target-model forward passes) could further improve final throughput, and we leave this architecture tuning as future work.

We plan to continue this effort by scaling up H-Spec training, evaluating against more open-source baselines, and releasing all checkpoints with the paper to facilitate future research. We hope a comparison with a widely used third-party release could help resolve the reviewer’s remaining concern.

We thank the reviewer again for their time and engagement.

[1] https://huggingface.co/RadixArk/Qwen3.8-27B-DSpark

Comments on the paper

2 comments
AuthorsOpenReview ↗

Uploaded revised PDF

We thank the reviewers for their time and thoughtful reviews, and we are encouraged by their positive assessment of our work. Overall, reviewers find our problem important (fGQg "a real serving overhead", z6VV "a practical improvement in industrial settings"), our solution effective (z6VV "effective cache-free conditioning", 4RuW "a useful way to reduce serving memory overhead"), and our evaluation rigorous (fGQg "clean, with matched parameter counts", z6VV "broad evaluation", 4RuW "support the effectiveness and practical relevance").

We have uploaded a revised PDF that incorporates all reviewer feedback items. For convenience, changes are highlighted in red; new tables are numbered R1 to R9, so old table numbers remain unchanged.

  • Section 4.1: added a case study showing that H-Spec increases serving capacity in memory-bound settings. The accompanying new Appendix F.3 extends this case study to more tasks and target models (fGQg W1/Q1, z6VV W1, 4RuW W1).
  • Section 4.3: added an ablation that compares Mamba with a simpler gated-fusion variant, including both MAL and throughput, and a pointer to the drafter runtime scaling analysis (4RuW W2/Q1).
  • Section 5: discussed GliDe, LongSpec, and KVShot in the related work section, and clarified H-Spec’s differences from those methods (z6VV W2). We also added a constraint regarding drafter location under pipeline parallelism to the Limitations paragraph (z6VV W3).
  • Appendix D: discussed implementation details of H-Spec’s in-place target KV reuse under paged memory management (z6VV W3), and H-Spec’s compatibility with P/D disaggregation (z6VV W3), distributed or offloaded KV storage (z6VV W1), and model sharding (z6VV W3).
  • Appendix F.4: added maximum token capacity reported by vLLM at startup, as evidence for H-Spec’s increased serving capacity (fGQg W1/Q1, z6VV W1, 4RuW W1).
  • Appendix F.6: added MAL and throughput evaluation on the Qwen3.8-27B target, comparing H-Spec with reproduced DSpark under a unified training recipe (fGQg W2, z6VV W4).
  • Appendix F.9: scaled-up training and public checkpoint comparison (fGQg W3/Q2):
    • trained H-Spec over PerfectBlend and compared with public third-party DSpark releases (due to limited official releases), for Qwen3-4B and Qwen3.8-27B targets respectively (the Qwen3.8-27B throughput eval also covers z6VV W4).
    • comparisons to official DFlash/DSpark releases, and consistency checks between reproduced and official baselines.

We thank the reviewers for their constructive feedback, which helped us improve this work. We are happy to answer any additional questions from the reviewers.

AuthorsOpenReview ↗

(follow-up) new version updated 11/18

We have uploaded another revision in response to the reviewers' follow-ups. The previous list has been updated. Additional changes include:

  • Section 4.3: now includes throughput numbers and a pointer to the drafter runtime scaling analysis.
  • Appendix F.6: now includes throughput for reproduced DSpark and H-Spec for Qwen3.8-27B target
  • Appendix F.9: now includes comparisons with public DSpark releases on Qwen3-4B and Qwen3.8-27B.

We sincerely thank the reviewers for their time and engagement, and we are happy to answer any additional questions during the remaining rebuttal period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.