- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper highlights the growing share of verification cost in speculative decoding (SD) under two conditions: (1) parallel drafters such as DFlash [R1] and DSpark [R2] reduce drafting cost, and (2) hybrid models combining GDN and MoE incur higher verification costs.
The authors propose HybridSpec, a two-stage verification framework combining draft-side selection and routing correction. Stage 1 selects a verification length that maximizes throughput based on estimated draft-token acceptance probabilities and verification cost. It also executes one additional token up to the first MoE router. Stage 2 updates the cost estimate based on the number of new experts activated by this token and decides whether to complete its execution.
To reduce intermediate-state storage, HybridSpec stores compact factors instead of full per-token GDN states and reconstructs only the required states after Stage 2.
[R1] Chen, J., Liang, Y., & Liu, Z. (2026). Dflash: Block diffusion for flash speculative decoding. arXiv preprint arXiv:2602.06036.
[R2] Cheng, X., Yu, X., Shao, C., Li, J., Xiong, Y., Qian, Y., ... & Liang, W. (2026). DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation. arXiv preprint arXiv:2607.05147.
Critical strengths
S1. The paper identifies that execution overhead in hybrid GDN–MoE models is more sensitive to verification length than in dense models. This is a timely observation. S2. Tentatively executing one additional token and revisiting the decision at the routing stage is an interesting idea that exploits the properties of GDN computation.
Critical weaknesses
W1. Limited novelty [+0.4 if resolved]
The paper should clearly distinguish its original contributions from prior work.
Relative to existing methods that adapt verification length to improve SD throughput [R2–R7], the paper should identify new findings or clearly establish its methodological novelty and scope.
Storing decomposed representations instead of full GDN states to reduce memory overhead appears to draw on prior work [R8-R12]. To establish this as a contribution, the authors should explain the challenges of integrating it into their framework and demonstrate a compelling solution.
W2. Component validity [+0.4 if resolved]
The paper should more clearly demonstrate whether each component works as intended.
In particular, it should evaluate how accurately draft-token probabilities estimate actual acceptance probabilities under the target model. For example, if DFlash is trained for greedy decoding and assumes sharply peaked token distributions, are its token probabilities unbiased estimators of acceptance probabilities? Analysis and experiments should establish when and how bias arises.
*W3. Limited evaluation [+0.4 if resolved]
Batched decoding is evaluated only at batch size 4, leaving batch-scaling behavior unclear.
This is particularly important because larger batches change expert overlap across requests and execution parallelism. These changes may affect both routing correction and the relationship between expert count and latency. The paper should evaluate each component across multiple batch sizes and explain the underlying causes and relationships.
The evaluation should establish HybridSpec’s practical benefits: under which production-serving conditions, such as high concurrency and latency SLOs, does it help, by how much, and where does it perform poorly?
[R2] Cheng, X., Yu, X., Shao, C., Li, J., Xiong, Y., Qian, Y., ... & Liang, W. (2026). DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation. arXiv preprint arXiv:2607.05147.
[R3] Huang, K., Guo, X., & Wang, M. (2024). Specdec++: Boosting speculative decoding via adaptive candidate lengths. arXiv preprint arXiv:2405.19715.
[R4] Kim, S., Kim, J., Yun, D., Shin, J., Lee, J., & Seo, J. (2026, July). Speculative Verification: Exploiting Information Gain for Speculative Decoding. In Findings of the Association for Computational Linguistics: ACL 2026 (pp. 36290-36307).
[R5] Liu, T., Shen, Y., Cen, R., Shi, J., Zhang, J., Qin, G., ... & Zhu, J. (2026). D-cut: Adaptive Verification Depth Pruning for Batched Speculative Decoding. arXiv preprint arXiv:2607.14647.
[R6] Pan, L., Tao, Z., Wang, X., Zhao, Z., Wang, M., & Zhang, Y. (2026). Making Every Verified Token Count: Adaptive Verification for MoE Speculative Decoding. arXiv preprint arXiv:2605.00342.
[R7] Chen, Z., May, A., Svirschevski, R., Huang, Y., Ryabinin, M., Jia, Z., & Chen, B. (2024). Sequoia: Scalable and robust speculative decoding. Advances in Neural Information Processing Systems, 37, 129531-129563.
[R8] Wang, Z., Han, X., Yang, Z., Liu, F., Li, X., Gu, R., ... & Tian, C. (2026). SpecLA: Efficient Speculative Decoding for Linear-Attention Models. arXiv preprint arXiv:2607.16673.
[R9] Wang, L., Su, Y., Wu, X., You, C., Liu, Y., Qiu, Z., ... & Huan, C. (2026). Bole: Efficient Tree Speculation for Hybrid-Attention Language Models. arXiv preprint arXiv:2608.01651.
[R10] Ghantasala, S. M. (2026). TreeWY: Speculative Verification for Gated DeltaNet Hybrids. arXiv preprint arXiv:2608.20961.
[R11] Liou, Z. W., Nover, H., Naik, A., Skliar, A., Shovkun, I., Shahaf, I., & Dao, T. (2026). ReplaySSM: Cache SSM inputs, not state. arXiv preprint arXiv:2610.13778.
[R12] KVBuffer Zou, L., & Zhong, L. (2026). KVBuffer: IO-aware Serving for Linear Attention. arXiv preprint arXiv:2605.19049.
Questions for authors
Q1. How much does routing correction reduce new expert loading?
Q2. How accurately does expert count predict latency?
Q3. How do joint producer kernel fusion and compact factors affect latency and throughput?
Q4. How much does Stage 2 improve performance over Stage 1 alone?
Q5. Tentatively verifying one additional token is a promising idea. Does it scale to multiple tokens? Under what conditions would adding two or three tokens be beneficial?
6 comments
Response to Reviewer FFRS (2/3)
W2: Are draft-token probabilities unbiased estimates of acceptance under the target model, and when and how does bias arise?
A: No. Raw draft probabilities are biased. HybridSpec therefore maps them to the acceptance observed in calibration runs at length 16. On held-out data, the average gap between calibrated and observed acceptance (ECE) is 0.0025, and the calibration slope is 0.97, where 1 means no systematic bias. The direction of the bias depends on the temperature. At T = 0, raw confidence underestimates acceptance, because greedy acceptance only needs the draft and target argmax tokens to agree, and they often agree even when the drafter's top-1 probability is well below 1. At T = 1, the greedy calibration becomes over-confident.
| Estimate | Bias | Effect |
|---|---|---|
| Raw top-1 confidence, T = 0 | 0.04–0.06 too low at every block position (18,317 verification windows) | Mostly removed by calibration |
| Calibrated, T = 0, lengths 2–12 | Live acceptance 4–9% above prediction | Small residual, estimates slightly conservative |
| Greedy calibration, T = 1 | Too high, 0.775 predicted vs 0.743 observed (ECE 0.032) | Selector leans long, yet still 1.32× faster than vanilla DFlash and 1.16× faster than static length 10 (3 sessions) |
At T = 0, calibration removes almost all of the bias. At T = 1, the greedy calibration overestimates acceptance, but HybridSpec still keeps a clear speedup. A calibration fit at T = 1 reduces ECE from 0.032 to 0.003 on the calibration data.
Q1: How much does routing correction reduce new expert loading?
A: The routing correction (your Stage 2) does not shrink the experts Stage 1 already reads. It reduces the new experts each extension brings, by preferring extra tokens whose experts are already loaded. On Qwen3.6 at bs = 1, an added token brings 2.7 new experts on average, out of the 8 it routes to. At c = 16, where one extension adds a token to every request, the extensions it accepts bring 12.1 new experts in the first MoE layer, 25% fewer than the 16.2 of an average candidate. The paper's Stage 0 and Stage 1 instead split the target pass at the first MoE router.
After our draft-side selector (your Stage 1), the routing correction extends the length on 65.4% of rounds on Qwen3.6-35B-A3B and 47.4% on Qwen3.5-122B-A10B.
Q2: How accurately does expert count predict latency?
A: The number of distinct experts selected in the first MoE layer predicts the measured latency of a decode step with R² 0.92–0.95 at c ≤ 4. At c = 16 the accuracy drops to R² 0.81, because the experts routed across the batch already cover nearly all 256, so the count changes little from step to step. Adding expert count to the length raises R² at every c, from 0.90–0.95 to 0.94–0.96, so the router output carries latency information that the length alone misses.
Q3: How do joint producer kernel fusion and compact factors affect latency and throughput?
A: To roll back after a partial acceptance, vanilla DFlash stores a full GDN state after every draft token. Our kernels instead keep compact factors, the small per-token terms from which the accepted state is rebuilt. They take 64× less memory than the stored states (about 0.015 GB vs 0.94 GB per request) and also save time:
| Comparison | Throughput | Setting |
|---|---|---|
| Our GDN kernels vs SGLang's, both DFlash at length 16 | 1.067×, 1.044× and 1.050× faster | c = 1, 2 and 4 |
| HybridSpec that also writes every per-token state, vs default | 3.0% slower | bs = 1 |
| One commit launch per GDN layer, vs our single launch for all layers | 1.5% slower | bs = 1 |
| Joint producer (paper §3.4) vs separate projection and convolution kernels | 2.4% faster | bs = 1, 4 sessions |
In a microbenchmark, the joint producer runs the input projections and convolution 1.14–1.30× faster than separate kernels. They take only a small part of each step, so the end-to-end gain is 2.4%.
For memory, compact factors remove the 4.69 GB per-token state buffer that vanilla DFlash allocates at c = 4. At c ≥ 8 this buffer leaves no room for the KV cache, so SGLang refuses to start vanilla DFlash on one A100-80GB. At bs = 1 our runtime skips the buffer entirely, saving 1.88 GB (paper App. D.4). App. D.4's ~40% larger KV pool assumes all of it goes to the pool, but SGLang also keeps part of GPU memory free for use at run time, and that would leave HybridSpec less free memory than vanilla DFlash. Keeping the same free memory as vanilla DFlash (4.93 GB after CUDA graph capture), the KV pool is 30.8% larger at bs = 1.
Response to Reviewer FFRS (3/3)
W3: How does HybridSpec scale across batch sizes, and under which production conditions, such as high concurrency and latency SLOs, does it help, by how much, and where does it perform poorly?
A: HybridSpec helps most at low concurrency and under tight latency SLOs, and AR wins under loose SLOs. Expert offload widens HybridSpec's lead. For the SLO rows, we sweep the concurrency (up to c = 16 for HybridSpec and c = 64 for AR) and report the highest throughput at which each system's p90 time per output token (TPOT) meets the SLO. Rows without a model name use Qwen3.6-35B-A3B (GDN) on one A100, and the GLM-5.3-Flash row shows that the batching gain also holds on KDA.
| Condition | HybridSpec | Compared with |
|---|---|---|
| c = 2, 4, 8 and 16 | 15.8%, 5.5%, 12.6% and 9.5% faster (3 sessions per point) | Kernel-only DFlash baseline (DFlash at length 16 with our GDN kernels) |
| GLM-5.3-Flash (KDA, 8× H100), c = 2, 4, 8 and 16 | 3.7–6.2% faster at each c (3 sessions) | Vanilla DFlash at block 8 |
| p90 TPOT SLO of 15 ms | 712 tok/s | 358 tok/s for AR |
| p90 TPOT SLO of 45.6 ms or looser | 1,009 tok/s (at c = 16) | 1,330 tok/s for AR (at c = 64) |
Under a 15 ms SLO, HybridSpec reaches 1.99× AR's throughput, because AR can only run a small batch at that speed. As the SLO loosens, AR can batch more requests, and from 45.6 ms on it overtakes HybridSpec.
At c ≤ 4 one length serves the whole batch, as in the paper (App. E.7). A shared length fits each request less well as the batch grows, so the gain falls from c = 2 to c = 4. From c = 8 we give each request its own length, as D-Cut and DSpark do (reply to mtL2, Q1), and the gain rises again. It stays below the c = 2 gain and dips at c = 16, because at larger c most experts are read anyway, so a shorter length saves fewer expert reads. At length 16, the first MoE layer reads 73 experts per verification step at c = 1 but 248 of 256 at c = 16.
Under expert offload, eight MoE layers read their routed experts from host memory over PCIe, so verification cost grows faster with length (Qwen3.6-35B-A3B, A100, 3 sessions for all offload results):
| HybridSpec vs | Result |
|---|---|
| Vanilla DFlash | 1.63× |
| Best static length on calibration prompts (5) | 1.25× |
| AR | 1.44× |
Without offload HybridSpec is 1.33× faster than vanilla DFlash. With offload the gain grows to 1.63× and the best static length drops from 7 to 5, because each extra verification token now also moves experts over PCIe.
A routing-aware expert cache makes HybridSpec 2.03× faster than plain host-memory offload. Its eviction policy scores each resident expert by its access history and by how likely it is to be routed to in the next step, given the experts routed in the current step. A small per-layer model, fit offline on calibration traces, provides that likelihood. At the same GPU memory budget, the cache fetches 7.7% fewer experts than LRU and gives 5.4% higher throughput.
Q4: How much does Stage 2 improve performance over Stage 1 alone? Does tentatively verifying one additional token scale to multiple tokens, and under what conditions would two or three tokens help?
A: Your Stage 2 (our routing correction) matters when your Stage 1 (our draft-side selector) misjudges cost without actual MoE routing information. Stage 2 re-prices extra tokens after the first MoE layer has routed them, so it can add back tokens that Stage 1 cut too early. On Qwen3.6-35B-A3B, Stage 2 raises throughput by 4.5% at batch size 1. On Qwen3.5-122B-A10B, it raises throughput by 4.8% (2 sessions).
On Qwen3.6-35B-A3B, Stage 2 extends the length on 65.4% of 188,019 rounds. Raising its cap from one extra token, as in the paper, to three leaves this share unchanged and splits the extensions as follows:
| Tokens added | Share of rounds |
|---|---|
| At least one | 65.4% |
| At least two | 32.1% |
| At least three | 16.9% |
So Stage 2 does use multiple tokens, but the second and third tokens do not pay off. Going from one to three extra tokens raises the accepted length by 2.7% and the cycle time by 2.9%, and the throughput difference is not measurable. Two or three tokens help when later draft tokens are accepted often and each extra token is cheap to verify. They help most with stronger drafters, high-acceptance content and large batches, where the experts an extra token needs are mostly loaded for other requests already.
Response to Reviewer FFRS (1/3)
Thank you for the detailed review and the references.
W1: Limited novelty. How do the contributions differ from prior adaptive-length and compact-factor methods, and what challenges does integrating compact factors raise and how are they solved?
A: HybridSpec targets hybrid GDN-MoE models. On these models each extra verification token adds expert reads and GDN recurrent work, so the cost-optimal length is short and shifts with content. The length our selector picks averages 6.2 on Qwen3.6-35B-A3B (GDN), from 4.5 on roleplay to 7.9 on RAG, and 5.0 on GLM-5.3-Flash (Kimi Delta Attention, KDA), from 3.7 on roleplay to 6.4 on coding and RAG. We contribute a length selector that prices both costs and corrects the length at the router boundary, a runtime inside SGLang that selects the length and verifies GDN from compact factors on the GPU, and an analysis of GDN state cost.
Adaptive length. SpecDec++ casts the length as a stopping problem, Sequoia optimizes the speculation tree from draft probabilities and hardware costs, and Cascade adapts verification budgets for MoE targets. DSpark, D-Cut, EVICT and EcoSpec bring this line to serving engines and MoE expert reuse. Speculative Verification (Kim et al.), which we will cite, also maximizes accepted tokens per unit of verification latency, using an extra companion model on dense targets. All of these fix the length before the target pass, and EVICT and D-Cut read it on the CPU every step, which D-Cut notes does not fit CPU-GPU overlap. HybridSpec maximizes pooled throughput with calibrated acceptance, keeps the length on the GPU so SGLang's overlap scheduler stays on, and corrects the length after the first MoE router using the target's own routing. Holding the rule fixed, our system makes EVICT's rule 1.171× faster, and our selector with routing correction is 1.038× faster than EVICT's rule on the same system (reply to wdvy, Q1).
Compact factors. As the paper discusses (§2.2), SpecLA, Bole and TreeWY also keep compact factors instead of per-token states. KVBuffer, which we will cite, also avoids per-token states, and ReplaySSM is concurrent work that we compare against directly (table below). What we add is making compact factors work in a verifier whose length is chosen on the GPU every round and can still grow at the router boundary. This raises two problems that a fixed-length verifier does not have. First, the length stays on the GPU, so we pre-capture one CUDA graph per length and launch the matching one through a conditional graph node that reads the length from GPU memory. Second, the length is final only after the first MoE router decides whether to keep the extra token, so our kernels verify from compact factors and commit the accepted states of all GDN layers in one launch after that decision.
The same kernels also serve batching. They accept a different length for each request, which we use at c ≥ 8 in the new batching runs (reply to mtL2, Q1). The DSpark paths in SGLang and vLLM do not support per-request lengths on GDN layers. Like the other compact-factor methods, our kernels need no per-token GDN state buffer, which takes 4.69 GB for vanilla DFlash at c = 4 (paper App. D.4). At c ≥ 8 this buffer leaves no GPU memory for the KV cache, so SGLang refuses to start vanilla DFlash on one A100-80GB, while HybridSpec runs at c = 8 and 16 (Q3).
Main speedups of HybridSpec on Qwen3.6-35B-A3B (one A100):
The first two rows keep our runtime and replace only the length selector. Choosing the length every round is 1.25× faster than always verifying 16 tokens and 1.19× faster than the best single length tuned on calibration prompts, so the gain comes from a length that is short and follows the content. The next two rows compare with ReplaySSM, the concurrent compact-state work, alone and combined with EVICT. The last row shows that the gain holds under batching, against a baseline that already runs our GDN kernels.