acceptodds
Under review as a conference paper at ICLR 2027

HybridSpec: Throughput-Aware Speculative Decoding for Hybrid GDN-MoE Models

Abstract

In speculative decoding, parallel drafters such as DFlash make drafting cheap, so target verification dominates each round. A central choice is the verification length, the number of tokens that the target processes in each round. Verifying more draft tokens can accept more of them, but work after the first rejection is wasted. For hybrid target models that combine Gated DeltaNet (GDN) and mixture-of-experts (MoE) layers, verification time grows much faster with this length than for dense models. In a single-request profile of Qwen3.6-35B-A3B with DFlash, it grew by 71% from length 4 to 16, almost entirely in GDN and MoE layers, compared with 4%–6% for a dense Qwen3-4B target. Each GDN layer must update a recurrent state at every token and recover it at the end of the accepted prefix. The MoE cost depends on the experts that the tokens activate, which are known only once verification runs. We present HybridSpec, which selects the verification length on the GPU from draft-side acceptance estimates and measured target latency. Verification then runs this length plus one extra token up to the first MoE router, before any expert runs. There, a lightweight routing correction uses the observed expert assignments to decide whether to keep that token. The rest of verification runs only on the selected tokens. Block-parallel GDN kernels avoid storing a state after every token and reconstruct the accepted state of every GDN layer in one launch. In isolated benchmarks at verification lengths 4 to 16 with every draft token accepted, these kernels, including accepted-state reconstruction, are – faster than the default kernels. With three GDN–MoE target models in single-request greedy decoding, HybridSpec is – faster than autoregressive decoding and – faster than DFlash.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Speculative Decoding, Linear Attention, GDN, Mixture of Experts

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_FFRSOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper highlights the growing share of verification cost in speculative decoding (SD) under two conditions: (1) parallel drafters such as DFlash [R1] and DSpark [R2] reduce drafting cost, and (2) hybrid models combining GDN and MoE incur higher verification costs.

The authors propose HybridSpec, a two-stage verification framework combining draft-side selection and routing correction. Stage 1 selects a verification length that maximizes throughput based on estimated draft-token acceptance probabilities and verification cost. It also executes one additional token up to the first MoE router. Stage 2 updates the cost estimate based on the number of new experts activated by this token and decides whether to complete its execution.

To reduce intermediate-state storage, HybridSpec stores compact factors instead of full per-token GDN states and reconstructs only the required states after Stage 2.

[R1] Chen, J., Liang, Y., & Liu, Z. (2026). Dflash: Block diffusion for flash speculative decoding. arXiv preprint arXiv:2602.06036.
[R2] Cheng, X., Yu, X., Shao, C., Li, J., Xiong, Y., Qian, Y., ... & Liang, W. (2026). DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation. arXiv preprint arXiv:2607.05147.
Critical strengths

S1. The paper identifies that execution overhead in hybrid GDN–MoE models is more sensitive to verification length than in dense models. This is a timely observation. S2. Tentatively executing one additional token and revisiting the decision at the routing stage is an interesting idea that exploits the properties of GDN computation.

Critical weaknesses

W1. Limited novelty [+0.4 if resolved]

The paper should clearly distinguish its original contributions from prior work.

Relative to existing methods that adapt verification length to improve SD throughput [R2–R7], the paper should identify new findings or clearly establish its methodological novelty and scope.

Storing decomposed representations instead of full GDN states to reduce memory overhead appears to draw on prior work [R8-R12]. To establish this as a contribution, the authors should explain the challenges of integrating it into their framework and demonstrate a compelling solution.

W2. Component validity [+0.4 if resolved]

The paper should more clearly demonstrate whether each component works as intended.

In particular, it should evaluate how accurately draft-token probabilities estimate actual acceptance probabilities under the target model. For example, if DFlash is trained for greedy decoding and assumes sharply peaked token distributions, are its token probabilities unbiased estimators of acceptance probabilities? Analysis and experiments should establish when and how bias arises.

*W3. Limited evaluation [+0.4 if resolved]

Batched decoding is evaluated only at batch size 4, leaving batch-scaling behavior unclear.

This is particularly important because larger batches change expert overlap across requests and execution parallelism. These changes may affect both routing correction and the relationship between expert count and latency. The paper should evaluate each component across multiple batch sizes and explain the underlying causes and relationships.

The evaluation should establish HybridSpec’s practical benefits: under which production-serving conditions, such as high concurrency and latency SLOs, does it help, by how much, and where does it perform poorly?

[R2] Cheng, X., Yu, X., Shao, C., Li, J., Xiong, Y., Qian, Y., ... & Liang, W. (2026). DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation. arXiv preprint arXiv:2607.05147.
[R3] Huang, K., Guo, X., & Wang, M. (2024). Specdec++: Boosting speculative decoding via adaptive candidate lengths. arXiv preprint arXiv:2405.19715.
[R4] Kim, S., Kim, J., Yun, D., Shin, J., Lee, J., & Seo, J. (2026, July). Speculative Verification: Exploiting Information Gain for Speculative Decoding. In Findings of the Association for Computational Linguistics: ACL 2026 (pp. 36290-36307).
[R5] Liu, T., Shen, Y., Cen, R., Shi, J., Zhang, J., Qin, G., ... & Zhu, J. (2026). D-cut: Adaptive Verification Depth Pruning for Batched Speculative Decoding. arXiv preprint arXiv:2607.14647.
[R6] Pan, L., Tao, Z., Wang, X., Zhao, Z., Wang, M., & Zhang, Y. (2026). Making Every Verified Token Count: Adaptive Verification for MoE Speculative Decoding. arXiv preprint arXiv:2605.00342.
[R7] Chen, Z., May, A., Svirschevski, R., Huang, Y., Ryabinin, M., Jia, Z., & Chen, B. (2024). Sequoia: Scalable and robust speculative decoding. Advances in Neural Information Processing Systems, 37, 129531-129563.
[R8] Wang, Z., Han, X., Yang, Z., Liu, F., Li, X., Gu, R., ... & Tian, C. (2026). SpecLA: Efficient Speculative Decoding for Linear-Attention Models. arXiv preprint arXiv:2607.16673.
[R9] Wang, L., Su, Y., Wu, X., You, C., Liu, Y., Qiu, Z., ... & Huan, C. (2026). Bole: Efficient Tree Speculation for Hybrid-Attention Language Models. arXiv preprint arXiv:2608.01651.
[R10] Ghantasala, S. M. (2026). TreeWY: Speculative Verification for Gated DeltaNet Hybrids. arXiv preprint arXiv:2608.20961.
[R11] Liou, Z. W., Nover, H., Naik, A., Skliar, A., Shovkun, I., Shahaf, I., & Dao, T. (2026). ReplaySSM: Cache SSM inputs, not state. arXiv preprint arXiv:2610.13778.
[R12] KVBuffer Zou, L., & Zhong, L. (2026). KVBuffer: IO-aware Serving for Linear Attention. arXiv preprint arXiv:2605.19049.
Questions for authors

Q1. How much does routing correction reduce new expert loading?

Q2. How accurately does expert count predict latency?

Q3. How do joint producer kernel fusion and compact factors affect latency and throughput?

Q4. How much does Stage 2 improve performance over Stage 1 alone?

Q5. Tentatively verifying one additional token is a promising idea. Does it scale to multiple tokens? Under what conditions would adding two or three tokens be beneficial?

6 comments
AuthorsOpenReview ↗

Response to Reviewer FFRS (1/3)

Thank you for the detailed review and the references.

W1: Limited novelty. How do the contributions differ from prior adaptive-length and compact-factor methods, and what challenges does integrating compact factors raise and how are they solved?

A: HybridSpec targets hybrid GDN-MoE models. On these models each extra verification token adds expert reads and GDN recurrent work, so the cost-optimal length is short and shifts with content. The length our selector picks averages 6.2 on Qwen3.6-35B-A3B (GDN), from 4.5 on roleplay to 7.9 on RAG, and 5.0 on GLM-5.3-Flash (Kimi Delta Attention, KDA), from 3.7 on roleplay to 6.4 on coding and RAG. We contribute a length selector that prices both costs and corrects the length at the router boundary, a runtime inside SGLang that selects the length and verifies GDN from compact factors on the GPU, and an analysis of GDN state cost.

Adaptive length. SpecDec++ casts the length as a stopping problem, Sequoia optimizes the speculation tree from draft probabilities and hardware costs, and Cascade adapts verification budgets for MoE targets. DSpark, D-Cut, EVICT and EcoSpec bring this line to serving engines and MoE expert reuse. Speculative Verification (Kim et al.), which we will cite, also maximizes accepted tokens per unit of verification latency, using an extra companion model on dense targets. All of these fix the length before the target pass, and EVICT and D-Cut read it on the CPU every step, which D-Cut notes does not fit CPU-GPU overlap. HybridSpec maximizes pooled throughput with calibrated acceptance, keeps the length on the GPU so SGLang's overlap scheduler stays on, and corrects the length after the first MoE router using the target's own routing. Holding the rule fixed, our system makes EVICT's rule 1.171× faster, and our selector with routing correction is 1.038× faster than EVICT's rule on the same system (reply to wdvy, Q1).

Compact factors. As the paper discusses (§2.2), SpecLA, Bole and TreeWY also keep compact factors instead of per-token states. KVBuffer, which we will cite, also avoids per-token states, and ReplaySSM is concurrent work that we compare against directly (table below). What we add is making compact factors work in a verifier whose length is chosen on the GPU every round and can still grow at the router boundary. This raises two problems that a fixed-length verifier does not have. First, the length stays on the GPU, so we pre-capture one CUDA graph per length and launch the matching one through a conditional graph node that reads the length from GPU memory. Second, the length is final only after the first MoE router decides whether to keep the extra token, so our kernels verify from compact factors and commit the accepted states of all GDN layers in one launch after that decision.

The same kernels also serve batching. They accept a different length for each request, which we use at c ≥ 8 in the new batching runs (reply to mtL2, Q1). The DSpark paths in SGLang and vLLM do not support per-request lengths on GDN layers. Like the other compact-factor methods, our kernels need no per-token GDN state buffer, which takes 4.69 GB for vanilla DFlash at c = 4 (paper App. D.4). At c ≥ 8 this buffer leaves no GPU memory for the KV cache, so SGLang refuses to start vanilla DFlash on one A100-80GB, while HybridSpec runs at c = 8 and 16 (Q3).

Main speedups of HybridSpec on Qwen3.6-35B-A3B (one A100):

BaselineHybridSpec speedupSetting
Fixed length 161.25×bs = 1
Best static length on calibration prompts (7)1.19×bs = 1
ReplaySSM tuned for bs = 11.235×bs = 1
Tuned EVICT + ReplaySSM1.120×bs = 1
Kernel-only DFlash baseline (length 16 with our GDN kernels)1.055–1.158×c = 2, 4, 8 and 16

The first two rows keep our runtime and replace only the length selector. Choosing the length every round is 1.25× faster than always verifying 16 tokens and 1.19× faster than the best single length tuned on calibration prompts, so the gain comes from a length that is short and follows the content. The next two rows compare with ReplaySSM, the concurrent compact-state work, alone and combined with EVICT. The last row shows that the gain holds under batching, against a baseline that already runs our GDN kernels.

AuthorsOpenReview ↗

Response to Reviewer FFRS (2/3)

W2: Are draft-token probabilities unbiased estimates of acceptance under the target model, and when and how does bias arise?

A: No. Raw draft probabilities are biased. HybridSpec therefore maps them to the acceptance observed in calibration runs at length 16. On held-out data, the average gap between calibrated and observed acceptance (ECE) is 0.0025, and the calibration slope is 0.97, where 1 means no systematic bias. The direction of the bias depends on the temperature. At T = 0, raw confidence underestimates acceptance, because greedy acceptance only needs the draft and target argmax tokens to agree, and they often agree even when the drafter's top-1 probability is well below 1. At T = 1, the greedy calibration becomes over-confident.

EstimateBiasEffect
Raw top-1 confidence, T = 00.04–0.06 too low at every block position (18,317 verification windows)Mostly removed by calibration
Calibrated, T = 0, lengths 2–12Live acceptance 4–9% above predictionSmall residual, estimates slightly conservative
Greedy calibration, T = 1Too high, 0.775 predicted vs 0.743 observed (ECE 0.032)Selector leans long, yet still 1.32× faster than vanilla DFlash and 1.16× faster than static length 10 (3 sessions)

At T = 0, calibration removes almost all of the bias. At T = 1, the greedy calibration overestimates acceptance, but HybridSpec still keeps a clear speedup. A calibration fit at T = 1 reduces ECE from 0.032 to 0.003 on the calibration data.

Q1: How much does routing correction reduce new expert loading?

A: The routing correction (your Stage 2) does not shrink the experts Stage 1 already reads. It reduces the new experts each extension brings, by preferring extra tokens whose experts are already loaded. On Qwen3.6 at bs = 1, an added token brings 2.7 new experts on average, out of the 8 it routes to. At c = 16, where one extension adds a token to every request, the extensions it accepts bring 12.1 new experts in the first MoE layer, 25% fewer than the 16.2 of an average candidate. The paper's Stage 0 and Stage 1 instead split the target pass at the first MoE router.

After our draft-side selector (your Stage 1), the routing correction extends the length on 65.4% of rounds on Qwen3.6-35B-A3B and 47.4% on Qwen3.5-122B-A10B.

Q2: How accurately does expert count predict latency?

A: The number of distinct experts selected in the first MoE layer predicts the measured latency of a decode step with R² 0.92–0.95 at c ≤ 4. At c = 16 the accuracy drops to R² 0.81, because the experts routed across the batch already cover nearly all 256, so the count changes little from step to step. Adding expert count to the length raises R² at every c, from 0.90–0.95 to 0.94–0.96, so the router output carries latency information that the length alone misses.

Q3: How do joint producer kernel fusion and compact factors affect latency and throughput?

A: To roll back after a partial acceptance, vanilla DFlash stores a full GDN state after every draft token. Our kernels instead keep compact factors, the small per-token terms from which the accepted state is rebuilt. They take 64× less memory than the stored states (about 0.015 GB vs 0.94 GB per request) and also save time:

ComparisonThroughputSetting
Our GDN kernels vs SGLang's, both DFlash at length 161.067×, 1.044× and 1.050× fasterc = 1, 2 and 4
HybridSpec that also writes every per-token state, vs default3.0% slowerbs = 1
One commit launch per GDN layer, vs our single launch for all layers1.5% slowerbs = 1
Joint producer (paper §3.4) vs separate projection and convolution kernels2.4% fasterbs = 1, 4 sessions

In a microbenchmark, the joint producer runs the input projections and convolution 1.14–1.30× faster than separate kernels. They take only a small part of each step, so the end-to-end gain is 2.4%.

For memory, compact factors remove the 4.69 GB per-token state buffer that vanilla DFlash allocates at c = 4. At c ≥ 8 this buffer leaves no room for the KV cache, so SGLang refuses to start vanilla DFlash on one A100-80GB. At bs = 1 our runtime skips the buffer entirely, saving 1.88 GB (paper App. D.4). App. D.4's ~40% larger KV pool assumes all of it goes to the pool, but SGLang also keeps part of GPU memory free for use at run time, and that would leave HybridSpec less free memory than vanilla DFlash. Keeping the same free memory as vanilla DFlash (4.93 GB after CUDA graph capture), the KV pool is 30.8% larger at bs = 1.

AuthorsOpenReview ↗

Response to Reviewer FFRS (3/3)

W3: How does HybridSpec scale across batch sizes, and under which production conditions, such as high concurrency and latency SLOs, does it help, by how much, and where does it perform poorly?

A: HybridSpec helps most at low concurrency and under tight latency SLOs, and AR wins under loose SLOs. Expert offload widens HybridSpec's lead. For the SLO rows, we sweep the concurrency (up to c = 16 for HybridSpec and c = 64 for AR) and report the highest throughput at which each system's p90 time per output token (TPOT) meets the SLO. Rows without a model name use Qwen3.6-35B-A3B (GDN) on one A100, and the GLM-5.3-Flash row shows that the batching gain also holds on KDA.

ConditionHybridSpecCompared with
c = 2, 4, 8 and 1615.8%, 5.5%, 12.6% and 9.5% faster (3 sessions per point)Kernel-only DFlash baseline (DFlash at length 16 with our GDN kernels)
GLM-5.3-Flash (KDA, 8× H100), c = 2, 4, 8 and 163.7–6.2% faster at each c (3 sessions)Vanilla DFlash at block 8
p90 TPOT SLO of 15 ms712 tok/s358 tok/s for AR
p90 TPOT SLO of 45.6 ms or looser1,009 tok/s (at c = 16)1,330 tok/s for AR (at c = 64)

Under a 15 ms SLO, HybridSpec reaches 1.99× AR's throughput, because AR can only run a small batch at that speed. As the SLO loosens, AR can batch more requests, and from 45.6 ms on it overtakes HybridSpec.

At c ≤ 4 one length serves the whole batch, as in the paper (App. E.7). A shared length fits each request less well as the batch grows, so the gain falls from c = 2 to c = 4. From c = 8 we give each request its own length, as D-Cut and DSpark do (reply to mtL2, Q1), and the gain rises again. It stays below the c = 2 gain and dips at c = 16, because at larger c most experts are read anyway, so a shorter length saves fewer expert reads. At length 16, the first MoE layer reads 73 experts per verification step at c = 1 but 248 of 256 at c = 16.

Under expert offload, eight MoE layers read their routed experts from host memory over PCIe, so verification cost grows faster with length (Qwen3.6-35B-A3B, A100, 3 sessions for all offload results):

HybridSpec vsResult
Vanilla DFlash1.63×
Best static length on calibration prompts (5)1.25×
AR1.44×

Without offload HybridSpec is 1.33× faster than vanilla DFlash. With offload the gain grows to 1.63× and the best static length drops from 7 to 5, because each extra verification token now also moves experts over PCIe.

A routing-aware expert cache makes HybridSpec 2.03× faster than plain host-memory offload. Its eviction policy scores each resident expert by its access history and by how likely it is to be routed to in the next step, given the experts routed in the current step. A small per-layer model, fit offline on calibration traces, provides that likelihood. At the same GPU memory budget, the cache fetches 7.7% fewer experts than LRU and gives 5.4% higher throughput.

Q4: How much does Stage 2 improve performance over Stage 1 alone? Does tentatively verifying one additional token scale to multiple tokens, and under what conditions would two or three tokens help?

A: Your Stage 2 (our routing correction) matters when your Stage 1 (our draft-side selector) misjudges cost without actual MoE routing information. Stage 2 re-prices extra tokens after the first MoE layer has routed them, so it can add back tokens that Stage 1 cut too early. On Qwen3.6-35B-A3B, Stage 2 raises throughput by 4.5% at batch size 1. On Qwen3.5-122B-A10B, it raises throughput by 4.8% (2 sessions).

On Qwen3.6-35B-A3B, Stage 2 extends the length on 65.4% of 188,019 rounds. Raising its cap from one extra token, as in the paper, to three leaves this share unchanged and splits the extensions as follows:

Tokens addedShare of rounds
At least one65.4%
At least two32.1%
At least three16.9%

So Stage 2 does use multiple tokens, but the second and third tokens do not pay off. Going from one to three extra tokens raises the accepted length by 2.7% and the cycle time by 2.9%, and the throughput difference is not measurable. Two or three tokens help when later draft tokens are accepted often and each extra token is cheap to verify. They help most with stronger drafters, high-acceptance content and large batches, where the experts an extra token needs are mostly loaded for other requests already.

Reviewer_wdvyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HybridSpec improves speculative decoding for hybrid GDN–MoE models. It selects verification lengths based on expected acceptance and execution cost, adjusts the choice using routing information, and optimizes GDN execution to reduce verification overhead.

Critical strengths
  1. The experiments use a common inference framework and include runtime overhead. Although the method accepts slightly shorter drafts, faster rounds lead to higher throughput, which is convincing.

  2. The paper clearly explains the conditions needed for correctness and when routing information can be used to adjust verification length.

Critical weaknesses
  1. The contributions of the individual changes are not fully separated. The comparison with EVICT changes length selection, acceptance estimates, and implementation together, leaving the benefit of the selection method unclear.

  2. The baseline configurations need clarification. Since some default settings perform poorly in the tested setting, it is important to confirm that the main comparisons use tuned MoE and ReplaySSM configurations.

  3. Most results use single-request decoding on A100. The additional gains shrink with concurrency, and differences in cache capacity may affect the results. The benefit of routing correction at higher concurrency and on the largest model also remains unclear.

Questions for authors
  1. Is the MoE configuration tuned in the main experiments? How much do results vary across runs, and do cache-capacity differences affect the concurrent tests?

  2. With the same implementation, how much advantage remains over D-Cut? How does tuned ReplaySSM perform?

  3. How do the gains change with tuned MoE kernels and higher concurrency? How much does routing correction help on the largest model?

3 comments
AuthorsOpenReview ↗

Response to Reviewer wdvy (1/2)

Thank you for the careful reading of our evaluation and correctness analysis. We group your comments into seven questions.

Q1: How much of the advantage over EVICT and D-Cut remains when they share HybridSpec's dispatch and GDN kernels, and how much do the objective and calibration changes contribute?

A: Over the EVICT baseline, HybridSpec is 1.216× faster in total (Qwen3.6-35B-A3B, one A100, bs = 1, 3 sessions), and over the D-Cut baseline it is 1.238× faster. To split this gain into the selection rule and the system, we moved the EVICT and D-Cut rules onto our system.

Selection rule on the same system. With dispatch and GDN kernels shared, our selector beats EVICT's rule by a 3.8% speedup gain on decoding throughput. Against D-Cut, our finer length set (any length from 2 to 16 instead of D-Cut's 4, 8, 12 and 16) and routing correction give 1.086×. Our lead widens when each verification token costs more. Under expert offload, eight MoE layers read their routed experts from host memory over PCIe, and our calibrated selector is 1.054× faster than EVICT's rule (3 sessions). EVICT's rule picks longer lengths (5.44 vs 4.95) and accepts about 6% more tokens per round, but the extra tokens do not repay their verification cost.

System with the same rule. Our system makes EVICT's own rule 1.171× faster. GPU-resident length dispatch with our cost tables gives 1.081×, the overlap scheduler it makes usable gives a further 1.042×, and our GDN kernels give 1.039×. Swapping our cost tables for EVICT's own startup profile, measured on our system, gives 0.991× (3 sessions), so the gain comes from the system and not from the tables. For D-Cut, our system and cost tables give 1.140×.

Calibration and objective have to change together. Starting from EVICT's rule on our full system, we changed one part at a time (over 3 sessions). With calibration alone, the acceptance estimates rise, so EVICT's per-round objective picks longer lengths (mean 6.35 → 7.18), and throughput drops to 0.979× because the extra verification tokens add expert reads. With our pooled-throughput objective alone, on raw draft probabilities that underestimate acceptance (FFRS, W2), throughput is 0.995×. With both, which is our selector, the objective charges the extra expert reads and the estimates are calibrated, and throughput beats EVICT's rule.

Q2: How does HybridSpec compare end to end with ReplaySSM tuned for batch size 1?

A: With ReplaySSM tuned for bs = 1, EVICT + ReplaySSM is the strongest baseline, and HybridSpec is 1.120× faster than it. This comes from a pre-registered run of 3 independent sessions (Qwen3.6-35B-A3B, A100, bs = 1). In the same run HybridSpec is 1.235× faster than tuned ReplaySSM alone.

By domain, HybridSpec beats tuned EVICT + ReplaySSM in every one (1.10–1.14×). Our selector is not told the domain, yet its mean length drops to 4.5–5.1 on writing and roleplay, where acceptance is low, against 7.9 on RAG and a fixed 16 for vanilla DFlash. Within roleplay it still selects a length of 9 or more on 12% of rounds.

DomainHybridSpec mean lengthvs tuned EVICT + ReplaySSM
coding7.41.104
math7.11.125
reasoning6.51.137
RAG7.91.097
summarization7.01.135
writing5.11.130
roleplay4.51.110
all (pooled)6.21.120

Q3: How large is the run-to-run variability behind the smaller margins?

A: The variability is small next to these margins. Each system in the paper runs in three sessions, each with a fresh server process. Pairing each HybridSpec session with each baseline session gives nine speedup ratios. For the smallest mean speedup over a baseline in the paper, 1.061× over EVICT on Qwen3.5-122B-A10B, all nine ratios lie between 1.056× and 1.063×. For the 1.103× over D-Cut on the same model, they lie between 1.098× and 1.108×. The session-to-session spread is therefore under 1%, well below these margins. A new 3-session run of this model on three different machines gives 1.345× over vanilla DFlash, close to the paper's 1.33×. In the batching runs (Q4), the per-session ratios agree within 1.3% at every concurrency. All new comparisons in this rebuttal also use 3 fresh-server sessions unless stated otherwise.

AuthorsOpenReview ↗

Response to Reviewer wdvy (2/2)

Q4: Does the main evaluation use the untuned MoE configuration, and how do the gains change with tuned MoE kernels, higher concurrency and hardware beyond A100?

A: Yes. The main evaluation uses SGLang's default (untuned) fused-MoE configuration for every system. In a paired run on Qwen3.6-35B-A3B, tuning changes the bs = 1 speedup over vanilla DFlash by less than 0.3%. Under batching, tuned kernels speed up HybridSpec by 1.7% at c = 4 and 2.6% at c = 16, and the kernel-only DFlash baseline (DFlash at length 16 with our GDN kernels) by 1.9% and 3.6%. On one A100 with tuned MoE in both systems, HybridSpec is faster than the kernel-only DFlash baseline by 15.8% at c = 2, 5.5% at c = 4, 12.6% at c = 8 and 9.5% at c = 16. On H100 at bs = 1 it is 1.28× faster than vanilla DFlash and 1.97× faster than AR.

Q5: Why do the KV-cache capacities differ at concurrency 4, and did either system experience memory pressure, preemption or queueing?

A: The capacities differ because of GDN state, not KV cache. To roll back after a partial acceptance, vanilla DFlash in SGLang stores the GDN recurrent state after every draft token, for every GDN layer and every running request. SGLang preallocates this per-token GDN state buffer (4.69 GB at c = 4 on Qwen3.6-35B-A3B, paper Appendix D.4) and gives the remaining memory to the KV pool. Our GDN kernels keep compact factors instead and rebuild only the accepted state, so they need no such buffer, and the freed memory goes to the KV pool. At c = 4 the larger pool does not explain the speedup: capping HybridSpec's pool at vanilla DFlash's 12,472 tokens gives 0.997×. Neither system was under memory pressure. No speculative system was preempted, 1–2% of admissions found a request already waiting (at most two), and no sampled decode step had a waiting request.

The freed memory pays off at higher concurrency. At c ≥ 8 the buffer leaves no GPU memory for the KV cache, so SGLang refuses to start vanilla DFlash on one A100-80GB, while HybridSpec runs at c = 8 and 16. At bs = 1 our runtime skips the buffer entirely, saving 1.88 GB (paper App. D.4). App. D.4's ~40% larger KV pool assumes all of it goes to the pool, but SGLang also keeps part of GPU memory free for use at run time, and that would leave HybridSpec less free memory than vanilla DFlash. Keeping the same free memory as vanilla DFlash (4.93 GB after CUDA graph capture), the KV pool is 30.8% larger at bs = 1.

Q6: How often does the routing correction extend the selected length, and what benefit does it provide on the largest target?

A: On Qwen3.5-122B-A10B the routing correction extends the length on 47.4% of rounds. It re-prices extra tokens at the router boundary, after the first MoE layer has routed them. It therefore pays off when the draft-side selector, which picks the nominal length from draft probabilities before the target pass, misestimates expert cost. The correction then improves the decoding throughput by 4.8% (2 sessions). On Qwen3.6-35B-A3B (bs = 1) it extends the length on 65.4% of rounds, and improves throughput by 4.5%.

Q7: How does HybridSpec differ from the close precedents for cost-aware length selection and compact state restoration?

A: On hybrid GDN-MoE targets, HybridSpec differs from these precedents in both length selection and state restoration, and each difference has a measured effect.

Length selection. EVICT, D-Cut and DSpark choose the length before verification from draft-side estimates, and EVICT and D-Cut read it on the CPU every step. D-Cut itself notes that its per-step lengths do not fit CPU-GPU overlap. HybridSpec prices expert reads and GDN work, selects and dispatches the length on the GPU so SGLang's overlap scheduler stays on, and re-prices extra tokens after the first MoE router with the target's own routing. Holding the rule fixed, our system makes EVICT's rule 1.171× faster. Holding the system fixed, our selector beats EVICT's rule at bs = 1 and is 1.038× faster (Q1).

State restoration. ReplaySSM, SpecLA, Bole and TreeWY also avoid storing every per-token state. Our GDN kernels verify directly from compact factors 64× smaller than per-token states, and rebuild the accepted states of all GDN layers in one launch. On their own they are 1.04–1.07× faster than vanilla DFlash at c = 1, 2 and 4, and they remove its 4.69 GB per-token GDN state buffer at c = 4 (Q5).

Combining both, HybridSpec is 1.120× faster than tuned EVICT + ReplaySSM, the strongest combination of these precedents, and 1.235× faster than tuned ReplaySSM alone.

Dear Reviewer wdvy,

Thank you again for your insightful feedback on our submission. With the discussion period drawing to a close, we wanted to check in to see if our responses have adequately addressed all of your concerns. Please let us know if any points require further clarification. If you find our revisions satisfactory, we would be grateful if you would consider updating your score accordingly.

We look forward to hearing from you.

Reviewer_mtL2OpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a speculative decoding method for GDN-MoE models, including approaches for verification-length selection and GDN kernel improvement. HybridSpec is 1.75×–1.88× faster than autoregressive decoding and 1.30×–1.33× faster than DFlash.

Critical strengths
  1. Sharp observation about the problem in GDN verification.
  2. Comparison with strong baselines: SGLang + DFlash2.
  3. Interesting mathematical analysis and a well-written paper.
Critical weaknesses
  1. Evaluations are limited to a batch size = 1.
  2. Evaluations are limited to A100.
Questions for authors
  1. What is the exact command to launch the SGLang baseline for your experiments?
  2. Any chance to see this method applied to GLM5.3-Flash?
8 comments

I disagree with your decision to give it a rating of 4, given the critical strengths and weakness you mentioned. Did you analyze the paper properly?

I don’t see any evidence that you carefully reviewed this paper, despite your claim of having substantial direct research experience.

AuthorsOpenReview ↗

Response to Reviewer mtL2 (1/2)

Thank you for the positive assessment and for highlighting the problem in GDN verification. We group your comments into four questions.

Q1: How does HybridSpec perform at batch sizes above 1?

A: On the paper's SPEED-Bench test prompts, HybridSpec is 1.158× faster at concurrency c = 2, 1.055× at c = 4, 1.126× at c = 8 and 1.095× at c = 16 (3 sessions per point) than the kernel-only DFlash baseline, our name for DFlash at length 16 running our GDN kernels. Both systems run Qwen3.6-35B-A3B on one A100-80GB with the same orthogonal system optimizations. The kernel-only baseline is faster than vanilla DFlash at c ≤ 4. Vanilla DFlash keeps a per-token GDN state buffer, and at c ≥ 8 this buffer leaves no GPU memory for the KV cache, so SGLang refuses to start it on one A100-80GB. At c ≤ 4, as in the paper (Appendix E.7), one length serves the whole batch.

A shared length fits each request less well as the batch grows, and the gain falls from c = 2 to c = 4. At c ≥ 8 we give each request its own length, as D-Cut and DSpark do. A GPU-side allocator first reserves 2 tokens per request, then spends the rest of a batch-wide budget on the draft tokens with the highest calibrated acceptance, and picks the budget that maximizes batch throughput under our cost model. Attention and MoE run only on the selected tokens, and the lengths stay on the GPU. On GDN layers this relies on our kernels. Vanilla DFlash stores a GDN state after every draft token, and the DSpark paths in SGLang and vLLM do not support per-request lengths on GDN layers. Our kernels keep compact factors instead, and a single commit launch advances each request's GDN state by its own accepted length.

Per-request lengths pay off from c = 8, and the gain rises again. It stays below the c = 2 gain and dips at c = 16, because in larger batches the experts touched in the first MoE layer approach the full set, so the verification length has little effect on expert weight reads. On KDA (GLM-5.3-Flash, 8× H100), HybridSpec is 3.7–6.2% faster than vanilla DFlash at block 8 at each of c = 2, 4, 8 and 16 over 3 sessions.

Q2: How does HybridSpec perform on GPUs other than A100?

A: On H100, HybridSpec is 1.279× faster than vanilla DFlash in SGLang (block 16) and 1.97× faster than autoregressive decoding (AR) at bs = 1 on Qwen3.6 (3 sessions). Recalibrating on H100 improves: in two further 3-session runs that measured both table sets, H100 calibrated parameters gave 1.279× and 1.270×, and A100 calibrated parameters 1.268× and 1.265×.

Q3: What is the exact command used to launch the SGLang baseline?

A: The baseline is the unmodified SGLang 0.5.16 DFlash server, launched with the command below. In the main evaluation, all Qwen3.6 systems use BF16, T = 0, max_new_tokens = 1024, thinking mode and SGLang's default fused-MoE configuration. Prefill CUDA graphs are off for every system, which does not affect our metric because it measures decode throughput after prefill.

python -m sglang.launch_server --model-path Qwen/Qwen3.6-35B-A3B \
  --revision 995ad96eacd98c81ed38be0c5b274b04031597b0 --trust-remote-code \
  --language-only --context-length 8192 --tp-size 1 --mem-fraction-static 0.9 \
  --max-running-requests 1 --random-seed 42 --attention-backend flashinfer \
  --linear-attn-backend triton --cuda-graph-backend-decode full \
  --cuda-graph-backend-prefill disabled --cuda-graph-bs-decode 1 \
  --disable-radix-cache --skip-server-warmup --speculative-algorithm DFLASH \
  --speculative-draft-model-path z-lab/Qwen3.6-35B-A3B-DFlash \
  --speculative-draft-model-revision f181eece646affea2c38b2765f1aaa01a9734ccd \
  --speculative-num-steps 1 --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 16 --speculative-dflash-block-size 16 \
  --speculative-draft-attention-backend triton

We adapt D-Cut and EVICT to the DFlash proposal block in SGLang 0.5.16, with the same drafter and target as HybridSpec. Each profiles the step latency of every verification length at startup, then in each round truncates the proposal block to the length that maximizes expected tokens per profiled step latency, using drafter confidences as acceptance estimates. EVICT + ReplaySSM adds SGLang's ReplaySSM GDN verification to EVICT.

AuthorsOpenReview ↗

Response to Reviewer mtL2 (2/2)

Q4: Can the method be applied to GLM-5.3-Flash?

A: Yes. GLM-5.3-Flash uses Kimi Delta Attention (KDA), so we ported our verification kernels to KDA as specKDA and ran it on 8× H100 (8-way tensor parallelism, bs = 1). With SGLang's default multi-token-prediction (MTP) drafter in both systems, specKDA kernel optimizations alone are 1.034× faster than SGLang's default MTP. With a DFlash drafter at block 8, HybridSpec is 1.10× faster than vanilla DFlash at block 8 (3 sessions), 1.08× faster than vanilla DFlash at its best calibrated static block size (6), and 1.26× faster than SGLang's default MTP setup on the paper's SPEED-Bench test prompts.

The table gives HybridSpec's speedup on GLM-5.3-Flash over each system by domain (bs = 1, 8× H100, 3 sessions). HybridSpec is faster than vanilla DFlash at block 8 in every domain (1.06–1.14×), with the largest gains on writing and roleplay (1.12–1.14×). It is 1.08× (roleplay) to 1.51× (coding) faster than MTP. Our selector is not told the domain, and its mean length ranges from 3.7 on roleplay to 6.4 on coding and RAG out of the block of 8 (5.0 overall).

DomainHybridSpec mean lengthvs MTPvs vanilla DFlash, block 8vs vanilla DFlash, best block (6)
coding6.41.5121.0631.139
math5.11.2821.0921.043
reasoning5.71.3511.0781.082
RAG6.41.4441.0741.104
summarization6.31.3341.0731.079
writing4.51.1621.1251.087
roleplay3.71.0841.1441.070
all (pooled)5.01.2571.1041.080

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank all the reviewers for the insightful comments and suggestions. Compared with dense targets, hybrid GDN-MoE targets make verification more expensive in two ways. Each extra verification token adds MoE expert reads and a step of GDN recurrence, and a verifier that keeps full states must store one GDN state per draft token. HybridSpec addresses both costs with several techniques: throughput-aware length selection, two-stage verification at the router boundary, parallel GDN verification and state reconstruction.

Unless noted otherwise, all speed results use the same test prompts as the paper (280 prompts from seven SPEED-Bench Qualitative categories). Each new comparison runs every system in 3 sessions. In each session, every system decodes the test prompts once on a freshly started server, and we pool decode throughput over all prompts and sessions. At bs = 1 on Qwen3.6-35B-A3B, HybridSpec is 1.33× faster than vanilla DFlash in SGLang (block 16) on A100, as reported in the paper, and 1.28× faster on H100. It is 1.120× faster than tuned EVICT + ReplaySSM, the strongest baseline. On the same runtime, our selector beats EVICT's rule (reply to wdvy, Q1). Under batching (c = 2–16, A100), HybridSpec is 5.5–15.8% faster than the kernel-only DFlash baseline, an equally optimized DFlash at length 16 that also runs our GDN kernels.

We also ported the design to Kimi Delta Attention (KDA) and ran it on GLM-5.3-Flash (8× H100). There HybridSpec is 1.10× faster than vanilla DFlash at block 8 and 1.08× faster than vanilla DFlash at its best calibrated static block size, and at c = 2–16 its lead over vanilla DFlash at block 8 is 3.7–6.2%. Batching is covered in our replies to mtL2 (Q1) and FFRS (W3), H100 in mtL2 (Q2), the EVICT and D-Cut attribution in wdvy (Q1), tuned ReplaySSM in wdvy (Q2) and component checks in FFRS (W2, Q1–Q4).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.