acceptodds
Under review as a conference paper at ICLR 2027

Hybrid Verified Decoding: Learning to Allocate Verification in Speculative Decoding

Abstract

Large language model (LLM) generation remains expensive because autoregressive decoding requires a model call for each new token. Speculative decoding reduces this cost by drafting multiple tokens and verifying them with the target model in one step, but its speedup depends on how many draft tokens are accepted. Parameter-free draft sources, such as suffix caches, can propose long continuations at low cost in structured and agentic workloads, yet their payoff varies across decoding steps. When a promising cache match yields only a short accepted prefix, most of the verification work is wasted. We propose Hybrid Verified Decoding, which uses a payoff predictor to estimate the accepted length of a cache draft before verification and then decides whether to verify the cache draft or fall back to a model-based drafter. Through experiments spanning three LLMs and sixteen datasets, we show that Hybrid Verified Decoding is especially effective on agentic workflows, where it outperforms EAGLE3 in every setting with speedups of 1.4× to 5.5×. Our analysis demonstrates that high-payoff cache drafts are rare and that the payoff predictor identifies most of them. We further show that the predictor transfers to unseen datasets and other target models without retraining.

Primary area:
foundation or frontier models, including LLMs
Keywords:
speculative decoding, LLM inference, efficient decoding, draft selection, cache-based drafting

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7WtKOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Hybrid Verified Decoding, which employs a payoff predictor to dynamically select between cache-based drafting (SuffixDecoding) and model-based drafting (EAGLE-3) based on the predicted accepted length. The authors evaluate the proposed method across multiple datasets and target models and implement it within the vLLM inference engine.

Critical strengths

Comprehensive evaluation and practical implementation. The paper evaluates the proposed method on a diverse set of benchmarks, particularly agentic and coding tasks, providing evidence of its effectiveness across different scenarios. Furthermore, the integration into vLLM demonstrates its practical applicability, and the overall speedup results are encouraging.

Critical weaknesses

Insufficient discussion and comparison with existing hybrid speculative decoding methods. Numerous existing studies have explored combining retrieval-based and model-based speculative decoding. However, the paper significantly lacks citations and comparisons with these closely related methods.

For example, SAM Decoding[1], CopySpec[2] have explored switching between retrieval-based and model-based drafting strategies. In addition, other studies, such as RASD[3] and Graft[4], have investigated integrating retrieval-based candidates directly into speculative decoding trees.

The authors should provide a more comprehensive discussion and experimental comparison with these existing hybrid approaches, rather than comparing only against standalone retrieval-based or model-based methods. In particular, the authors should clarify the advantages of dynamically switching between two drafting methods over combining them within a single speculative decoding tree. Such comparisons are important for establishing the novelty and effectiveness of the proposed method.

[1] SAM Decoding: Speculative Decoding via Suffix Automaton

[2] CopySpec: Accelerating LLMs with Speculative Copy-and-Paste

[3] RASD: Retrieval-Augmented Speculative Decoding

[4] Draft Less, Retrieve More: Hybrid Tree Construction for Speculative Decoding

Questions for authors

Please refer to the questions and concerns raised in the Weaknesses section.

4 comments

We thank the reviewer for recognizing our comprehensive evaluation on agentic and coding tasks, the integration into vLLM, and the encouraging speedup results. We address your concerns below.

W1 (a): "Numerous existing studies have explored combining retrieval-based and model-based speculative decoding. However, the paper significantly lacks citations and comparisons with these closely related methods."

We thank the reviewer for pointing us to these additional works. Section 2 cites SAM Decoding [1] as a parameter-free draft source, and we will expand it to also cover hybrid methods that combine retrieval with a model-based drafter (CopySpec [2], RASD [3], and Graft [4]). Table 1 of the paper already includes a switching hybrid baseline, Frequency-Based Hybrid Decoding (Section 4.1), and the table below summarizes how our method relates to these hybrid designs:

MethodHybrid structureSelection before verificationModel-based drafter
SAM Decoding [1], CopySpec [2]Per-step switchingFixed match-length thresholdSkipped on cache steps
Frequency-Based Hybrid (Table 1 of paper)Per-step switchingFixed frequency thresholdSkipped on cache steps
RASD [3], Graft [4]Draft-tree fusionHeuristic tree pruning / graftingRuns every step
OursPer-step switchingLearned accepted-length predictorSkipped on cache steps

Our method estimates how many tokens of the current cache draft the target model will accept from match statistics, draft structure, decoding position, and recent cache outcomes (Section 3.3), and falls back to the model-based drafter when this estimate is below the threshold. Whereas RASD and Graft run the model-based drafter at every step, this allows our method to skip neural drafting altogether on high-payoff cache cycles. At the same time, SAM Decoding and CopySpec verify every match above a fixed length cutoff even when the continuation diverges early, while our predictor filters out these low-payoff matches before verification. Our method thus delivers large speedups when long continuations recur and preserves the speed of the model-based drafter when they do not.

W1 (b): "The authors should provide a more comprehensive discussion and experimental comparison with these existing hybrid approaches, rather than comparing only against standalone retrieval-based or model-based methods."

Beyond the Frequency-Based Hybrid baseline in Table 1 of the paper, we also evaluated CopySpec [2] and SAM Decoding [1] on Qwen3-8B with EAGLE3 as the fallback drafter.

For CopySpec [2], we implemented its context-matching copy rule at match thresholds γ=3 and γ=5 in our evaluation framework. The table below reports speedup over EAGLE3:

DatasetCopySpec (γ=3)CopySpec (γ=5)OursOurs vs. CopySpec
InstructEdit/FineEdit1.21x1.34x1.56x1.17x to 1.29x
InfiniteBench1.04x1.16x2.58x2.22x to 2.47x
Delegate-521.18x1.24x1.49x1.20x to 1.26x
CNN/DailyMail0.92x0.99x1.00x1.01x to 1.08x
MT-Bench0.81x0.88x1.00x1.14x to 1.24x

Our method is 1.17x to 2.47x faster than CopySpec on the three workloads with reusable context. On CNN/DailyMail and MT-Bench, CopySpec verifies many false-positive matches and drops up to 19% below EAGLE3, whereas our method matches EAGLE3 (1.00x on both).

To compare the selection rules of CopySpec [2] and SAM Decoding [1] directly against our payoff predictor under identical candidates, we also evaluated them on the same cache drafts across all sixteen datasets on Qwen3-8B in the replay setup of Appendix C:

Selection ruleReplay tokens/sSpeedup over EAGLE3
EAGLE3109.51.00x
Match-length threshold 3 (CopySpec [2])70.10.64x
Match-length threshold 5 (SAM Decoding [1])103.80.95x
Payoff predictor, threshold 6 (Ours)145.41.33x

Fixed match-length cutoffs verify too many low-payoff matches across diverse tasks and fall below EAGLE3 overall (0.64x and 0.95x), while our payoff predictor reaches 1.33x. In end-to-end decoding with SAM Decoding's own suffix automaton, adding a payoff predictor on top of its fixed rule also raises throughput on held-out InstructEdit/FineEdit prompts from 210.5 to 227.1 tokens/s on the same H100 GPU (1.08x, see Reviewer yRDh, Q2). We will add these comparisons to the revised paper.

W1 (c): "In particular, the authors should clarify the advantages of dynamically switching between two drafting methods over combining them within a single speculative decoding tree."

The table below summarizes the advantages of dynamic per-step switching over single-tree fusion (RASD [3] and Graft [4]):

AspectSingle-Tree FusionDynamic Switching (Ours)
Neural drafting costInvokes the model-based drafter at every stepSkips neural drafting on cache steps and cuts verification cycles by 21% to 65% (Section 5.4)
Retrieved span lengthMerges retrieved tokens as tree branches (RASD [3]) or into pruned tree slots (Graft [4])Verifies a full contiguous cache span directly (23 to 26 accepted tokens per cache step, see Reviewer 4qeD, W2 & Q2)
Drafter compatibilityRequires custom candidate-merging logic for each drafter familyUses any fallback drafter as a black box and transfers directly to EAGLE3, DFlash (2.25x to 2.44x), and DSpark (2.50x to 2.51x, see Reviewer 4qeD, W2 & Q2)

While our pre-verification payoff predictor could also gate when to add retrieved branches into a fused tree, pairing it with dynamic switching captures the full benefit of long cached continuations without running or modifying the fallback drafter. This makes our method well suited to agentic and editing workflows and directly applicable to stronger drafters as they emerge. We will add this discussion to the revised paper.

[1] Yuxuan Hu, Ke Wang, Xiaokang Zhang, Fanjin Zhang, Cuiping Li, Hong Chen, and Jing Zhang. 2025. SAM Decoding: Speculative Decoding via Suffix Automaton. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, pages 12187–12204.

[2] Razvan-Gabriel Dumitru, Minglai Yang, Vikas Yadav, and Mihai Surdeanu. 2025. CopySpec: Accelerating LLMs with Speculative Copy-and-Paste. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

[3] Guofeng Quan, Wenfeng Feng, Chuzhan Hao, Guochao Jiang, Yuewei Zhang, and Hao Wang. 2025. RASD: Retrieval-Augmented Speculative Decoding. In Findings of the Association for Computational Linguistics: ACL 2025.

[4] Yuhao Shen, Tianyu Liu, Xinyi Hu, Quan Kong, Baolin Zhang, et al. 2026. Draft Less, Retrieve More: Hybrid Tree Construction for Speculative Decoding. arXiv:2605.20104.

Thank you for the additional comments. I will raise my score to 2.

Dear Reviewer,

Thank you for reading our response and for raising your score. We hope the new comparisons with CopySpec and SAM Decoding and the discussion of dynamic switching versus tree fusion have addressed your concerns. If there is anything else we can clarify, we would greatly appreciate the opportunity to do so before the discussion period ends.

Best regards, The Authors

Reviewer_yRDhOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Hybrid Verified Decoding, which trains a lightweight Payoff Predictor to estimate the acceptance length of Cache Drafts and dynamically selects between Cache-based (SuffixDecoding) and Model-based (Eagle3) drafting using a fixed threshold, thereby reducing low-yield verification overhead. The predictor leverages features such as cache matching, decoding states, and historical acceptance statistics, and is trained through offline trajectory replay. The experiments cover three Target Models and 16 datasets. On Agentic tasks, HVD achieves 1.44×–5.49× speedups over EAGLE3. The paper also demonstrates cross-task and cross-model generalization, along with preliminary deployment results on vLLM.

Critical strengths
  1. The algorithm is simple yet practically valuable.
  2. The experimental evaluation is relatively comprehensive.
Critical weaknesses
  1. The predictions of the Payoff Predictor are not fully utilized. The predicted acceptance length is only compared against a fixed threshold of 6, effectively reducing the problem to binary classification. The paper does not explain why Regression is preferred over Binary Classification, nor does it explore dynamically adjusting the verification length based on the predicted acceptance length. The lack of relevant ablation studies makes the necessity of acceptance-length regression unclear.
  2. It seems that this method merely resembles a ``two-stage'' speculative decoding process: first attempting cache-based drafting, and then deciding whether to invoke model-based drafting based on the prediction. The overall speedup largely depends on the efficiency and complementarity of SuffixDecoding and EAGLE3. In fact, the acceleration rate is not competitive against Eagle3 on most tasks.
Questions for authors
  1. Since the predicted acceptance length is used only for a binary routing decision, why is regression adopted instead of binary classification? Could the authors compare direct classification against an approach that dynamically adjusts the verification length based on the predicted acceptance length?
  2. The proposed method does not seem to require the fast and slow algorithms to be cache-based and model-based, respectively. Could the authors evaluate a broader range of algorithm choices to explore the extensibility of this ``two-level speculative sampling'' mechanism? Furthermore, could this mechanism be extended to three or more levels?
  3. The predicted acceptance length of a Cache Draft serves only as an indirect and potentially biased proxy for selecting the optimal Draft Source, as it does not directly capture the actual speedup relative to EAGLE3. Why not directly optimize for the relative computational costs or throughput gains of the two approaches? Could this training–inference objective mismatch negatively affect the optimality of the routing decisions?
4 comments

We thank the reviewer for recognizing that our algorithm is simple yet practically valuable and that our experimental evaluation is comprehensive. We address your concerns below.

W1 & Q1: "The paper does not explain why Regression is preferred over Binary Classification, nor does it explore dynamically adjusting the verification length based on the predicted acceptance length."

We use regression instead of binary classification so that a single predictor can support different decision thresholds. As discussed in Section 3.3, the minimum accepted length worth verifying depends on the cost of the model-based drafting cycle and varies across systems. Because regression outputs the accepted length in tokens, adapting to a new system requires only changing the threshold, whereas a binary classifier is tied to the threshold used for its training labels. To compare the two directly, we trained a binary classifier with the same features and MLP architecture to predict whether the accepted length is at least 6. The table below reports held-out F1 for identifying cache drafts whose accepted length reaches each threshold on Qwen3-8B.

ThresholdRegressionBinary classification (trained at 6)
668.265.8
875.971.7
1282.672.0
1686.068.9

Regression achieves higher F1 at every threshold, and the margin grows as the threshold moves away from 6. In practice, when we replace EAGLE3 with DFlash or DSpark (see our response to Reviewer 4qeD), the threshold shifts from 6 to 8 and the same predictor delivers 2.25x to 2.51x speedups without retraining.

Dynamically adjusting the verification length lowers throughput. Because target verification runs in one parallel pass, verifying 17 tokens on Qwen3-8B takes 35.2 ms and verifying 33 tokens takes 36.5 ms. Truncating selected cache drafts at their predicted accepted length therefore saves almost no time per cycle, yet it discards 14.9% of the tokens that would otherwise be accepted. We will add both comparisons to the revised paper.

W2: "It seems that this method merely resembles a 'two-stage' speculative decoding process... The overall speedup largely depends on the efficiency and complementarity of SuffixDecoding and EAGLE3. In fact, the acceleration rate is not competitive against Eagle3 on most tasks."

While SuffixDecoding and EAGLE3 are complementary, pairing them in a two-stage pipeline without selective verification is slower than EAGLE3 alone. As shown in Section 5.1, only 4.8% of cache drafts on Qwen3-8B are accepted for at least six tokens, so verifying most cache matches wastes verification passes. We tested this directly by lowering the threshold to 1, where a cache draft is verified whenever the predictor expects at least one accepted token. The table below reports aggregate throughput over all sixteen datasets on Qwen3-8B.

MethodCache cyclesTokens/sSpeedup over EAGLE3
EAGLE30.0%109.51.00x
Unselective two-stage (threshold 1)68.1%85.90.78x
Ours (threshold 6)9.7%133.01.21x

At threshold 1, 48.2% of verified cache drafts yield zero accepted tokens, and throughput falls below EAGLE3 on 14 of the 16 datasets. By restricting cache verification to 9.7% of cycles, the payoff predictor raises aggregate throughput from 0.78x to 1.21x of EAGLE3.

In Table 1 of the paper, our method improves over EAGLE3 in 28 of the 48 model-dataset settings and achieves average speedups of 1.15x, 1.29x, and 2.04x over all sixteen datasets on the three target models. On agentic and workflow-editing workloads where context can be reused, the speedup over EAGLE3 reaches 1.44x to 5.49x. On tasks with little reusable text, such as summarization and open-ended chat, high-payoff cache drafts are scarce (below 2.5% on open-ended chat in Section 5.1). As discussed in Section 4.2, the predictor falls back to EAGLE3 on nearly all steps in these tasks so that throughput stays on par with EAGLE3 (1.00x on both CNN/DailyMail and MT-Bench across repeated same-GPU runs). We will add the threshold-1 comparison to the revised paper.

Q2: "Could the authors evaluate a broader range of algorithm choices to explore the extensibility of this 'two-level speculative sampling' mechanism? Furthermore, could this mechanism be extended to three or more levels?"

Our mechanism supports replacing either drafting stage across different algorithm families. On the model-based side, we replaced the autoregressive tree drafter EAGLE3 with the block-diffusion drafter DFlash and the semi-autoregressive drafter DSpark on Qwen3-8B (see our response to Reviewer 4qeD). With the same payoff predictor and no retraining (threshold 8), our method achieves 2.25x to 2.51x speedups over each drafter alone (throughput in tokens/s):

DatasetDFlashOurs + DFlashDSparkOurs + DSpark
InstructEdit/FineEdit157.9354.9 (2.25x)154.5388.0 (2.51x)
Delegate-52124.6304.1 (2.44x)124.7311.4 (2.50x)

As noted in Section 3.2, the suffix cache can likewise be replaced by any parameter-free drafter that provides candidate drafts and match statistics. We evaluated this on Qwen3-8B by replacing SuffixDecoding with the suffix automaton of SAM Decoding [1]. SAM Decoding proposes dynamic and static candidates and filters them with a fixed match-length rule. When we add a payoff predictor trained on SAM candidates to score static drafts rejected by that rule, throughput on held-out InstructEdit/FineEdit prompts rises from 210.5 to 227.1 tokens/s (1.08x) on the same H100 GPU, and verification cycles per 1K tokens drop from 101.9 to 95.9.

The mechanism also extends naturally to three or more levels. If multiple parameter-free sources propose candidates at the same step (such as the request and global suffix trees in Section 3.2, or the dynamic and static automata above), predicting accepted length in tokens puts every candidate on a common scale, so the decoder can select the candidate with the largest predicted payoff and verify it when it reaches the threshold. Additional intermediate drafters (such as a lightweight draft model between cache lookup and a stronger model-based drafter) can also be cascaded sequentially by assigning each stage its own break-even threshold relative to the next stage. We will add the SAM-based results and this discussion to the revised paper.

Q3: "The predicted acceptance length of a Cache Draft serves only as an indirect and potentially biased proxy... Why not directly optimize for the relative computational costs or throughput gains of the two approaches? Could this training–inference objective mismatch negatively affect the optimality of the routing decisions?"

Our choice to predict accepted length while accounting for system cost through the decision threshold (Section 3.3) is motivated by two practical considerations. First, accepted-length labels can be read directly from recorded generations of the target model without additional model calls or per-step timing runs. Second, accepted length depends only on the target model and decoding state, whereas runtime cost varies across hardware, serving engines, and fallback drafters: in the cross-GPU runs of Appendix E, EAGLE3 mean accepted length differs by 1.7% across RTX 5090, H100, and B200 (2.85 to 2.89 tokens), while cycle time differs by 31.6% (31.3 to 41.2 ms). Keeping system cost in the threshold therefore lets one predictor transfer across GPUs, model sizes, and drafters without retraining.

In the offline replay setup of Appendix C across all sixteen datasets on Qwen3-8B, we compared our predictor against a throughput oracle that knows the true accepted length of every cache draft in advance. At each step, this oracle verifies the cache draft whenever doing so yields higher throughput than EAGLE3:

Routing policyReplay tokens/sSpeedup over EAGLE3Share of oracle
EAGLE3109.51.00x-
Ours (threshold 6)145.41.33x97.4%
Ours (thresholds 5 to 10)144.3 to 145.41.32x to 1.33x96.7% to 97.4%
Throughput oracle149.21.36x100.0%

Our accepted-length predictor reaches 97.4% of the throughput oracle, and performance stays within 0.8% of its peak across thresholds from 5 to 10. We will add this oracle comparison to the revised paper.

[1] Yuxuan Hu, Ke Wang, Xiaokang Zhang, Fanjin Zhang, Cuiping Li, Hong Chen, and Jing Zhang. 2025. SAM Decoding: Speculative Decoding via Suffix Automaton. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, pages 12187–12204.

Thanks for your reply and further analysis. I would raise the score to 3.

Dear Reviewer,

Thank you for reading our response and for raising your score. We appreciate your constructive feedback, and we will incorporate the new analyses into the revised paper. If there is anything else we can clarify, we would greatly appreciate the opportunity to do so before the discussion period ends.

Best regards, The Authors

Reviewer_4qeDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Hybrid Verified Decoding, which uses a payoff predictor to estimate the accepted length of a cache draft before verification and then decides whether to verify the cache draft or fall back to a model-based drafter. Through experiments spanning three LLMs and sixteen datasets, we show that Hybrid Verified Decoding is especially effective on agentic workflows, where it outperforms EAGLE3 in every setting with speedups of 1.4× to 5.5×.

Critical strengths
  1. Clear performance improvement.
  2. Evaluation on multiple datasets.
  3. Hybrid methods of cache drafter and model drafter are novel.
Critical weaknesses
  1. Small-scale evaluations: only an 8B-level model.
  2. Do not consider SOTA methods like DFlash/DSpark.
Questions for authors
  1. How is the method applied to larger-scale models (like >= 30B)?
  2. Is the method compatible with DFlash and DSpark?
2 comments

We thank the reviewer for recognizing the clear performance improvement of our work, its evaluation across multiple datasets, and the novelty of combining a cache drafter with a model-based drafter. We address your concerns below.

W1 & Q1: "Small-scale evaluations: only an 8B-level model... How is the method applied to larger-scale models (like >= 30B)?"

Our method applies to 30B-scale models under the same decoding pipeline, and because the predictor's input features do not depend on target-model size, the predictor itself can be reused directly. We evaluated our method on Qwen3-32B with the payoff predictor trained on Qwen3-8B at the same threshold of 6 without retraining. The table below compares throughput (tokens/s) against Greedy and EAGLE3 on the same H200 GPU:

DatasetGreedyEAGLE3OursOurs vs. EAGLE3
InstructEdit/FineEdit20.536.262.91.74x
InfiniteBench15.626.538.91.47x
Delegate-5218.232.653.71.65x

Across all three datasets, our method achieves speedups of 1.47x to 1.74x over EAGLE3 and 2.50x to 3.07x over Greedy. These results show that a single predictor trained at 8B scale transfers effectively to a model four times larger. We will add these results to the revised paper.

W2 & Q2: "Do not consider SOTA methods like DFlash/DSpark... Is the method compatible with DFlash and DSpark?"

Our method is directly compatible with both. Because the payoff predictor estimates the accepted length of the cache draft before invoking the fallback drafter, this estimate does not depend on which model-based drafter is used. On Qwen3-8B, we replaced EAGLE3 with the official implementations and released checkpoints of DFlash [1] and DSpark [2], and reused the same Qwen3-8B payoff predictor without retraining (threshold 8, selected on cache warm-up prompts). The table below reports throughput in tokens/s on agentic and workflow-editing workloads on the same H100 GPU (numbers in parentheses are speedups over each drafter alone):

DatasetGreedyDFlashOurs + DFlashDSparkOurs + DSpark
InstructEdit/FineEdit57.0157.9354.9 (2.25x)154.5388.0 (2.51x)
Delegate-5258.8124.6304.1 (2.44x)124.7311.4 (2.50x)

Whereas DFlash and DSpark alone commit about 5 tokens per cycle on these workloads, our method selects the cache draft in 35% to 37% of cycles and accepts 23 to 26 draft tokens per cache cycle on average. It delivers an additional 2.25x to 2.51x speedup over DFlash and DSpark alone (5.17x to 6.81x over Greedy), so payoff-guided cache verification continues to bring substantial gains even on top of strong block-diffusion and semi-autoregressive drafters. We will add these results to the revised paper.

[1] Jian Chen, Yesheng Liang, and Zhijian Liu. 2026. DFlash: Block Diffusion for Flash Speculative Decoding. arXiv:2602.06036.

[2] Xin Cheng, Xingkai Yu, Chenze Shao, et al. 2026. DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation. arXiv:2607.05147.

Dear Reviewer,

Thank you again for your review. We would like to check whether our response has addressed your concerns. We evaluated our method on Qwen3-32B (1.47x to 1.74x over EAGLE3) and combined it with DFlash and DSpark (2.25x to 2.51x over each drafter alone). If any questions remain, we would be very happy to address them before the discussion period ends.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.