- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Hybrid Verified Decoding, which employs a payoff predictor to dynamically select between cache-based drafting (SuffixDecoding) and model-based drafting (EAGLE-3) based on the predicted accepted length. The authors evaluate the proposed method across multiple datasets and target models and implement it within the vLLM inference engine.
Critical strengths
Comprehensive evaluation and practical implementation. The paper evaluates the proposed method on a diverse set of benchmarks, particularly agentic and coding tasks, providing evidence of its effectiveness across different scenarios. Furthermore, the integration into vLLM demonstrates its practical applicability, and the overall speedup results are encouraging.
Critical weaknesses
Insufficient discussion and comparison with existing hybrid speculative decoding methods. Numerous existing studies have explored combining retrieval-based and model-based speculative decoding. However, the paper significantly lacks citations and comparisons with these closely related methods.
For example, SAM Decoding[1], CopySpec[2] have explored switching between retrieval-based and model-based drafting strategies. In addition, other studies, such as RASD[3] and Graft[4], have investigated integrating retrieval-based candidates directly into speculative decoding trees.
The authors should provide a more comprehensive discussion and experimental comparison with these existing hybrid approaches, rather than comparing only against standalone retrieval-based or model-based methods. In particular, the authors should clarify the advantages of dynamically switching between two drafting methods over combining them within a single speculative decoding tree. Such comparisons are important for establishing the novelty and effectiveness of the proposed method.
[1] SAM Decoding: Speculative Decoding via Suffix Automaton
[2] CopySpec: Accelerating LLMs with Speculative Copy-and-Paste
[3] RASD: Retrieval-Augmented Speculative Decoding
[4] Draft Less, Retrieve More: Hybrid Tree Construction for Speculative Decoding
Questions for authors
Please refer to the questions and concerns raised in the Weaknesses section.
4 comments
W1 (c): "In particular, the authors should clarify the advantages of dynamically switching between two drafting methods over combining them within a single speculative decoding tree."
The table below summarizes the advantages of dynamic per-step switching over single-tree fusion (RASD [3] and Graft [4]):
| Aspect | Single-Tree Fusion | Dynamic Switching (Ours) |
|---|---|---|
| Neural drafting cost | Invokes the model-based drafter at every step | Skips neural drafting on cache steps and cuts verification cycles by 21% to 65% (Section 5.4) |
| Retrieved span length | Merges retrieved tokens as tree branches (RASD [3]) or into pruned tree slots (Graft [4]) | Verifies a full contiguous cache span directly (23 to 26 accepted tokens per cache step, see Reviewer 4qeD, W2 & Q2) |
| Drafter compatibility | Requires custom candidate-merging logic for each drafter family | Uses any fallback drafter as a black box and transfers directly to EAGLE3, DFlash (2.25x to 2.44x), and DSpark (2.50x to 2.51x, see Reviewer 4qeD, W2 & Q2) |
While our pre-verification payoff predictor could also gate when to add retrieved branches into a fused tree, pairing it with dynamic switching captures the full benefit of long cached continuations without running or modifying the fallback drafter. This makes our method well suited to agentic and editing workflows and directly applicable to stronger drafters as they emerge. We will add this discussion to the revised paper.
[1] Yuxuan Hu, Ke Wang, Xiaokang Zhang, Fanjin Zhang, Cuiping Li, Hong Chen, and Jing Zhang. 2025. SAM Decoding: Speculative Decoding via Suffix Automaton. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, pages 12187–12204.
[2] Razvan-Gabriel Dumitru, Minglai Yang, Vikas Yadav, and Mihai Surdeanu. 2025. CopySpec: Accelerating LLMs with Speculative Copy-and-Paste. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.
[3] Guofeng Quan, Wenfeng Feng, Chuzhan Hao, Guochao Jiang, Yuewei Zhang, and Hao Wang. 2025. RASD: Retrieval-Augmented Speculative Decoding. In Findings of the Association for Computational Linguistics: ACL 2025.
[4] Yuhao Shen, Tianyu Liu, Xinyi Hu, Quan Kong, Baolin Zhang, et al. 2026. Draft Less, Retrieve More: Hybrid Tree Construction for Speculative Decoding. arXiv:2605.20104.
Thank you for the additional comments. I will raise my score to 2.
Dear Reviewer,
Thank you for reading our response and for raising your score. We hope the new comparisons with CopySpec and SAM Decoding and the discussion of dynamic switching versus tree fusion have addressed your concerns. If there is anything else we can clarify, we would greatly appreciate the opportunity to do so before the discussion period ends.
Best regards, The Authors
We thank the reviewer for recognizing our comprehensive evaluation on agentic and coding tasks, the integration into vLLM, and the encouraging speedup results. We address your concerns below.
W1 (a): "Numerous existing studies have explored combining retrieval-based and model-based speculative decoding. However, the paper significantly lacks citations and comparisons with these closely related methods."
We thank the reviewer for pointing us to these additional works. Section 2 cites SAM Decoding [1] as a parameter-free draft source, and we will expand it to also cover hybrid methods that combine retrieval with a model-based drafter (CopySpec [2], RASD [3], and Graft [4]). Table 1 of the paper already includes a switching hybrid baseline, Frequency-Based Hybrid Decoding (Section 4.1), and the table below summarizes how our method relates to these hybrid designs:
Our method estimates how many tokens of the current cache draft the target model will accept from match statistics, draft structure, decoding position, and recent cache outcomes (Section 3.3), and falls back to the model-based drafter when this estimate is below the threshold. Whereas RASD and Graft run the model-based drafter at every step, this allows our method to skip neural drafting altogether on high-payoff cache cycles. At the same time, SAM Decoding and CopySpec verify every match above a fixed length cutoff even when the continuation diverges early, while our predictor filters out these low-payoff matches before verification. Our method thus delivers large speedups when long continuations recur and preserves the speed of the model-based drafter when they do not.
W1 (b): "The authors should provide a more comprehensive discussion and experimental comparison with these existing hybrid approaches, rather than comparing only against standalone retrieval-based or model-based methods."
Beyond the Frequency-Based Hybrid baseline in Table 1 of the paper, we also evaluated CopySpec [2] and SAM Decoding [1] on Qwen3-8B with EAGLE3 as the fallback drafter.
For CopySpec [2], we implemented its context-matching copy rule at match thresholds γ=3 and γ=5 in our evaluation framework. The table below reports speedup over EAGLE3:
Our method is 1.17x to 2.47x faster than CopySpec on the three workloads with reusable context. On CNN/DailyMail and MT-Bench, CopySpec verifies many false-positive matches and drops up to 19% below EAGLE3, whereas our method matches EAGLE3 (1.00x on both).
To compare the selection rules of CopySpec [2] and SAM Decoding [1] directly against our payoff predictor under identical candidates, we also evaluated them on the same cache drafts across all sixteen datasets on Qwen3-8B in the replay setup of Appendix C:
Fixed match-length cutoffs verify too many low-payoff matches across diverse tasks and fall below EAGLE3 overall (0.64x and 0.95x), while our payoff predictor reaches 1.33x. In end-to-end decoding with SAM Decoding's own suffix automaton, adding a payoff predictor on top of its fixed rule also raises throughput on held-out InstructEdit/FineEdit prompts from 210.5 to 227.1 tokens/s on the same H100 GPU (1.08x, see Reviewer yRDh, Q2). We will add these comparisons to the revised paper.