- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Caduceus is a training-free speculative decoding system that intertwines two complementary drafters around a persistent draft tree: fast n-gram retrieval and a small language model that scores and expands branches, producing anchors for further retrieval. A three-level n-gram store (global, optional domain, and prompt-local online) supplies candidates, while drafting overlaps asynchronously with target verification and compatible branches survive across rounds. On Spec-Bench and HumanEval with targets from 8B to 70B, it reports 2.13×–3.70× and 2.27×–3.86× speedups over autoregressive decoding and the best full-benchmark throughput among reported baselines in each setting; repeated sampling on SpecTTS-Bench further benefits from the growing prompt-local store. The design aims to balance proposal quality against drafting cost without target-specific training.
Critical strengths
Clear systems recipe. Alternating n-gram and small-model growth on a persistent tree, plus draft–verify overlap, directly targets the quality–latency trade-off of single drafters.
Broad empirical comparison. Spec-Bench, HumanEval, and SpecTTS-Bench across four draft–target pairs (up to 70B), with strong baselines including EAGLE, PipeSpec/PEARL, and retrieval methods, plus ablations isolating each component.
Training-free practicality. No target-specific draft head is required, and online L3 adaptation helps under repeated sampling.
Critical weaknesses
Incremental novelty. Core pieces—n-gram retrieval, small-model drafting, trees, and async overlap—already exist; Caduceus mainly composes and schedules them, so algorithmic novelty is limited.
No mainstream serving validation. Experiments use a custom runtime, not vLLM/SGLang. The persistent tree, multi-process overlap, and CPU/GPU intertwining are hard to port; paper speedups may not transfer to production engines.
Limited scale. The largest target is 70B (AWQ, two A100s). Despite being training-free—and thus easier to try on 405B-class models—no larger evaluation is shown.
Incomplete dialogue with related lines. It does not engage methods such as FLy [1], which is also a training free method using n-gram+draft.
[1] Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Match.
Questions for authors
If the author can solve the problem mentioned above, I will consider raising my score.
2 comments
Post-rebuttal decision
After reading the author's rebuttal, I decide to keep my score unchanged.
We thank you for your careful review and constructive feedback, and apologize for the late response: the additional experiments took longer than anticipated. We have repeated most experiments with colocated drafting and improved implementation details without changing Caduceus's algorithmic workflow. The revised manuscript includes these results and the comparisons below.
Q1. Contribution beyond existing components.
Caduceus's contribution is the interaction between retrieval, model expansion, and asynchronous verification. Retrieval extends a persistent tree; the small model scores branches from both sources and supplies new anchors for further retrieval. Target observations update L3 online, while verifier-side snapshot alignment allows verification to proceed without waiting for drafter alignment when compatible candidates are available.
Appendix B.5 isolates repeated alternation from retrieving only at initialization and after each alignment, with the same tree budgets, branching settings, stores, and asynchronous execution:
These gains show the value of repeatedly coupling the two proposal sources. Section 4.4 further shows that the complete system preserves the n-gram drafter's ability to improve across repeated samples as L3 accumulates observations.
Q2. Transfer beyond our original runtime.
Appendix B.6 evaluates Caduceus using SwiftSpec-derived model kernels and shared SDPA tree attention. Caduceus and the adapted SwiftSpec-style baseline use the same Llama-70B target, Llama-1B drafter, and three A100 GPUs:
Caduceus thus provides additional gains on SwiftSpec-derived execution infrastructure, supporting transfer beyond our original model-execution backend.
Q3. Model scale.
Our evaluation covers targets from 8B to 70B. Across these targets, speedups over AR increase with model size under both deployment modes (Section 4.2; Appendix B.1):
On 70B, Caduceus achieves 3.98×/3.96× speedup on Spec-Bench/HumanEval with colocated drafting and 4.01×/4.16× with dedicated drafting. Slower target verification offers more opportunity to overlap drafting and amortize coordination costs. We currently lack sufficient GPU resources to evaluate a 405B target model.
Q4. Relationship to FLy.
Thank you for pointing out FLy (Li et al., ICLR 2026). We have added its citation and discussed its use of prompt lookup to accelerate drafting in Section 2.
FLy's multi-level drafting is conceptually closer to our serial variant. Caduceus further couples retrieval and small-model expansion within a persistent tree and overlaps drafting with target verification. Its retrieval draws on global public-corpus L1, optional domain-specific L2, and prompt-local L3 updated from target-model distributions.
Retrieval rapidly adds multi-token branches at multiple model-selected anchors; the small model scores and extends these branches, retains promising children, and selects the next retrieval anchors. Retrieval thus supplies inexpensive growth, while the small model guides where the tree grows next. Compatible branches persist across verification rounds, sustaining this interaction.