acceptodds
Under review as a conference paper at ICLR 2027

Caduceus: Parallel Speculative Decoding with Two Simple Drafters Intertwined

Abstract

Speculative decoding can reduce target-model passes by verifying multiple draft tokens at once, but its end-to-end speedup depends on balancing proposal quality against drafting and verification cost. Small language models offer high-quality drafts at the cost of latency, whereas n-gram retrieval is fast but depends on close matches. We introduce Caduceus, a training-free speculative decoding framework that combines these complementary drafters around a persistent draft tree. Within the compound drafter, n-gram retrieval and small-model expansion alternate: retrieval adds candidate paths to the persistent tree, and the small model scores and extends the tree to produce new anchors for subsequent retrieval. Target observations also update a prompt-local online n-gram store. At runtime, the drafter continues growing the tree while the target model verifies a snapshot, and compatible branches persist across verification rounds. Across four target models from 8B to 70B, Caduceus achieves 2.13–3.70× speedup on Spec-Bench and 2.27–3.86× on HumanEval over target-only autoregressive decoding, attaining the highest full-benchmark throughput among the baselines reported for each of the eight benchmark–model settings. On repeated-sampling SpecTTS-Bench with a Llama-70B target, Caduceus achieves 3.99× speedup on the first rollout and 4.39× on the third.

Primary area:
foundation or frontier models, including LLMs
Keywords:
speculative decoding, large language models, inference acceleration, n-gram retrieval, tree decoding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_exorOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Caduceus is a training-free speculative decoding system that intertwines two complementary drafters around a persistent draft tree: fast n-gram retrieval and a small language model that scores and expands branches, producing anchors for further retrieval. A three-level n-gram store (global, optional domain, and prompt-local online) supplies candidates, while drafting overlaps asynchronously with target verification and compatible branches survive across rounds. On Spec-Bench and HumanEval with targets from 8B to 70B, it reports 2.13×–3.70× and 2.27×–3.86× speedups over autoregressive decoding and the best full-benchmark throughput among reported baselines in each setting; repeated sampling on SpecTTS-Bench further benefits from the growing prompt-local store. The design aims to balance proposal quality against drafting cost without target-specific training.

Critical strengths

Clear systems recipe. Alternating n-gram and small-model growth on a persistent tree, plus draft–verify overlap, directly targets the quality–latency trade-off of single drafters.

Broad empirical comparison. Spec-Bench, HumanEval, and SpecTTS-Bench across four draft–target pairs (up to 70B), with strong baselines including EAGLE, PipeSpec/PEARL, and retrieval methods, plus ablations isolating each component.

Training-free practicality. No target-specific draft head is required, and online L3 adaptation helps under repeated sampling.

Critical weaknesses

Incremental novelty. Core pieces—n-gram retrieval, small-model drafting, trees, and async overlap—already exist; Caduceus mainly composes and schedules them, so algorithmic novelty is limited.

No mainstream serving validation. Experiments use a custom runtime, not vLLM/SGLang. The persistent tree, multi-process overlap, and CPU/GPU intertwining are hard to port; paper speedups may not transfer to production engines.

Limited scale. The largest target is 70B (AWQ, two A100s). Despite being training-free—and thus easier to try on 405B-class models—no larger evaluation is shown.

Incomplete dialogue with related lines. It does not engage methods such as FLy [1], which is also a training free method using n-gram+draft.

[1] Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Match.

Questions for authors

If the author can solve the problem mentioned above, I will consider raising my score.

2 comments

We thank you for your careful review and constructive feedback, and apologize for the late response: the additional experiments took longer than anticipated. We have repeated most experiments with colocated drafting and improved implementation details without changing Caduceus's algorithmic workflow. The revised manuscript includes these results and the comparisons below.

Q1. Contribution beyond existing components.

Caduceus's contribution is the interaction between retrieval, model expansion, and asynchronous verification. Retrieval extends a persistent tree; the small model scores branches from both sources and supplies new anchors for further retrieval. Target observations update L3 online, while verifier-side snapshot alignment allows verification to proceed without waiting for drafter alignment when compatible candidates are available.

Appendix B.5 isolates repeated alternation from retrieving only at initialization and after each alignment, with the same tree budgets, branching settings, stores, and asynchronous execution:

Target / benchmarkOnce TPSAlternating TPSGain
8B / Spec-Bench57.1464.5012.9%
8B / HumanEval58.7267.0114.1%
70B / Spec-Bench23.0425.068.8%
70B / HumanEval22.5025.2012.0%

These gains show the value of repeatedly coupling the two proposal sources. Section 4.4 further shows that the complete system preserves the n-gram drafter's ability to improve across repeated samples as L3 accumulates observations.

Q2. Transfer beyond our original runtime.

Appendix B.6 evaluates Caduceus using SwiftSpec-derived model kernels and shared SDPA tree attention. Caduceus and the adapted SwiftSpec-style baseline use the same Llama-70B target, Llama-1B drafter, and three A100 GPUs:

BenchmarkSwiftSpec-style TPSCaduceus TPSGain
Spec-Bench50.2453.857.2%
HumanEval44.3947.186.3%

Caduceus thus provides additional gains on SwiftSpec-derived execution infrastructure, supporting transfer beyond our original model-execution backend.

Q3. Model scale.

Our evaluation covers targets from 8B to 70B. Across these targets, speedups over AR increase with model size under both deployment modes (Section 4.2; Appendix B.1):

TargetBenchmarkAR TPSColocated TPSDedicated TPS
Llama-8BSpec-Bench28.4855.6264.50
Llama-8BHumanEval28.1860.0767.01
Qwen-32BSpec-Bench9.8927.0029.56
Qwen-32BHumanEval9.6229.9633.18
Llama-70BSpec-Bench6.2624.8825.06
Llama-70BHumanEval6.0523.9525.20

On 70B, Caduceus achieves 3.98×/3.96× speedup on Spec-Bench/HumanEval with colocated drafting and 4.01×/4.16× with dedicated drafting. Slower target verification offers more opportunity to overlap drafting and amortize coordination costs. We currently lack sufficient GPU resources to evaluate a 405B target model.

Q4. Relationship to FLy.

Thank you for pointing out FLy (Li et al., ICLR 2026). We have added its citation and discussed its use of prompt lookup to accelerate drafting in Section 2.

FLy's multi-level drafting is conceptually closer to our serial variant. Caduceus further couples retrieval and small-model expansion within a persistent tree and overlaps drafting with target verification. Its retrieval draws on global public-corpus L1, optional domain-specific L2, and prompt-local L3 updated from target-model distributions.

Retrieval rapidly adds multi-token branches at multiple model-selected anchors; the small model scores and extends these branches, retains promising children, and selects the next retrieval anchors. Retrieval thus supplies inexpensive growth, while the small model guides where the tree grows next. Compatible branches persist across verification rounds, sustaining this interaction.

Reviewer_exorOpenReview ↗

Post-rebuttal decision

After reading the author's rebuttal, I decide to keep my score unchanged.

Reviewer_9xEFOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes Caduceus, a training-free speculative decoding system that combines n-gram retrieval with an off-the-shelf small draft model around a persistent draft tree. Within the drafter, retrieval from a three-level n-gram store alternates with small-model tree expansion; the small model scores retrieved branches and selects anchors for the next retrieval. The store consists of static offline L1/L2 and an online, prompt-local L3 built from target distributions. At the system level, drafting runs in a separate process on a dedicated GPU and overlaps with target verification; snapshot alignment lets verification proceed without waiting for the drafter. Across four targets (8B–70B) on A100 PCIe GPUs, the paper reports 2.13–3.70× speedup on Spec-Bench and 2.27–3.86× on HumanEval over target-only autoregressive (AR) decoding. It also reports gains on SpecTTS-Bench and under small fixed batches.

Critical strengths
  1. The compound drafter is simple and requires no draft-model training. Using the small model to score retrieved branches and select retrieval anchors is a reasonable way to combine the two sources.
  2. The ablations isolating the n-gram sources, the small model, and draft–verify overlap (Tables 3–6) are informative. The appendix provides detailed pseudocode and cost measurements.
  3. The gains on the 70B target, and the improvement across rollouts from L3 on SpecTTS-Bench, are notable.
Critical weaknesses
  1. Unequal hardware. Caduceus runs its drafter on an extra GPU, whereas AR, the retrieval baselines, and EAGLE use none. Normalized by GPU count, its Spec-Bench speedups (1.07×, 1.53×, 1.39×, 2.47×) fall below EAGLE-3 (2.08×, 2.89×) and EAGLE-2 (2.31×) wherever those are reported. Neither equal-hardware alternatives (e.g., a tensor-parallel target with a trained drafter) nor a single-GPU Caduceus are evaluated.
  2. Missing strong baselines. EAGLE-3 is absent for two of the four targets. DFlash is not compared, although it has a released drafter for the paper’s 8B target, where Caduceus leads EAGLE-3 by only 2.13× vs. 2.08×. The paper also does not explain why asynchronous overlap still helps when drafting takes a single forward pass.
  3. Domain-matched corpus. L1 contains the source datasets of several evaluation domains (CNN/DailyMail, GSM8K, WMT14, MBPP, Magicoder), and L1b requires target-generated responses. Without L1 (Table 6), Caduceus falls below EAGLE-3 on Llama-8B and below EAGLE-2 on Vicuna-33B HumanEval. The out-of-domain RareArena experiment reports no AR or baseline numbers.
Questions for authors
  1. Can the authors report GPU-normalized throughput, and an equal-GPU-count latency comparison against a tensor-parallel target (TP=2) with AR, EAGLE-3, or DFlash? Ideally this would also be run on an NVLink system.
  2. What speedup does Caduceus achieve with the drafter colocated on the target GPU? How does Caduceus compare with DFlash on Llama-3.1-8B-Instruct? Does n-gram retrieval with a persistent tree add gains on top of a trained drafter?
  3. What are the main results, per domain, after removing benchmark-source datasets from L1a/L1b? What are the AR and EAGLE-3 throughputs on RareArena?
1 comment

We thank you for your careful review and constructive feedback, and apologize for the late response: the additional experiments took longer than anticipated. We have repeated most experiments with colocated drafting and improved implementation details without changing Caduceus's algorithmic workflow. The revised manuscript includes these results and the comparisons below.

Q1. Equal GPU resources and colocated drafting.

We now evaluate colocated Caduceus using the same GPU count as AR and colocated EAGLE-3. At matched GPU counts, relative TPS gains also equal relative TPS-per-GPU gains (Section 4.2; Appendix B.1).

TargetGPUsBenchmarkAR TPSCaduceus TPSEAGLE-3 TPS
Llama-8B1Spec-Bench28.4855.6268.46
Llama-8B1HumanEval28.1860.0768.11
Qwen-32B1Spec-Bench9.8927.0027.02
Qwen-32B1HumanEval9.6229.9624.40
Llama-70B2Spec-Bench6.2624.8819.97
Llama-70B2HumanEval6.0523.9519.29

On greedy Spec-Bench and HumanEval, Caduceus trails EAGLE-3 on 8B but achieves 24.6% and 24.2% higher throughput on 70B, respectively.

On 70B SpecTTS-Bench, colocated Caduceus leads colocated EAGLE-3 throughout, with its advantage growing from 55.9% to 69.9% across rollouts (Section 4.4; Appendix B.2):

RolloutCaduceus TPSEAGLE-3 TPS
132.0220.54
233.4320.31
334.1020.24
435.0020.60

Q2. EAGLE-3, DFlash, and repeated sampling.

On Llama-8B, both trained drafters outperform Caduceus on greedy Spec-Bench and HumanEval. Caduceus is more effective for accelerating larger targets in our experiments. However, their advantages over Caduceus narrow substantially on SpecTTS-Bench, even on the first rollout, before observations accumulate across samples. All entries below use one A100 GPU:

BenchmarkCaduceus TPSEAGLE-3 TPSDFlash TPS
Spec-Bench55.6268.4678.93
HumanEval60.0768.1180.56
SpecTTS-Bench, rollout 166.8569.9671.03

With colocated drafting, Caduceus overtakes both methods from rollout 2. By rollout 4, it leads EAGLE-3 by 8.5% and DFlash by 5.0%. With dedicated drafting, it exceeds DFlash on all four rollouts, with its advantage growing from 8.5% to 22.5% (Figures 7 and 13):

RolloutColocated Caduceus TPSColocated EAGLE-3 TPSColocated DFlash TPSDedicated Caduceus TPSDedicated DFlash TPS
166.8569.9671.0372.1666.49
272.8370.4271.7679.4467.37
375.0469.9571.8481.2567.69
475.7669.8472.1382.7167.52

Caduceus's n-gram store accumulates target observations across samples, enabling adaptation without retraining. Draft performance can also depend on how well training data matches the downstream workload (TAPS: Task Aware Proposal Distributions for Speculative Sampling, Zbib et al., 2026), which may contribute to the observed differences across benchmarks.

Many trained drafters, including EAGLE-3 and DFlash, depend on target-model hidden states, making direct integration with Caduceus's independently running drafter nontrivial.

Q3. Datastore dependence and RareArena.

Removing the heuristically added target-generated L1b changes throughput by less than 1%, so we removed it from the main configuration and retain public-corpus L1 plus online L3.

In a separate public-corpus ablation, removing CNN/DailyMail, GSM8K, WMT14, MBPP, and Magicoder retains 98.2%–99.1% of throughput:

Target / benchmarkFull public L1 TPSAfter removal TPSChange
8B / Spec-Bench64.5063.46−1.6%
8B / HumanEval67.0165.80−1.8%
70B / Spec-Bench25.0624.83−0.9%
70B / HumanEval25.2024.88−1.3%

RareArena provides a supplementary ablation of the optional L2 layer. Between global static L1 and prompt-local, online L3, L2 supplies domain-specific information reusable across requests. Adding L2 to L1+L3 improves throughput by 9.0–16.4%, supporting its role as a natural, complementary layer. Table 14 includes AR references (Appendix B.3):

TargetAR TPSL1+L3 TPSL1+L2+L3 TPS
Llama-8B28.2257.15366.508
Qwen-32B9.7726.67130.016
Llama-70B6.1518.72320.416
Reviewer_nXAnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Caduceus is a training-free speculative decoding framework that alternates n-gram retrieval and small-model expansion within a persistent draft tree, while overlapping drafting with target verification. Experiments demonstrate throughput improvements across multiple models and tasks, with ablations supporting the benefits of compound drafting and asynchronous execution.

Critical strengths
  • Effective retrieval augmentation. Ablations show that n-gram retrieval improves throughput beyond the asynchronous small-model baseline.

  • Training-free design. Retrieval and small-model expansion complement each other within a reusable draft tree.

  • Practical value. Despite its simplicity, the approach offers useful inference acceleration without specialized training.

Critical weaknesses
  • The improvements over SwiftSpec are unclear.

    SwiftSpec[1] already runs tree drafting and verification in parallel and reuses valid branches and their KV caches. Although Caduceus cites SwiftSpec, it does not provide an experimental comparison. Consequently, the current results do not establish how much improvement comes specifically from alternating retrieval–model expansion or verifier-side local snapshot alignment over an existing persistent-tree asynchronous system.

  • The evaluation does not fully separate gains from offline target-specific data and additional hardware resources.

    L1 contains data generated by the target model (L1b, as described in Appendix), but the ablations do not isolate its contribution from public data (L1a). It is therefore unclear how much L1b improves performance and whether that benefit justifies its preparation cost. I am concerned that the effectiveness of n-gram retrieval may depend heavily on L1b.

    Caduceus also uses an extra draft GPU compared with AR and EAGLE[2]. Higher throughput under this setup does not necessarily mean better resource efficiency. Reporting resource-normalized metrics, such as throughput per GPU, or comparisons under matched hardware budgets would make the paper's practical contribution more convincing.

[1] Zhang, et. al, SwiftSpec: Disaggregated Speculative Decoding and Fused Kernels for Low-Latency LLM Inference. ASPLOS 2026

[2] Li, et. al, EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test. NeurIPS 2025.

Questions for authors
  • Q1. L1b ablation

    How much throughput remains when L1b is removed while retaining L1a, L3, the small model, and the same runtime configuration? Please report this comparison across representative targets. Could you also vary L1b size and its mixing weight separately, and provide a size-matched public-data control to distinguish target-specific data benefits from increased datastore capacity?

  • Q2. SwiftSpec comparison

    Why was SwiftSpec omitted from the experimental baselines despite its close architectural relationship and inclusion in Related Work? Could you provide a comparison on at least one representative model pair and benchmark under matched total GPU resources, with inference-backend and kernel differences controlled or explicitly quantified? If direct reproduction is infeasible, a clearly specified SwiftSpec-style baseline in the same backend would still help assess the algorithmic increment.

  • Q3. Resource efficiency

    Please add comparisons under a fixed total GPU budget, or report resource-aware metrics, such as throughput per GPU, alongside raw speedup. I recognize the value of asynchronous CPU–GPU collaboration and system-level optimization. However, Caduceus uses more GPUs than some baselines, including AR and EAGLE. Without accounting for this difference, the reported gains provide less convincing evidence of its practical advantage.

  • Q4. Attribution to the proposed composition and scheduling

    How much improvement comes from repeatedly alternating retrieval and small-model expansion, compared with a simpler one-shot retrieval augmentation under comparable candidate and compute budgets?

I see this paper as borderline between weak accept and weak reject, and currently lean toward weak reject. I would be happy to raise my rating to weak accept if the authors can more clearly establish Caduceus's contribution, particularly through an L1b ablation and a comparison with SwiftSpec under matched resource budgets.

4 comments

As I have not received a rebuttal from the authors, I will retain my original score for the time being.

We thank you for your careful review and constructive feedback, and apologize for the late response: the additional experiments took longer than anticipated. We have repeated most experiments with colocated drafting and improved implementation details without changing Caduceus's algorithmic workflow. The revised manuscript includes these results and the comparisons below.

Q1. L1b.

Our experiments show that removing the heuristically added L1b changes throughput by less than 1%. We therefore removed L1b, retaining public-corpus L1 and online, prompt-local L3 alongside the small-model drafter. Across three targets and two benchmarks, this configuration achieves 2.26–4.16× speedup over AR with dedicated drafting and 1.95–3.98× with colocated drafting (Section 4.2; Appendix B.1).

Q2. SwiftSpec comparison.

Caduceus is compatible with SwiftSpec's execution infrastructure. Using SwiftSpec-derived model kernels and shared SDPA tree attention, it achieves 1.063–1.072× the throughput of our adapted SwiftSpec baseline, a further 6.3–7.2% improvement. Both use the same Llama-70B target, Llama-1B drafter, and three A100 GPUs (Appendix B.6).

BenchmarkSwiftSpec TPSCaduceus TPSGain
Spec-Bench50.2453.857.2%
HumanEval44.3947.186.3%

Q3. Equal GPU resources.

With colocated drafting, Caduceus, EAGLE-3, and AR use the same GPU count. At matched GPU counts, relative TPS gains also equal relative TPS-per-GPU gains (Section 4.2; Appendix B.1).

TargetGPUsBenchmarkAR TPSCaduceus TPSEAGLE-3 TPS
Llama-8B1Spec-Bench28.4855.6268.46
Llama-8B1HumanEval28.1860.0768.11
Qwen-32B1Spec-Bench9.8927.00—
Qwen-32B1HumanEval9.6229.96—
Llama-70B2Spec-Bench6.2624.8819.97
Llama-70B2HumanEval6.0523.9519.29

On greedy Spec-Bench and HumanEval, Caduceus trails EAGLE-3 on 8B but achieves 24.6% and 24.2% higher throughput on 70B, respectively. On 8B, target verification is relatively fast, and the 1B drafter offers only a limited forward-latency advantage in our tree-based execution, leaving less room to overlap drafting and amortize tree-management and coordination overhead.

On SpecTTS-Bench with colocated drafting, EAGLE-3's advantage over Caduceus on 8B is already smaller on the first rollout than on Spec-Bench. Caduceus overtakes it from rollout 2 and leads by 8.5% on rollout 4. On 70B, Caduceus leads throughout, with its advantage growing from 55.9% to 69.9% as L3 accumulates observations (Section 4.4; Appendix B.2):

Rollout8B Caduceus TPS8B EAGLE-3 TPS70B Caduceus TPS70B EAGLE-3 TPS
166.8569.9632.0220.54
272.8370.4233.4320.31
375.0469.9534.1020.24
475.7669.8435.0020.60

Q4. Alternating versus once-per-alignment retrieval.

With the same tree budgets, branching settings, stores, and asynchronous execution, repeated alternation improves throughput by 8.8–14.1% over retrieving only at initialization and after each alignment (Appendix B.5).

Target / benchmarkOnce TPSAlternating TPSGain
8B / Spec-Bench57.1464.5012.9%
8B / HumanEval58.7267.0114.1%
70B / Spec-Bench23.0425.068.8%
70B / HumanEval22.5025.2012.0%

Small-model expansion supplies new retrieval anchors. On 8B Spec-Bench, this reduces mean target forward calls from 60.3 to 51.0.

Thank you for the additional experiments. They address several concerns, but three points still need clarification:

  1. Q1: Please provide per-setting TPS with and without L1b under identical configurations, rather than only stating that the difference is below 1%.

  2. Q2: Please define the adapted SwiftSpec-style baseline more precisely: which original mechanisms are retained, what modifications were made, and how its tree construction, scheduling, alignment, and budgets differ from Caduceus. Shared kernels and matched hardware alone do not establish that this is a representative comparison.

  3. Q3: The colocated experiments address the GPU-count concern, but weaken the case for a general advantage over EAGLE-3: Caduceus trails on 8B greedy decoding and leads on 70B. Please narrow the contribution and performance claims accordingly, and explain the missing Qwen-32B EAGLE-3 results. Universal superiority is unnecessary, but the claims should accurately reflect the demonstrated advantages and limitations.

These clarifications would help me assess the remaining concerns and the strength of the revised contribution.

Thank you for reviewing our additional experiments and for these follow-up questions. We address the three points below.

Q1. Per-setting L1b ablation.

TargetDeploymentBenchmarkWith L1b TPSWithout L1b TPSChange after removal
Llama-8BColocatedSpec-Bench55.8755.62−0.45%
Qwen-32BColocatedSpec-Bench26.8427.00+0.59%
Llama-70BColocatedSpec-Bench24.9324.88−0.21%

Q2. Clarification of the adapted SwiftSpec baseline.

Thank you for requesting clarification. By “SwiftSpec-style,” we mean SwiftSpec adapted to our local hardware and models. We compile for A100, replace the Hopper-specific RMSNorm kernel, use separate GEMM and NCCL all-reduce because GPU P2P is unavailable in our setup, and load the 70B AWQ target and 1B FP16 drafter with the required weight-layout conversion. We replace the native custom tree-attention kernel with shared SDPA tree attention in both methods because the locally adapted native path did not pass numerical consistency checks. SwiftSpec’s core tree-construction, candidate-selection, and scheduling logic remains unchanged.

Caduceus uses this same model-execution backend, including target TP=2 and CUDA Graph support. Its adapter passes tree tokens, positions, and attention masks to the backend and maps cache-retention decisions to backend KV-cache operations; its compound drafting algorithm is retained. Both methods use Llama-3.3-70B-Instruct-AWQ and Llama-3.2-1B-Instruct on three A100 GPUs.

AspectAdapted SwiftSpecCaduceus
Tree constructionSmall-model expansion with score-based candidate selectionAlternating retrieval and model expansion: retrieval adds multi-token branches, and the model scores and extends them to supply new retrieval anchors
SchedulingVerifies a transmitted candidate tree while drafting continues; the next candidate tree is sent after the configured expansion rounds and processing of verified tokensPublishes updated snapshots during compound drafting; the verifier can reuse an already-published snapshot compatible with the committed prefix without waiting for drafter completion, when sufficient candidates remain

Both methods were tuned on 24 Spec-Bench prompts, selecting the highest-throughput configuration tested for each method. Under this shared adapted backend, Caduceus improves mean per-request throughput over SwiftSpec by 7.2% on Spec-Bench and 6.3% on HumanEval.

Q3. Scope of performance claims and Qwen-32B results.

We agree that the results do not establish a universal advantage over EAGLE-3. Our contribution is a training-free framework combining retrieval and small-model drafting, with performance depending on the target and workload. Section 4.2 now explicitly states that Caduceus trails EAGLE-3 on 8B greedy decoding and leads on 70B under both deployments. The Conclusion also notes that tree-management and coordination costs limit speedups for smaller targets. With matched-GPU colocated deployment, its 70B throughput gains are 24.6% on Spec-Bench and 24.2% on HumanEval. On SpecTTS-Bench with colocated drafting, Caduceus overtakes EAGLE-3 from rollout 2 on 8B and leads throughout on 70B.

In our initial experiments, we mistakenly believed that no suitable EAGLE-3 draft model was publicly available for Qwen2.5-32B-Instruct. We have now completed the evaluation and added the results to the revised manuscript (Appendix B.1, Tables 3 and 4). The completed matched-GPU comparison below uses colocated drafting:

TargetGPUsBenchmarkAR TPSCaduceus TPSEAGLE-3 TPS
Llama-8B1Spec-Bench28.4855.6268.46
Llama-8B1HumanEval28.1860.0768.11
Qwen-32B1Spec-Bench9.8927.0027.02
Qwen-32B1HumanEval9.6229.9624.40
Llama-70B2Spec-Bench6.2624.8819.97
Llama-70B2HumanEval6.0523.9519.29

On Qwen-32B, colocated throughput is nearly identical on Spec-Bench, while Caduceus is faster on HumanEval.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.