acceptodds
Under review as a conference paper at ICLR 2027

SpecRouter: Reusable Payoff Prediction for Cost-Aware Speculative Decoding

25 views

Abstract

Speculative decoding can obtain draft tokens in very different ways—copying from the context, n-gram lookup, a small draft model, or a learned tree drafter—and which drafter is fastest depends both on the text being generated and on the hardware and batch size that execute it. We introduce , which separates these two factors. A 0.58M-parameter predictor reads the target model's hidden state and estimates how many tokens each drafter would commit; a separately calibrated cost model prices each drafter on the current hardware; only the drafter with the lowest predicted cost per committed token runs. The predictor needs no exploration to train: with deterministic drafters and greedy verification, the output is the same whichever drafter runs, so every drafter's committed length at every prefix can be computed exactly offline. We evaluate on sixteen workloads with 100 held-out requests each. Averaged over workloads, speeds up decoding by on Llama-3.1-8B and on Qwen3-8B, which is 30% and 27% higher than the speedup of EAGLE-3, the strongest fixed drafter. It also exceeds the best fixed drafter chosen separately for each workload in hindsight. Comparing costs explicitly is the largest single source of this gain. also outperforms online methods that learn drafter choice during generation, reaching versus on a pool of domain-specialized drafters. A single predictor, never retrained, is the fastest policy on A40 and H200 GPUs at batch sizes 1, 4, and 16; only its cost model is recalibrated.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Speculative Decoding, router, inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RhTSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the problem of dynamically selecting the most efficient drafter in speculative decoding, as different drafting strategies exhibit varying efficiency across generation contexts and hardware configurations. It proposes SpecRouter, which predicts the accepted length of each drafter using target model hidden states and selects the one with the lowest estimated cost per committed token. Experiments on 16 workloads show 2.75× and 2.95× speedups on Llama-3.1-8B and Qwen3-8B, respectively, outperforming EAGLE-3.

Critical strengths
  1. Practical problem. The paper addresses the practical challenge of dynamically selecting among heterogeneous speculative decoding strategies, whose efficiency varies across generation contexts and deployment configurations. This is a relevant problem for improving inference efficiency.
  2. Simple cost-aware routing framework. SpecRouter separates drafter payoff prediction from execution cost estimation, enabling dynamic selection based on predicted cost per committed token. The design is intuitive and easy to follow, and allows the payoff predictor to be reused across different deployment configurations without retraining.
Critical weaknesses
  1. The core idea of predicting drafter payoffs and selecting the action with the lowest estimated cost per committed token is relatively straightforward, while adaptive drafter selection has already been extensively explored in prior work, so the novelty is limited. Moreover, the ablation studies show that using only simple prefix statistics achieves performance close to the full method (2.59× vs. 2.75×), while a static cost model performs almost identically to the full cost model. These results suggest that the additional components provide limited incremental benefits, weakening the evidence for the paper's claimed methodological contributions.
  2. The proposed method achieves relatively modest improvements over existing hybrid routing baselines (2.75× vs. 2.55× on Llama and 2.95× vs. 2.76× on Qwen). Furthermore, the evaluation only includes two 8B target models, without experiments on larger backbones. It remains unclear whether these improvements can be maintained at larger model scales, where the relative costs and effectiveness of different drafting strategies may change.
Questions for authors
  1. Given the limited gains from target hidden states and dynamic cost modeling, how does SpecRouter compare with a simpler lightweight cost-aware routing baseline?
  2. I want to see experimental results on larger target models (e.g., 32B or 70B) to evaluate whether the performance gains generalize beyond 8B models.
4 comments
AuthorsOpenReview ↗

Reply to Reviewer RhTS (1/2)

We thank the reviewer for the review, and for recognizing the practical problem and the separation that lets one payoff predictor be reused across deployments.

W1. Novelty

"The core idea of predicting drafter payoffs and selecting the action with the lowest estimated cost per committed token is relatively straightforward, while adaptive drafter selection has already been extensively explored in prior work, so the novelty is limited."

We agree the decision rule is simple, and we see that as a strength: it is the right rule once both of its inputs are available. What prior selectors lacked is a way to supply the payoff term for drafters of different families without running them. Online methods learn it from verification feedback and pay for exploration; threshold hybrids approximate it for a single pair of drafters. SpecRouter obtains it exactly offline: deterministic proposals make every drafter's committed tokens at every prefix an exact label (Proposition 1; Proposition 2 under sampling), and keeping payoff separate from deployment cost lets the learned part transfer while only a cost table is recalibrated. If the reviewer has specific works in mind, we would be glad to add a direct comparison. The closest work we know:

WorkDateChooses amongHow
SAM decodingarXiv, Nov 2024Copying vs an auxiliary drafterMatch-length rule
SuffixDecodingNeurIPS 2025Copying vs a neural fallbackMatch-score threshold
BanditSpecICML 2025Configurations of one speculative methodOnline bandit
HedgeSpecICLR 2026Seven domain-tuned EAGLE-3 draftersOnline hedging on accepted length
MetaSDarXiv, Apr 2026Domain-specialized draftersOnline bandit
HVDarXiv, Jun 2026 (concurrent)A suffix-cache draft vs EAGLE-3Fixed threshold on a learned accepted length

SpecRouter differs from all of them: it routes per block across three or more drafter families with different proposal mechanisms, it is trained offline on exact labels with no exploration cost, and it compares drafters by predicted time per committed token, so one predictor carries across deployments.

The two established learned selectors were already baselines in the submission, and SpecRouter beats both. BanditSpec's UCB and EXP3 reach 1.80× and 1.73× against SpecRouter's 2.77× (Table 3); with its throughput reward added, the best of eight variants still reaches only 1.84× against 2.56× under sampling. Measured live on HedgeSpec's own expert pool, HedgeSpec reaches 3.68× and SpecRouter 4.14×.

W1. Prefix statistics and static costs

"Moreover, the ablation studies show that using only simple prefix statistics achieves performance close to the full method (2.59× vs. 2.75×), while a static cost model performs almost identically to the full cost model. These results suggest that the additional components provide limited incremental benefits, weakening the evidence for the paper's claimed methodological contributions."

The reviewer is right that target states and state-dependent costs are not where most of the gain comes from, and our new results confirm it: static costs match fitted ones, and target states add 4–6% in four of five settings and nothing in one. The camera-ready says this plainly. The gain comes from what we claim as the contribution, exact supervision and explicit cost comparison:

  • Removing the cost comparison loses 16% (Table 4).
  • Every lightweight router trained on the same labels with the same cost rule beats EAGLE-3 (Q1 below).
  • Context-free online selectors, which lack exact supervision, fall below fixed EAGLE-3 (best of eight BanditSpec variants: 1.84× vs 2.03×).
SettingPrefix statistics only+ Target states
Llama-3.1-8B, greedy (submitted)2.59×2.75×
Qwen3-8B, T = 0.72.61×2.73×
Qwen3-32B, greedy2.51×2.66×
Llama-3.3-70B, greedy3.40×3.61×
Llama-3.1-8B, T = 0.62.53×2.52×

We therefore present target states as an optional input with a consistent but modest benefit, not as the source of the method's value. Costs still have to be compared and recalibrated per deployment, even when they are static: at batch size 16, where every single drafter slows decoding down (fixed EAGLE-3: 0.78×), recalibrated costs let SpecRouter reach 1.08× (Table 18).

AuthorsOpenReview ↗

Reply to Reviewer RhTS (2/2)

W2. Gains over hybrids, and scale

"The proposed method achieves relatively modest improvements over existing hybrid routing baselines (2.75× vs. 2.55× on Llama and 2.95× vs. 2.76× on Qwen). Furthermore, the evaluation only includes two 8B target models, without experiments on larger backbones. It remains unclear whether these improvements can be maintained at larger model scales, where the relative costs and effectiveness of different drafting strategies may change."

The SuffixDecoding-style threshold is the existing method in our comparison; the HVD-style row reuses our own SAM payoff head, since HVD's code was not public when we ran our experiments. Against SuffixDecoding-style, SpecRouter is 9% faster on Llama (2.75× vs 2.52×) and 8% faster on Qwen (2.95× vs 2.74×) at batch size 1, closing roughly 40% of the remaining gap between the threshold and the local hindsight reference (Table 1). The margin widens where thresholds are brittle:

  • Unseen workloads. With each workload held out of training entirely, SpecRouter reaches 2.59×, above both thresholds (2.52× and 2.55×), which were tuned on validation data that includes those workloads (Tables 1 and 4).
  • New deployments. Frozen at A40 batch size 1, the SuffixDecoding-style threshold slows decoding at batch size 16 (0.90× on A40, 0.93× on H200), while SpecRouter's unchanged predictor speeds it up (1.04× with the same frozen costs, 1.08× after recalibrating costs; Table 18).
  • More than two drafters. A threshold arbitrates exactly one pair of drafters; SpecRouter extends to any pool, and adding a parallel drafter (PARD) gives a further 9.3% on Llama.

On a two-drafter pool with thresholds retuned for each sampling setting, the learned router also wins in all four settings (2.54× vs 2.43× on Llama and 2.71× vs 2.64× on Qwen at the recommended settings). SpecRouter's gains over EAGLE-3 also carry over to larger targets (Q2 below).

Q1. Simpler cost-aware routers

"Given the limited gains from target hidden states and dynamic cost modeling, how does SpecRouter compare with a simpler lightweight cost-aware routing baseline?"

Every lighter router built on our labels and cost rule beats EAGLE-3 (2.20× on Qwen3-8B, 2.13× on Qwen3-32B), which confirms that exact supervision plus cost comparison carries most of the value. A learned per-block predictor with target states is the best of them:

RouterQwen3-8B, T = 0.7Qwen3-32B
Running average of each drafter's payoff2.28×2.26×
Best fixed drafter per workload, picked on validation2.45×2.39×
Gradient-boosted trees, prefix statistics2.49×2.35×
MLP, prefix statistics2.61×2.51×
SpecRouter2.73×2.66×

Q2. Larger targets

"I want to see experimental results on larger target models (e.g., 32B or 70B) to evaluate whether the performance gains generalize beyond 8B models."

The gains generalize to both scales the reviewer names, with a new predictor per target trained by the same recipe:

TargetSetupEAGLE-3SpecRouter
Qwen3-32B1× H200, 16 workloads, 1,600 requests2.13×2.66× (+24.9%)
Llama-3.3-70B2× H200 tensor parallel, 8 workloads, 800 requests2.64×3.61× (+36.7%)

The margin at 32B matches the 8B results (24–30%), and at 70B it is larger still.

Closing

We believe these results address the concerns about novelty and scale, and we respectfully ask the reviewer to reconsider the score.

Thank you for the detailed response. The additional experiments on larger models and comparisons with simpler routers substantially address my concerns about scalability and practical effectiveness.

While I still consider the methodological novelty relatively incremental, I acknowledge the practical value and consistent efficiency gains of the proposed framework. Given the clear presentation and sufficient experimental evidence, I am inclined to raise my score from 2 to 3 .

I encourage the authors to further highlight the key methodological contributions and clarify the distinctions from existing adaptive routing methods.

Thank you for raising your score, and for the constructive suggestion.

"I encourage the authors to further highlight the key methodological contributions and clarify the distinctions from existing adaptive routing methods."

The revision does both.

Key contributions, stated up front. The introduction now leads with the two ideas that make SpecRouter work, rather than with the decision rule:

  1. Exact offline supervision for every drafter. Deterministic proposals make each drafter's committed tokens at every prefix an exact label (Proposition 1). Proposition 2 extends this to sampling, where the label is the expected committed length. Routing across drafter families therefore becomes supervised learning: there is no exploration while serving, and every drafter is labeled, including ones a deployed policy would rarely try.
  2. Separating payoff from cost. What a drafter would commit depends on the text; what it costs depends on the deployment. Keeping the two apart lets one payoff predictor serve every deployment we tested, with only costs re-profiled.

Distinctions from adaptive routing methods. The related work now includes the comparison from our response, organized by what each approach requires:

  • Online selectors (BanditSpec, MetaSD, HedgeSpec) learn drafter choice from verification feedback while serving. Bandits pay to explore and cannot condition on context; HedgeSpec needs every drafter's counterfactual payoff online. SpecRouter learns offline, so exactly one drafter runs per block.
  • Threshold hybrids (SuffixDecoding, SAM decoding, HVD) arbitrate one fixed pair of drafters with a threshold tuned for a given deployment. SpecRouter routes over any pool and adapts to a new deployment by re-profiling costs.

Evidence that these distinctions matter in practice, also added to the revision:

  • Adding a drafter costs one output head. Four new EAGLE-3 configurations took 7.4 GPU-hours of labeling on 2,500 training requests and 11 seconds of head training in total, with the existing heads unchanged.
  • One deployment covers settings that no fixed method does. In batched serving (our responses to Reviewers a127 and C5J5), the fastest fixed method changes with traffic, latency target and batch size, and every one of the eight fixed methods we measured fails at least one latency target outright. SpecRouter, as a single deployment, meets all of them and stays within 7% of the fastest fixed method for each.

Together with the larger-model and simpler-router results you highlighted, these results make the paper's practical case, and the revision presents them that way.

Reviewer_C5J5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SpecRouter selects between N-gram lookup, SAM copying, a draft LM, EAGLE-3, and target-only decoding at each verification block. A 0.58M-parameter predictor estimates each action's committed tokens from target states and prefix statistics, while a separately calibrated cost model estimates execution time. The router chooses the lowest predicted cost per committed token. Deterministic greedy verification supplies complete offline labels for all actions. On Llama-3.1-8B and Qwen3-8B across 16 workloads, it reports 2.75×/2.95× macro decode speedup at batch one. One Llama predictor is also reused across A40/H200 and batch sizes 1/4/16 by recalibrating costs.

Critical strengths
  • Useful offline supervision. Under greedy verification, every action can be labeled at each prefix, avoiding inference-time exploration.
  • A lightweight predictor. The 0.58M-parameter network reuses target hidden states to score all actions without running every drafter.
  • Reuse across deployments. One Llama predictor transfers across A40/H200 and batch sizes 1/4/16 with cost recalibration alone.
Critical weaknesses
  • Limited novelty in the cost objective. Optimizing expected progress per unit execution time is well established (e.g., Sequoia). What does this formulation add beyond the learned payoff predictor?
  • The drafter choices need justification. Native MTP (e.g., Qwen3.5) and parallel drafting (e.g., DSpark) have different cost structures from EAGLE-3 and may change the value of routing. Why choose SAM over alternatives such as Token Recycling?
  • Evaluation is greedy-only. Copying-based proposals may lose acceptance at higher temperatures, changing their advantage over neural drafters. Neither the offline-label construction nor the routing gains are validated under sampling.
  • Serving benefits remain unclear under optimized deployment. The deployment executor is eager, with no CUDA Graph results. Although extra drafter weights occupy 3.3 GB, their effect on KV capacity/concurrency is unmeasured. There is also no evaluation with prefill–decode disaggregation.
Questions for authors

See weaknesses above.

6 comments
AuthorsOpenReview ↗

Reply to Reviewer C5J5 (1/2)

We thank the reviewer for an expert review and for recognizing the offline supervision, the lightweight predictor and reuse across deployments. We address each concern below with new experiments; the camera-ready will include all of them.

W1. Novelty

"Limited novelty in the cost objective. Optimizing expected progress per unit execution time is well established (e.g., Sequoia). What does this formulation add beyond the learned payoff predictor?"

We agree that time per committed token is the standard objective: Sequoia optimizes it, and every speculative decoding method is judged by it. We do not claim the objective. Our contribution is cross-family drafter routing, and the formulation answers a different question: how to evaluate that objective at every block across drafters that propose in entirely different ways (copying, n-gram lookup, a standalone draft model, EAGLE-3 trees, and in the new experiments parallel drafting and native MTP), without running them. Its two terms need different information:

  • Payoff depends on the text. Under deterministic proposals, every drafter's committed tokens at every prefix are exact offline labels (Proposition 1; Proposition 2 under sampling). One predictor is therefore supervised for all drafters, including ones a behavior policy would rarely pick, with no exploration at serving time. Context-free online bandits, which lack this supervision, fall below fixed EAGLE-3 on our pool (best of eight variants: 1.84× vs 2.03× on Llama), while SpecRouter reaches 2.56×.
  • Cost depends on the deployment, and we keep it separate. At batch size 16, recalibrating costs alone moves the same predictions from 78% EAGLE-3 to 83% SAM (Figure 3b). The original 8B predictor, never retrained, stays 24–29% faster than EAGLE-3 under sampling, CUDA Graphs and re-tuned drafters, with only costs refit.

The learned payoff predictor is one component of this interface, not the contribution on its own: removing the cost comparison loses 16% (Table 4), while target states add a further 4–6% in most settings.

Prior cross-family selectors switch between one copy mechanism and one neural drafter with a threshold (SuffixDecoding, SAM decoding, and the concurrent HVD); prior learned selectors are online and choose within one family (BanditSpec, HedgeSpec, MetaSD). Sequoia applies the same objective at a different level: it picks one tree structure per hardware setting from acceptance statistics estimated offline, whereas SpecRouter picks a proposal mechanism per block from predictions conditioned on the current prefix. A Sequoia tree could be one of SpecRouter's actions, as two EAGLE-3 tree sizes already are (Table 9).

W2. Drafter choices

"The drafter choices need justification. Native MTP (e.g., Qwen3.5) and parallel drafting (e.g., DSpark) have different cost structures from EAGLE-3 and may change the value of routing. Why choose SAM over alternatives such as Token Recycling?"

Routing keeps its value with every drafter the reviewer names. The recipe is not tied to the drafters we chose; testing every combination of every known drafter is out of scope, but we added each kind requested. A new drafter needs only its labels, one output head and one cost, and existing heads stay frozen.

PoolTargetBest single drafterSpecRouter
Original five draftersLlama-3.1-8B / Qwen3-8BEAGLE-3 2.17× / 2.31×2.76× / 2.93×
+ PARD (parallel drafting)Llama-3.1-8B / Qwen3-8BPARD 2.40× / EAGLE-3 2.31×3.02× / 2.99×
Native MTP, N-gram, SAM, draft modelQwen3.5-9B (end-to-end)Native MTP 1.75×2.02×

Native MTP changes the cost structure, and routing still wins. Integrated into SGLang's hybrid-state runtime, SpecRouter is 15% faster than native MTP alone: it sends 79% of tokens through MTP and the rest through copying, which MTP cannot do. Our fixed-MTP policy reproduces SGLang's native MTP token for token; numerical validation of the full integration is ongoing.

Parallel drafting. DSpark appeared on arXiv in July 2026, two months before the deadline, so we evaluated PARD, a public parallel drafter released for both our targets. On Llama, PARD becomes the best single drafter, and routing still adds 25.9% on top of it.

Why SAM. SAM's state depends only on the committed prefix, which exact labels require (Proposition 1). For the comparison we implemented Token Recycling with the same committed-prefix property. Replacing SAM with it lowers SpecRouter from 2.76× to 2.54× on Llama and from 2.93× to 2.53× on Qwen, and adding it alongside SAM gives no gain: SAM is the better copy drafter.

AuthorsOpenReview ↗

Reply to Reviewer C5J5 (2/2)

W3. Sampling

"Evaluation is greedy-only. Copying-based proposals may lose acceptance at higher temperatures, changing their advantage over neural drafters. Neither the offline-label construction nor the routing gains are validated under sampling."

The routing gains hold at every temperature. All 16 workloads, 1,600 requests:

TargetTemperatureSAMEAGLE-3SpecRouterMargin over EAGLE-3
Llama-3.1-8B0 (greedy)1.70×2.17×2.80×+29.2%
Llama-3.1-8B0.6 (recommended)1.63×2.03×2.56×+26.4%
Llama-3.1-8B1.01.49×1.80×2.24×+24.6%
Qwen3-8B0 (greedy)1.82×2.30×2.95×+28.1%
Qwen3-8B0.7 (recommended)1.80×2.20×2.82×+28.0%
Qwen3-8B1.01.77×2.20×2.82×+27.9%

The reviewer is right that copying loses acceptance as temperature rises (on Llama, SAM commits 1.59 tokens per block at T = 0.6 and 1.48 at T = 1.0), but neural drafting loses as much (EAGLE-3: 2.75 to 2.46), so the router's margin barely moves. Shifting trade-offs like this are exactly what a router handles.

The label construction also extends exactly (new Proposition 2). With deterministic proposals, each draft token is accepted with exactly the target's probability, so an action's expected committed tokens follow from one target pass; Monte Carlo verification matches them within 0.9%. A predictor trained only on these sampled labels also beats EAGLE-3 by 22–25%.

W4. Serving under optimized deployment

"Serving benefits remain unclear under optimized deployment. The deployment executor is eager, with no CUDA Graph results. Although extra drafter weights occupy 3.3 GB, their effect on KV capacity/concurrency is unmeasured. There is also no evaluation with prefill–decode disaggregation."

CUDA Graphs. The relative gain is unchanged under CUDA Graphs. A40, batch size 1, all 16 workloads, each row relative to autoregressive decoding in the same runtime ("Runtime" refers to target verification):

TargetRuntimeEAGLE-3SpecRouterMargin
Llama-3.1-8BEager2.17×2.80×+29.2%
Llama-3.1-8BCUDA Graphs2.08×2.69×+29.0%
Qwen3-8BEager2.30×2.95×+28.1%
Qwen3-8BCUDA Graphs2.23×2.87×+28.3%

In the submission's Llama runs, the standalone draft model's decode path and the payoff MLP already used CUDA Graphs; target verification remained eager. We now capture target verification too, with the payoff predictor unchanged and only costs refit. The five-drafter router reaches 2.80× here and 2.76× in the W2 table; these are separate runs, and only the runs here graph-capture the draft model's decode path.

KV capacity. Maximum batch on one A40 at 2k / 8k context, without paging:

TargetEAGLE-3 onlyLean router pool (target-only, SAM, EAGLE-3)Full router pool
Llama-3.1-8B56 / 1456 / 1444 / 11
Qwen3-8B51 / 1351 / 1334 / 8

The capacity cost comes entirely from the standalone draft model's weights and KV cache. The lean pool has exactly EAGLE-3's capacity and keeps 98.5–99% of the full router's speed (2.76× vs EAGLE-3's 2.17× on Llama; 2.92× vs 2.30× on Qwen), so memory-bound deployments get SpecRouter's gain at no capacity cost.

Prefill–decode disaggregation. With prefill on one GPU and decode on another in the same host, SpecRouter's decode speedup is unchanged (2.77–2.78× vs 2.78× colocated). The first committed block arrives 9–15 ms later (about 1%), and the router adds about 7 MiB of state per request to the transfer.

Closing

We believe these results address every concern, and we respectfully ask the reviewer to reconsider the score.

Thank you for the thorough rebuttal. The new sampling results (W3), PARD / Token Recycling comparisons (W2), and batch-1 CUDA Graph and lean-pool experiments (W4) address my single-request questions well.

However, the multi-batch SLO results in your follow-up to Reviewer a127 ("Throughput under latency SLOs") reinforce my concern about practical batched serving (W4):

  1. Multi-batch performance and EAGLE-3 configurations. Under a 50 ms TPOT budget at batch sizes 8–32, SpecRouter already trails fixed EAGLE-3 (16/4/4) by 24%–30% on short contexts and falls behind fixed SAM and Target-only on H200 long contexts. Moreover, both 32/8/4 (which caused EAGLE-3's 0.78×–0.81× slowdown at B=16B=16 in Table 18) and 16/4/4 (tuned at B=1B=1 and unmeasured at B=1B=1–44 in C1) are unusually large multi-step trees for batched serving, where production engines typically use chain or small-tree EAGLE-3 configurations (top-k 1–2, 3–6 draft tokens).
  2. Structural limit under concurrency. In a concurrent batch, only a small subset of requests hits copyable spans at any step: a shared batch action penalizes the remaining requests, whereas per-request actions would still pay the GPU neural-drafting latency of EAGLE-3 / PARD for the rest of the batch, eroding SAM's drafting-time advantage.

I therefore remain unconvinced of the practical serving benefit beyond B=1B=1 and will maintain my score for now, though I welcome your thoughts.

Reviewer_a127OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes selecting the best of drafting methods at each speculative decoding (SD) step. This is motivated by two factors: (1) computational costs vary across methods and execution environments, and (2) token acceptance depends on the generated content.

The approach uses two components for online method selection: (1) a lightweight 0.58M-parameter model that predicts each method’s accepted token count before drafting, and (2) profiled latency estimates to calculate expected throughput and select the method that maximizes it.

The evaluation covers 16 workloads with 100 requests each at batch size 1. The approach improves speed over EAGLE-3 by 30% on Llama-3.1-8B and 27% on Qwen3-8B. However, its speedup drops to approximately 1.08× on Llama-3.1-8B at batch size 16.

Critical strengths

S1. Predicting all methods’ payoffs before drafting distinguishes this approach from prior work. Throughput-aware scheduling further improves execution efficiency.

Critical weaknesses

W1. Limited applicability [+0.3 if resolved]

  • W1-1. Training and evaluation use greedy decoding, which is rarely used in practical LLM serving. The core label-collection and prediction methods remain unvalidated under sampling.

  • W1-2. Thinking mode is not addressed. Whether thinking is enabled can substantially change what the 256 output tokens represent. The paper should specify the generation configuration and discuss how thinking mode may affect the results, given current LLM usage patterns.

W2. Weak baselines [+0.4 if resolved]

  • W2-1. Comparisons with cost-aware baselines are needed. BanditSpec also evaluates throughput-based rewards, which should be considered.

  • W2-2. Table 14 suggests that most gains come from choosing between SAM and EAGLE-3. A controlled comparison with a model restricted to these two methods is therefore needed.

W3. Potentially unfair evaluation [+0.3 if resolved]

  • W3-1. The drafting configurations appear unfair. Draft lengths are typically chosen with each method’s average acceptance length in mind. However, the paper uses draft lengths substantially larger than the accepted token counts reported in Appendix [TODO], which may disadvantage the baselines.

  • W3-2. The HedgeSpec comparison omits key experimental settings, including batch size and draft length. Its output length also differs from the main evaluation without explanation.

Questions for authors

Q1. Please report acceptance length for all experiments so readers can assess both drafting accuracy and efficiency.

Q2. How does performance change with generation position as outputs grow longer? Analyzing this trend could motivate an important research direction.

Q3. Were the experiments run with CUDA Graph capture? If so, how was capture implemented under its constraints? If not, it remains unclear whether the relative gains would hold in an optimized runtime.

16 comments
AuthorsOpenReview ↗

Reply to Reviewer a127 (1/3)

We thank the reviewer for a careful review and for stating what would resolve each weakness. We ran every requested experiment on both 8B targets (1,600 test requests across 16 workloads); the camera-ready will include all of them. As a frame for the results below: SpecRouter routes across drafter families using exact offline labels for every drafter and a separation of payoff from deployment cost, so one predictor carries across sampling settings, runtimes and batch sizes with only its costs recalibrated.

Summary

"However, its speedup drops to approximately 1.08× on Llama-3.1-8B at batch size 16."

At batch size 16, every single drafter slows decoding down: fixed EAGLE-3 runs at 0.78× on A40 and 0.81× on H200, and the best fixed policy is not to speculate at all (1.00×; Table 18). Only routing yields a speedup, and SpecRouter yields the largest (1.08× on both GPUs, vs 1.01× for a threshold router retuned for that setting), with the same payoff predictor and recalibrated costs. No single drafter works across batch sizes; SpecRouter does.

W1-1. Sampling

"Training and evaluation use greedy decoding, which is rarely used in practical LLM serving. The core label-collection and prediction methods remain unvalidated under sampling."

SpecRouter's gains hold under sampling. On all 16 workloads at each model's recommended settings:

SettingEAGLE-3SpecRouter, original predictorSpecRouter, trained on sampled labels
Llama-3.1-8B, T = 0.6, top-p 0.92.03×2.56× (+26.4%)2.52× (+24.1%)
Qwen3-8B, T = 0.7, top-p 0.8, top-k 202.20×2.82× (+28.0%)2.73× (+23.9%)

The margin also holds at T = 1.0 (+22% to +28%). The label construction extends exactly (new Proposition 2). Our proposals are deterministic, so under speculative sampling each draft token is accepted with exactly the target's probability for it, and an action's expected committed tokens follow from one target pass:

E[La(x1:n)]=1+∑k=1K∏i=1kpT(di∣x1:n,r,d<i)\mathbb{E}[L_a(x_{1:n})] = 1 + \sum_{k=1}^{K} \prod_{i=1}^{k} p_T\left(d_i \mid x_{1:n}, r, d_{<i}\right)

For trees the sum runs over nodes. Monte Carlo verification (4 settings × 4 drafters × 200 prefixes × 32 draws) matches these labels within 0.9%, and a predictor trained only on them beats EAGLE-3 by 22–25% in every setting, which validates label collection and prediction under sampling end to end. The original greedy-trained predictor transfers unchanged and is slightly faster still, so in practice one greedy labeling pass serves every temperature.

W1-2. Thinking mode

"Thinking mode is not addressed. Whether thinking is enabled can substantially change what the 256 output tokens represent. The paper should specify the generation configuration and discuss how thinking mode may affect the results, given current LLM usage patterns."

SpecRouter's advantage holds with thinking enabled. Our submitted Qwen3 evaluation used non-thinking prompts, which we verified from the frozen prompt token IDs, and the camera-ready states the full generation configuration for every run. We now also run Qwen3-8B with thinking on and off, with a 2,048-token cap at the model card's sampling settings, on all 16 workloads:

Qwen3-8B, 2,048 tokensBest single drafterSpecRouter
Thinking onEAGLE-3 2.44×2.79× (+14.3%)
Thinking offSAM 2.34×3.28× (+40.0%)

Thinking changes which drafter is best (EAGLE-3 with thinking on, SAM with it off), so any single-drafter choice is wrong for one of the two modes. SpecRouter follows the shift and beats the best single drafter in both.

AuthorsOpenReview ↗

Reply to Reviewer a127 (2/3)

W2-1. Cost-aware baselines

"Comparisons with cost-aware baselines are needed. BanditSpec also evaluates throughput-based rewards, which should be considered."

BanditSpec was already a baseline in the submission: Table 3 reports its UCB and EXP3 algorithms at 1.80× and 1.73×, against 2.77× for SpecRouter. As requested, we now add its throughput reward (committed tokens per second), with statistics either reset per request or carried across a workload. The best of all eight variants (UCB, throughput reward, carried) reaches 1.84× on Llama and 1.67× on Qwen under sampling, below even fixed EAGLE-3 (2.03× / 2.20×) and far below SpecRouter (2.56× / 2.82×).

The gap is structural rather than a matter of bandit tuning. A context-free bandit's arms are fixed drafters, and it cannot condition on whether the current block lies inside a copied span, so at best it converges to the best fixed drafter for the traffic it sees. Even that ceiling, chosen in hindsight with zero exploration cost, is below SpecRouter: 2.53× for the best fixed drafter per workload and 2.68× per request, against 2.75× (Table 13). In practice bandits also pay to explore the draft model, whose blocks take about twice as long as EAGLE-3's (Table 16).

W2-2. SAM/EAGLE-3 control

"Table 14 suggests that most gains come from choosing between SAM and EAGLE-3. A controlled comparison with a model restricted to these two methods is therefore needed."

Agreed: the copy-versus-neural decision carries most of the gain, and the camera-ready says so directly. A learned router still adds to a tuned threshold on that same decision in three ways.

Same workloads. Restricted to target-only, SAM and EAGLE-3, a retrained SpecRouter beats every threshold tuned on validation data and all eight bandits, and also wins at T = 1.0 on both targets:

Pool: target-only, SAM, EAGLE-3Best tuned thresholdBest banditSpecRouter
Llama-3.1-8B, T = 0.62.43×2.23×2.54×
Qwen3-8B, T = 0.72.64×2.35×2.71×

Unseen workloads. With each workload held out of training entirely, SpecRouter reaches 2.59×, above both thresholds (2.52× and 2.55×), which were tuned on validation data that includes those workloads (Tables 1 and 4).

New deployments. A threshold frozen at A40 batch size 1 slows decoding at batch size 16 (0.90× on A40, 0.93× on H200). SpecRouter's unchanged predictor speeds it up: 1.04× with the same frozen costs and 1.08× after recalibrating costs on 160 requests, against 1.01× for a threshold retuned for that setting (Table 18).

The router also extends past two drafters with no retuning: adding a parallel drafter (PARD) gives a further 9.3% on Llama.

W3-1. Draft lengths

"The drafting configurations appear unfair. Draft lengths are typically chosen with each method's average acceptance length in mind. However, the paper uses draft lengths substantially larger than the accepted token counts reported in Appendix [TODO], which may disadvantage the baselines."

With every drafter tuned, SpecRouter is still 27.5% faster than tuned EAGLE-3 on Llama and 24.2% on Qwen. We tuned each drafter on validation requests (N-gram length, SAM cap, draft-model length, and EAGLE-3 trees including the official 60/7/10) and froze the fastest:

TargetN-gramSAM capDraft-model lengthEAGLE-3 nodes/depth/top-kTuned EAGLE-3SpecRouter
Llama-3.1-8B864316/4/42.22×2.83× (+27.5%)
Qwen3-8B832248/6/82.53×3.14× (+24.2%)

Tuning every drafter shrinks SpecRouter's margin over EAGLE-3 only slightly, from 29.2% to 27.5% on Llama and from 28.1% to 24.2% on Qwen. Shorter drafts helped the draft model and, on Llama, EAGLE-3. For SAM, the 64-token cap never binds: in our new greedy runs no SAM proposal reached it, and SAM proposes 1.4–1.5 tokens on average as a fixed drafter and 3.0 (Llama) to 6.2 (Qwen) when the router selects it. Validation accordingly kept the 64-token cap on Llama. The camera-ready reports the headline against these tuned baselines.

AuthorsOpenReview ↗

Reply to Reviewer a127 (3/3)

W3-2. HedgeSpec settings

"The HedgeSpec comparison omits key experimental settings, including batch size and draft length. Its output length also differs from the main evaluation without explanation."

We re-measured the comparison live under one explicit contract. Settings: batch size 1, Llama-3.1-8B in FP16, 280 matched prompts over HedgeSpec's seven released experts. HedgeSpec runs its public code with its published settings and expert trees, with only its stopping rule patched to stop exactly at the cap; our runtime uses a 60/7/10 tree per expert. The 512-token cap followed HedgeSpec's own evaluation protocol. Speedups are request-inclusive, with each runtime normalized to its own paired autoregressive baseline:

Output capHedgeSpecSpecRouterDomain-matched expert (same 60/7/10 trees, uses domain labels)
2563.68×4.14×4.22×
5123.79×4.16×4.26×

SpecRouter beats HedgeSpec at both caps. Within our runtime, where every expert uses the same 60/7/10 tree, SpecRouter comes within 2.2% of an oracle that picks each prompt's domain-matched expert using domain labels, which SpecRouter never sees. Unlike the replay-based numbers in Table 3, these runs execute the full expert pool live, including the router's controller and the catch-up cost of inactive experts; SpecRouter's margin over HedgeSpec holds (+12.5% at 256 tokens and +9.8% at 512, against +9.7% in Table 3).

Q1. Acceptance length

"Please report acceptance length for all experiments so readers can assess both drafting accuracy and efficiency."

Llama-3.1-8B, greedy, A40, pooled over all blocks (total time divided by total committed tokens):

PolicyCommitted tokens per blockms per committed token
Draft model5.0324.8
EAGLE-32.7824.6
SAM1.6324.9
SpecRouter3.7315.6

Acceptance length alone misranks drafters: the draft model commits the most tokens per block yet is no faster per token than EAGLE-3. SpecRouter optimizes time per committed token directly and spends 37% less of it than EAGLE-3. The camera-ready reports both columns beside every speedup.

Q2. Generation position

"How does performance change with generation position as outputs grow longer? Analyzing this trend could motivate an important research direction."

SpecRouter's advantage grows as outputs get longer. On Llama-3.1-8B (greedy), its margin over EAGLE-3 rises from +29% at 256 tokens to +58% at 2,048 tokens. Fixed EAGLE-3 stays flat (2.17× vs 2.15×), while copying becomes more valuable as the output repeats its own content (SAM: 1.70× to 2.38×), and the router captures that value. We agree this points to an important direction: payoff and cost models that account for generation position.

Q3. CUDA Graphs

"Were the experiments run with CUDA Graph capture? If so, how was capture implemented under its constraints? If not, it remains unclear whether the relative gains would hold in an optimized runtime."

The relative gain is unchanged under CUDA Graphs. A40, batch size 1, all 16 workloads, each row relative to autoregressive decoding in the same runtime ("Runtime" refers to target verification):

TargetRuntimeEAGLE-3SpecRouterMargin
Llama-3.1-8BEager2.17×2.80×+29.2%
Llama-3.1-8BCUDA Graphs2.08×2.69×+29.0%
Qwen3-8BEager2.30×2.95×+28.1%
Qwen3-8BCUDA Graphs2.23×2.87×+28.3%

In the submission's Llama runs, the standalone draft model's incremental decode path and the payoff MLP used CUDA Graphs; target verification remained eager. We now capture target verification too: static KV capacity, one graph per token-count bucket with proposals padded to the bucket, and explicit prefix and tree attention masks; EAGLE-3 tree expansion has its own graphs. N-gram and SAM run on the CPU, and the router's choice reuses the host synchronization that acceptance already requires, so routing adds no extra sync. The payoff predictor is unchanged; only costs are refit.

Closing

We believe these results resolve W1–W3 and all three questions, and we respectfully ask the reviewer to revisit the score in light of the experiments described in the review.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Revised Draft has been uploaded

We thank all three reviewers for a constructive discussion. We have uploaded a revised PDF. Main-text changes present the contributions more directly and summarize the new evidence. Many of the new results are in the appendix. Below is where each discussion point now appears.

Main text

  • Contributions (Section 1). The introduction now leads with the two ideas that make SpecRouter work: exact offline supervision for every drafter (Proposition 1, and the new Proposition 2 for sampling), and separating payoff from cost, so one payoff predictor serves every deployment with only costs recalibrated.
  • Which drafters qualify (Section 3). Any proposal mechanism whose payoff can be computed offline. A new action needs its labels, one output head and a cost, with existing heads unchanged.
  • Robustness of the gains (Section 5.2, Tables 1 and 4). SpecRouter's margin over EAGLE-3 holds with every drafter tuned on validation data (+27.5% on Llama, +24.2% on Qwen), under sampling at each model's recommended settings (+26–28%), with thinking mode, and on Qwen3-32B (+24.9%) and Llama-3.3-70B (+36.7%).
  • Batched serving under latency targets (Section 5.4, Table 31). We measure throughput at p90 TPOT limits of 25–100 ms against eight fixed policies, including small-tree and chain EAGLE-3 configurations. The fastest fixed policy changes with traffic, latency target and batch size, and each fails at least one limit outright. A single SpecRouter deployment meets every limit that any fixed policy meets, leads at the tightest, and stays within 7% of the per-setting best.
  • Related work (Section 6, Table 6). Adaptive drafter selection methods, organized by what each requires in a new deployment.

Appendix

  • D: proof of Proposition 2, label validation and results under sampling, thinking mode, output length.
  • E: time per committed token, CUDA Graphs, larger targets, and actions added through the same recipe (Table 16: PARD, native MTP in SGLang, Token Recycling, domain experts, EAGLE-3 configurations).
  • F: restricted pools versus tuned thresholds, lighter cost-aware routers, and target states from 8B to 70B (Tables 21–23).
  • G: throughput-reward bandits, and HedgeSpec executed end to end (Table 28).
  • H: serving under latency targets (H.1), KV capacity and prefill–decode disaggregation (H.2).

By reviewer

  • a127: sampling and thinking mode (D); cost-aware bandits and the restricted pool (F, G); tuned drafters (B.3); HedgeSpec settings (G); committed tokens per block (E.1); output length (D.4); CUDA Graphs (E.2); the SLO evaluation and bucketed costs (5.4, H.1); generality (Section 3, E.4).
  • C5J5: positioning against prior selectors (Section 1, Table 6); native MTP, parallel drafting and Token Recycling (E.4); sampling (D); CUDA Graphs, KV capacity and prefill–decode disaggregation (E.2, H.2); batched serving with production-style EAGLE-3 configurations (5.4, H.1).
  • RhTS: contributions and distinctions from adaptive routing methods (Section 1, Table 6); larger targets (E.3); simpler cost-aware routers (Table 22).
AuthorsOpenReview ↗

Summary for the Area Chair

We thank the AC and the reviewers for a substantive discussion. In brief:

Where the reviewers landed

  • RhTS raised their score from 2 to 3: the larger-model and simpler-router experiments "substantially address my concerns about scalability and practical effectiveness."
  • a127 raised their score from 2 to 3, writing that "W1–W3 of my original review are resolved," and listed camera-ready items, all of which we have committed to (below).
  • C5J5 confirmed that the sampling, PARD/Token Recycling, CUDA Graph and lean-pool results "address my single-request questions well," and acknowledged the batched-serving, PARD and smaller EAGLE-3 experiments. They maintain their score, asking for continuous-batching comparisons with mixed traffic against native MTP and newer parallel drafters (addressed below).

What the discussion added

  • Robustness: the margin over EAGLE-3 holds with every drafter tuned (+24–28%), under sampling at every temperature tested (+25–28%), with thinking mode, and on Qwen3-32B (+25%) and Llama-3.3-70B (+37%).
  • Baselines: SpecRouter beats BanditSpec with throughput rewards, tuned thresholds on the copy-versus-neural decision at every temperature, lighter cost-aware routers, and HedgeSpec executed end to end on its own expert pool with identical trees (4.16× vs. 3.79×, ahead in all seven domains).
  • Generality: parallel drafting (PARD), native MTP in SGLang, Token Recycling and four EAGLE-3 configurations joined the pool through the same recipe, each EAGLE-3 configuration with one new output head and existing heads unchanged.
  • Deployment: the margin is unchanged under CUDA Graphs at batch size 1; a lean pool keeps EAGLE-3's KV capacity; prefill–decode disaggregation leaves decode speedup intact.

Batched serving With three timing runs per point and every policy swept to its own memory limit (Section 5.4, Table 32, Figure 7), every fixed policy, including the small-tree and chain EAGLE-3 configurations C5J5 recommended, fails at least one latency limit at every measured batch size. A single SpecRouter deployment meets all five limits, leads the fastest fixed policy in three settings, and is within 3.5% in the other two, with overlapping 95% intervals in every setting. On long contexts, SAM is the fastest fixed policy and SpecRouter matches or exceeds it; no EAGLE-3 configuration meets 50 ms there. The drafters C5J5 names are actions in SpecRouter's pool rather than baselines: native MTP and PARD are already integrated, and SpecRouter exceeds each alone. We agree that continuous batching with per-request actions is the next test, and the paper states this scope.

On novelty

  • Prior hybrids are drafters, not routers. SuffixDecoding and SAM decoding contribute a copy drafter whose adaptivity is internal, plus one fixed rule that falls back to a single neural drafter.
  • Online selectors learn which arm suits a prompt while serving. None predicts from the current context which arm will win the next block.
  • SpecRouter routes entire methods per block from the target's state. The levers prior methods tune (speculation depth, tree shape, which expert) become further actions in its pool. Exact offline labels (Propositions 1 and 2) remove exploration at serving time, and separating payoff from cost lets one predictor transfer across GPUs, batch sizes, runtimes and sampling settings. SpecRouter exceeds even the best fixed method per request chosen in hindsight (2.75× vs. 2.68×).

Camera-ready commitments

  • Appendix G: discussion of per-runtime normalization, which does not favor SpecRouter (our autoregressive baseline is 0.7–1.3% faster).
  • Revised claims in Contribution 3, Section 5.1 and the Table 1 caption, noting that SpecRouter's interval overlaps the HVD-style threshold's on Qwen3-8B; the HVD-style row with three seeds and three timing repetitions.
  • The serving evaluation re-run with CUDA Graph capture, with the eager-executor scope stated in the Limitations until then; a per-workload breakdown of serving routing choices from the recorded runs.

Scope of the changes The method, its training and the primary results are unchanged. The new experiments were requested by the reviewers and test the existing claims under more conditions (sampling, thinking mode, tuned drafters, larger targets, additional baselines, batched serving); they are in the appendix, with short summaries in Sections 5.2 and 5.4, and the other main-text changes clarify the contributions and related work. The corrections made during the discussion (the HedgeSpec per-domain table, the HVD-style row, and single-run serving results, now three runs) adjust specific numbers. None reverses a conclusion; the one claim we soften, at a127's request, is SpecRouter's lead over the HVD-style threshold on Qwen3-8B, where the intervals overlap.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.