- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses the problem of dynamically selecting the most efficient drafter in speculative decoding, as different drafting strategies exhibit varying efficiency across generation contexts and hardware configurations. It proposes SpecRouter, which predicts the accepted length of each drafter using target model hidden states and selects the one with the lowest estimated cost per committed token. Experiments on 16 workloads show 2.75× and 2.95× speedups on Llama-3.1-8B and Qwen3-8B, respectively, outperforming EAGLE-3.
Critical strengths
- Practical problem. The paper addresses the practical challenge of dynamically selecting among heterogeneous speculative decoding strategies, whose efficiency varies across generation contexts and deployment configurations. This is a relevant problem for improving inference efficiency.
- Simple cost-aware routing framework. SpecRouter separates drafter payoff prediction from execution cost estimation, enabling dynamic selection based on predicted cost per committed token. The design is intuitive and easy to follow, and allows the payoff predictor to be reused across different deployment configurations without retraining.
Critical weaknesses
- The core idea of predicting drafter payoffs and selecting the action with the lowest estimated cost per committed token is relatively straightforward, while adaptive drafter selection has already been extensively explored in prior work, so the novelty is limited. Moreover, the ablation studies show that using only simple prefix statistics achieves performance close to the full method (2.59× vs. 2.75×), while a static cost model performs almost identically to the full cost model. These results suggest that the additional components provide limited incremental benefits, weakening the evidence for the paper's claimed methodological contributions.
- The proposed method achieves relatively modest improvements over existing hybrid routing baselines (2.75× vs. 2.55× on Llama and 2.95× vs. 2.76× on Qwen). Furthermore, the evaluation only includes two 8B target models, without experiments on larger backbones. It remains unclear whether these improvements can be maintained at larger model scales, where the relative costs and effectiveness of different drafting strategies may change.
Questions for authors
- Given the limited gains from target hidden states and dynamic cost modeling, how does SpecRouter compare with a simpler lightweight cost-aware routing baseline?
- I want to see experimental results on larger target models (e.g., 32B or 70B) to evaluate whether the performance gains generalize beyond 8B models.
4 comments
Reply to Reviewer RhTS (2/2)
W2. Gains over hybrids, and scale
"The proposed method achieves relatively modest improvements over existing hybrid routing baselines (2.75× vs. 2.55× on Llama and 2.95× vs. 2.76× on Qwen). Furthermore, the evaluation only includes two 8B target models, without experiments on larger backbones. It remains unclear whether these improvements can be maintained at larger model scales, where the relative costs and effectiveness of different drafting strategies may change."
The SuffixDecoding-style threshold is the existing method in our comparison; the HVD-style row reuses our own SAM payoff head, since HVD's code was not public when we ran our experiments. Against SuffixDecoding-style, SpecRouter is 9% faster on Llama (2.75× vs 2.52×) and 8% faster on Qwen (2.95× vs 2.74×) at batch size 1, closing roughly 40% of the remaining gap between the threshold and the local hindsight reference (Table 1). The margin widens where thresholds are brittle:
- Unseen workloads. With each workload held out of training entirely, SpecRouter reaches 2.59×, above both thresholds (2.52× and 2.55×), which were tuned on validation data that includes those workloads (Tables 1 and 4).
- New deployments. Frozen at A40 batch size 1, the SuffixDecoding-style threshold slows decoding at batch size 16 (0.90× on A40, 0.93× on H200), while SpecRouter's unchanged predictor speeds it up (1.04× with the same frozen costs, 1.08× after recalibrating costs; Table 18).
- More than two drafters. A threshold arbitrates exactly one pair of drafters; SpecRouter extends to any pool, and adding a parallel drafter (PARD) gives a further 9.3% on Llama.
On a two-drafter pool with thresholds retuned for each sampling setting, the learned router also wins in all four settings (2.54× vs 2.43× on Llama and 2.71× vs 2.64× on Qwen at the recommended settings). SpecRouter's gains over EAGLE-3 also carry over to larger targets (Q2 below).
Q1. Simpler cost-aware routers
"Given the limited gains from target hidden states and dynamic cost modeling, how does SpecRouter compare with a simpler lightweight cost-aware routing baseline?"
Every lighter router built on our labels and cost rule beats EAGLE-3 (2.20× on Qwen3-8B, 2.13× on Qwen3-32B), which confirms that exact supervision plus cost comparison carries most of the value. A learned per-block predictor with target states is the best of them:
| Router | Qwen3-8B, T = 0.7 | Qwen3-32B |
|---|---|---|
| Running average of each drafter's payoff | 2.28× | 2.26× |
| Best fixed drafter per workload, picked on validation | 2.45× | 2.39× |
| Gradient-boosted trees, prefix statistics | 2.49× | 2.35× |
| MLP, prefix statistics | 2.61× | 2.51× |
| SpecRouter | 2.73× | 2.66× |
Q2. Larger targets
"I want to see experimental results on larger target models (e.g., 32B or 70B) to evaluate whether the performance gains generalize beyond 8B models."
The gains generalize to both scales the reviewer names, with a new predictor per target trained by the same recipe:
| Target | Setup | EAGLE-3 | SpecRouter |
|---|---|---|---|
| Qwen3-32B | 1× H200, 16 workloads, 1,600 requests | 2.13× | 2.66× (+24.9%) |
| Llama-3.3-70B | 2× H200 tensor parallel, 8 workloads, 800 requests | 2.64× | 3.61× (+36.7%) |
The margin at 32B matches the 8B results (24–30%), and at 70B it is larger still.
Closing
We believe these results address the concerns about novelty and scale, and we respectfully ask the reviewer to reconsider the score.
Thank you for the detailed response. The additional experiments on larger models and comparisons with simpler routers substantially address my concerns about scalability and practical effectiveness.
While I still consider the methodological novelty relatively incremental, I acknowledge the practical value and consistent efficiency gains of the proposed framework. Given the clear presentation and sufficient experimental evidence, I am inclined to raise my score from 2 to 3 .
I encourage the authors to further highlight the key methodological contributions and clarify the distinctions from existing adaptive routing methods.
Thank you for raising your score, and for the constructive suggestion.
"I encourage the authors to further highlight the key methodological contributions and clarify the distinctions from existing adaptive routing methods."
The revision does both.
Key contributions, stated up front. The introduction now leads with the two ideas that make SpecRouter work, rather than with the decision rule:
- Exact offline supervision for every drafter. Deterministic proposals make each drafter's committed tokens at every prefix an exact label (Proposition 1). Proposition 2 extends this to sampling, where the label is the expected committed length. Routing across drafter families therefore becomes supervised learning: there is no exploration while serving, and every drafter is labeled, including ones a deployed policy would rarely try.
- Separating payoff from cost. What a drafter would commit depends on the text; what it costs depends on the deployment. Keeping the two apart lets one payoff predictor serve every deployment we tested, with only costs re-profiled.
Distinctions from adaptive routing methods. The related work now includes the comparison from our response, organized by what each approach requires:
- Online selectors (BanditSpec, MetaSD, HedgeSpec) learn drafter choice from verification feedback while serving. Bandits pay to explore and cannot condition on context; HedgeSpec needs every drafter's counterfactual payoff online. SpecRouter learns offline, so exactly one drafter runs per block.
- Threshold hybrids (SuffixDecoding, SAM decoding, HVD) arbitrate one fixed pair of drafters with a threshold tuned for a given deployment. SpecRouter routes over any pool and adapts to a new deployment by re-profiling costs.
Evidence that these distinctions matter in practice, also added to the revision:
- Adding a drafter costs one output head. Four new EAGLE-3 configurations took 7.4 GPU-hours of labeling on 2,500 training requests and 11 seconds of head training in total, with the existing heads unchanged.
- One deployment covers settings that no fixed method does. In batched serving (our responses to Reviewers a127 and C5J5), the fastest fixed method changes with traffic, latency target and batch size, and every one of the eight fixed methods we measured fails at least one latency target outright. SpecRouter, as a single deployment, meets all of them and stays within 7% of the fastest fixed method for each.
Together with the larger-model and simpler-router results you highlighted, these results make the paper's practical case, and the revision presents them that way.
Reply to Reviewer RhTS (1/2)
We thank the reviewer for the review, and for recognizing the practical problem and the separation that lets one payoff predictor be reused across deployments.
W1. Novelty
We agree the decision rule is simple, and we see that as a strength: it is the right rule once both of its inputs are available. What prior selectors lacked is a way to supply the payoff term for drafters of different families without running them. Online methods learn it from verification feedback and pay for exploration; threshold hybrids approximate it for a single pair of drafters. SpecRouter obtains it exactly offline: deterministic proposals make every drafter's committed tokens at every prefix an exact label (Proposition 1; Proposition 2 under sampling), and keeping payoff separate from deployment cost lets the learned part transfer while only a cost table is recalibrated. If the reviewer has specific works in mind, we would be glad to add a direct comparison. The closest work we know:
SpecRouter differs from all of them: it routes per block across three or more drafter families with different proposal mechanisms, it is trained offline on exact labels with no exploration cost, and it compares drafters by predicted time per committed token, so one predictor carries across deployments.
The two established learned selectors were already baselines in the submission, and SpecRouter beats both. BanditSpec's UCB and EXP3 reach 1.80× and 1.73× against SpecRouter's 2.77× (Table 3); with its throughput reward added, the best of eight variants still reaches only 1.84× against 2.56× under sampling. Measured live on HedgeSpec's own expert pool, HedgeSpec reaches 3.68× and SpecRouter 4.14×.
W1. Prefix statistics and static costs
The reviewer is right that target states and state-dependent costs are not where most of the gain comes from, and our new results confirm it: static costs match fitted ones, and target states add 4–6% in four of five settings and nothing in one. The camera-ready says this plainly. The gain comes from what we claim as the contribution, exact supervision and explicit cost comparison:
We therefore present target states as an optional input with a consistent but modest benefit, not as the source of the method's value. Costs still have to be compared and recalibrated per deployment, even when they are static: at batch size 16, where every single drafter slows decoding down (fixed EAGLE-3: 0.78×), recalibrated costs let SpecRouter reach 1.08× (Table 18).