acceptodds
Withdrawn from ICLR 2027

SpecPilot: Guiding Speculation with Online Adaptive Cost Prediction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Speculative decoding speeds up autoregressive inference by having a cheap pro- poser draft several tokens that the target model then verifies in a single forward pass. Whether a step pays off depends on how far acceptance holds, what verify- ing a given tree costs on the machine at hand, and what drafting itself costs, and all three change during a run. Fixed policies therefore win on some workloads and lose on others: the best draft length for llama.cpp’s speculative decoding is the worst on prose, where it runs 0.69× as fast as not speculating at all. We present SpecPilot, a planner that decides at every step how many drafter levels to buy and which tree to verify, judging each choice against the throughput the run has actually achieved. SpecPilot learns the verify cost surface online, including kernel-switch discontinuities whose direction differs between GPUs, and can pad a tree past one when the larger tree is cheaper. It prices each node by a path probability calibrated against observed acceptance, and requests another drafter level only while its expected yield covers the measured cost of drafting. With a single untuned configuration on an RTX 3090, SpecPilot reaches a 1.78× geometric-mean speedup over plain decoding across five workloads, 14.1% faster than llama.cpp's builtin speculative decoding and 3.7% faster than CAST, each tuned per cell in hindsight. CAST’s best thresh- old changes with the workload and the GPU: across 23 cells on three GPUs, SpecPilot matches CAST’s best fixed threshold (+0.5%) and is within 0.3% of CAST retuned for every cell, while CAST’s worst threshold falls up to 22.7% below it.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
llm, speculation, speculative-decoding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_f3ftOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SpecPilot is an online planner that adjusts drafting depth and the verification tree using calibrated acceptance estimates and measured execution costs. It is implemented in llama.cpp and evaluated with quantized Qwen models on three GPUs.

Critical strengths
  1. It is good to count drafting and planning time as well as verification. A longer accepted draft is only useful if it actually saves time.

  2. The kernel-switch examples are interesting. They show why adding one more verification position can sometimes make a step cheaper.

  3. I appreciate that the authors report where the planner loses and which components do not consistently help. That makes the tradeoffs easier to judge.

Critical weaknesses
  1. I find the technical contribution incremental. Choosing speculative budgets by balancing acceptance against execution cost is already well studied, including in CAST and SMART. Online calibration and cost fitting add complexity, but the evidence for their value is limited: the run-level objective contributes about 1%, padding has no consistent benefit, and the overall gain over CAST's best fixed threshold across GPUs is only 0.5%.

  2. The deployment setting is not well motivated. Why is llama.cpp with Q4 quantization on an RTX 3090, V100, and Titan V the right setting for this work? These are older GPUs, and several observations depend on llama.cpp-specific kernel switches. If local or resource-constrained inference is the target, the paper should explain the relevant challenges and evaluate them. Otherwise, it is unclear how well the findings transfer to current serving hardware and engines.

  3. The related-work coverage and comparisons are incomplete. DSpark, D-cut, and ECHO are neither discussed nor evaluated. Their deployment regimes differ, but they are relevant to confidence- and cost-aware speculative scheduling, so the paper should explain where its contribution fits and whether it applies to modern parallel drafters. More directly, SMART is described as the closest method but is not included in the experiments.

Questions for authors

Same as the weaknesses above.

Reviewer_p1AKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces speculative decoding with online guidance through adaptive cost prediction. The main motivation of this paper is driven from the fact that verification cost can vary across devices and task types, therefore to maintain cost vs throughput. The main solution proposed by this paper is to select how far along to speculate given the current context, draft model, hardware and verifier model.

The paper proposes to use the drafter to span out multiple candidate decodings based on top-k sampling per token. Each of these decodes will have a per token predicted total probability. The authors introduce a beam search based technique to decide the subtree to keep based on draft confidence and cost of verification until the current tree depth vs running another draft step and verifying then.

This is a classic dynamic programming problem under beam search with a modified cost model which has been well studied in the past and the application of the same in speculative decoding is interesting.

The authors have built their harness on top of llama.cpp and have run their experiment over prose, technical Q&A, math, information ex-traction and JSON generation, with three prompts each repeated twice. The teacher forcing part was unclear.

The main takeaway is that depending on tasks the tree depth can dynamically change using SpecPilot which results in better verification cost and acceptance rates. The improvements compared to llama.cpp seems promising but with respect to CAST seemed marginal.

Critical strengths
  1. Identifies the variance in true serving cost due to hardware and task type
  2. Contrary to common belief that more speculation is free, this paper shows that depending on hardware (block size, tensor core size) sometimes it might not be true.
  3. Introduces a novel planner to account drafting and verification cost, drafter confidence per decoding tree to select only those which meet an expected verification cost.
Critical weaknesses
  1. The evaluation section is difficult to follow.
  2. The evaluation examples might be too small to comment on the significance of the method. (3 prompts per task is too small)
  3. The paper shows that adaptive drafting is helpful but fails to show which component of the introduced costs helps what.
  4. The relative improvements are marginal.
Questions for authors
  1. What is a cell? I assume it is one run config for combined, drafter, verifier and task type.
  2. How do you isolate the contribution of each cost type?
  3. The reported numbers are percentage improvements. Can you add a baseline runtime to establish the significance of the relative numbers?
  4. Have you validated the method on any other model family?
Reviewer_AYHCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Speculative Decoding is a technique that speeds up autoregressive inference by using a cheap draft model that drafts multiple tokens and then the target model verified in a single forward pass. It is based on the idea that the verification step on the larger target model can happen in parallel and hence, we can save computation time. The actual speedup achieved depends on a lot of factors like what draft length we use, how cheap is the draft model, how costly is the verification and what is the expected acceptance length. This paper mentions that fixed policies can work well in some settings but can be actually slower in others. This work proposes dynamic method SpecPilot. which learns all these decision factors based on the actual workload, the gpus used, the drafter model and all the other important criteria. It learns verification cost during generation, adjusts the drafter’s confidence estimates using observed acceptance, and accounts for the time spent drafting. They also conduct experiments to indicate that their proposed method achieves superior performance compared to baseline methods in the literature so far.

Critical strengths
  • This work studies an important problem on how to setup the actual different parameters for the speculative decoding algorithm. Speculative decoding is used widely to speed up inference and the ways to speed it up in practice depends a lot on the workload, the hardware as this paper mentions and this work proposes a method to improve this which is an important problem and would be relevant in practice.
  • The paper is overall well written and easy to follow. The motivation section 3 is well presented and highlights the limitations of the previous work.
  • The authors report where the method loses, include negative ablation results, and explain several performance differences through drafting depth and step cost.
Critical weaknesses
  • Each workload contains only three prompts, with two repeats per prompt so it not clear how would the results hold in realistic settings.
  • While the authors mention SMART baseline is most similar to their setup, this is not included in the experimental results. There are other weaknesses with the experimental section. See below.
Questions for authors
  • It would be nice to include more agentic workload comparison between the algorithms which is the a very relevant workload. Do the authors have thoughts on that?
  • Do the authors have comparisons on the benefits of including the different parts of the algorithm like - estimating drafter cost vs. not, estimating verifier cost vs. not, predicting acceptance rate vs. not?
  • Could the authors give some intuition of why their method does not perform well on Volta GPUs whereas it does on RTX 3090?
    • The experiments use a warmup of 1024 tokens and then a prediction of 1024 tokens. Do the speedup results include the warmup phase also? How would the results look like in real settings and what warmup would be good to use?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.