- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SpecPilot is an online planner that adjusts drafting depth and the verification tree using calibrated acceptance estimates and measured execution costs. It is implemented in llama.cpp and evaluated with quantized Qwen models on three GPUs.
Critical strengths
-
It is good to count drafting and planning time as well as verification. A longer accepted draft is only useful if it actually saves time.
-
The kernel-switch examples are interesting. They show why adding one more verification position can sometimes make a step cheaper.
-
I appreciate that the authors report where the planner loses and which components do not consistently help. That makes the tradeoffs easier to judge.
Critical weaknesses
-
I find the technical contribution incremental. Choosing speculative budgets by balancing acceptance against execution cost is already well studied, including in CAST and SMART. Online calibration and cost fitting add complexity, but the evidence for their value is limited: the run-level objective contributes about 1%, padding has no consistent benefit, and the overall gain over CAST's best fixed threshold across GPUs is only 0.5%.
-
The deployment setting is not well motivated. Why is llama.cpp with Q4 quantization on an RTX 3090, V100, and Titan V the right setting for this work? These are older GPUs, and several observations depend on llama.cpp-specific kernel switches. If local or resource-constrained inference is the target, the paper should explain the relevant challenges and evaluate them. Otherwise, it is unclear how well the findings transfer to current serving hardware and engines.
-
The related-work coverage and comparisons are incomplete. DSpark, D-cut, and ECHO are neither discussed nor evaluated. Their deployment regimes differ, but they are relevant to confidence- and cost-aware speculative scheduling, so the paper should explain where its contribution fits and whether it applies to modern parallel drafters. More directly, SMART is described as the closest method but is not included in the experiments.
Questions for authors
Same as the weaknesses above.