- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes CurveK, an algorithm to adaptively choose the draft length in speculative decoding based on an estimate of the throughput at each draft length (where throughput is defined abstractively as the ratio of expected token count divided by a relative draft cost, i.e. the cost of generating draft tokens relative to the target model cost). This estimate is recomputed at every batch at inference. Since it's estimated from previous iterations, its computation is cheap. Compared against other adaptive baselines (OTF, TurboSpec, vanilla SD), CurevK achieves measured throughput speedups of between 3-11%.
Critical strengths
- the method is easy to implement and (presumably) cheap
- the gains seem to hold across datasets and over a variety of baselines
- the paper is generally pretty well-written with only minor exceptions
Critical weaknesses
- novelty is arguably somewhat limited. Baselines like OTF and TurboSpec also feature adaptive draft length based on online-estimates
- there is a substantive gap between the formula used for the algorithm, and the actual throughput, stemming from taking the mean across within a sample/across samples. The authors acknowledge this (it's the big equation in the middle of page 29), but it remains an unresolved gap. In particular, proposition 3 seems to just assume concentration, without showing this is warranted empirically, and furthermore features bias and online-loss terms that are not discussed and could be large from what I'm seeing
- a big amount of the gains over vanilla speculative decoding (although not all of it) disappears when picking "adaptively chosen, but then kept fixed throughput generation" draft lengths (table 8). This shows that a well-calibrated fixed draft length could achieve almost the same speedup
- potential issue with the oracle evaluations, see "questions" below
Questions for authors
- in the "oracle" experiments (section 6.4, appendix I) there's potentially an issue with how CurveK estimates are updated: from my reading, in these experiments, information from the full-length draft step is used to update the CurveK depth estimates for the next iteration, which would "leak" information from the full-length draft step (say, CurveK would've picked draft length=2 but the full-length step shows 5 tokens were accepted, then it seems that the CurveK depth estimates use 5 for their next update, instead of 2). Could the authors clarify if this is the case? If it is, it seems to me the experimental setup should be changed to be fairer (and CurveK should only use its own values)
- not a question, but I feel like the experimental results could be a bit more extensive (not that many datasets), and presented a bit better: e.g. it's not clear at first what the red boxes in tables mean (they look like something "negative"); some tables are a bit hard to read, especially when only reading their captions; some of them could benefit from better formatting
3 comments
Part 2/2
: In the "oracle" experiments there's potentially an issue with how CurveK estimates are updated: information from the full-length draft step is used to update the depth estimates for the next iteration, which would "leak" information (say, CurveK would've picked draft length=2 but the full-length step shows 5 tokens were accepted, then the depth estimates use 5 for their next update, instead of 2). Could the authors clarify if this is the case? If it is, the experimental setup should be changed to be fairer.
A: Thank you for the careful reading; your example describes the diagnostic accurately. It updates with 5, whereas a deployed CurveK that executed length 2 and saw both tokens accepted would update with 3, since a fully accepted draft of length is recorded as . The end-to-end throughput tables are unaffected: those runs execute CurveK's chosen length and update only from the feedback it reveals.
To make the oracle evaluation fair, as you suggest, we reran it with a second controller that chooses its own length and updates only from its own values, . Both controllers are scored on the same full-depth curve.
Tab. R1. Retained oracle-best score in the paired full- vs. censored-feedback replay.
| Workload | Full feedback | Censored feedback | Gap | |
|---|---|---|---|---|
| GSM8K | 16 | 96.5% | 96.0% | −0.5 pp |
| GSM8K | 64 | 97.1% | 96.7% | −0.4 pp |
| HumanEval | 16 | 95.8% | 95.1% | −0.7 pp |
| HumanEval | 64 | 94.2% | 93.9% | −0.3 pp |
| SpecBench-Mix | 16 | 96.3% | 95.7% | −0.6 pp |
| SpecBench-Mix | 64 | 97.4% | 96.5% | −0.9 pp |
| Mean | — | 96.2% | 95.6% | −0.6 pp |
Using only its own feedback, CurveK retains 93.9–96.7% of the oracle-best score, 0.3–0.9 points below the full-feedback controller. Both columns come from this new run. We will add Tab. R1 to App. I and describe both feedback protocols in Sec. 6.4.
: Not a question, but the experimental results could be a bit more extensive (not that many datasets), and presented a bit better: it's not clear at first what the red boxes in tables mean (they look like something "negative"); some tables are hard to read, especially from their captions alone.
A: Thank you for both suggestions.
On presentation: the peach cells mark the second-highest mean in a row, or the strongest alternative within each ablation group in Tab. 3, and the red values in Tabs. 7–8 mark throughput below the per-setting best fixed length. We agree the shading can read as negative. In the revision we will replace it with bold for the best and underline for the second-best, add a legend to each table, and state the model, batch size, decoding mode, and comparison reference in every caption.
On coverage: the single-benchmark tables use GSM8K (math), HumanEval (code), AlpacaEval (instructions), and SpecBench-Mix, which combines eight task groups (coding, math, QA, dialogue, RAG, summarization, translation, writing). LMSYS-Chat-1M dialogue is reported in App. A.8, and the mixed serving trace adds CNN/DailyMail summarization and Natural Questions/DPR retrieval QA. These are evaluated on 8B and 70B targets, A100 and H20, EAGLE-3 and Medusa drafters, greedy and sampling decoding, –, and –. We will summarize this coverage at the start of Sec. 6 so that it is visible without the appendix.
Acknowledging author's response
Thank you for your detailed response, which strengthens the paper. I will maintain my positive rating and recommend acceptance of the paper.
Part 1/2
Thank you for the thoughtful review. We address your points in order and report the fairer oracle replay you asked for in Tab. R1.
A: Thank you for the comment. OTF and TurboSpec also adapt k online; the difference is what is estimated. Both fit one geometric acceptance rate to the whole batch. CurveK keeps one accepted depth per request and evaluates each candidate through B1∑imin(k,μi), so differences between requests are kept until each candidate length is scored.
In Tab. 1, CurveK is 9.9% and 8.5% higher than TurboSpec and OTF on average across the six 8B settings. A controlled ablation links this to the aggregation choice: MeanDepth-K uses the same depth states, cost model, and selection rule and only averages before truncating, and CurveK is 6.8% higher across the same six settings (Tab. 3).
A: Thank you for the careful reading. Prop. 3 is meant as a decomposition that identifies where selection loss can come from, under the concentration condition in App. D.1, with the Jensen gap and online mismatch kept explicit. We examine each term empirically and will add this discussion to Sec. 5.4.
The decomposition names the error sources; these measurements, together with the end-to-end throughput tables, evaluate their effect on the selected lengths.
A: Thank you for this point. A fixed length chosen for each setting is indeed a strong reference, and CurveK reaches about the same throughput without that calibration. The per-setting reference requires a seven-length sweep for each workload, batch size, and decoding mode, and is chosen after seeing the results. We compare against both this reference and the best single fixed length across settings:
The percentages compare mean throughput across the 21 settings.
The distinction matters because the best fixed length moves with the setting. On GSM8K under greedy decoding it is k=4 at B=8 and k=2 at B=64, and using either at the other batch size loses 9–10%. The best single fixed length (k=3 for greedy, k=2 for sampling) loses up to 6.9% and 9.3% in individual settings (App. A.2). A table of fixed lengths indexed by batch size would also need to know the workload, which a server handling mixed traffic does not know in advance. On the mixed-workload trace (Tab. 2), where request types change during serving, CurveK is ahead of the strongest dynamic baseline by 4.7%, 5.3%, and 4.9% at B=32, 64, and 128, and within 0.5% of it at B=16.