- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper asks whether signals available during serving, such as draft confidence, acceptance history, and the target model's hidden state, can guide speculative decoding decisions. It studies three decisions: how many tokens to draft (horizon), how wide a draft tree to verify (width), and how many drafted tokens to verify (depth).
Width and depth are evaluated through offline replay of recorded drafts, using learned predictors to allocate a fixed verification budget across requests. Live serving experiments evaluate adaptive horizon policies.
Width: Learned reallocation gains almost nothing. Confidence predicts when verification will fail much better than whether a wider tree would repair the failure.
Depth: Adaptive allocation improves accepted length in offline experiments. A predictor using only pre-draft information achieves around half the gain of one that can also inspect the current draft, across three text benchmarks.
Horizon: In live tests, adjusting the number of tokens drafted gave small improvements when serving one request at a time, but choosing a fixed setting for each type of task worked better. Under heavier load, the adaptive methods were generally slower than a fixed setting.
Critical strengths
- Important problem. Deciding how much to draft and verify under a fixed compute is an important problem which matters for serving large models efficiently.
- Interesting observation on width. Across most settings, giving a request a wider draft tree buys little extra acceptance, and a model’s confidence predicts whether a draft will fail much better than whether widening would fix it. This finding is possibly a useful caution for work that assumes confidence is enough to determine budget allocation.
Critical weaknesses
1. Poor writing and structure make the paper hard to follow.
The writing is often dense and terse, with technical terminology introduced without sufficient explanation. For example, "Recorded lattices support counterfactual accepted-length measurements for the latter two at fixed round starts, with replayed width labels checked against execution on the primary text system" is unnecessarily difficult to parse.
The paper has no background section, and the introduction provides little explanation of key concepts such as speculative decoding, DFlash, lattices, and tree verification. Terms such as "saturation" are also not clearly defined. An overview figure explaining the setup and showing what draft horizon, verification width, and verification depth control would help considerably.
Section 2 introduces experimental minutiae before the main ideas are clear. Details such as "Replay reproduces 169,610 of 169,611 video request-rounds" could go in the appendix, leaving more room to explain the setup and findings. The main text is over a page below the nine-page limit, so there is little excuse for the lack of clear exposition. As written, understanding the experiments takes more effort than it should.
2. The interpretation of the main results is unclear.
For verification width, learned allocators achieve gains of only −0.4% to +0.5% in the lower-saturation settings, despite hindsight gains of 6–11%. This gap is interesting, but the experiments do not establish whether it reflects a fundamental difficulty or simply limitations of the predictors tested.
Questions for authors
Width: weak learner or limited predictability? The learned width allocator recovers almost none of the 6–11% hindsight gain. What evidence suggests that this reflects a genuine difficulty in predicting the benefits of wider trees, rather than limitations of the chosen predictors and features? In particular, did you test richer inputs, such as target hidden states, which proved useful for depth allocation?