acceptodds
Under review as a conference paper at ICLR 2027

Beyond Acceptance Prediction: When Adaptivity Pays Off in Speculative Decoding

Abstract

Adaptive speculative decoding uses serving-time signals to choose how much to draft or verify. When do these signals identify useful budget changes,and when does that knowledge improve throughput? We study these questions through per-round counterfactual replay and controlled serving experiments.In the evaluated block-diffusion lattices away from draft-window saturation, confidence predicts accepted length but weakly distinguishes rounds that benefit from one additional beam width. Frozen learned allocators change accepted length by −0.4 to +0.5% at matched node budgets, despite hindsight gains of 6–11%. A failure–repair decomposition localizes this gap: the evaluated signals predict whether verification fails much better than whether a wider tree repairs that failure. Positive results at smaller budgets, saturated windows, and with an official EAGLE-3 head delimit this finding. For verification depth, the tested survival predictors yield useful allocation gains. With the decision and learner held fixed, target-side features available before drafting recover approximately half of the post-draft learner's allocation gain on three homogeneous text benchmarks; discarded drafts also retain useful signal after prefix correction. Separate live draft-horizon experiments expose two further constraints: execution cost and the static baseline. In one controlled stream at concurrency one, a target-state rule improves the measured token rate by 1.33% over the best global constant but loses to a frozen workload-specific constant. These measurements distinguish acceptance prediction, actionable budget information, and realized serving benefit. They support decision-specific evaluation rather than a general limit on adaptive decoding.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Speculative decoding, Adaptive inference, Acceptance prediction, Inference-time resource allocation, Large language model serving

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_J71UOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper asks whether signals available during serving, such as draft confidence, acceptance history, and the target model's hidden state, can guide speculative decoding decisions. It studies three decisions: how many tokens to draft (horizon), how wide a draft tree to verify (width), and how many drafted tokens to verify (depth).

Width and depth are evaluated through offline replay of recorded drafts, using learned predictors to allocate a fixed verification budget across requests. Live serving experiments evaluate adaptive horizon policies.

Width: Learned reallocation gains almost nothing. Confidence predicts when verification will fail much better than whether a wider tree would repair the failure.

Depth: Adaptive allocation improves accepted length in offline experiments. A predictor using only pre-draft information achieves around half the gain of one that can also inspect the current draft, across three text benchmarks.

Horizon: In live tests, adjusting the number of tokens drafted gave small improvements when serving one request at a time, but choosing a fixed setting for each type of task worked better. Under heavier load, the adaptive methods were generally slower than a fixed setting.

Critical strengths
  • Important problem. Deciding how much to draft and verify under a fixed compute is an important problem which matters for serving large models efficiently.
  • Interesting observation on width. Across most settings, giving a request a wider draft tree buys little extra acceptance, and a model’s confidence predicts whether a draft will fail much better than whether widening would fix it. This finding is possibly a useful caution for work that assumes confidence is enough to determine budget allocation.
Critical weaknesses

1. Poor writing and structure make the paper hard to follow.

The writing is often dense and terse, with technical terminology introduced without sufficient explanation. For example, "Recorded lattices support counterfactual accepted-length measurements for the latter two at fixed round starts, with replayed width labels checked against execution on the primary text system" is unnecessarily difficult to parse.

The paper has no background section, and the introduction provides little explanation of key concepts such as speculative decoding, DFlash, lattices, and tree verification. Terms such as "saturation" are also not clearly defined. An overview figure explaining the setup and showing what draft horizon, verification width, and verification depth control would help considerably.

Section 2 introduces experimental minutiae before the main ideas are clear. Details such as "Replay reproduces 169,610 of 169,611 video request-rounds" could go in the appendix, leaving more room to explain the setup and findings. The main text is over a page below the nine-page limit, so there is little excuse for the lack of clear exposition. As written, understanding the experiments takes more effort than it should.

2. The interpretation of the main results is unclear.

For verification width, learned allocators achieve gains of only −0.4% to +0.5% in the lower-saturation settings, despite hindsight gains of 6–11%. This gap is interesting, but the experiments do not establish whether it reflects a fundamental difficulty or simply limitations of the predictors tested.

Questions for authors

Width: weak learner or limited predictability? The learned width allocator recovers almost none of the 6–11% hindsight gain. What evidence suggests that this reflects a genuine difficulty in predicting the benefits of wider trees, rather than limitations of the chosen predictors and features? In particular, did you test richer inputs, such as target hidden states, which proved useful for depth allocation?

Reviewer_FuMvOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether acceptance signals help allocate speculative decoding budgets. It uses offline replay to examine verification width and depth, then runs live experiments on draft-horizon selection and scheduling overhead.

Critical strengths
  1. I like the question being asked. Knowing that a draft will fail is not the same as knowing that a larger verification budget will help.

  2. The authors include cases where their findings do not hold. That helps show how much depends on the particular budget and drafter.

  3. The execution controls are useful. Measuring the cost of reading a signal and comparing against workload-specific constants makes the evaluation more informative.

Critical weaknesses
  1. The paper feels underdeveloped and is difficult to follow. The main text stops at eight pages despite the nine-page allowance, yet key connections between the experiments and details of the serving implementation remain unclear. It moves between verification width, verification depth, and draft horizon while changing predictors and protocols. I would have preferred the remaining space to be used to explain these choices and build a more complete argument.

  2. The offline and online evidence do not connect cleanly. The offline study evaluates verification allocation, while the main live experiments change the draft horizon and use different predictors. The separate verification-depth prototype runs only in eager mode. These results do not establish how much of the offline allocation benefit survives in a matched, optimized serving implementation.

  3. I am not convinced that the negative findings provide enough guidance beyond the tested setup. Weak local-width gains depend on the candidate lattice, budget, and predictors; the paper itself reports positive results with smaller budgets and an official EAGLE-3 head. Likewise, the host-read penalty measures one execution path. Without matched tests of approaches such as DSpark or D-cut, it remains unclear how these observations translate to existing adaptive systems.

Questions for authors

Same as the weaknesses above.

Reviewer_C16eOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks when predicting how many draft tokens a model will accept helps decide where to spend a limited verification budget. The authors replay alternatives from the same recorded generation step, mainly on 27B and 4B models, using text and video workloads. They vary either the beam width (which branches to verify) or the depth (how far to verify). In the settings grouped in Table 2A, the learned policies change accepted length by −0.4% to +0.5%; choosing budgets after seeing the outcomes would gain 6.9–10.6%. Smaller budgets and a separate EAGLE-3 test give positive results. For depth, both previous target states and discarded draft tokens carry useful information. A separate live experiment varies how many tokens to draft in the first place. The replay gains are not speedups.

Critical strengths
  1. The paper tests whether extra width fixes a failure, not just whether a failure is likely. On 27B text with 15 draft positions, confidence predicts whether the width-4 tree falls short (AUROC 0.95). Among those failures, it barely predicts whether width 5 gets past the first missed token (0.55; chance is 0.50). The binary replay labels also agree with actual execution in 99.3–99.9% of comparisons on 1,414 greedy 27B text starts (Tables 5 and 9). Earlier work allocates verification slots across requests [1] or shows that better prediction need not improve a policy that skips verification [2]. The new evidence here is the failure-versus-repair test with target verification kept in place.

  2. The live controls change the practical conclusion. On 400 held-out requests, served one at a time, the target-state rule beats the best single fixed horizon by 1.33% in request token rate. Reconstructing a lookup from the constant-horizon runs reverses that result: the target-state rule loses by 0.88% to one calibrated constant for chat and another for math (Tables 3–4). A later two-engine test also favors the simple rule. Separately, holding the action and accepted length fixed shows that reading confidence back from the GPU adds 0.746 ms per round. This measures the cost of the read without confusing it with a change in the number of accepted tokens.

Critical weaknesses
  1. The repair test does not explain the full gap to hindsight. Equation 2 labels whether widening gains any tokens; the main test is width 4 to 5. A one-token repair and a five-token repair get the same label. The allocator in Table 2, by contrast, predicts how many tokens each change gains and chooses among widths 1–12 (Appendix C). Its small measured gain is credible. What remains unclear is how much of its shortfall comes from that one hard decision. DDTree [3] and BASTION [4] suggest a direct check: sum the drafter's prefix probabilities for each of the two actual beam trees, then take the difference. This scores paths gained as well as paths displaced. The best-first result in §3 changes the tree itself; it does not report this comparison on the original beam trees.

  2. “About half” compares two policies; it does not isolate the value of target state. In Figure 2, the pre-draft predictor sees history, the previous outcome and target features. The post-draft predictor sees history and the current draft, but not the other two inputs. On 27B GSM8K, history and outcome alone gain 2.9% over uniform depth. Adding target features brings this to 8.1%, an increase of 5.2 percentage points. The post-draft policy gains 15.4% (Table 12). The reported ratio is meaningful for those two policies on the three named benchmarks. It does not show how much one gains by waiting for the draft while keeping the other inputs fixed. Appendix A's check on real batches compares history, confidence and hindsight, not this pair of predictors.

References:

[1] Zhaoxuan Wu et al. TETRIS: Optimal Draft Token Selection for Batch Speculative Decoding

[2] Haoxuan Luo, Jameson Sandler, and Ferdinando Fioretto. From Positionwise Confidence to Prefix Scheduling: Verifier Skipping in Speculative Decoding

[3] Liran Ringel and Yaniv Romano. Accelerating Speculative Decoding with Block Diffusion Draft Trees

[4] Soowon Oh et al. BASTION: Budget-Aware Speculative Decoding with Tree-structured Block Diffusion Drafting

Questions for authors
  1. On the 27B, 15-position text trace, how much of the learned-versus-hindsight gap remains if both allocators can use only widths 3, 4 and 5 at the total node budget of uniform width 4? The existing width curves should let you separate nearby decisions from large reallocations.
  2. Do you have a depth result where both predictors receive the same history, previous outcome and target state, and only the second one also sees the current draft? If not, please distinguish the extra gain from target state from the total pre-draft policy gain in the abstract.
  3. Table 1 says the horizon is chosen before drafting. In Table 3, does confidence read after round r choose the horizon for round r+1? Does the fixed-horizon prediction score in Table 15 use that same time lag? The answer changes how I interpret the live comparison.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.