acceptodds
Under review as a conference paper at ICLR 2027

Where to Speculate: Cross-Request Survival-Value Allocation for Batched Speculative Decoding

Abstract

Many batched speculative decoders apply a common proposal width to requests whose candidates have sharply different survival probabilities. PACE (Prefix-Aware Compute Allocation) estimates prefix survival before proposal generation, allocates a shared candidate budget across requests, and materializes request-specific widths without batch-wide padding. The verifier remains unchanged. For equal-cost, prefix-consistent proposals, PACE optimizes the supplied survival objective under a fixed budget. The fused runtime is evaluated as an empirical realization of this contract. On the controlled heterogeneous workload, PACE supports – more concurrent requests than the best-static capacity envelope at per-request rate floors of – tok/s. Across a seven-point sweep that varies within-batch survival separation, pre-generation estimates improve equal-budget progress by – over the offline per-budget best-fixed envelope. In the fused I-DLM-8B runtime, materializing request-specific widths within the same N8 container improves throughput by – at concurrency –, showing the effect of omitting unselected candidate positions. Offline replay on external proposal stacks confirms that the allocation relationship transfers beyond the fused decoder; these replay results do not measure physical work removal. Serving gains emerge when the saved candidate computation is large enough to amortize ragged-execution overhead.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Speculative Decoding, Batched Inference, LLM Serving, Adaptive Computation, Resource Allocation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YjkhOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper targets batched speculative decoding (SD). It identifies wasted computation caused by using the same draft and verification lengths for every request, even though their expected acceptance lengths differ. To address this, it predicts each request’s expected acceptance length before drafting and selects the length expected to yield the greatest throughput gain. The authors argue that, as long as the current prefixes remain unchanged, this scheduling maximizes the expected number of accepted tokens. With the draft token budget held constant, the method improves accepted tokens by about 12–13% over the best fixed-length method. With a maximum block size of 8 and 80–128 concurrent requests, it also increases throughput by about 30%.

Critical strengths

S1. The method predicts the expected number of accepted tokens for each request before generating draft tokens. This distinguishes it from methods that adjust each request’s verification length after drafting to improve computational efficiency.

Critical weaknesses

W1. Serving performance gains appear limited [+0.5 if resolved]

  • W1-1. The serving experiments do not compare PACE with other SD methods, so its performance gains in serving are unclear. The offline inference setup uses a fixed output length and does not account for system load.

  • W1-2. The evaluation uses greedy decoding and non-thinking mode, which may not reflect realistic serving scenarios.

  • W1-3. Scout compares probabilities across requests, so calibration is necessary. As the paper notes, calibration error can increase substantially on some new tasks. The recalibration experiment shows recovery on a fixed calibration set, but the paper should provide guidance on how often recalibration is needed during serving.

  • W1-4. The performance gain in open-loop experiments is smaller than in closed-loop experiments. The paper should explain the cause and clarify how the measurement setup affects this difference.

  • W1-5. Performance gains depend heavily on the workload and hardware. Performance drops on mixed workloads, and throughput decreases by about 2% in the H100 experiment. A clearer analysis and broader measurements are needed to identify when PACE provides practical gains over a well-tuned fixed configuration.

W2. Comparisons with adaptive drafting and verification methods are insufficient [+0.5 if resolved]

  • W2-1. Comparisons with TETRIS, AngelSpec, DSpark, and SpecDec++ are based on simulation. This does not establish the performance difference between fully implemented systems.

  • W2-2. In controlled replay, PACE performs slightly worse than TETRIS with dFlash and EAGLE-3. This may be because reducing drafting cost makes the benefit of reducing verification cost relatively smaller. The paper should analyze this effect in more depth and identify the conditions under which PACE helps or hurts.

Minor weakness: Writing. The section structure is unclear. The entire experimental setup is described in Section 4, while evaluation results are discussed in Sections 5 and 6. The paper would benefit from restructuring.

Questions for authors

Q1. Why is the performance gain smaller in the open-loop setting?

Q2. How much does Scout improve execution performance over other acceptance prediction methods, such as draft-token entropy or a moving average? The comparison should include prediction and calibration costs, while accounting for the drafting cost savings.

Q4. How do acceptance prediction and throughput vary with generation position (prefix length)? The analysis should also consider long-reasoning workloads, given the recent use of LLMs for such tasks.

Q5. Does allocating more budget to requests with higher expected acceptance lengths cause starvation? Please evaluate this by reporting per-request token generation rates and tail latency.

Q6. Did the evaluation use CUDA Graph? If so, how were variable request lengths handled? Since CUDA Graphs are commonly used in serving systems such as vLLM and SGLang, how does their use affect PACE’s performance gains?

Reviewer_Dv2jOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a prefix-aware compute allocation method that estimates prefix survival before generation. In experiments, throughput can be improved, or the batch size can be increased.

Critical strengths

The results are clear, and the experiments are performed using a real model and real hardware.

Critical weaknesses

The experiments in the main paper are limited to relatively small-sized LLMs.

Questions for authors

Can you provide more experiments on multiple LLMs?

Reviewer_pgrwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a compute allocation algorithm for batched speculative decoding. Instead of allocating the same amount of compute budget to all requests, PACE estimates the prefix survival so that more budget can be allocated to useful work. The experimental results using a fused bidirectional block-diffusion decoder and Qwen3 model show ~13% better progress for a given budget.

Critical strengths

(+) The paper addresses an important and impactful problem of improving the performance of a decoder. The main idea of allocating the compute budget based on the prefix survival estimate appears to be novel.

(+) The experimental results show the proposed idea can improve the performance of the system at least for small models on an older GPU.

(+) The paper provides a theoretical analysis of the proposed algorithm, which provides a better foundation and justification.

Critical weaknesses

(-) Experimental setup is somewhat limited. The main results come from one particular speculative decoding implementation using a relatively small/old model, Qwen3, on an old-generation GPU, A100. In that sense it is not clear how much benefit this idea will provide for more recent and future-generation hardware and models.

(-) The experiments are also performed on a synthetic controlled workload. Because the performance of speculative decoding heavily depends on the distribution of requests, the benefit in real-world systems will need to be evaluated under a more realistic workload.

(-) The paper is difficult to read and assumes deep prior knowledge and familiarity with the terminology in speculative decoding. I found it also difficult to fully understand the details of exactly how the prefix survival rates are estimated and how the overhead of ragged execution was estimated. The paper also does not clearly state its novelty compared to the state of the art.

Questions for authors
  • How does PACE perform for newer generation hardware and more modern and larger models? If the real-world experiments are difficult please provide estimates/projections with supporting technical justifications.

  • How will PACE perform for a more realistic workload?

  • What are the main technical novelty/contributions of PACE compared to the prior work, especially considering the high-level idea of allocating different amount of resources to different reqeusts within a batch has been explored? For example, how does PACE compare with ASPIRE and Nightjar?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.