- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper targets batched speculative decoding (SD). It identifies wasted computation caused by using the same draft and verification lengths for every request, even though their expected acceptance lengths differ. To address this, it predicts each request’s expected acceptance length before drafting and selects the length expected to yield the greatest throughput gain. The authors argue that, as long as the current prefixes remain unchanged, this scheduling maximizes the expected number of accepted tokens. With the draft token budget held constant, the method improves accepted tokens by about 12–13% over the best fixed-length method. With a maximum block size of 8 and 80–128 concurrent requests, it also increases throughput by about 30%.
Critical strengths
S1. The method predicts the expected number of accepted tokens for each request before generating draft tokens. This distinguishes it from methods that adjust each request’s verification length after drafting to improve computational efficiency.
Critical weaknesses
W1. Serving performance gains appear limited [+0.5 if resolved]
-
W1-1. The serving experiments do not compare PACE with other SD methods, so its performance gains in serving are unclear. The offline inference setup uses a fixed output length and does not account for system load.
-
W1-2. The evaluation uses greedy decoding and non-thinking mode, which may not reflect realistic serving scenarios.
-
W1-3. Scout compares probabilities across requests, so calibration is necessary. As the paper notes, calibration error can increase substantially on some new tasks. The recalibration experiment shows recovery on a fixed calibration set, but the paper should provide guidance on how often recalibration is needed during serving.
-
W1-4. The performance gain in open-loop experiments is smaller than in closed-loop experiments. The paper should explain the cause and clarify how the measurement setup affects this difference.
-
W1-5. Performance gains depend heavily on the workload and hardware. Performance drops on mixed workloads, and throughput decreases by about 2% in the H100 experiment. A clearer analysis and broader measurements are needed to identify when PACE provides practical gains over a well-tuned fixed configuration.
W2. Comparisons with adaptive drafting and verification methods are insufficient [+0.5 if resolved]
-
W2-1. Comparisons with TETRIS, AngelSpec, DSpark, and SpecDec++ are based on simulation. This does not establish the performance difference between fully implemented systems.
-
W2-2. In controlled replay, PACE performs slightly worse than TETRIS with dFlash and EAGLE-3. This may be because reducing drafting cost makes the benefit of reducing verification cost relatively smaller. The paper should analyze this effect in more depth and identify the conditions under which PACE helps or hurts.
Minor weakness: Writing. The section structure is unclear. The entire experimental setup is described in Section 4, while evaluation results are discussed in Sections 5 and 6. The paper would benefit from restructuring.
Questions for authors
Q1. Why is the performance gain smaller in the open-loop setting?
Q2. How much does Scout improve execution performance over other acceptance prediction methods, such as draft-token entropy or a moving average? The comparison should include prediction and calibration costs, while accounting for the drafting cost savings.
Q4. How do acceptance prediction and throughput vary with generation position (prefix length)? The analysis should also consider long-reasoning workloads, given the recent use of LLMs for such tasks.
Q5. Does allocating more budget to requests with higher expected acceptance lengths cause starvation? Please evaluate this by reporting per-request token generation rates and tail latency.
Q6. Did the evaluation use CUDA Graph? If so, how were variable request lengths handled? Since CUDA Graphs are commonly used in serving systems such as vLLM and SGLang, how does their use affect PACE’s performance gains?