acceptodds
Under review as a conference paper at ICLR 2027

Cadence: Regulating the Speculation Externality in LLM Serving

Abstract

Speculative decoding (SD) speeds up LLM inference by verifying draft tokens in parallel. SD applies to every request, but its effectiveness, how many draft tokens are accepted, varies across requests and over time. Requests in a batch share each step, so chasing mean throughput can slow requests that accept little: in one step at batch 112, SGLang's default SD (SGL-SD) raises throughput 13% while 26% of requests run slower than autoregressive (AR) decoding. Cadence, our per-step controller, caps each request's delay, its time lost relative to AR steps. Every step commits at least one token per request, so the caps reduce to one allowed step latency, known before drafting. Within this latency, an all-or-none floor gives every request a draft token whenever the latency allows, and Cadence picks the verification budget that an operator's utility scores highest. On Qwen3-32B under high load, Cadence improves goodput by 1.29 over AR decoding and by 1.80 over SGL-SD, while 99.5% of its requests run no slower than AR decoding, against 69% under SGL-SD.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM serving, LLM inference efficiency, Speculative Decoding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_soS4OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CADENCE studies how speculative decoding can improve batch throughput while slowing individual requests with low acceptance. It tracks each request's cumulative delay relative to AR steps and chooses the batch-wide draft depth and per-request verification lengths within a latency allowance.

Critical strengths
  1. I like the speculation externality framing. Everyone waits through the same step, but not everyone gets enough accepted tokens to make that wait worthwhile.

  2. High-concurrency serving is a worthwhile setting to study. Looking at which requests get slower adds something that average throughput alone misses.

  3. The controller works without retraining the drafter or changing the target model. That makes it easier to consider as a serving-side optimization.

Critical weaknesses
  1. I found the introduction hard to follow. Too many concepts appear before the basic idea is clear. A simpler explanation of the problem and how the controller addresses it would help.

  2. I am still unsure how variable-length verification is implemented. SGLang's standard fixed-shape speculative verification path cannot simply assign a different verification length to each request without additional support. This needs a ragged verification implementation, as in DSpark, together with a clear strategy for CUDA graph replay. The paper says CUDA graphs and overlap scheduling are enabled, but does not explain how unequal prefixes are packed, how graph shapes are selected, or how KV-cache updates are handled. For a method built around per-request verification lengths, this is a central systems detail, and I would like to see it explained rather than left implicit.

  3. DSpark, D-cut, and ECHO are cited but not compared experimentally. Beating default speculation and fixed budgets leaves me unsure how much CADENCE adds over existing adaptive methods. A comparison under the same serving setup would be more convincing.

Questions for authors

Same as the weaknesses above.

4 comments

We thank you for your detailed review. Our responses are as follows:

[W1] Introduction

We agree, and in the final version we will restructure Section 1 around one idea: every decode step commits at least one token per request, so per-request delay caps reduce to one step latency known before drafting. Section 1 will state the problem (one request's speculation lengthens every co-batched request's step), then this observation, then Cadence's three stages tied to Figure 2, and then the contributions, moving the remaining concepts to Sections 2 and 3.

[W2] Variable-length verification under CUDA graphs and overlap

Packing. Each request rr verifies its first 1+ℓr1+\ell_r tokens. Cadence packs them into one FlashAttention-3 varlen launch of B+∑rℓrB+\sum_r \ell_r positions and rewrites cu_seqlens_q, cache lengths, page table and KV slots to match. Acceptance is a masked chain walk. With deterministic inference on, outputs are identical across policies in 656 comparisons.

Graph shapes. Cadence runs on SGLang's ragged target-verify path (compact mode, the path SGLang uses for DSpark [2]), extended to EAGLE-3 chains:

  1. Graphs keyed by token count. One target-verify graph is captured per token tier T=b(d+1)T = b(d+1), over the captured batch sizes bb and draft depth dd. Before replay, the live per-request lengths and query offsets are copied into the tier's captured buffers. FA-3 reads per-request extents from device buffers, so only TT, the slot count and the maximum query length are fixed at capture.
  2. Budgets on the tiers. Cadence restricts its candidate budgets KK to these tiers (at B=112B = 112 and depth 7, 20 candidates instead of 57). Fill places exactly KK tokens, so every partial budget replays a captured shape with no dummy tokens, and unused slots are zero-length rows. A full draft at an uncaptured batch size rounds up to the next tier, exactly as stock SGLang pads it. Restricting KK to the tiers costs at most 0.03% goodput.
  3. Depth. Each draft depth d∈{0,1,3,7}d \in \lbrace 0, 1, 3, 7 \rbrace has its own runtime state and graphs (SGLang's adaptive-mode mechanism), and Cadence selects one per step from its plan rather than by a moving average of acceptance.
  4. Each step. The host picks dd, KK and ℓ\ell and copies them asynchronously. The device builds positions and KV slots from its own sequence lengths, replays the tier of KK, scatters logits and hidden states back per request, and stops each chain walk at ℓr\ell_r.

KV cache. SGLang's Spec v2 already reserves each request's committed length plus twice the largest draft length, so allocation never depends on KK or ℓ\ell. The next step's writes overwrite rejected slots, and nothing is reclaimed.

Overlap. SGLang launches step t+1t+1 before it reads step tt's acceptance. Cadence's plan needs only slack, the shared acceptance curve and KK, so it plans step t+1t+1 by charging step tt as one committed token per request at its profiled latency, and accepted tokens earn their credit one step later. The bound then holds with two steps' overrun instead of one, under Proposition A.2's condition shifted by one step. Planning one step late costs at most 0.002% goodput per seed. Under tripled step-time noise, the tiers and the one-step-late plan put no request over the cap at ϵ=0.25\epsilon = 0.25, and one request in ten seeds at ϵ=0.1\epsilon = 0.1 (two without them), at most 19.2 ms past its cap (14.1 ms without them).

[W3] DSpark, D-cut and ECHO

We compare Cadence against the D-cut [1] and DSpark [2] allocation rules on the same drafter, latency profile, workloads, seeds and metrics (Table R1 in the general response). Both were designed for drafters that emit a whole block in about one pass. On EAGLE-3 at batch 112, their full depth 7 costs 5.5 ms more per step than depth 1, so they reach 0.89× and 0.91× Cadence's throughput and put up to 57.3% of slow requests over the ϵ=0.1\epsilon = 0.1 cap. Given Cadence's per-step depth choice they match its throughput, which isolates the allocation rule. Neither rule reads delay: ranked by true acceptance, DSpark's rule puts 40.4% of slow requests over the ϵ=0.1\epsilon = 0.1 cap, while Cadence puts none over at either tolerance. For CUDA graphs, D-cut limits KK to four fractions of the full draft, the role Cadence's tiers play above.

ECHO [3] treats the batch as one draft tree under a fixed global verification budget set near hardware saturation and, at high load, moves budget from low-confidence to high-confidence requests, as the DSpark rows do, reading no per-request delay. The closest policies in our evaluation are SGL-SD-C320, the best of nine fixed budgets (1.14× SGL-AR's throughput with 14.7% of requests slower than AR, against 1.30× and 0.53% for Cadence), and the DSpark rows of Table R1. ECHO's tree shaping is complementary: Cadence's bound needs only that each step fit the allowed step latency, whatever fills it (Lemma A.1), so ECHO's trees can fill Cadence's budget.

References.

[1] Liu et al., "D-cut: Adaptive Verification Depth Pruning for Batched Speculative Decoding," arXiv:2607.14647, 2026.

[2] Cheng et al., "DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation," arXiv:2607.05147, 2026.

[3] Hu et al., "ECHO: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarios," arXiv:2604.09603, 2026.

Thanks. I will raise my score.

Thank you for raising your score, and for the detailed comments that improved the paper.

Reviewer_Wv1nOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work studies the speculation externality problem, where speculative decoding improves aggregate throughput while increasing the serving latency of a subset of requests within a batch. The authors track each request's cumulative delay relative to standard autoregressive (AR) decoding and propose Cadence, a framework that dynamically adjusts the speculation budget based on the measured cumulative delay. Through experiments, the authors demonstrate that Cadence improves the throughput and goodput of speculative decoding while effectively reducing the fraction of requests that experience latency degradation.

Critical strengths
  • Meaningful problem formulation. This work clearly identifies the problem that improvements in aggregate throughput do not necessarily translate into latency improvements for individual requests. This is a practically relevant and potentially significant problem for real-world LLM serving systems.

  • Simple and intuitive methodology. The proposed approach of translating per-request cumulative delay constraints into a shared step latency budget is straightforward and intuitive. It also appears relatively easy to implement and integrate into existing LLM serving frameworks.

Critical weaknesses
  • Insufficient differentiation from existing adaptive speculative decoding methods. My primary concern is the lack of a clear distinction between the proposed method and existing adaptive speculative decoding approaches. Adaptive speculation budget selection is a well-established technique that has already been explored in several prior studies. However, this work does not provide explicit comparisons with relevant methods such as D-Cut, DSpark, and AdaServe. The static baselines considered in the paper are insufficient to clearly establish the benefits of the proposed approach over existing adaptive methods.

  • Weak connection between the proposed components and the experimental results. The delay regulation mechanism, which is presented as a core component of the proposed method, does not actually affect the scheduling decisions in the reported experiments. Most of the throughput gains appear to come from adaptive speculation budget selection. However, it remains unclear which aspects of the proposed approach, beyond those already explored in prior work, account for these improvements. This raises concerns about whether the experimental results adequately support the claimed contributions.

  • Limited generality and questionable evaluation methodology. The proposed latency model does not account for context length (i.e., KV cache length), and all experiments focus on workloads with short input prompts and short generations. Furthermore, the definition of goodput, which excludes tokens from requests that are slower than under AR decoding, may disproportionately penalize the baselines. It is therefore unclear to what extent the reported improvements would translate into practical benefits in real-world LLM serving systems.

Questions for authors
  • What are the key advantages of the proposed method over existing adaptive speculative decoding approaches, such as D-Cut, DSpark, and AdaServe? Could the authors provide direct experimental comparisons and ablation studies to demonstrate the effectiveness of each core component separately?

  • If the delay regulation mechanism does not affect speculation budget selection in the reported experiments, how should its role as a core contribution of this work be understood?

  • What evidence supports the validity of the proposed latency model and goodput definition under representative LLM serving workloads? In particular, do the reported improvements consistently hold under long-context workloads and evaluations based on absolute latency SLOs?

5 comments

We thank you for your thoughtful review. Our responses are as follows:

[W1 & Q1] Advantages over adaptive speculative decoding (D-cut, DSpark, AdaServe)

Adaptive methods tune how much to speculate, to raise throughput (D-cut [1], DSpark [2]) or to meet per-request deadlines (AdaServe [3]). Cadence answers a different question: how much may the batch speculate, given the delay each request has already absorbed from others' speculation? It answers before drafting, from each request's slack and without drafter confidence. As long as no step overruns its profile by more than ϵ\epsilon AR steps, every capped request stays within its cap up to one step's overrun (Proposition A.2). None of the three methods provides such a bound. Tables R1 and R2 in the general response (same drafter, profile, workloads, seeds and metrics) show why it matters:

  • D-cut's and DSpark's published rules draft the full depth 7 every step, which is nearly free on the block-parallel drafters they target but costs 5.5 ms per step on EAGLE-3 at batch 112. They reach 0.89× and 0.91× Cadence's throughput, slow 8.7% and 10.7% of requests, and put up to 57.3% of a slow class over the ϵ=0.1\epsilon = 0.1 cap.
  • Given Cadence's per-step depth choice, they match its throughput: depth sets throughput, and the ranking moves it by under 2%. The ranking decides who pays, and nothing bounds it. Ranked by true acceptance, DSpark's rule slows the same low-acceptance requests every step and puts 40.4% of them over the ϵ=0.1\epsilon = 0.1 cap. Cadence puts none over at either tolerance.
  • At full throughput (budgets 224 and 320), AdaServe's deadline-first rule meets a 20 ms deadline for fewer tight requests than Cadence (41.3% and 36.7% against 42.2%) and slows more best-effort requests. Its best budget reaches 72.2% only at 0.94× throughput, where Cadence with priority weight 4 reaches 83.7% by moving delay onto best-effort requests that carry no cap.

[W2 & Q2] Which components drive the gains, and the role of delay regulation

The per-step depth choice earns the throughput. Delay regulation is the safety property: it keeps low-acceptance requests from paying more than their cap when the operating point shifts. One-at-a-time ablations, paired by seed:

  • Depth. Fixing depth 3 costs 12.6% throughput and slows 6.9 points more requests. At saturation depth 1 is the right choice and Cadence makes it. At lighter load, fixing depth 1 raises median TPOT from 8.6 to 11.0 ms.
  • The cap supplies the bound, and the depth choice makes it nearly free. At ϵ=0.25\epsilon = 0.25 the depth choice keeps every request inside its cap, so the cap never overrides a plan. At ϵ=0.1\epsilon = 0.1 it acts on 0.6% of steps and costs 0.08% throughput. When depth cannot adapt, the cap does the work: with depth fixed at SGL-SD's 3 and no cap, 64.7% of the slow class ends over its cap at ϵ=0.1\epsilon = 0.1, and 27.2% of a 10% low-acceptance minority at ϵ=0.25\epsilon = 0.25. With the cap, 0.12% and 0.13%. Under Proposition A.2's condition this protection holds whatever the acceptance, which is what the adaptive methods above lack.
  • Budget. The budget choice adds 0.6% throughput on the slow-class workload and slows 2.5 points fewer of the slow class (ϵ=0.1\epsilon = 0.1).

[W3 & Q3] Latency model and context length

The profile L(B,d,K)L(B, d, K) (batch size, draft depth, verified tokens) is fitted to SGLang step-latency measurements with about 740-token prompts and 64 output tokens. At these lengths context matters little: verify time varies by 0.9 to 4.8% across 512 to 1,856 context tokens (batch 32). Cadence tracks measured step time, so profile error appears as overrun, which the bound tolerates up to ϵ\epsilon AR steps per step (Proposition A.2). At ϵ=0.25\epsilon = 0.25, a profile up to 10% off in either direction keeps throughput at 1.30× SGL-AR with no request over the cap (Appendix D.9). For long contexts, a context dimension in the profile changes no part of the algorithm. With long, unequal contexts, Fill extends directly to per-request verify costs.

[W3 & Q3] Goodput definition and absolute SLOs

Our goodput targets exactly the externality the paper studies: a token counts only if speculation did not leave its request worse off than no speculation. Cadence's lead does not depend on it. Goodput with TPOT measured from admission, trace at high load:

TPOT targetover SGL-SDover SGL-SD-C320over SGL-AR
25 ms2.50×1.65×10.6×
30 ms1.65×1.23×1.30×
40 ms1.30×1.14×1.30×
50 ms1.27×1.14×1.30×

Cadence leads every SGLang baseline at every target. The 10.6× over SGL-AR at 25 ms reflects SGL-AR's 28.8 ms median TPOT at high load, just above the target. Measured from arrival, queueing included, Cadence's lead over SGL-SD grows to 2.66×, 2.54×, 2.25× and 2.04×. Under the delay cap itself, Cadence's goodput is 1.29× SGL-SD's and 1.30× SGL-AR's.

References.

[1] Liu et al., "D-cut: Adaptive Verification Depth Pruning for Batched Speculative Decoding," arXiv:2607.14647, 2026.

[2] Cheng et al., "DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation," arXiv:2607.05147, 2026.

[3] Li et al., "AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding," EuroSys 2026.

[Q] Long contexts (8K to 32K tokens)

Context length enters Cadence in one place, the step-latency profile, and our measurements show how it enters. At batch 32, moving from 512 to 1,856 context tokens shifts step latency by a nearly constant amount at every verify width: 2.6 ms with no draft tokens, and 2.7 ms with 31 requests verifying seven extra tokens each. The latency jumps at kernel tile boundaries stay at the same verify width at every context. Context therefore shifts the profile rather than changing its shape, and indexing the profile by context, L(B,d,K,C)L(B, d, K, C) with CC the batch's context, captures it.

Three properties carry over to long contexts unchanged:

  1. The cap scales with context. Each request's cap is (1+ϵ)(1+\epsilon) times the time AR steps would take for its tokens, plus β\beta, and AR steps lengthen with context. A long-context request gets a proportionally larger delay budget, with no retuning of ϵ\epsilon or β\beta.
  2. The ledger charges measured time. Every step is charged at its measured latency, so any context effect the profile misses appears as overrun, which the bound tolerates up to ϵ\epsilon AR steps per step (Proposition A.2).
  3. Fill prices each request separately. With unequal contexts, Fill ranks draft tokens by benefit per unit of each request's own verify cost, read from its context bin.

Where long contexts land. A fixed KV pool holds 10× fewer requests at 8K tokens than at the roughly 800 tokens our profile was measured at, and 40× fewer at 32K. Long-context serving on one GPU therefore runs at small batch, where the externality is weak and Cadence drafts exactly as SGL-SD does. In the light-load phase of our trace (mean batch 8.8), both verify three draft tokens per request and reach a p99 TPOT of 14.7 ms, against 23.0 ms for SGL-AR, with no request slowed. Cadence's cap engages when a large batch makes verification expensive, which is the regime Section 4 targets. Where long contexts are served at large batch, the profile indexed by context prices that cost directly.

We will add a context-indexed profile at 8K, 16K and 32K tokens, with the corresponding trace results, to the final version.

Reviewer_pB2EOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Speculative decoding when done in batched mode only helps those generations which get more token accepted. Therefore, while one batch element might be producing 2-3 tokens per step the other might be lagging behind because of its acceptance rate while paying for the additional speculation time. Cadence proposes a method to control the amount of speculative decoding a batch should do so that faster requests don’t gain at the expense of slower ones.

The central idea is to give each batch element a delay budget called slack. A request that gets many accepted tokens earns slack while requests that accepts little spends slack because it waits through longer steps without making much progress. Additionally, the other constraint is that the slack of any batch element should be >0>0. The number of speculation steps is determined by the slowest budgeting relaxation threshold around it.

The only thing I am unable to understand is why would anyone want to reduce the throughout of some batch elements using this approach.

Critical strengths
  1. Good problem identification.
  2. The proposed algorithm to improve average throughput during batched decoding while using speculative decoding seems to be working better than native SGLang SD.
  3. Measurements under various loads, models and hardware constraints.
Critical weaknesses
  1. Comparisons with other drafting techniques like Mirros-SD, DSpark etc
  2. Slow-down of some prompts
Questions for authors

In distributed serving with batched inference the verifier model might be sharded. Verifying multiple tokens at once gives the experts a better chance to be distributed well decreasing the probability of scenarios where a single GPU have more expert load than others. Going back to single token verification might shake this balance. Any comments on that?

3 comments

We thank you for your thoughtful review and positive assessment. Our responses are as follows:

[Summary] Why would anyone reduce the throughput of some batch elements?

Cadence does not take throughput from any request. Speculation does, and Cadence limits it: 0.53% of requests end slower than AR decoding, against 31.4% under SGLang's default speculation (SGL-SD). Every request in a batch waits for the same decode step. Once a large batch has used up the GPU's spare compute, each verified draft token lengthens that step for everyone, but only the request whose token is accepted commits more. A request whose drafts are rejected waits through a longer step for the one token a plain autoregressive (AR) step would have given it. Section 2 measures this directly: on an H200 at batch 48 (deterministic-inference mode), when one request verified seven more draft tokens in a step, an unchanged co-batched request took a median 31.9 ms longer to commit its eight tokens.

Cadence drafts less deeply when the batch is large, so the step shortens for everyone (28 ms against 40 ms for SGL-SD at high load). Both slow and fast requests gain over SGL-SD: p99 TPOT falls from 50.4 to 29.8 ms for the lowest-acceptance quarter, within 1 ms of no speculation, and from 27.3 to 22.3 ms for the highest. Trace of Section 4.2 at high load:

PolicyThroughput vs SGL-ARRequests slower than ARWorst slowdownp99 TPOT, lowest-acceptance quarterp99 TPOT, highest-acceptance quarter
SGL-AR (no speculation)1.00×0%1.0029.1 ms29.1 ms
SGL-SD1.03×31.4%2.0850.4 ms27.3 ms
SGL-SD-C3201.14×14.7%1.7140.9 ms27.2 ms
Cadence1.30×0.53%1.2129.8 ms22.3 ms

The last column answers your question: the requests that accept the most also run faster under Cadence. Favoring them further buys almost nothing. Given Cadence's depth choice and a ranking by true acceptance, DSpark's rule ties Cadence's throughput, gains the highest-acceptance quarter 0.5 ms at p99 and costs the lowest-acceptance quarter 8.8 ms (38.6 against 29.8 ms). When an operator does want to favor a class, Cadence does it with a priority weight: with weight 4, 83.7% of a latency-sensitive class meets a 20 ms deadline, against 42.2% with equal weights, at 0.94× throughput.

[W1] Mirror-SD and DSpark Cadence decides how many draft tokens to verify for a given drafter, so it is orthogonal to how drafts are produced. Mirror-SD drafts on an NPU in parallel with GPU verification, which changes what a step costs, and Cadence would read that cost from its profile. DSpark [1] pairs a block drafter with a confidence-ranked verification scheduler. We compare Cadence against its scheduling rule on the same drafter and workloads (Table R1 in the general response). As published, it drafts the full depth every step and reaches 0.91× Cadence's throughput, with 13.4% of slow requests over the cap at ϵ=0.25\epsilon = 0.25 and 57.3% at ϵ=0.1\epsilon = 0.1. Given Cadence's depth choice it matches Cadence's throughput, but it has no per-request bound: ranked by true acceptance, it puts 40.4% of slow requests over the cap at ϵ=0.1\epsilon = 0.1. Cadence puts none over at either tolerance.

[Q] Sharded MoE verifiers and expert balance On our main trace Cadence never fell back to single-token verification: it ran no AR step at any load. It shortens drafts only when the batch is large, which is when routing already sees the most tokens. At a full batch of 112 it verifies 224 tokens per step (two per request), against 112 for AR decoding and 448 for SGL-SD, and at light load four per request, as SGL-SD does. Appendix D.4 adds that verified draft tokens stop reaching new experts once the batch itself reaches nearly every expert (B=95B = 95 on gpt-oss-120b, 382 on Llama-4-Maverick). Cadence also reads step cost from a profile taken on the deployment and charges every request the measured time of each step. Under expert parallelism, imbalance shows up in both: the profile steers Cadence to the budgets that cost least per committed token, and a slow step counts against every request's cap. Our multi-GPU MoE results use tensor parallelism (Appendix D.2 and D.4).

[W2] Slowdown of some prompts

Cadence bounds it. Each request carries a delay cap: it may take at most 25% longer than AR steps would take for its tokens, plus 40 ms (ϵ=0.25\epsilon = 0.25, β=40\beta = 40 ms). As long as no step overruns its profile by more than ϵ\epsilon AR steps, every capped request finishes within its cap up to one step's overrun, whatever its acceptance (Proposition A.2). At high load, only 0.53% of requests run slower than AR decoding, all within their cap, with a worst slowdown of 1.21 (SGL-SD: 31.4% and 2.08, with 2.25% over the cap). When 10% of requests rarely accept their drafts, none exceeds its cap, while SGL-SD puts 77% of that class over it.

References.

[1] Cheng et al., "DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation," arXiv:2607.05147, 2026.

Thank you for addressing my questions.

Comments on the paper

1 comment

We thank all reviewers for their thoughtful feedback.

The main requests were comparisons with adaptive speculative decoding (D-cut [1], DSpark [2], AdaServe [3]), ablations, absolute latency SLOs, CUDA graphs and overlap, and a revised introduction. We address each below and in the per-reviewer responses.

What Cadence adds. Every decode step commits at least one token per request. Cadence uses this to turn per-request delay caps, measured against AR steps, into one allowed step latency known before drafting, and chooses the draft depth and per-request verification lengths inside it. As long as no step overruns its profile by more than ϵ\epsilon AR steps, every capped request finishes within its cap up to one step's overrun, whatever its acceptance (Proposition A.2). Adaptive speculation methods tune speculation for throughput or deadlines. None of them bounds the delay that speculation imposes on co-batched requests.

[R1] Comparison with D-cut and DSpark. We compare Cadence against each method's verification-allocation rule on the same EAGLE-3 drafter, latency profile, workloads, seeds and metrics (no released DSpark drafter targets Qwen3-32B). Columns 2 to 4 are the trace of Section 4.2 at high load. The last two are the slow-class runs of Section 4.2 (Table 10), where 10% of requests rarely have their drafts accepted. Throughput and goodput are relative to Cadence.

PolicyThroughputGoodputRequests slower than ARSlow class over cap (ϵ=0.25\epsilon = 0.25)Slow class over cap (ϵ=0.1\epsilon = 0.1)
SGL-SD0.79×0.56×31.4%76.8%84.7%
D-cut rule, full depth 70.89×0.83×8.7%1.2%35.0%
DSpark rule, full depth 70.91×0.82×10.7%13.4%57.3%
D-cut rule + Cadence's depth choice1.00×1.00×0.53%0%0.03%
DSpark rule + Cadence's depth choice1.01×1.01×0.75%0%0%
... ranked by true acceptance1.00×0.97×3.5%0%40.4%
Cadence1.00×1.00×0.53%0%0%

Three results. (1) Both rules draft the full depth every step, which is nearly free on the block-parallel drafters they target but costs 5.5 ms per step on EAGLE-3 at batch 112. They lose 9 to 11% throughput and put up to 57% of slow requests over the ϵ=0.1\epsilon = 0.1 cap. (2) Adding Cadence's per-step depth choice closes the throughput gap (DSpark's fill then leads by 1%), so the depth choice, not the ranking, sets throughput. (3) Those rows stay inside the cap on this workload, but nothing in either rule keeps them there. When the ranking tracks each request's true acceptance, the same DSpark rule starves the same requests every step and puts 40.4% of the slow class over the ϵ=0.1\epsilon = 0.1 cap. Cadence keeps every slow request within its cap at both tolerances, with no confidence signal.

[R2] Comparison with AdaServe. AdaServe's rule fills each step deadline-first under a fixed verification budget. We run it at four budgets on classed traffic at saturation (25% tight requests with a 20 ms TPOT SLO, 75% best effort), on the same drafter, profile, seeds and metrics as R1. Throughput is relative to Cadence with equal weights.

PolicyTight requests meeting 20 msThroughputBest-effort requests slower than AR
AdaServe rule, budget 44819.9%0.79×31.3%
AdaServe rule, budget 32036.7%1.00×0.81%
AdaServe rule, budget 22441.3%1.00×0.40%
AdaServe rule, budget 16272.2%0.94×5.4%
Cadence, equal weights42.2%1.00×0.37%
Cadence, priority weight 483.7%0.94×19.7%

At full throughput, Cadence meets the deadline for more tight requests than any AdaServe budget (42.2% against at most 41.3%) and slows the fewest best-effort requests. AdaServe reaches 72.2% only by giving up 6% of throughput. At that same throughput, Cadence with a priority weight reaches 83.7%, moving delay onto best-effort requests, which carry no cap.

[R3] Ablations. The per-step depth choice drives throughput (fixing depth 3 costs 12.6%). The cap bounds delay when depth cannot adapt: with depth fixed at 3 and no cap, 64.7% of the slow class ends over its cap at ϵ=0.1\epsilon = 0.1, and with the cap, 0.12%. Details in the response to Wv1n.

[R4] Absolute TPOT SLOs. Cadence's goodput over SGL-SD is 2.50×, 1.65×, 1.30× and 1.27× at TPOT targets of 25, 30, 40 and 50 ms, and over the best static budget, SGL-SD-C320, 1.65×, 1.23×, 1.14× and 1.14×.

References.

[1] Liu et al., "D-cut: Adaptive Verification Depth Pruning for Batched Speculative Decoding," arXiv:2607.14647, 2026.

[2] Cheng et al., "DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation," arXiv:2607.05147, 2026.

[3] Li et al., "AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding," EuroSys 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.