- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CADENCE studies how speculative decoding can improve batch throughput while slowing individual requests with low acceptance. It tracks each request's cumulative delay relative to AR steps and chooses the batch-wide draft depth and per-request verification lengths within a latency allowance.
Critical strengths
-
I like the speculation externality framing. Everyone waits through the same step, but not everyone gets enough accepted tokens to make that wait worthwhile.
-
High-concurrency serving is a worthwhile setting to study. Looking at which requests get slower adds something that average throughput alone misses.
-
The controller works without retraining the drafter or changing the target model. That makes it easier to consider as a serving-side optimization.
Critical weaknesses
-
I found the introduction hard to follow. Too many concepts appear before the basic idea is clear. A simpler explanation of the problem and how the controller addresses it would help.
-
I am still unsure how variable-length verification is implemented. SGLang's standard fixed-shape speculative verification path cannot simply assign a different verification length to each request without additional support. This needs a ragged verification implementation, as in DSpark, together with a clear strategy for CUDA graph replay. The paper says CUDA graphs and overlap scheduling are enabled, but does not explain how unequal prefixes are packed, how graph shapes are selected, or how KV-cache updates are handled. For a method built around per-request verification lengths, this is a central systems detail, and I would like to see it explained rather than left implicit.
-
DSpark, D-cut, and ECHO are cited but not compared experimentally. Beating default speculation and fixed budgets leaves me unsure how much CADENCE adds over existing adaptive methods. A comparison under the same serving setup would be more convincing.
Questions for authors
Same as the weaknesses above.
4 comments
[W3] DSpark, D-cut and ECHO
We compare Cadence against the D-cut [1] and DSpark [2] allocation rules on the same drafter, latency profile, workloads, seeds and metrics (Table R1 in the general response). Both were designed for drafters that emit a whole block in about one pass. On EAGLE-3 at batch 112, their full depth 7 costs 5.5 ms more per step than depth 1, so they reach 0.89× and 0.91× Cadence's throughput and put up to 57.3% of slow requests over the cap. Given Cadence's per-step depth choice they match its throughput, which isolates the allocation rule. Neither rule reads delay: ranked by true acceptance, DSpark's rule puts 40.4% of slow requests over the cap, while Cadence puts none over at either tolerance. For CUDA graphs, D-cut limits to four fractions of the full draft, the role Cadence's tiers play above.
ECHO [3] treats the batch as one draft tree under a fixed global verification budget set near hardware saturation and, at high load, moves budget from low-confidence to high-confidence requests, as the DSpark rows do, reading no per-request delay. The closest policies in our evaluation are SGL-SD-C320, the best of nine fixed budgets (1.14× SGL-AR's throughput with 14.7% of requests slower than AR, against 1.30× and 0.53% for Cadence), and the DSpark rows of Table R1. ECHO's tree shaping is complementary: Cadence's bound needs only that each step fit the allowed step latency, whatever fills it (Lemma A.1), so ECHO's trees can fill Cadence's budget.
References.
[1] Liu et al., "D-cut: Adaptive Verification Depth Pruning for Batched Speculative Decoding," arXiv:2607.14647, 2026.
[2] Cheng et al., "DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation," arXiv:2607.05147, 2026.
[3] Hu et al., "ECHO: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarios," arXiv:2604.09603, 2026.
Thanks. I will raise my score.
Thank you for raising your score, and for the detailed comments that improved the paper.
We thank you for your detailed review. Our responses are as follows:
[W1] Introduction
We agree, and in the final version we will restructure Section 1 around one idea: every decode step commits at least one token per request, so per-request delay caps reduce to one step latency known before drafting. Section 1 will state the problem (one request's speculation lengthens every co-batched request's step), then this observation, then Cadence's three stages tied to Figure 2, and then the contributions, moving the remaining concepts to Sections 2 and 3.
[W2] Variable-length verification under CUDA graphs and overlap
Packing. Each request r verifies its first 1+ℓr tokens. Cadence packs them into one FlashAttention-3 varlen launch of B+∑rℓr positions and rewrites
cu_seqlens_q, cache lengths, page table and KV slots to match. Acceptance is a masked chain walk. With deterministic inference on, outputs are identical across policies in 656 comparisons.Graph shapes. Cadence runs on SGLang's ragged target-verify path (compact mode, the path SGLang uses for DSpark [2]), extended to EAGLE-3 chains:
KV cache. SGLang's Spec v2 already reserves each request's committed length plus twice the largest draft length, so allocation never depends on K or ℓ. The next step's writes overwrite rejected slots, and nothing is reclaimed.
Overlap. SGLang launches step t+1 before it reads step t's acceptance. Cadence's plan needs only slack, the shared acceptance curve and K, so it plans step t+1 by charging step t as one committed token per request at its profiled latency, and accepted tokens earn their credit one step later. The bound then holds with two steps' overrun instead of one, under Proposition A.2's condition shifted by one step. Planning one step late costs at most 0.002% goodput per seed. Under tripled step-time noise, the tiers and the one-step-late plan put no request over the cap at ϵ=0.25, and one request in ten seeds at ϵ=0.1 (two without them), at most 19.2 ms past its cap (14.1 ms without them).