- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
Summary. The paper studies speculative decoding with block-parallel drafters (mainly DFlash) in batched serving. At high concurrency, verification becomes compute-bound. Every request still verifies a fixed depth of positions, and about 60% of those draft tokens get rejected. As a result, long-draft DFlash can end up slower than plain autoregressive (AR) decoding; for example, Qwen3-8B on H20 at bs=64 (Fig. 1a). The authors propose D-cut, a training-free scheduler that runs between drafting and verification. It decides how deep to verify each request.
Method. (1) Cross-request pruning. Each draft position gets a score equal to the prefix product of drafter confidences, . This estimates the expected marginal tokens gained from verifying that position. All positions in the batch are then ranked together and the global top- are kept. Because does not increase with , each request keeps a contiguous prefix, so only suffixes are pruned. (2) Runtime-adaptive budget. is limited to a few ratio buckets of . This keeps CUDA-graph shapes fixed. At startup, a cost table of end-to-end step latency is profiled. Each step then picks , which approximates maximizing speedup over AR. The selector costs about 0.56 ms per step (2–3% of a step). Appendix B discusses lossless behavior under rejection sampling, which requires a shifted confidence score plus causal or history-based budget selection.
Contributions. (i) Identifying the "verification cost explosion" of long block-parallel drafts under batched serving. (ii) A simple confidence × profiled-cost scheduler that prunes across requests, needs no change to the target model, and preserves the target distribution in the evaluated setting (deterministic drafts, target-only verification). (iii) An empirical study across dense and MoE models.
Validation. Experiments are built on vLLM with 8×H20, on six targets: Llama-3.1-8B, Qwen3-4B/8B, Qwen3.5-27B, Qwen3.5-35B-A3B, and Hy3-295B-A21B. Benchmarks are GSM8K, Math500, HumanEval, MBPP, and MT-Bench, at concurrency 4–64. Baselines are DFlash with block sizes 8 and 16, plus EAGLE-3 or MTP and AR.
- At concurrency 32, D-cut (16) beats DFlash (16) in 29 of 30 model–dataset pairs. Average speedup over AR rises from 1.26× to 1.65×, and the dense-model configs where DFlash is slower than AR become faster than AR again. With block size 8, the average goes from 1.56× to 1.67×.
- The largest MoE gains are on Hy3: up to about 3.0× over AR. On Qwen3.5-35B-A3B, D-cut and DFlash are close.
- A budget ablation on Qwen3-8B (H20 vs. H800) shows the best fixed ratio changes with hardware and dataset. The adaptive version matches or beats the best fixed ratio. Averaged over concurrency 16–64, it is 1.58× faster than DFlash on H20 and 1.25× on H800.
- With target temperature 1, DFlash drops to 0.68× of AR, while D-cut stays at 1.15×.
- Draft confidence predicts acceptance well (AUROC 0.96–0.97).
The authors state that the experiments use piecewise CUDA graphs only, without full-graph capture or CPU–GPU overlap, and argue the reported throughput is therefore conservative.
Critical strengths
1. It solves a problem that block-parallel drafting must address to be practical at scale. Speculative decoding is splitting into two lines: sequential drafters (EAGLE, MTP) and the more recent block-parallel drafters (DFlash, DSpark), which propose a fixed block of 4/8/16 tokens per forward pass. Block-parallel drafters give the longest acceptance lengths at small batch sizes. However, fixed-size blocks make the target verify positions per step. Once verification becomes compute-bound, most of that work is spent on tokens that will be rejected. The paper makes this concrete. At concurrency 32, DFlash (16) is slower than AR on average on all four dense targets (0.67×, 0.98×, 0.80×, 0.94×). D-cut (16) turns these into clear speedups (1.12×, 1.35×, 1.21×, 1.25×), with a similar gain on the 295B MoE target (1.83× → 2.48×). Why it matters: without this kind of verification-depth control, block-parallel drafting loses its advantage in exactly the high-concurrency regime that matters for real serving. This paper gives a simple and effective answer, and this is the main reason I support acceptance.
2. It is training-free and drop-in. D-cut reuses the confidences the existing drafter already outputs. It needs no additional training and no change to the target or the drafter, and it keeps the target output distribution in the evaluated setting. By contrast, the concurrent DSpark tackles a similar cross-request problem but relies on a trained confidence head. The only setup step is a one-time cost-table profiling at server startup, which also lets the same method adapt to different hardware. The budget ablation shows the best fixed pruning ratio shifts between H20 and H800, while the adaptive rule stays at or near the best without manual tuning. The runtime overhead is small (about 0.56 ms per step), and no new CUDA-graph shapes are needed. Why it matters: training drafters or confidence heads for large targets is costly and must be redone per model. D-cut works directly on the released DFlash checkpoints, which the paper uses for all six targets. This makes it cheap to adopt and easy to reproduce.
Critical weaknesses
1. The benefit of cross-request allocation over simpler pruning strategies is not shown. D-cut's central idea is to rank draft positions globally across the batch, rather than pruning each request on its own (motivated by Fig. 2a). However, the experiments compare D-cut only with unpruned DFlash, and the fixed-ratio variants in Table 2 also use global top-. It therefore remains unclear how much of the gain is specific to global allocation. A comparison at an equal budget would answer this: (a) uniform truncation of every request to depth , (b) per-request confidence-threshold truncation, with the threshold set to match the budget, and (c) D-cut's global top-. Why it matters: cross-request allocation is the paper's central algorithmic idea, so evidence against these simpler alternatives is needed to fully support the main claim.
2. Per-request variable verification lengths make real-system integration hard. Under cross-request scheduling, each request verifies a different number of tokens, and this number changes every step, while current engines rely on a fixed per-request shape. Accordingly, full CUDA-graph capture, CPU–GPU overlap (Spec-V2), and asynchronous scheduling are all disabled in the reported runs (Appendices C–D). The proposed fix for overlap predicts the budget from previous steps' confidence, which removes the very signal D-cut relies on. Why it matters: in production, DFlash would run with these optimizations while D-cut could not, or only with stale confidences. The reported gains may therefore be optimistic rather than "conservative" as claimed, and the paper should discuss this trade-off concretely.
Questions for authors
-
Equal-budget ablation of cross-request allocation. At the same budget , how does D-cut's global top- compare in MAT and throughput with (a) uniform truncation of every request to depth , and (b) per-request confidence-threshold truncation with the threshold matched to the budget? This is the main factor in my assessment (Weakness 1).
-
Implementation in production engines. Verification lengths differ across requests and change every step. This conflicts with optimizations that assume a fixed per-request shape, such as full CUDA-graph capture and CPU–GPU overlap (Spec-V2). Appendix C sketches possible solutions. However, even with a history-based , the per-request depths still depend on the current step's confidences, so the next step's metadata cannot be fully prepared in advance. Could the authors explain how they expect D-cut to be integrated into an engine with these optimizations enabled, and what they see as the main implementation difficulties?
-
Cost table and context length. The table is profiled once at startup with dummy inputs. At what context length is it profiled? Verification attention cost grows with KV length, so does the best change for long-context requests, and is the table ever refreshed?
If the authors can address the questions and concerns above, I would be willing to raise my score.
4 comments
Response to Reviewer XZJJ (Part 2 / 2)
Q2: "Implementation in production engines. Verification lengths differ across requests and change every step. This conflicts with optimizations that assume a fixed per-request shape, such as full CUDA-graph capture and CPU–GPU overlap (Spec-V2). ... Could the authors explain how they expect D-cut to be integrated into an engine with these optimizations enabled, and what they see as the main implementation difficulties?"
A2: Integrating D-cut with these optimizations relies on separating its decision into two stages. Total budget (CPU, before drafting) determines the CUDA-graph shape and is fixed on the CPU in advance. Cross-request allocation of (GPU, after drafting) waits until the current step's draft is generated, ranks the current step's confidences on the GPU to allocate across requests, and directly populates the attention metadata before the captured target graph launches.
vLLM Model Runner V2's adaptive verification (AV) follows this exact decoupled structure under full CUDA-graph capture and asynchronous scheduling with CPU–GPU overlap. It also largely handles the engine-side work this requires: building variable-length attention metadata on the GPU, reclaiming the KV slots of pruned positions, and supporting variable-length verification in the rejection sampler. With both optimizations enabled across all configurations, AV achieves 1.07–1.89× the throughput of DFlash-16 (0.99–1.18× vs. DFlash-8) on H800, and 1.41–2.20× the throughput of DFlash-16 (1.01–1.30× vs. DFlash-8) on H20. AV differs from D-cut in confidence calibration and budget granularity, so these results serve to demonstrate that this class of design runs effectively on a production engine path.
The main remaining difficulty is the one you raised: under CPU–GPU overlap, can only use historical information, and the impact of this lag under real traffic requires further evaluation. One concrete way to shorten it is to estimate at an intermediate drafter layer. Allocation needs final-layer, per-position confidences, but is a single batch-level number and should be predictable earlier. After layer , the GPU estimates from intermediate confidences (e.g., by reusing the drafter's LM head) and copies this one integer to the CPU while the remaining layers run, so the target graph is prepared for the current step's .
Q3: "Cost table and context length. The table is profiled once at startup with dummy inputs. At what context length is it profiled? Verification attention cost grows with KV length, so does the best change for long-context requests, and is the table ever refreshed?"
A3: The cost table is profiled at a KV length of 2,048, with 3 warmup steps and the mean of 5 measurements per cell. Consistent with your intuition, the best does change with context length: a 32k-profiled table agrees with the 2k table on only 69.6% of bucket selections, shifting the optimal budget upward and raising the share of the 50% bucket from 42% to 70%.
The direct remedy is a length-bucketed cost table: profile at several KV lengths (e.g., 2k / 8k / 16k / 32k) and, at each step, look up the table matching the batch's average KV length. This only adds startup profiling time; the per-step cost remains a table lookup. Our length-matched runs already correspond to this setting: on 8k / 16k-prefix workloads, they change end-to-end throughput (including prefill) by −0.6% to +0.5% relative to the 2k table, so the default table is adequate at these lengths, while bucketing guards against larger mismatches at longer contexts. For drift that is not tied to length (e.g., contention or throttling), the table can also be refreshed online: step latency is already measured every step, so the selected bucket's entry can be updated via an exponential moving average (EMA), unselected buckets extrapolated via with occasional probing, and the refresh interval shortened whenever measured latency persistently deviates from the table. We will add both designs to the revised manuscript.
In light of these new baselines, production-path evaluations, and sensitivity analyses, we kindly invite you to reconsider your score.
We hope the response has addressed your concerns. If you have any additional concerns or comments that we may have missed, we would be most grateful for any further feedback to help us further enhance our work.
Thank you for the detailed response and the new experiments. They address my main concerns, and I have raised my score from 3 to 4.
Q1 (equal-budget ablation). This was the main factor in my assessment, and the new results resolve it. At the same budget, D-cut is about 10–13% faster than uniform truncation (R3-Table1), and it matches or outperforms a per-request threshold tuned on the same data. The margin over the tuned threshold is smaller (about 1–7% in R3-Table1). This suggests that most of the gain comes from confidence-based, variable-depth allocation, which both methods share. D-cut's further advantage is that its budget comes from the runtime cost table, so no threshold needs to be tuned for each deployment. I suggest stating this clearly in the revision. Please also specify the target model used in R3-Table1, and report MAT and the spread over the repeated measurement windows for all three policies. If the target is Qwen3-8B, the D-cut entries are 5–8% higher than the corresponding fixed-ratio entries in Table 2 (e.g., 3217 vs. 3014 for GSM8K at 25%), and a short explanation would help.
Q2 (production engines). The two-stage design is convincing. The AV results on vLLM Model Runner V2 show that this class of method keeps clear gains with full CUDA-graph capture and asynchronous scheduling. Since D-cut itself has not yet been measured on that path, I suggest revising the claim in Appendix C that the reported throughput is a conservative estimate. Measuring D-cut with the budget chosen from the previous step's confidences, in the current setup, would directly show the cost of this lag. A brief discussion of how D-cut differs from AV would also be useful.
Q3 (cost table). Thank you for the profiling details and the context-length analysis. At 32k, where the bucket selections differ most, it would help to report decode-only throughput with the 2k table and with a length-matched table, because end-to-end throughput that includes prefill can hide differences in the decode phase.
These suggestions are for the revised manuscript and do not affect my updated score.
Thank You and Follow-up Response to Reviewer XZJJ
Dear Reviewer XZJJ,
We sincerely thank you for reading our response so carefully, for raising your score, and for offering such thoughtful suggestions. We truly appreciate your time and constructive engagement throughout the review process.
To briefly clarify your question on R3-Table1: the target model is Qwen3-8B with DFlash-b16 (on H20, c=32). The 5%–8% higher absolute throughput compared to Table 2 is because all H20 rebuttal runs were re-measured together on a different H20 node with an updated software stack, which shifted throughput upward uniformly across all methods while keeping relative speedups unchanged.
Your remaining suggestions for the revised manuscript are very helpful and well taken, and we will gladly incorporate them in our revision. Thank you once again for your generous support and valuable feedback!
Best regards, Authors of D-cut
Response to Reviewer XZJJ (Part 1 / 2)
We thank Reviewer XZJJ for the thorough and constructive review, and are encouraged that you see D-cut as solving "a problem that block-parallel drafting must address to be practical at scale" with a "training-free and drop-in" design. In response to your three questions, we added an equal-budget ablation (Q1), an evaluation on a production engine path with full CUDA-graph capture and asynchronous scheduling (Q2), and a cost-table sensitivity analysis across context lengths (Q3).
Q1: "Equal-budget ablation of cross-request allocation. At the same budget Kρ(B), how does D-cut's global top-K compare in MAT and throughput with (a) uniform truncation of every request to depth ≈ρ(D+1), and (b) per-request confidence-threshold truncation with the threshold matched to the budget? This is the main factor in my assessment (Weakness 1)."
A1: Starting from the same 16-token draft block, we compare three ways of allocating verification positions across the batch:
R3-Table1: Equal-budget throughput comparison (H20, c=32, tokens/s)
At the same verification budget, global top-K outperforms both per-request baselines. Across c∈{16,32,64} on H20 and H800, D-cut improves throughput over uniform truncation by +9.8% to +13.1% at a 25% budget and by +3.1% to +14.4% at a 50% budget, holding steady across three repeated measurement windows at c=32. The gain is smallest on MT-Bench at 50%, where accepted prefixes are short and uniform truncation already reaches a MAT of 3.09—close to unpruned DFlash's 3.23—leaving limited room for cross-request reallocation. Against the closest-budget fixed threshold on H20 (c∈{16,32,64}), D-cut matches or exceeds throughput in every setting (−0.1% to +8.0%; MT-Bench has no 50%-budget entry because even the lowest threshold in our sweep realizes at most a 31% keep ratio).
R3-Table1compares allocation policies under a fixed, pre-specified budget. In deployment, however, the budget itself must be chosen, and the optimal choice shifts with hardware and load. A fixed τ cannot track this shift because its keep ratio depends only on drafter confidence and dataset statistics.R3-Table2illustrates how the optimal budget moves with concurrency on H800 MT-Bench.R3-Table2: The best budget shifts with load (from Table 2; H800, MT-Bench, tokens/s)
The optimal fixed budget reverses as load grows—0.5 is 5.2% faster than 0.25 at c=16 but 15.2% slower at c=64—while D-cut (auto) tracks this shift without tuning, staying within 2.6%–4.4% of the best fixed budget in all three columns. In our rebuttal measurements, the average bucket selected by D-cut (auto) on H800 MT-Bench moves from about 49% at c=16 to about 31% at c=32 and 25% at c=64, avoiding the up to 15% throughput loss of a mismatched fixed ratio.
On the weaker H20, where verification is the bottleneck, the optimal budget does not shift: in Table 2, D-cut (auto) slightly edges out D-cut (0.25) across all three concurrencies (2169 vs. 2152 at c=16, 2449 vs. 2377 at c=32, and 2600 vs. 2564 at c=64), and in our rebuttal measurements D-cut (auto) keeps its average budget at 25.5% / 25.0% / 25.0% across c=16/32/64. Meanwhile, a fixed τ=0.2 retains 23.1% / 23.2% / 23.2% of positions at all three concurrencies; it performs close to D-cut (auto) on H20 because that keep ratio happens to sit near H20's best budget—and we chose this τ by sweeping on the same data. A fixed τ therefore has to be re-tuned for each hardware and load regime, whereas D-cut derives the budget from runtime cost.
We will add both baselines, the full τ sweep, and the throughput-vs.-budget curves to the revised manuscript.