acceptodds
Under review as a conference paper at ICLR 2027

D-cut: Adaptive Verification Depth Pruning for Batched Speculative Decoding

19 views

Abstract

Speculative decoding has become a leading approach for accelerating large language model (LLM) inference without compromising output quality. Recent methods further improve single-request speedup by decoupling draft length from drafting latency via parallel generation, which produces longer drafts to achieve higher mean accepted tokens (MAT). However, as request concurrency grows, these long drafts waste more compute on rejected tokens, inflating the *verification cost* and making speculative decoding suboptimal in high-concurrency scenarios. To address this challenge, we propose D-cut, an adaptive pruning strategy that selects draft tokens across the entire batch, concentrating verification budget on tokens most likely to be accepted. D-cut is motivated by two observations: (1) acceptance length varies significantly across concurrent requests, so D-cut performs *cross-request pruning* that adaptively allocates verification budget based on draft confidence; (2) verification cost largely depends on the runtime environment (GPU, parallelism, etc.), so we incorporate a *runtime cost model* into the selection, allowing D-cut to adapt its pruning depth to the actual deployment. Experiments across dense and MoE models show that D-cut improves the average speedup from 1.26× to 1.65× under high concurrency, restores acceleration for dense-model configurations where long-draft baselines fall below autoregressive decoding, and achieves up to 3.0× speedup over autoregressive decoding on MoE models.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Speculative Decoding, LLM inference, large language models

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_XZJJOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

Summary. The paper studies speculative decoding with block-parallel drafters (mainly DFlash) in batched serving. At high concurrency, verification becomes compute-bound. Every request still verifies a fixed depth of D+1D+1 positions, and about 60% of those draft tokens get rejected. As a result, long-draft DFlash can end up slower than plain autoregressive (AR) decoding; for example, Qwen3-8B on H20 at bs=64 (Fig. 1a). The authors propose D-cut, a training-free scheduler that runs between drafting and verification. It decides how deep to verify each request.

Method. (1) Cross-request pruning. Each draft position (i,k)(i,k) gets a score equal to the prefix product of drafter confidences, si,k=∏t≤kci,ts_{i,k}=\prod_{t\le k}c_{i,t}. This estimates the expected marginal tokens gained from verifying that position. All B(D+1)B(D+1) positions in the batch are then ranked together and the global top-KK are kept. Because si,ks_{i,k} does not increase with kk, each request keeps a contiguous prefix, so only suffixes are pruned. (2) Runtime-adaptive budget. KK is limited to a few ratio buckets ρ∈{0.25,0.5,0.75,1.0}\rho\in\{0.25,0.5,0.75,1.0\} of B(D+1)B(D+1). This keeps CUDA-graph shapes fixed. At startup, a cost table C(B,ρ)C(B,\rho) of end-to-end step latency is profiled. Each step then picks ρ⋆=arg⁡max⁡ρ∑SKρsi,k/C(B,ρ)\rho^\star=\arg\max_\rho \sum_{S_{K_\rho}} s_{i,k}/C(B,\rho), which approximates maximizing speedup over AR. The selector costs about 0.56 ms per step (2–3% of a step). Appendix B discusses lossless behavior under rejection sampling, which requires a shifted confidence score plus causal or history-based budget selection.

Contributions. (i) Identifying the "verification cost explosion" of long block-parallel drafts under batched serving. (ii) A simple confidence × profiled-cost scheduler that prunes across requests, needs no change to the target model, and preserves the target distribution in the evaluated setting (deterministic drafts, target-only verification). (iii) An empirical study across dense and MoE models.

Validation. Experiments are built on vLLM with 8×H20, on six targets: Llama-3.1-8B, Qwen3-4B/8B, Qwen3.5-27B, Qwen3.5-35B-A3B, and Hy3-295B-A21B. Benchmarks are GSM8K, Math500, HumanEval, MBPP, and MT-Bench, at concurrency 4–64. Baselines are DFlash with block sizes 8 and 16, plus EAGLE-3 or MTP and AR.

  • At concurrency 32, D-cut (16) beats DFlash (16) in 29 of 30 model–dataset pairs. Average speedup over AR rises from 1.26× to 1.65×, and the dense-model configs where DFlash is slower than AR become faster than AR again. With block size 8, the average goes from 1.56× to 1.67×.
  • The largest MoE gains are on Hy3: up to about 3.0× over AR. On Qwen3.5-35B-A3B, D-cut and DFlash are close.
  • A budget ablation on Qwen3-8B (H20 vs. H800) shows the best fixed ratio changes with hardware and dataset. The adaptive version matches or beats the best fixed ratio. Averaged over concurrency 16–64, it is 1.58× faster than DFlash on H20 and 1.25× on H800.
  • With target temperature 1, DFlash drops to 0.68× of AR, while D-cut stays at 1.15×.
  • Draft confidence predicts acceptance well (AUROC 0.96–0.97).

The authors state that the experiments use piecewise CUDA graphs only, without full-graph capture or CPU–GPU overlap, and argue the reported throughput is therefore conservative.

Critical strengths

1. It solves a problem that block-parallel drafting must address to be practical at scale. Speculative decoding is splitting into two lines: sequential drafters (EAGLE, MTP) and the more recent block-parallel drafters (DFlash, DSpark), which propose a fixed block of 4/8/16 tokens per forward pass. Block-parallel drafters give the longest acceptance lengths at small batch sizes. However, fixed-size blocks make the target verify B×(D+1)B\times(D+1) positions per step. Once verification becomes compute-bound, most of that work is spent on tokens that will be rejected. The paper makes this concrete. At concurrency 32, DFlash (16) is slower than AR on average on all four dense targets (0.67×, 0.98×, 0.80×, 0.94×). D-cut (16) turns these into clear speedups (1.12×, 1.35×, 1.21×, 1.25×), with a similar gain on the 295B MoE target (1.83× → 2.48×). Why it matters: without this kind of verification-depth control, block-parallel drafting loses its advantage in exactly the high-concurrency regime that matters for real serving. This paper gives a simple and effective answer, and this is the main reason I support acceptance.

2. It is training-free and drop-in. D-cut reuses the confidences the existing drafter already outputs. It needs no additional training and no change to the target or the drafter, and it keeps the target output distribution in the evaluated setting. By contrast, the concurrent DSpark tackles a similar cross-request problem but relies on a trained confidence head. The only setup step is a one-time cost-table profiling at server startup, which also lets the same method adapt to different hardware. The budget ablation shows the best fixed pruning ratio shifts between H20 and H800, while the adaptive rule stays at or near the best without manual tuning. The runtime overhead is small (about 0.56 ms per step), and no new CUDA-graph shapes are needed. Why it matters: training drafters or confidence heads for large targets is costly and must be redone per model. D-cut works directly on the released DFlash checkpoints, which the paper uses for all six targets. This makes it cheap to adopt and easy to reproduce.

Critical weaknesses

1. The benefit of cross-request allocation over simpler pruning strategies is not shown. D-cut's central idea is to rank draft positions globally across the batch, rather than pruning each request on its own (motivated by Fig. 2a). However, the experiments compare D-cut only with unpruned DFlash, and the fixed-ratio variants in Table 2 also use global top-KK. It therefore remains unclear how much of the gain is specific to global allocation. A comparison at an equal budget Kρ(B)K_\rho(B) would answer this: (a) uniform truncation of every request to depth ≈ρ(D+1)\approx\rho(D+1), (b) per-request confidence-threshold truncation, with the threshold set to match the budget, and (c) D-cut's global top-KK. Why it matters: cross-request allocation is the paper's central algorithmic idea, so evidence against these simpler alternatives is needed to fully support the main claim.

2. Per-request variable verification lengths make real-system integration hard. Under cross-request scheduling, each request verifies a different number of tokens, and this number changes every step, while current engines rely on a fixed per-request shape. Accordingly, full CUDA-graph capture, CPU–GPU overlap (Spec-V2), and asynchronous scheduling are all disabled in the reported runs (Appendices C–D). The proposed fix for overlap predicts the budget from previous steps' confidence, which removes the very signal D-cut relies on. Why it matters: in production, DFlash would run with these optimizations while D-cut could not, or only with stale confidences. The reported gains may therefore be optimistic rather than "conservative" as claimed, and the paper should discuss this trade-off concretely.

Questions for authors
  1. Equal-budget ablation of cross-request allocation. At the same budget Kρ(B)K_\rho(B), how does D-cut's global top-KK compare in MAT and throughput with (a) uniform truncation of every request to depth ≈ρ(D+1)\approx\rho(D+1), and (b) per-request confidence-threshold truncation with the threshold matched to the budget? This is the main factor in my assessment (Weakness 1).

  2. Implementation in production engines. Verification lengths differ across requests and change every step. This conflicts with optimizations that assume a fixed per-request shape, such as full CUDA-graph capture and CPU–GPU overlap (Spec-V2). Appendix C sketches possible solutions. However, even with a history-based KK, the per-request depths nin_i still depend on the current step's confidences, so the next step's metadata cannot be fully prepared in advance. Could the authors explain how they expect D-cut to be integrated into an engine with these optimizations enabled, and what they see as the main implementation difficulties?

  3. Cost table and context length. The table C(B,ρ)C(B,\rho) is profiled once at startup with dummy inputs. At what context length is it profiled? Verification attention cost grows with KV length, so does the best ρ\rho change for long-context requests, and is the table ever refreshed?

If the authors can address the questions and concerns above, I would be willing to raise my score.

4 comments
AuthorsOpenReview ↗

Response to Reviewer XZJJ (Part 1 / 2)

We thank Reviewer XZJJ for the thorough and constructive review, and are encouraged that you see D-cut as solving "a problem that block-parallel drafting must address to be practical at scale" with a "training-free and drop-in" design. In response to your three questions, we added an equal-budget ablation (Q1), an evaluation on a production engine path with full CUDA-graph capture and asynchronous scheduling (Q2), and a cost-table sensitivity analysis across context lengths (Q3).


Q1: "Equal-budget ablation of cross-request allocation. At the same budget Kρ(B)K_\rho(B), how does D-cut's global top-KK compare in MAT and throughput with (a) uniform truncation of every request to depth ≈ρ(D+1)\approx\rho(D+1), and (b) per-request confidence-threshold truncation with the threshold matched to the budget? This is the main factor in my assessment (Weakness 1)."

A1: Starting from the same 16-token draft block, we compare three ways of allocating verification positions across the batch:

  • (a) Uniform truncation: every request verifies its first ρ(D+1)\rho(D{+}1) positions.
  • (b) Per-request confidence-threshold truncation (fixed τ\tau): each request retains positions with si,k≥τs_{i,k} \ge \tau. We sweep τ∈{0.8,0.6,0.4,0.2,0.1,0.05}\tau \in \lbrace 0.8, 0.6, 0.4, 0.2, 0.1, 0.05 \rbrace and pair each target budget (25% or 50%) with the τ\tau whose realized keep ratio is closest.
  • (c) D-cut global top-KK.

R3-Table1: Equal-budget throughput comparison (H20, c=32c=32, tokens/s)

DatasetBudgetUniformFixed Threshold (τ\tau, realized budget)D-cut
GSM8K25%28923092 (τ=0.8\tau=0.8, 23.5%)3217
GSM8K50%30083308 (τ=0.1\tau=0.1, 51.5%)3367
HumanEval25%28402936 (τ=0.8\tau=0.8, 22.4%)3132
HumanEval50%29533312 (τ=0.1\tau=0.1, 49.9%)3347
MT-Bench25%23112542 (τ=0.2\tau=0.2, 23.2%)2569

At the same verification budget, global top-KK outperforms both per-request baselines. Across c∈{16,32,64}c \in \lbrace 16, 32, 64 \rbrace on H20 and H800, D-cut improves throughput over uniform truncation by +9.8% to +13.1% at a 25% budget and by +3.1% to +14.4% at a 50% budget, holding steady across three repeated measurement windows at c=32c=32. The gain is smallest on MT-Bench at 50%, where accepted prefixes are short and uniform truncation already reaches a MAT of 3.09—close to unpruned DFlash's 3.23—leaving limited room for cross-request reallocation. Against the closest-budget fixed threshold on H20 (c∈{16,32,64}c \in \lbrace 16, 32, 64 \rbrace), D-cut matches or exceeds throughput in every setting (−0.1% to +8.0%; MT-Bench has no 50%-budget entry because even the lowest threshold in our sweep realizes at most a 31% keep ratio).

R3-Table1 compares allocation policies under a fixed, pre-specified budget. In deployment, however, the budget itself must be chosen, and the optimal choice shifts with hardware and load. A fixed τ\tau cannot track this shift because its keep ratio depends only on drafter confidence and dataset statistics. R3-Table2 illustrates how the optimal budget moves with concurrency on H800 MT-Bench.

R3-Table2: The best budget shifts with load (from Table 2; H800, MT-Bench, tokens/s)

ccD-cut (0.25)D-cut (0.5)D-cut (auto)
16338735623469
32549551855253
64712660416894

The optimal fixed budget reverses as load grows—0.5 is 5.2% faster than 0.25 at c=16c=16 but 15.2% slower at c=64c=64—while D-cut (auto) tracks this shift without tuning, staying within 2.6%–4.4% of the best fixed budget in all three columns. In our rebuttal measurements, the average bucket selected by D-cut (auto) on H800 MT-Bench moves from about 49% at c=16c=16 to about 31% at c=32c=32 and 25% at c=64c=64, avoiding the up to 15% throughput loss of a mismatched fixed ratio.

On the weaker H20, where verification is the bottleneck, the optimal budget does not shift: in Table 2, D-cut (auto) slightly edges out D-cut (0.25) across all three concurrencies (2169 vs. 2152 at c=16c=16, 2449 vs. 2377 at c=32c=32, and 2600 vs. 2564 at c=64c=64), and in our rebuttal measurements D-cut (auto) keeps its average budget at 25.5% / 25.0% / 25.0% across c=16/32/64c = 16 / 32 / 64. Meanwhile, a fixed τ=0.2\tau=0.2 retains 23.1% / 23.2% / 23.2% of positions at all three concurrencies; it performs close to D-cut (auto) on H20 because that keep ratio happens to sit near H20's best budget—and we chose this τ\tau by sweeping on the same data. A fixed τ\tau therefore has to be re-tuned for each hardware and load regime, whereas D-cut derives the budget from runtime cost.

We will add both baselines, the full τ\tau sweep, and the throughput-vs.-budget curves to the revised manuscript.


AuthorsOpenReview ↗

Response to Reviewer XZJJ (Part 2 / 2)

Q2: "Implementation in production engines. Verification lengths differ across requests and change every step. This conflicts with optimizations that assume a fixed per-request shape, such as full CUDA-graph capture and CPU–GPU overlap (Spec-V2). ... Could the authors explain how they expect D-cut to be integrated into an engine with these optimizations enabled, and what they see as the main implementation difficulties?"

A2: Integrating D-cut with these optimizations relies on separating its decision into two stages. Total budget KK (CPU, before drafting) determines the CUDA-graph shape and is fixed on the CPU in advance. Cross-request allocation of KK (GPU, after drafting) waits until the current step's draft is generated, ranks the current step's confidences on the GPU to allocate KK across requests, and directly populates the attention metadata before the captured target graph launches.

vLLM Model Runner V2's adaptive verification (AV) follows this exact decoupled structure under full CUDA-graph capture and asynchronous scheduling with CPU–GPU overlap. It also largely handles the engine-side work this requires: building variable-length attention metadata on the GPU, reclaiming the KV slots of pruned positions, and supporting variable-length verification in the rejection sampler. With both optimizations enabled across all configurations, AV achieves 1.07–1.89× the throughput of DFlash-16 (0.99–1.18× vs. DFlash-8) on H800, and 1.41–2.20× the throughput of DFlash-16 (1.01–1.30× vs. DFlash-8) on H20. AV differs from D-cut in confidence calibration and budget granularity, so these results serve to demonstrate that this class of design runs effectively on a production engine path.

The main remaining difficulty is the one you raised: under CPU–GPU overlap, KK can only use historical information, and the impact of this lag under real traffic requires further evaluation. One concrete way to shorten it is to estimate KK at an intermediate drafter layer. Allocation needs final-layer, per-position confidences, but KK is a single batch-level number and should be predictable earlier. After layer ℓ\ell, the GPU estimates KK from intermediate confidences (e.g., by reusing the drafter's LM head) and copies this one integer to the CPU while the remaining layers run, so the target graph is prepared for the current step's KK.


Q3: "Cost table and context length. The table C(B,ρ)C(B,\rho) is profiled once at startup with dummy inputs. At what context length is it profiled? Verification attention cost grows with KV length, so does the best ρ\rho change for long-context requests, and is the table ever refreshed?"

A3: The cost table is profiled at a KV length of 2,048, with 3 warmup steps and the mean of 5 measurements per cell. Consistent with your intuition, the best ρ\rho does change with context length: a 32k-profiled table agrees with the 2k table on only 69.6% of bucket selections, shifting the optimal budget upward and raising the share of the 50% bucket from 42% to 70%.

The direct remedy is a length-bucketed cost table: profile C(B,ρ)C(B,\rho) at several KV lengths (e.g., 2k / 8k / 16k / 32k) and, at each step, look up the table matching the batch's average KV length. This only adds startup profiling time; the per-step cost remains a table lookup. Our length-matched runs already correspond to this setting: on 8k / 16k-prefix workloads, they change end-to-end throughput (including prefill) by −0.6% to +0.5% relative to the 2k table, so the default table is adequate at these lengths, while bucketing guards against larger mismatches at longer contexts. For drift that is not tied to length (e.g., contention or throttling), the table can also be refreshed online: step latency is already measured every step, so the selected bucket's entry can be updated via an exponential moving average (EMA), unselected buckets extrapolated via C(B,K)≈a(B)+b(B)KC(B,K) \approx a(B) + b(B)K with occasional probing, and the refresh interval shortened whenever measured latency persistently deviates from the table. We will add both designs to the revised manuscript.


In light of these new baselines, production-path evaluations, and sensitivity analyses, we kindly invite you to reconsider your score.

We hope the response has addressed your concerns. If you have any additional concerns or comments that we may have missed, we would be most grateful for any further feedback to help us further enhance our work.

Thank you for the detailed response and the new experiments. They address my main concerns, and I have raised my score from 3 to 4.

Q1 (equal-budget ablation). This was the main factor in my assessment, and the new results resolve it. At the same budget, D-cut is about 10–13% faster than uniform truncation (R3-Table1), and it matches or outperforms a per-request threshold tuned on the same data. The margin over the tuned threshold is smaller (about 1–7% in R3-Table1). This suggests that most of the gain comes from confidence-based, variable-depth allocation, which both methods share. D-cut's further advantage is that its budget comes from the runtime cost table, so no threshold needs to be tuned for each deployment. I suggest stating this clearly in the revision. Please also specify the target model used in R3-Table1, and report MAT and the spread over the repeated measurement windows for all three policies. If the target is Qwen3-8B, the D-cut entries are 5–8% higher than the corresponding fixed-ratio entries in Table 2 (e.g., 3217 vs. 3014 for GSM8K at 25%), and a short explanation would help.

Q2 (production engines). The two-stage design is convincing. The AV results on vLLM Model Runner V2 show that this class of method keeps clear gains with full CUDA-graph capture and asynchronous scheduling. Since D-cut itself has not yet been measured on that path, I suggest revising the claim in Appendix C that the reported throughput is a conservative estimate. Measuring D-cut with the budget chosen from the previous step's confidences, in the current setup, would directly show the cost of this lag. A brief discussion of how D-cut differs from AV would also be useful.

Q3 (cost table). Thank you for the profiling details and the context-length analysis. At 32k, where the bucket selections differ most, it would help to report decode-only throughput with the 2k table and with a length-matched table, because end-to-end throughput that includes prefill can hide differences in the decode phase.

These suggestions are for the revised manuscript and do not affect my updated score.

AuthorsOpenReview ↗

Thank You and Follow-up Response to Reviewer XZJJ

Dear Reviewer XZJJ,

We sincerely thank you for reading our response so carefully, for raising your score, and for offering such thoughtful suggestions. We truly appreciate your time and constructive engagement throughout the review process.

To briefly clarify your question on R3-Table1: the target model is Qwen3-8B with DFlash-b16 (on H20, c=32). The 5%–8% higher absolute throughput compared to Table 2 is because all H20 rebuttal runs were re-measured together on a different H20 node with an updated software stack, which shifted throughput upward uniformly across all methods while keeping relative speedups unchanged.

Your remaining suggestions for the revised manuscript are very helpful and well taken, and we will gladly incorporate them in our revision. Thank you once again for your generous support and valuable feedback!

Best regards, Authors of D-cut

Reviewer_LLF4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tries to address the problem that speculative decoding wastes substantial computational resources in high-concurrency scenarios, degrading its speedup. The paper proposes D-cut to adaptively select draft tokens across the entire batch. D-cut ranks candidate tokens in the batch based on the drafter's confidence, so that the verification budget is spent on relatively higher-quality tokens. Experiments show that D-cut achieves effective speedups across multiple models.

Critical strengths
  1. The paper is well written and easy to follow
  2. The motivation is convincing, and the experimental results show that the proposed D-cut effectively solves the problem.
  3. The authors conduct experiments on models of different scales and multiple datasets, validating the improvements in mean accepted tokens and output-token throughput speedup brought by D-cut.
Critical weaknesses
  1. The number of baselines is relatively small. Methods such as ECHO could be compared in terms of speedup performance under the same model and the same hardware.
  2. It currently appears that D-cut can only be applied to DFlash, so its generality may be insufficient.
Questions for authors
  1. Please see weaknesses.
  2. On the H800, D-cut (auto) is not optimal in most cases (5 out of 9). Could this be because the ratio options are too few? What is the actual distribution of ratios selected by 'D-cut(auto)'? When computing resources are abundant, could the range of ratio options for auto be expanded?
3 comments
AuthorsOpenReview ↗

Response to Reviewer LLF4 (Part 1 / 2)

Thank you for your review and for recognizing that "The paper is well written and easy to follow" and that "The motivation is convincing, and the experimental results show that the proposed D-cut effectively solves the problem." In response to your three comments, we clarify the relation to ECHO with new equal-budget baselines (Q1), evaluate cross-request adaptive verification on DFlash2 (Q2), and report the selected-ratio distributions, selector overhead, and an 8-bucket ablation on H800 (Q3). We kindly ask you to re-evaluate our paper and consider raising your score.


Q1 (W1): "The number of baselines is relatively small. Methods such as ECHO could be compared in terms of speedup performance under the same model and the same hardware."

A1: ECHO and D-cut target different draft structures: ECHO schedules tree depth and width for EAGLE-style tree drafters, whereas DFlash emits a single linear block per step with no tree structure to schedule.

On a linear block, the natural counterpart to ECHO is confidence-based depth truncation. Fixing the same 16-token draft block, we added two equal-budget baselines: (a) uniform truncation, where every request verifies its first ρ(D+1)\rho(D{+}1) positions; and (b) per-request confidence-threshold truncation (fixed τ\tau), where each request retains positions with si,k≥τs_{i,k} \ge \tau (sweeping τ∈{0.8,0.6,0.4,0.2,0.1,0.05}\tau \in \lbrace 0.8, 0.6, 0.4, 0.2, 0.1, 0.05 \rbrace and pairing each target budget with the τ\tau whose realized keep ratio is closest).

R2-Table1: Equal-budget throughput comparison (H20, c=32c=32, tokens/s)

DatasetBudgetUniformFixed Threshold (τ\tau, realized budget)D-cut
GSM8K25%28923092 (τ=0.8\tau=0.8, 23.5%)3217
GSM8K50%30083308 (τ=0.1\tau=0.1, 51.5%)3367
HumanEval25%28402936 (τ=0.8\tau=0.8, 22.4%)3132
HumanEval50%29533312 (τ=0.1\tau=0.1, 49.9%)3347
MT-Bench25%23112542 (τ=0.2\tau=0.2, 23.2%)2569

Under matched verification budgets, D-cut outperforms uniform truncation across both platforms and matches or exceeds the closest-budget fixed threshold in every setting on H20. Across c∈{16,32,64}c \in \lbrace 16, 32, 64 \rbrace, D-cut improves throughput by 3.1% to 14.4% over uniform truncation on H20 and H800, and by −0.1% to +8.0% relative to the closest-budget fixed threshold on H20 (MT-Bench has no 50%-budget row because the τ\tau sweep realizes at most a 31% keep ratio). In addition, Table 1 in the paper compares D-cut directly against EAGLE-3 and MTP on the same models and hardware.


Q2 (W2): "It currently appears that D-cut can only be applied to DFlash, so its generality may be insufficient."

A2: D-cut requires only three conditions:

  • A linear draft block per request
  • Per-position confidence scores
  • Batched target verification

Beyond DFlash, we evaluated the same class of cross-request adaptive verification on the public DFlash2 drafter (target Qwen3.8-27B, H20, BF16, 7 draft tokens per step, full CUDA graph + async scheduling, effective running concurrency about 17):

R2-Table2: Cross-request adaptive verification on DFlash2 (Qwen3.8-27B, H20, BF16)

DatasetDFlash2 tokens/s (MAT)DFlash2 + Adaptive Verification tokens/s (MAT)Speedup
GSM8K1192 (6.16)1349 (6.01)1.13×
HumanEval1071 (6.70)1159 (6.53)1.08×
MT-Bench831 (4.00)1150 (3.38)1.38×

Cross-request adaptive verification accelerates DFlash2 by 1.08–1.38× across all three benchmarks. D-cut also applies to sequential drafters such as EAGLE-3 in chain mode, though with smaller gains because sequential drafting cost grows with depth.


AuthorsOpenReview ↗

Response to Reviewer LLF4 (Part 2 / 2)

Q3 (Q2): "On the H800, D-cut (auto) is not optimal in most cases (5 out of 9). Could this be because the ratio options are too few? What is the actual distribution of ratios selected by 'D-cut(auto)'? When computing resources are abundant, could the range of ratio options for auto be expanded?"

A3: Empirical bucket distribution. On H20, GSM8K and HumanEval alternate between the 25% and 50% buckets, with 50% selected in 55%–78% of steps, whereas MT-Bench selects 25% in over 98% of steps. On H800, GSM8K and HumanEval select 50% in over 85% of steps (plus 9%–15% at 75% when c=16c=16), while MT-Bench shifts from 50% toward 25% as concurrency increases. Across both platforms, the 75% and 100% buckets are rarely selected. The full distribution table will be included in the revised manuscript.

Why D-cut (auto) is not first in every cell. D-cut (auto) is designed to stay near-optimal across hardware and workloads without manual tuning, rather than beating an oracle fixed ratio in every cell. In Table 2, D-cut (auto) stays within 1.2%–4.4% of the per-cell best; over three repeated windows at c=32c=32, the gap to the best fixed ratio is at most 2.6% on H800 and 0.9% on H20. Any single fixed ratio, by contrast, loses heavily on mismatched cells: 0.25 loses 33% on H800 HumanEval at c=16c=16, and 0.5 loses 22% on H20 MT-Bench at c=32c=32. Part of the small gap to the per-cell best comes from selector overhead itself: D-cut (auto) adds 0.29–0.31 ms per step over a fixed ratio, accounting for 1.1%–2.5% of a step on H800 (highest at c=16c=16), on the same scale as the throughput gap above.

Expanding to 8 buckets. On H800, we doubled the bucket set to {12.5%, 25%, …, 100%}.

R2-Table3: 4 vs. 8 buckets (H800, tokens/s, one measurement window per cell)

Datasetcc4 buckets8 bucketsMain 8-bucket choices (% of steps)
GSM8K1674687483 (+0.2%)50% (63), 62.5% (35)
GSM8K321137711334 (−0.4%)50% (96)
GSM8K641371013815 (+0.8%)37.5% (78), 50% (22)
HumanEval1675037508 (+0.1%)50% (73), 62.5% (26)
HumanEval321127211233 (−0.3%)50% (91), 37.5% (9)
HumanEval641343613423 (−0.1%)37.5% (89), 50% (11)
MT-Bench1641684131 (−0.9%)50% (91), 37.5% (7)
MT-Bench3267246759 (+0.5%)25% (52), 37.5% (48)
MT-Bench6492159160 (−0.6%)25% (98)

With 8 buckets, the selector does use the new intermediate buckets (e.g., 37.5% in most steps at c=64c=64 on GSM8K and HumanEval), yet throughput changes by only −0.9% to +0.8%. Throughput is flat around the optimum, so bucket granularity is not the main reason D-cut (auto) is not first in every cell. Averaged over three windows at c=32c=32, 8 buckets gain +0.1%, +1.1%, and +1.6%. Finer buckets bring this modest gain at the cost of capturing more CUDA-graph shapes and using more GPU memory. We default to 4 configurable buckets, which can be expanded when compute and memory are abundant.


In light of these results, we kindly ask you to re-evaluate our work and raise your score.

We hope the response has addressed your concerns. If you have any additional concerns or comments that we may have missed, we would be most grateful for any further feedback to help us further enhance our work.

AuthorsOpenReview ↗

Gentle Follow-up to Reviewer LLF4

Dear Reviewer LLF4,

As the discussion period will conclude in a few days, we wanted to gently follow up to see if our response and new experiments—covering the equal-budget baselines (R2-Table1), generality on DFlash2 (R2-Table2), and the H800 bucket distribution and 8-bucket ablation (R2-Table3)—have addressed your concerns. Please let us know if there is anything else we can clarify. Thank you very much again for your time and helpful feedback!

Best regards, Authors of D-cut

Reviewer_YdANOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes an adaptive pruning strategy for batched speculative decoding by ranking the draft tokens across the entire batch and then selecting top K tokens for verification based on a compute budget determined at runtime from the GPU hardware.

Critical strengths

I find the key contribution to be the runtime computation of the verification budget, while the other contributions could be considered largely overlapping with existing literature that the authors have mentioned in the related work. The results demonstrate the benefit of the approach though and the overall work is sound.

Critical weaknesses

While the runtime characterization of the GPU to calculate the ‘budget’ available for verification is a great idea in my opinion, the approach itself can be improved further. At present the authors do this one time at the start of execution and assume that this is constant throughout, but this may not be a realistic assumption. In practice, the token verification latency may change based on factors such as context length and GPU contentions and throttling. Can the authors comment on a possibility of doing this characterization step periodically, the period itself determined dynamically during the course of execution? They could simply measure the execution time periodically to not introduce too much additional overhead for this step.

Questions for authors

See above

3 comments
AuthorsOpenReview ↗

Response to Reviewer YdAN

We thank Reviewer YdAN for the strong support and constructive feedback, and are glad that you find the runtime characterization of the verification budget "a great idea" and the overall work "sound". Below, we discuss which runtime factors shift bucket selection (supported by long-context cost-table experiments), outline a periodic refresh design for the revised manuscript, and clarify how batch-wide budget allocation differs from prior single-request methods.


Q1: "At present the authors do this one time at the start of execution and assume that this is constant throughout, but this may not be a realistic assumption. In practice, the token verification latency may change based on factors such as context length and GPU contentions and throttling. Can the authors comment on a possibility of doing this characterization step periodically, the period itself determined dynamically during the course of execution?"

A1: When runtime changes affect bucket selection. Uniform slowdowns and relative cost shifts affect the selector differently. If throttling or contention slows down all buckets by the same factor, the arg⁡max⁡\arg\max does not change; a suboptimal bucket is chosen only when the relative cost across buckets changes. Context length belongs to the latter case, and it has a direct remedy: profile one cost table per KV-length bucket and look up the table matching the batch's average KV length at each step. Our measurements show that a 32k-profiled table agrees with the default 2k table on 69.6% of bucket selections (raising the share of the 50% bucket from 42% to 70%), while on 8k / 16k prefix workloads, using the length-matched table changes end-to-end throughput (including prefill) by within ±0.6%, so the default table is adequate at these lengths and bucketing guards against larger mismatches at longer contexts.

Periodic refresh design. Adopting your suggestion, we will incorporate an online refresh design into the revised manuscript, targeting drift that is not tied to length (e.g., contention or throttling). Because actual step latency is already measured at every step, we can apply an exponential moving average (EMA) update to the currently selected bucket's table entry online, while extrapolating unselected buckets at the same batch size via the linear fit C(B,K)≈a(B)+b(B)KC(B,K) \approx a(B) + b(B)K (supplemented by low-frequency probing when needed). Whenever the deviation between measured latency and the table entry persistently exceeds a threshold, the refresh interval is shortened—realizing the dynamic period you proposed.


Q2: "... the other contributions could be considered largely overlapping with existing literature that the authors have mentioned in the related work."

A2: Most existing adaptive methods adjust each request's draft length independently. D-cut instead shares a single verification budget across the entire batch and reallocates positions across requests: under matched verification budgets (25% and 50%), D-cut's global top-KK improves throughput by +9.8%–13.1% (at 25% budget) and +3.1%–14.4% (at 50% budget) over uniform per-request truncation across H20 and H800, and matches or exceeds per-request confidence-threshold truncation in every setting on H20 (−0.1% to +8.0%).


We hope the response has addressed your concerns. If you have any additional concerns or comments that we may have missed, we would be most grateful for any further feedback to help us further enhance our work.

Thank you for the additional comments. I will maintain my positive score.

Dear Reviewer YdAN,

Thank you very much for taking the time to read our response and for your continued support of our work! We truly appreciate your constructive feedback, which will help us further strengthen the revised manuscript.

Best regards, Authors of D-cut

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

For clarity, we refer to Reviewers YdAN, LLF4, and XZJJ as R1, R2, and R3, respectively.

We sincerely thank all reviewers for their thoughtful feedback and constructive suggestions. We are encouraged by their recognition of the practical importance of reducing verification waste in high-concurrency speculative decoding and the breadth of our evaluation across model scales and benchmarks (R1, R2, and R3). In particular, R1 and R3 highlight how runtime hardware cost characterization motivates adaptive verification-budget selection, and R3 notes that controlling verification depth is essential for block-parallel drafting to be practical at scale. R2 and R3 recognize the effectiveness of cross-request confidence-guided pruning to concentrate verification compute on higher-quality tokens across the batch. We also thank R3 for highlighting the training-free and drop-in nature of D-cut, which reuses existing drafter confidences without adding new CUDA-graph shapes or altering the target output distribution. R1, R2, and R3 found the presentation clear and easy to follow. Finally, we are glad that R1, R2, and R3 all found the paper informative and gained useful insights into how batched verification costs affect speculative decoding under high concurrency and how adaptive depth pruning addresses this bottleneck.

We have addressed each reviewer’s questions in our individual responses. Below, we summarize our core contributions and updated experimental results.

Core Contributions of Our Work

  • Verification cost explosion in batched block-parallel speculative decoding. We show that under high concurrency, verifying B(D+1)B(D{+}1) draft tokens per step becomes compute-bound and roughly 60% of verified tokens are rejected, causing long-draft DFlash to fall below autoregressive (AR) decoding.
  • Training-free cross-request pruning and runtime-adaptive budget selection. D-cut ranks candidate positions across the batch by prefix-product confidence and selects a discrete global budget KK from a one-time startup cost table C(B,ρ)C(B,\rho). This design requires no extra training or new CUDA-graph shapes and preserves the target output distribution.
  • D-cut accelerates serving across dense and MoE models and hardware platforms. At concurrency 32 across six target models and five benchmarks, D-cut (16) beats DFlash (16) in 29 of 30 configurations, lifting average speedup over AR from 1.26× to 1.65×, restoring acceleration on all four dense targets, and reaching up to 3.04× speedup over AR on MoE models. Across concurrencies 16–64 on Qwen3-8B, D-cut (auto) achieves 1.58× speedup over DFlash on H20 and 1.25× on H800, and sustains 1.15× speedup over AR under temperature-1 sampling where DFlash drops to 0.68×.

Updates of experimental results during rebuttal

  • Equal-budget allocation ablations. We compared uniform truncation, per-request confidence-threshold truncation, and D-cut's global top-KK under matched 25% and 50% budgets across H20 and H800, and showed how D-cut (auto) tracks load-induced shifts in the optimal budget without manual tuning (Response to R1, Q2; Response to R2, R2-Table1; Response to R3, R3-Tables 1–2).
  • Bucket distribution, selector overhead, and 8-bucket ablation. We reported the step-level ratio distributions selected by D-cut (auto) on H20 and H800, measured per-step selector latency, and evaluated an 8-bucket expansion on H800 showing that throughput is flat around the optimum (Response to R2, Q3 and R2-Table3).
  • Cost-table context-length sensitivity and online refresh. We compared cost tables profiled at 2k and 32k KV lengths, evaluated length-matched tables on 8k- and 16k-prefix workloads, and detailed length-bucketed lookup and dynamic-period online EMA refresh designs (Response to R1, Q1; Response to R3, Q3).
  • Production engine path and DFlash2 generality. We evaluated decoupled cross-request adaptive verification under full CUDA-graph capture and asynchronous scheduling in vLLM Model Runner V2 on H20 and H800, and validated generality on the public DFlash2 drafter (Response to R2, R2-Table2; Response to R3, Q2).

We have carefully addressed each individual comment provided by the reviewers and look forward to further discussion! We thank the reviewers again for their constructive feedback and will incorporate all new experimental results and clarifications into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.