acceptodds
Under review as a conference paper at ICLR 2027

Training-Free Dynamic Verification Budget for Parallel Speculative Decoding

Abstract

Parallel speculative decoding reduces drafting latency through one-shot block generation, but its end-to-end efficiency increasingly depends on how target-model verification is budgeted. A fixed verification budget wastes computation on difficult prompts with limited acceptance potential while restricting acceleration on easy ones. We characterize this mismatch through the trade-off between affine verification costs and strict diminishing marginal acceptance gains, yielding a unique interior optimum that varies dynamically with prompt difficulty. To address this, we propose TDVB, a training-free dynamic verification budget controller built upon a dual channel architecture. Specifically, a feed-forward Shape module constructs a position-wise acceptance prior from draft confidence margins to capture local spatial uncertainty, while a feedback Scale module estimates global prompt difficulty through an exponential moving average of observed acceptance lengths across rounds. The controller integrates these signals to adapt verification budgets online, effectively converting extra candidate capacity into accepted depth without offline calibration or additional training, while strictly preserving the target model’s output distribution. Extensive experiments show that TDVB achieves an average acceptance length of up to 10.18, improves end-to-end speedup by 14%–44% over strong static baselines, and reaches up to 7.40× over autoregressive decoding. Performance profiling reveals that these efficiency gains stem primarily from eliminating redundant verification rounds rather than merely reducing per-round execution overheads. Furthermore, the performance advantages scale robustly with model size, demonstrating that adaptive verification budgeting is essential to translating faster draft generation into sustained end-to-end acceleration and unlocking the true scaling potential of parallel speculative decoding.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Speculative Decoding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fD69OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to allocate the target model's verification budget in parallel speculative decoding. In this scenario, a block drafter proposes all candidate tokens in a single forward pass, so latency is primarily determined by the verification process rather than the drafting process. The authors observe that, given affine verification costs and diminishing acceptance gains, the optimal number of candidates to verify depends on the difficulty of the prompt, so a fixed budget is not the optimal solution. They propose TDVB, a training-free controller that sets candidate quotas per position by combining a Shape term derived from the drafter's Top-1/Top-2 confidence difference with a Scale term derived from the EMA of recent acceptance lengths. Experiments on Qwen3-4B/8B across seven benchmarks show that TDVB achieves longer acceptance lengths and higher speedups than EAGLE-3, DFlash, and DSpark, with up to a 7.40× speedup over autoregressive decoding.

Critical strengths
  1. The method is simple and practical to deploy. TDVB requires no training and only adjusts the number of draft candidates verified at each position. It can be directly integrated into existing parallel drafters without modifying the target model.

  2. The paper identifies a relevant optimization target and provides a clear analysis to support it. In parallel drafting, verification dominates round latency, and the latency decomposition shows that the gains mainly come from fewer verification rounds (120 vs 157 per 1k tokens for TDVB and DFlash), with the per-round cost nearly unchanged.

Critical weaknesses
  1. The main comparison doesn't fully isolate the contribution of the dynamic controller. In the ablation experiments (Table 2, Qwen3-4B, T=0), the average speedup of Static without Shape or Scale is already about 25% higher than DFlash (block-16), while the full TDVB improves over Static by about 8%, so it remains unclear how much of the reported performance gain over DFlash can be attributed specifically to the dynamic budget itself.

  2. The model-scaling claim may require further support. Although the abstract states that the advantages scale robustly as model size increases, the speedup gains of TDVB over DFlash decrease from Qwen3-4B to Qwen3-8B across all four settings (Appendix E) and in most per-task comparisons in Table 1.

Questions for authors
  1. Which parallel drafter is TDVB built on, what block size, candidate budget, and tree structure do DFlash and Static use, and why does Static outperform DFlash by about 25% on average (Qwen3-4B, T=0)? Clarifying these details would help separate the effects of candidate-tree configuration and dynamic budget. I would be happy to raise my score if TDVB shows a clear advantage over Static under a matched configuration.

  2. Although the paper states that dynamic budget is intended to reduce verification costs for difficult prompts and increase capacity for easy prompts, Appendix A suggests that Shape dominates in practice, so difficult prompts may actually have more candidates than easy prompts. Can the authors clarify how these two channels interact and what the average candidate budgets are for difficult and easy prompts?

3 comments
AuthorsOpenReview ↗

# Reply to Reviewer fD69 - Part 1

We sincerely thank the reviewer for the insightful analysis and constructive suggestions. We address the questions below.

[W1 / Q1] — Isolating dynamic controller gains under matched configurations

To isolate the contribution of our dynamic controller from the underlying tree structure, we introduce a Uniform baseline sharing the identical causal candidate pool and hard budget ceiling Bmax⁡=128B_{\max}=128 with TDVB, which allocates candidate quotas uniformly across decoding positions. The causal candidate pool fixes the inherent lack of causal dependencies among parallel drafted tokens in parallel drafters such as DFlash, supporting causal structures comparable to those of trained causal parallel methods like DSpark. Since official block-16 checkpoints of DSpark are not released, we adopt Domino [1] as our competing causal tree baseline with public block-16 weights.

All compared methods are built on the block-size-16 DFlash parallel drafter:

  • Domino [1]: the competing causal tree baseline using official block-16 checkpoints.
  • TDVB: DFlash + our causal candidate pool + our dynamic budget controller.
  • Uniform: DFlash + our causal candidate pool, uniform position-wise budget allocation.
  • Static: DFlash + our causal candidate pool, uniform position-wise budget allocation, with a globally optimized fixed budget searched exhaustively per prompt.

Results are shown in Table 1.

Table 1: Matched-configuration comparison on Qwen3-4B (block-16). Entries are speedup × average acceptance length.

MethodHumanEvalMBPPSWE-LiteGSM8KMATH-500AIME24MT-BenchMean
T=0T=0
DFlash4.75× (6.64)4.36× (6.17)2.57× (3.59)4.62× (6.55)5.50× (7.72)5.46× (7.68)3.11× (4.39)4.34× (6.11)
Domino4.77× (7.02)4.70× (7.11)2.68× (4.00)6.66× (10.10)5.83× (9.07)4.69× (7.62)2.65× (5.17)4.57× (7.16)
TDVB6.47× (9.06)5.97× (8.55)3.71× (5.32)6.58× (8.73)7.40× (10.06)6.81× (10.00)4.25× (6.22)5.88× (8.28)
Uniform4.43× (8.14)5.50× (7.59)3.48× (4.76)4.81× (7.94)6.68× (9.17)6.67× (9.09)3.46× (5.57)5.00× (7.47)
Static5.94× (8.60)5.55× (8.09)3.35× (4.91)5.81× (8.34)6.67× (9.72)6.63× (9.76)4.05× (6.00)5.43× (7.92)
T=1T=1
DFlash4.07× (5.77)3.70× (5.28)2.08× (2.95)3.96× (5.65)4.58× (6.51)3.59× (5.08)2.65× (3.78)3.52× (5.00)
Domino3.94× (6.26)3.73× (6.17)2.02× (3.09)5.54× (8.72)4.66× (7.14)2.95× (4.67)2.44× (4.56)3.61× (5.80)
TDVB5.91× (8.43)5.42× (7.99)3.24× (4.76)5.83× (8.28)6.17× (9.00)5.10× (7.63)3.87× (5.81)5.08× (7.41)
Uniform3.57× (7.59)5.02× (7.09)3.01× (4.18)4.09× (7.60)5.85× (8.24)4.84× (6.77)3.12× (5.19)4.22× (6.67)
Static5.38× (8.03)5.06× (7.60)2.86× (4.25)5.27× (7.87)5.67× (8.55)4.68× (7.04)3.70× (5.61)4.66× (7.00)

Our TDVB dynamic controller delivers pure speedup gains of +17.6% (T=0T=0) to +20.4% (T=1T=1) over Uniform. TDVB also outperforms the globally optimized Static baseline by +8.3% (T=0T=0) to +9.0% (T=1T=1), while Static requires global tuning. Relative to Domino, TDVB achieves an overall speedup gain of +28.7% at T=0T=0 (+15.6% in acceptance length) and +40.7% overall at T=1T=1 (+27.8% in acceptance length). This validates the advantages of adaptive internal budget scheduling. Our results also show that budget scheduling is a notable factor in speculative-decoding performance.

[1] Huang, Jianuo, et al. "Domino: Decoupling Causal Modeling from Autoregressive Drafting in Speculative Decoding." Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing. EMNLP, 2026.

AuthorsOpenReview ↗

# Reply to Reviewer fD69 - Part 2

[W2] — Clarifying cross-scale robustness

We appreciate this careful observation and would like to clarify our notion of robust scaling, which covers two distinct dimensions.

For a fixed target model, increasing the draft block size expands the pool of candidate tokens. TDVB is able to translate this expanded candidate capacity into longer accepted prefixes, which leads to a growing performance advantage over static baselines as the block size increases from 8 to 16, for instance from 14% to over 35% on Qwen3-4B at T=0T=0.

Separately, our method exhibits zero-shot cross-target-model adaptability. When moving from Qwen3-4B to Qwen3-8B, generation becomes inherently harder and drafting approximation bounds tighten for all speculative decoding approaches. This effect reduces the achievable performance margin for every method. Even under these more challenging conditions, TDVB preserves meaningful speedup gains without requiring offline recalibration or retraining for the new target model. For example, at T=1T=1 with block-16, TDVB maintains more than 25% speedup gain when scaling from Qwen3-4B to Qwen3-8B.

[Q2] — Dual-channel interaction between Shape and Scale

We would like to clarify a potential subtle misreading of our framing. As highlighted in our abstract: "A fixed verification budget wastes computation on difficult prompts with limited acceptance potential while restricting acceleration on easy ones." In the introduction, we further note that fixed budgets introduce redundant verification overhead on difficult prompts while prematurely truncating and capping throughput on easy ones.

Our unified objective is to minimize cumulative verification cost over the full sequence while maximizing per-step acceptance length, which motivates our two-channel design. Rather than focusing solely on cutting verification cost for hard prompts or expanding budget capacity for easy prompts, our design pursues this combined objective. The Scale channel regulates the global compute budget, while the Shape channel identifies local uncertainty and reallocates compute toward positions prone to rejection. These two signals jointly determine the per-round budget.

For hard prompts, Scale tends to shrink the global quota, yet local uncertainty margins are small, so Shape assigns large local weights and dominates the final allocation. The resulting candidate count often hits the Bmax⁡=128B_{\max}=128 cap to explore more continuations and rescue valid prefixes. For easy prompts, Shape automatically reduces local weights, and the average budget stays well below the cap to save redundant computation.To fully address this concern, we present these fine-grained budget statistics in Table 2 below, while both the table and the corresponding per-round trajectory visualization of the two example prompts are included in Appendix A.

Table 2: Average verification budget ∣B∣\vert B\vert (GSM8K, Qwen3-4B, T=0T=0).

GroupMean candidates ∣B∣\vert B\vert
Hard rounds124.6 (near the 128 cap)
Easy rounds75.5
Hard prompts117.4
Easy prompts102.6

Hard decoding steps consume budget close to the upper limit to extend valid exploration depth, whereas easy steps automatically avoid redundant verification. Per-round verification budget of one hard and one easy GSM8K prompt (Qwen3-4B, T=0T=0; all-prompt mean 111).(a) Hard prompt #63 (τp=5.4\tau_p=5.4, 39 rounds, mean 115). (b) Easy prompt #35 (τp=8.0\tau_p=8.0, 30 rounds, mean 102).

Thank you again for your constructive feedback. We would be happy to address any further concerns.

Reviewer_fD69OpenReview ↗

Official Comment by Reviewer fD69

The responses to W1/Q1 demonstrate that TDVB has certain advantages over the static method, and the responses to W2 and Q2 have provided me with a deeper understanding of this work. Thank you for the clarification, and I will raise my score from 2 to 3.

Reviewer_suaQOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies parallel speculative decoding. Its main motivation is that a fixed verification budget may be inefficient because the optimal amount of verification computation depends on prompt difficulty and acceptance behavior. To address this issue, the paper proposes TDVB, a training-free controller that dynamically allocates the verification budget during decoding without requiring additional controller training.

Experimentally, the paper reports improvements in both average acceptance length and end-to-end generation latency across several benchmarks and model configurations. The reported results include a peak speedup of 7.40× over standard autoregressive decoding.

Critical strengths
  1. Practical, training-free adaptive verification. TDVB adjusts the verification budget online using signals that are already available during decoding:

  2. Strong empirical gains across multiple benchmarks and decoding settings. TDVB consistently reports higher acceptance length and end-to-end speedup than the compared baselines, with a peak speedup of 7.40× over autoregressive decoding.

Critical weaknesses

1. Hallucinations in reference

A. Stern et al. (2018): an incorrect author has been added

Location: p. 11, lines 558–560.

The manuscript lists:

Mitchell Stern, Noam Shazeer, Jakob Uszkoreit, and Mohammad Norouzi.

The official NeurIPS proceedings list exactly three authors:

Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit.

B. EAGLE-3: the year differs from the official citation export

Location: p. 10, lines 538–539; also the corresponding “Li et al., 2026” citations in the text.

The manuscript gives:

Advances in Neural Information Processing Systems 38:136737–136756, 2026.

The official NeurIPS record identifies the paper as NeurIPS 2025.

C. SpecInfer: the citation combines the arXiv title with the conference publication details

Location: p. 11, lines 548–552.

The manuscript’s title includes “Generative”:

SpecInfer: Accelerating Generative Large Language Model Serving …

That matches the arXiv record. However, the final ACM proceedings title omits that word. Because the manuscript cites the ASPLOS 2024 publication, it should use the final publisher title:

SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification.

The ASPLOS volume and 932–949 page range are confirmed by ACM’s publisher-deposited metadata.

Classification: A minor mismatch between publication versions. The work is real. arxiv.org

D. Two entries omit their proceedings name

  • Vaswani et al. (2017), Attention Is All You Need — p. 11, lines 562–563: Missing the proceedings name.

  • Zheng et al. (2023), Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — p. 11, lines 570–572: Missing the proceedings name.

2. Improper use of math definitions

  • p. 3, lines 148–153 — Eq. (1): throughput numerator is questionable.

    $ S(B_t)=\frac{\mathbb E[\tau_t]}{C_{\text{draft}}+C_{\text{verify}}(B_t)}. $ The notations on the right-hand side are not defined, and why the equation is defined like this is not explained.

  • p. 4, lines 167–179 — “concave gain + affine cost ⇒ unique interior optimum” is mathematically unjustified. How come you can directly draw a conclusion without careful analysis?

  • p. 6, lines 283–313 — Eqs. (5)–(7) have inconsistent time indices.

    The paper writes $ \bar\tau_{t+1}=(1-\gamma)\bar\tau_t+\gamma\tau_t, $ then $ s_{t+1}=\operatorname{clip}\left(\frac{\bar\tau_t}{\tau_{\rm ref}},\ldots\right), $ and finally uses st+1s_{t+1} to construct the current-round $ W_{t,i}. $ These indices are inconsistent.

  • p. 6, lines 291–292 — EMA initialization conflicts with the algorithm.

    The text says $ \bar\tau_1\equiv\tau_1, $ but τ1\tau_1 is not known until the first verification round has completed.

  • p. 18, equation 9, the printed indicator symbol “⊮” should be replaced by a standard 1{⋅}\mathbf1\{\cdot\}.

Questions for authors
  • Could you provide the exact mathematical definition of GatherTopCandidates, TargetVerify, and Commit, and explain why selecting a dynamically truncated/top-WiW_i candidate set preserves the target distribution?

  • Please provide the definition of acceptance length τt\tau_t.*

  • What additional assumptions guarantee the claimed unique interior optimum B∗B^*?

3 comments
AuthorsOpenReview ↗

# Reply to Reviewer suaQ

[W1/W2-P18] — Reference and notation updates

We have updated our manuscript to align with recent literature and standardized all mathematical notations.

[W2-P3/Q2]— Throughput definition (Eq. 1) & acceptance length (τt\boldsymbol{\tau_t})

Following the renewal-reward formulation established in speculative decoding [1]( Theorem 3.8). τt\boldsymbol{\tau_t} denotes the length of the accepted token prefix in the tt-th decoding round. E[τt]\mathbb{E}[\tau_t] is its expectation. The cycle cost consists of the latency incurred by the draft model CdraftC_{\text{draft}}, plus the target-model verification latency for BtB_t parallel candidates Cverify(Bt)C_{\text{verify}}(B_t). The steady-state throughput (tokens per cycle) is written as

S(Bt)=E[τt]Cdraft+Cverify(Bt)S(B_t) = \frac{\mathbb{E}[\tau_t]}{C_{\text{draft}} + C_{\text{verify}}(B_t)}

[W2-P4/Q3]— Unique interior optimum B∗\boldsymbol{B^*}

Setting ∂S(Bt)∂Bt=0\frac{\partial S(B_t)}{\partial B_t} = 0 yields the first-order marginal equilibrium condition for the optimal budget: ∂E[τt]∂Bt=E[τt]κCdraft+κBt+β.\frac{\partial \mathbb{E}[\tau_t]}{\partial B_t} = \frac{\mathbb{E}[\tau_t]\kappa}{C_{\text{draft}} + \kappa B_t + \beta}. The numerator E[τt]\mathbb{E}[\tau_t] is strictly concave and monotonically increasing in BtB_t. Furthermore, E[τt]\mathbb{E}[\tau_t] saturates to an upper bound, and its marginal gain ∂E[τt]/∂Bt\partial \mathbb{E}[\tau_t]/\partial B_t is monotonically decreasing and converges to zero. The verification cost is an affine function of BtB_t, written as Cverify(Bt)=κBt+βC_{\text{verify}}(B_t)=\kappa B_t+\beta.

To establish uniqueness, we analyze the numerator h(Bt)=∂E[τt]∂Bt(Cdraft+κBt+β)−κE[τt]h(B_t) = \frac{\partial \mathbb{E}[\tau_t]}{\partial B_t}\big(C_{\text{draft}}+\kappa B_t+\beta\big)-\kappa \mathbb{E}[\tau_t] from the quotient rule. Differentiating h(Bt)h(B_t) gives ∂h(Bt)∂Bt=∂2E[τt]∂Bt2⋅(Cdraft+κBt+β).\frac{\partial h(B_t)}{\partial B_t} = \frac{\partial^2 \mathbb{E}[\tau_t]}{\partial B_t^2} \cdot \big(C_{\text{draft}}+\kappa B_t+\beta\big). Given the affine verification cost, strict concavity ∂2E[τt]∂Bt2<0\frac{\partial^2 \mathbb{E}[\tau_t]}{\partial B_t^2}<0 implies h(Bt)h(B_t) is strictly decreasing. Since ∂S(Bt)∂Bt=h(Bt)/(Cdraft+κBt+β)2\frac{\partial S(B_t)}{\partial B_t} = h(B_t)/\big(C_{\text{draft}}+\kappa B_t+\beta\big)^2 and the denominator is always positive, ∂S(Bt)∂Bt\frac{\partial S(B_t)}{\partial B_t} is strictly decreasing. Under these properties, ∂S(Bt)∂Bt\frac{\partial S(B_t)}{\partial B_t} crosses zero exactly once from positive to negative, which guarantees a unique interior optimum B∗B^* via the Intermediate Value Theorem.

[W2-P6]— Time index consistency & cold-start initialization

We have standardized all indices to distinguish the current round tt and the previous round t−1t-1. For cold start at (t=1)(t=1), we set τˉ1=∅\bar{\tau}_1 = \emptyset and s1=1.0s_1 = 1.0.

[Q1] — Definitions of the three primitives

GatherTopCandidates\texttt{GatherTopCandidates}. At each position ii, this function selects the Wt,iW_{t,i} candidate tokens whose ancestor path carries the largest cumulative draft log-probability, retaining the complete ancestor prefix.

TargetVerify\texttt{TargetVerify} performs parallel checking, while Commit\texttt{Commit} appends the first τ\tau accepted tokens and advances the KV-cache by τ\tau positions. Together they realize standard lossless speculative sampling under the ancestral visibility mask.

Why truncation preserves the target distribution?

Losslessness arises from the acceptance rule itself, rather than the specific construction of BB. For any valid candidate set, the committed tokens remain distributed according to the target distribution.

[1] Leviathan, et al. "Fast inference from transformers via speculative decoding." International conference on machine learning. PMLR, 2023.

Thank you again for your constructive feedback. We would be happy to address any further concerns.

Please note that ICLR’s stated policy treats hallucinated references as a Code of Ethics violation and grounds for desk rejection. My comments raise substantive concerns about the validity and accuracy of the cited references; they should not be treated as requests to correct minor typographical or grammatical errors.

Please see the ICLR 2026 policy statement.

AuthorsOpenReview ↗

### Detailed Clarification and Corrections on Cited References (W1)

We sincerely thank the reviewer for the careful reading of our paper and for raising these important concerns regarding our references. We fully respect ICLR’s Code of Ethics and its strict stance against fabricated or hallucinated citations, and we sincerely apologize for both our oversight in the initial submission and our overly brief explanation in the earlier response.

To clarify clearly for both the reviewer and the Area Chair: all four works listed in [W1] are genuine, well-known publications in the field, and none were fabricated. The metadata discrepancies stemmed from unverified third-party BibTeX exports (e.g., Google Scholar) and version/template mismatches during compilation, which we mistakenly failed to cross-check against official publisher records prior to submission. Below is a transparent breakdown of each item and how we have resolved it in our revised manuscript:

  • A. Stern et al. (2018) — Blockwise Parallel Decoding for Deep Autoregressive Models

    • Cause & Correction: The erroneous extra author (Mohammad Norouzi) originated from an unverified BibTeX entry imported from an external search aggregator. We cross-checked the official NeurIPS 2018 proceedings (neurips.cc) and removed the incorrect name so that the entry strictly lists Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit.
  • B. Li et al. (2025) — EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test

    • Cause & Correction: Upon checking the official DOI and Curran Associates publication metadata, the publication year field for Advances in Neural Information Processing Systems Volume 38 is recorded as 2026 (reflecting the formal volume publication and DOI assignment date, even though the NeurIPS conference took place in late 2025). To align strictly with the standard NeurIPS 2025 conference record and avoid any confusion, we have updated the year to 2025 in both the bibliography and all in-text citations (Li et al., 2025).
  • C. Miao et al. (2024) — SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification

    • Cause & Correction: We originally combined the earlier arXiv preprint title (which includes the word "Generative") with the ASPLOS 2024 publication metadata. Following the official ACM Digital Library record (dl.acm.org), we updated the title to match the final published proceedings version: "SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification".
  • D. Vaswani et al. (2017) & Zheng et al. (2023) — Missing Proceedings Names

    • Cause & Correction: Google Scholar's default BibTeX export formatted these conference papers as @article entries rather than @inproceedings, which caused the venue names to be suppressed during compilation under the ICLR bibliography style. We have converted both entries to proper @inproceedings records with the complete official proceedings name so they render properly.

Full Bibliography Audit: Following your feedback, we conducted a complete, line-by-line audit of every single reference in our manuscript against official publisher databases (NeurIPS Proceedings, ACM Digital Library, IEEE Xplore, DBLP, and arXiv) and updated all entries to their latest official published versions. We deeply regret this carelessness in our initial submission and thank you again for helping us improve the rigor and quality of our manuscript.

Reviewer_ttP7OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies dynamic verification-budget allocation for parallel speculative decoding and proposes TDVB, a controller requiring no additional training. It uses draft confidence margins to allocate candidates across positions and historical acceptance lengths to adjust a global budget scale, adapting verification to prompt difficulty. Experiments on mathematical reasoning, code, and dialogue benchmarks report longer accepted prefixes and improved end-to-end speedups, clearly demonstrating the work is practically relevant.

Critical strengths
  1. As parallel drafting reduces generation costs, verification allocation becomes an important optimization target. Adapting budgets to prediction difficulty can reduce computation wasted by fixed configurations.

  2. The controller combines local confidence and historical acceptance feedback without training an additional budget predictor. Its adaptive state and computation are small, making the approach potentially useful in deployment.

  3. The method shows positive results across model sizes, tasks, and sampling settings, while ablations support combining the two signals. Analyzing verification-round counts and per-round latency helps explain the acceleration mechanism.

Critical weaknesses
  1. Evaluation on long-horizon agentic benchmarks is missing. The evaluation mainly follows the mathematical reasoning, code, and dialogue benchmarks traditionally used for speculative decoding. As LLM applications expand from standalone short-horizon responses to multi-step agentic tasks, these benchmarks provide limited evidence about performance on mainstream applications of LLMs. Prediction difficulty may change substantially across task stages and tool responses, leaving the effectiveness of dynamic budgeting in this setting unclear. A representative agentic benchmark should report both generation time and complete task latency to establish how decoding gains translate into task-level acceleration.

  2. Comparisons with DSpark and related methods at block-16 would be valuable. DOMINO, a similar work to dspark reports block-16 results suggesting a smaller gap between accepted length and speedup in some settings. This raises the question of whether TDVB could achieve greater acceleration at comparable accepted lengths. Under the same acceptance length as in that paper, a higher speedup ratio of this method and dspark should be achievable. Could you explore the reasons for this?

  3. Speedups across different concurrency levels are not evaluated. The experiments focus on a single request and do not establish performance in high-concurrency serving. When concurrent prompts differ in prediction difficulty, adaptive budgets may have greater opportunities to reduce redundant verification and improve resource utilization, making this an especially relevant setting to test. However, concurrency also changes computational bottlenecks and batching efficiency, so the actual benefit requires measurement.

Questions for authors

Please see the weakness.

3 comments
AuthorsOpenReview ↗

# Reply to Reviewer ttP7 - Part 1

We sincerely thank the reviewer for acknowledging our contributions and the insightful comments. We address the raised questions in detail below.

W1 — Evaluation on long-horizon agentic benchmarks is missing

Our study focuses on accelerating response generation, and we follow established speculative-decoding benchmarks covering mathematical reasoning, code generation, and dialogue to enable direct comparison with prior methods. We agree that long-horizon agentic evaluation would provide additional evidence of robustness to tool-induced context changes and clarify how decoding gains translate into complete task acceleration. Our current results establish generation-level benefits within the evaluated settings; they do not establish end-to-end agentic speedups. We clarify this scope and discuss full agentic evaluation as an important extension for future work.

AuthorsOpenReview ↗

# Reply to Reviewer ttP7 - Part 2

W2 — Comparisons with DSpark at block-16

We thank the reviewer for this insightful suggestion. Since official block-16 checkpoints of DSpark are not released, we compare with Domino [1] , which provides official block-16 checkpoints, under matched device and experimental settings. Full results are provided in Table 1.

Table 1: Full block-16 results on Qwen3-4B and Qwen3-8B with T ∈ {0,1}. Entries are speedup × average acceptance length; in each setting the last row is TDVB vs Domino (speedup gain / length gain).

Table 1a — Qwen3-4B, T=0

MethodHumanEvalMBPPSWE-LiteGSM8KMATH-500AIME24MT-BenchMean
DFlash (block-16)4.75× (6.64)4.36× (6.17)2.57× (3.59)4.62× (6.55)5.50× (7.72)5.46× (7.68)3.11× (4.39)4.34× (6.11)
Domino (block-16)4.77× (7.02)4.70× (7.11)2.68× (4.00)6.66× (10.10)5.83× (9.07)4.69× (7.62)2.65× (5.17)4.57× (7.16)
TDVB (block-16)6.47× (9.06)5.97× (8.55)3.71× (5.32)6.58× (8.73)7.40× (10.06)6.81× (10.00)4.25× (6.22)5.88× (8.28)
TDVB vs Domino↑35.6% (+29.1%)↑27.0% (+20.3%)↑38.4% (+33.0%)↓1.2% (−13.6%)↑26.9% (+10.9%)↑45.2% (+31.2%)↑60.4% (+20.3%)↑28.7% (+15.6%)

Table 1b — Qwen3-4B, T=1

MethodHumanEvalMBPPSWE-LiteGSM8KMATH-500AIME24MT-BenchMean
DFlash (block-16)4.07× (5.77)3.70× (5.28)2.08× (2.95)3.96× (5.65)4.58× (6.51)3.59× (5.08)2.65× (3.78)3.52× (5.00)
Domino (block-16)3.94× (6.26)3.73× (6.17)2.02× (3.09)5.54× (8.72)4.66× (7.14)2.95× (4.67)2.44× (4.56)3.61× (5.80)
TDVB (block-16)5.91× (8.43)5.42× (7.99)3.24× (4.76)5.83× (8.28)6.17× (9.00)5.10× (7.63)3.87× (5.81)5.08× (7.41)
TDVB vs Domino↑50.0% (+34.7%)↑45.3% (+29.5%)↑60.4% (+54.0%)↑5.2% (+4.1%)↑32.4% (+26.1%)↑72.9% (+63.4%)↑58.6% (+27.4%)↑40.7% (+27.8%)

Table 1c — Qwen3-8B, T=0

MethodHumanEvalMBPPSWE-LiteGSM8KMATH-500AIME24MT-BenchMean
DFlash (block-16)4.58× (6.48)4.17× (5.92)2.53× (3.59)4.61× (6.54)5.56× (7.87)5.16× (7.35)2.99× (4.30)4.23× (6.01)
Domino (block-16)4.70× (7.32)4.54× (7.01)2.61× (4.14)6.38× (9.99)5.87× (9.45)4.68× (7.85)2.64× (5.19)4.49× (7.28)
TDVB (block-16)5.46× (8.96)5.32× (8.35)3.14× (5.34)5.62× (8.85)6.32× (10.18)5.90× (9.76)3.70× (6.08)5.07× (8.22)
TDVB vs Domino↑16.2% (+22.4%)↑17.2% (+19.1%)↑20.3% (+29.0%)↓11.9% (−11.4%)↑7.7% (+7.7%)↑26.1% (+24.3%)↑40.2% (+17.1%)↑12.9% (+12.9%)

Table 1d — Qwen3-8B, T=1

MethodHumanEvalMBPPSWE-LiteGSM8KMATH-500AIME24MT-BenchMean
DFlash (block-16)3.72× (5.33)3.43× (4.94)2.00× (2.87)3.80× (5.46)4.34× (6.24)3.56× (5.13)2.47× (3.59)3.33× (4.79)
Domino (block-16)3.63× (5.75)3.83× (5.95)1.80× (2.88)5.09× (8.15)4.24× (7.12)2.80× (4.91)2.14× (4.26)3.36× (5.57)
TDVB (block-16)4.82× (8.03)4.72× (7.72)2.49× (4.40)5.13× (8.38)5.45× (8.92)4.26× (7.36)3.29× (5.58)4.31× (7.20)
TDVB vs Domino↑32.8% (+39.7%)↑23.2% (+29.7%)↑38.3% (+52.8%)↑0.8% (+2.8%)↑28.5% (+25.3%)↑52.1% (+49.9%)↑53.7% (+31.0%)↑28.3% (+29.3%)

TDVB outperforms Domino by +28.7% (T=0) and +40.7% (T=1) on Qwen3-4B, and by +12.9% (T=0) and +28.3% (T=1) on Qwen3-8B. GSM8K at T=0 is the sole exception: Domino achieves a higher acceptance length and a marginally better speedup.

Domino uses a trained drafter to improve candidate quality, which comes with increased draft computation and higher per-step verification latency. TDVB instead translates candidates into deeper effective acceptance, with negligible controller overhead and lightweight parallel drafting, yielding stronger speedup at comparable acceptance lengths across most benchmarks. Our results also show that budget scheduling is a notable factor in speculative-decoding performance.

[1] Huang, Jianuo, et al. "Domino: Decoupling Causal Modeling from Autoregressive Drafting in Speculative Decoding." Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing. EMNLP, 2026.

AuthorsOpenReview ↗

# Reply to Reviewer ttP7 - Part 3

W3 — Speedups across concurrency levels

Thank you for highlighting the importance of evaluating TDVB under concurrent workloads. Although TDVB has not yet been integrated into production serving engines such as vLLM or SGLang, we provide preliminary results using a lightweight synchronous batching implementation built on Hugging Face model forward passes.

We conduct experiments under matched device and experimental settings and concurrency c ∈ {1, 2, 4, 8, 16, 32}. Detailed numbers are in Table 2.

Table 2: Batch-concurrency results (Qwen3-4B, block-8, GSM8K, T=0).

cτLatency (s)Speedup vs ARBatch efficiency
16.841.64.66×100.0%
26.823.54.15×40.0%
46.824.83.20×26.0%
86.838.12.13×15.4%
166.8215.61.24×7.7%
326.8729.51.00×4.1%

The adaptive budget controller is concurrency-robust. The acceptance length stays stable within 6.82–6.87 (fluctuation < 0.05), indicating that draft quality does not degrade under batching. As concurrency grows, GPU saturation reduces batch efficiency from 100.0% to 4.1%. The declining speedup arises from the well-known transition from the memory-bound to the compute-bound regime.

Thank you again for your constructive feedback. We would be happy to address any further concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

# Rebuttal Summary for Submission #25505

We sincerely thank all reviewers for their thoughtful and constructive feedback. We are encouraged by their recognition of the practical relevance of dynamic verification budgeting for parallel speculative decoding, the simplicity of our training‑free controller, and its potential to unlock sustained end‑to‑end acceleration.

In response to the reviewers’ comments, we have supplemented new controlled comparisons and concurrency profiling, updated the citation version, and clarified the scaling interpretation and dual‑channel behavior of TDVB. All corresponding revisions are incorporated into the manuscript. Below, we summarize the core contributions of TDVB and the main revisions made during the rebuttal.

Core Contributions of TDVB

Adaptive verification budgeting for parallel speculative decoding: We identify that fast block drafting makes candidate generation cheap, yet target‑model verification becomes the dominant bottleneck. Fixed verification budgets waste computation on difficult prompts while limiting acceleration on easy prompts. We characterize this trade‑off under affine verification costs and strictly diminishing marginal acceptance gains, proving a unique prompt‑dependent interior optimum for the verification budget.

Training‑free dual‑channel budget control: TDVB requires no extra training or offline recalibration. The Shape module constructs position‑wise acceptance priors from draft confidence margins to capture local uncertainty; the Scale module estimates global prompt difficulty via an exponential moving average of observed acceptance lengths. TDVB reallocates candidate quota across positions and rounds while preserving the target model’s output distribution.

Efficiency gains via fewer verification rounds: Empirical profiling shows TDVB improves average acceptance length and end‑to‑end speedup mainly by eliminating redundant verification rounds instead of cutting per‑round overhead. Furthermore, these performance advantages scale robustly, demonstrating that adaptive verification budgeting is essential to translating faster draft generation into sustained end‑to‑end acceleration and unlocking the true scaling potential of parallel speculative decoding.

Manuscript Updates during Rebuttal

Main Text: Added block‑16 comparisons against Domino and discussion of synchronous concurrency findings within Section 5.2; full concurrency tables and raw measurements are provided in Appendix L (ttP7). TDVB improves mean generation speedup by 12.9%–40.7% across Qwen3‑4B and Qwen3‑8B at the two evaluated temperatures. In batching tests, mean acceptance length remained stable at 6.82–6.87 tokens as concurrency increased from 1 to 32.

We performed a full bibliography audit against official publication records (suaQ). Where published versions are available, we have updated reference entries to align with official releases. We take official publisher records as the source of citation metadata, and cross‑check both imported BibTeX sources and the compiled LaTeX reference list to catch metadata mismatches.

Added Uniform and Static controlled ablations to isolate adaptive‑allocation benefits in Section 5.3. In Section 5.4, we clarify our two‑dimensional notion of robust scaling: (1) for a fixed target model, enlarging draft block size expands candidate capacity and yields growing advantages over static baselines, e.g., gains rise from 14% to over 35% on Qwen3‑4B at T=0T=0 as block size increases from 8 to 16; (2) TDVB exhibits zero‑shot adaptability across target‑model sizes. When switching from Qwen3‑4B to Qwen3‑8B, generation becomes inherently harder for all speculative‑decoding schemes and compresses performance margins, yet TDVB still retains meaningful speedup gains without recalibration or retraining, for instance delivering over 25% speedup gain at block‑16 when scaling from Qwen3-4B to Qwen3-8B. We also elaborate the interaction between Shape and Scale channels. Supplementary fine‑grained round‑level budget statistics and generation traces for easy‑hard prompt pairs can be found in Appendix A (fD69). Across seven benchmarks on Qwen3‑4B, TDVB improves mean end‑to‑end speedup over Uniform by 17.6% at T=0T=0 and 20.4% at T=1T=1, and outperforms the tuned Static control by 8.3% and 9.0%, respectively. In those hard rounds, Shape’s response to local uncertainty can outweigh Scale’s tendency to reduce the global budget, prompting the controller to allocate more candidates to explore potential valid continuations.

All revisions to the manuscript are highlighted in blue for ease of reference.

We hope that these revisions and additional empirical evidence address the concerns raised and assist the review committee in their further assessment. We sincerely appreciate their time and effort in evaluating our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.