- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to allocate the target model's verification budget in parallel speculative decoding. In this scenario, a block drafter proposes all candidate tokens in a single forward pass, so latency is primarily determined by the verification process rather than the drafting process. The authors observe that, given affine verification costs and diminishing acceptance gains, the optimal number of candidates to verify depends on the difficulty of the prompt, so a fixed budget is not the optimal solution. They propose TDVB, a training-free controller that sets candidate quotas per position by combining a Shape term derived from the drafter's Top-1/Top-2 confidence difference with a Scale term derived from the EMA of recent acceptance lengths. Experiments on Qwen3-4B/8B across seven benchmarks show that TDVB achieves longer acceptance lengths and higher speedups than EAGLE-3, DFlash, and DSpark, with up to a 7.40× speedup over autoregressive decoding.
Critical strengths
-
The method is simple and practical to deploy. TDVB requires no training and only adjusts the number of draft candidates verified at each position. It can be directly integrated into existing parallel drafters without modifying the target model.
-
The paper identifies a relevant optimization target and provides a clear analysis to support it. In parallel drafting, verification dominates round latency, and the latency decomposition shows that the gains mainly come from fewer verification rounds (120 vs 157 per 1k tokens for TDVB and DFlash), with the per-round cost nearly unchanged.
Critical weaknesses
-
The main comparison doesn't fully isolate the contribution of the dynamic controller. In the ablation experiments (Table 2, Qwen3-4B, T=0), the average speedup of Static without Shape or Scale is already about 25% higher than DFlash (block-16), while the full TDVB improves over Static by about 8%, so it remains unclear how much of the reported performance gain over DFlash can be attributed specifically to the dynamic budget itself.
-
The model-scaling claim may require further support. Although the abstract states that the advantages scale robustly as model size increases, the speedup gains of TDVB over DFlash decrease from Qwen3-4B to Qwen3-8B across all four settings (Appendix E) and in most per-task comparisons in Table 1.
Questions for authors
-
Which parallel drafter is TDVB built on, what block size, candidate budget, and tree structure do DFlash and Static use, and why does Static outperform DFlash by about 25% on average (Qwen3-4B, T=0)? Clarifying these details would help separate the effects of candidate-tree configuration and dynamic budget. I would be happy to raise my score if TDVB shows a clear advantage over Static under a matched configuration.
-
Although the paper states that dynamic budget is intended to reduce verification costs for difficult prompts and increase capacity for easy prompts, Appendix A suggests that Shape dominates in practice, so difficult prompts may actually have more candidates than easy prompts. Can the authors clarify how these two channels interact and what the average candidate budgets are for difficult and easy prompts?
3 comments
# Reply to Reviewer fD69 - Part 2
[W2] — Clarifying cross-scale robustness
We appreciate this careful observation and would like to clarify our notion of robust scaling, which covers two distinct dimensions.
For a fixed target model, increasing the draft block size expands the pool of candidate tokens. TDVB is able to translate this expanded candidate capacity into longer accepted prefixes, which leads to a growing performance advantage over static baselines as the block size increases from 8 to 16, for instance from 14% to over 35% on Qwen3-4B at .
Separately, our method exhibits zero-shot cross-target-model adaptability. When moving from Qwen3-4B to Qwen3-8B, generation becomes inherently harder and drafting approximation bounds tighten for all speculative decoding approaches. This effect reduces the achievable performance margin for every method. Even under these more challenging conditions, TDVB preserves meaningful speedup gains without requiring offline recalibration or retraining for the new target model. For example, at with block-16, TDVB maintains more than 25% speedup gain when scaling from Qwen3-4B to Qwen3-8B.
[Q2] — Dual-channel interaction between Shape and Scale
We would like to clarify a potential subtle misreading of our framing. As highlighted in our abstract: "A fixed verification budget wastes computation on difficult prompts with limited acceptance potential while restricting acceleration on easy ones." In the introduction, we further note that fixed budgets introduce redundant verification overhead on difficult prompts while prematurely truncating and capping throughput on easy ones.
Our unified objective is to minimize cumulative verification cost over the full sequence while maximizing per-step acceptance length, which motivates our two-channel design. Rather than focusing solely on cutting verification cost for hard prompts or expanding budget capacity for easy prompts, our design pursues this combined objective. The Scale channel regulates the global compute budget, while the Shape channel identifies local uncertainty and reallocates compute toward positions prone to rejection. These two signals jointly determine the per-round budget.
For hard prompts, Scale tends to shrink the global quota, yet local uncertainty margins are small, so Shape assigns large local weights and dominates the final allocation. The resulting candidate count often hits the cap to explore more continuations and rescue valid prefixes. For easy prompts, Shape automatically reduces local weights, and the average budget stays well below the cap to save redundant computation.To fully address this concern, we present these fine-grained budget statistics in Table 2 below, while both the table and the corresponding per-round trajectory visualization of the two example prompts are included in Appendix A.
Table 2: Average verification budget (GSM8K, Qwen3-4B, ).
| Group | Mean candidates |
|---|---|
| Hard rounds | 124.6 (near the 128 cap) |
| Easy rounds | 75.5 |
| Hard prompts | 117.4 |
| Easy prompts | 102.6 |
Hard decoding steps consume budget close to the upper limit to extend valid exploration depth, whereas easy steps automatically avoid redundant verification. Per-round verification budget of one hard and one easy GSM8K prompt (Qwen3-4B, ; all-prompt mean 111).(a) Hard prompt #63 (, 39 rounds, mean 115). (b) Easy prompt #35 (, 30 rounds, mean 102).
Thank you again for your constructive feedback. We would be happy to address any further concerns.
Official Comment by Reviewer fD69
The responses to W1/Q1 demonstrate that TDVB has certain advantages over the static method, and the responses to W2 and Q2 have provided me with a deeper understanding of this work. Thank you for the clarification, and I will raise my score from 2 to 3.
# Reply to Reviewer fD69 - Part 1
We sincerely thank the reviewer for the insightful analysis and constructive suggestions. We address the questions below.
[W1 / Q1] — Isolating dynamic controller gains under matched configurations
To isolate the contribution of our dynamic controller from the underlying tree structure, we introduce a
Uniformbaseline sharing the identical causal candidate pool and hard budget ceiling Bmax=128 with TDVB, which allocates candidate quotas uniformly across decoding positions. The causal candidate pool fixes the inherent lack of causal dependencies among parallel drafted tokens in parallel drafters such as DFlash, supporting causal structures comparable to those of trained causal parallel methods like DSpark. Since official block-16 checkpoints of DSpark are not released, we adopt Domino [1] as our competing causal tree baseline with public block-16 weights.All compared methods are built on the block-size-16 DFlash parallel drafter:
Uniform: DFlash + our causal candidate pool, uniform position-wise budget allocation.Static: DFlash + our causal candidate pool, uniform position-wise budget allocation, with a globally optimized fixed budget searched exhaustively per prompt.Results are shown in Table 1.
Table 1: Matched-configuration comparison on Qwen3-4B (block-16). Entries are speedup × average acceptance length.
UniformStaticUniformStaticOur TDVB dynamic controller delivers pure speedup gains of +17.6% (T=0) to +20.4% (T=1) over
Uniform. TDVB also outperforms the globally optimizedStaticbaseline by +8.3% (T=0) to +9.0% (T=1), whileStaticrequires global tuning. Relative to Domino, TDVB achieves an overall speedup gain of +28.7% at T=0 (+15.6% in acceptance length) and +40.7% overall at T=1 (+27.8% in acceptance length). This validates the advantages of adaptive internal budget scheduling. Our results also show that budget scheduling is a notable factor in speculative-decoding performance.[1] Huang, Jianuo, et al. "Domino: Decoupling Causal Modeling from Autoregressive Drafting in Speculative Decoding." Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing. EMNLP, 2026.