acceptodds
Under review as a conference paper at ICLR 2027

Cannot Wait to Verify: Parallel Drafting to Eliminate Verifier Idle Time in Speculative Decoding

Abstract

Autoregressive inference in large transformer models is limited by token-level sequential execution, leading to high latency and poor GPU utilization. Speculative decoding (SD) mitigates this bottleneck by introducing a draft and verifier model. However, the drafting stage remains sequential, leaving the verifier idle and underutilizing compute. To address this, we propose a plug-and-play drafting adaptive drafting method that parallelizes inference across transformer layers during draft generation. Starting from a shared hidden state, our approach evaluates multiple layers in parallel and verifies consistency by measuring divergence in hidden-state space. Layer-wise draft states are accepted when deviations are small; otherwise, a correction step is applied. By exploiting redundancy across transformer layers, our method accelerates draft generation and improves hardware utilization without modifying architectures or requiring retraining. Experimental results on NLP tasks show that our method speeds up the drafting stage by up to with minimal degradation in accuracy, while maintaining the same performance as SD.

Primary area:
generative models
Keywords:
Large Language Models, Speculative Decoding, Efficient LLM Inference

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_4yPkOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies the drafting bottleneck in speculative decoding. It proposes Adaptive Layer-parallel Drafting (ALD), a training-free method for accelerating an existing multi-layer draft model. ALD evaluates several future transformer layers concurrently from a shared proxy input. It uses cosine similarity to select how far to advance before refreshing that input. The method preserves standard target verification and adopts bonus calibration, which reprocesses confirmed tokens to reconstruct the draft model's key–value cache. The paper also provides a bound on the propagation of hidden-state approximation errors. Experiments compare ALD with standard speculative decoding, EasySpec, and LayerSkip on Qwen-4B/30B and LLaMA-3B/70B draft–target pairs across various tasks. Table 1 reports drafting speedups of approximately 1.40$\times$ to 1.81$\times$ over standard drafting and end-to-end speedups of 2.21$\times$ to 3.75$\times$ over autoregressive target-model decoding.

Critical strengths

S1. The paper reduces the draf cost without requiring draft-model training. ALD operates on an existing draft model and leaves target verification unchanged. The design also separates approximate proposal generation from output-distribution preservation: the latter comes from the standard verification procedure, rather than requiring the approximate drafter to reproduce its original outputs exactly.

S2. The reported results show a reduction in drafting costs with marginal decrease in acceptance. For example, on MGSM with the Qwen-4B/30B draft–target pair, drafting time decreases from 30.8s to 19.5s per 100 draft tokens while the reported acceptance rate remains 0.35. End-to-end speedup over target-only decoding increases from 1.85$\times$ to 2.61$\times$. Within the evaluated setting, this is a meaningful improvement without a substantial loss in draft–target agreement.

Critical weaknesses

W1. The parallel execution mechanism is insufficiently specified. Section 5 reports four A6000 GPUs and a maximum parallel window of ten layers, but does not explain how candidate layers, weights, and caches are distributed across devices. Appendix A.8 includes single-T4 profiling, but its relationship to the main implementation/evaluation remains unclear. Reducing sequential dependencies alone does not explain lower latency when concurrent layers compete for the same resources. The reported component execution times do not explain the four-GPU execution path, and CUDA Graph usage is not specified. It is therefore difficult to distinguish gains from additional hardware parallelism, reduced launch overhead, or differences in baseline optimization.

W2. The baseline set is limited even within the training-free setting. PEARL (ICLR, 2025) and REST (NAACL, 2024) reduce or avoid neural drafting overhead, while OPT-Tree (TACL, 2025) improves accepted length through adaptive candidate trees. These methods are discussed in Section 2 but are not evaluated. TETRIS (ACL, 2025) and AdaSpec (SoCC, 2025) address wasted verification overhead by filtering which draft tokens to verify. All these methods have public implementations. The current comparisons do not establish whether ALD remains competitive against alternatives targeting drafting cost, accepted length, and wasted verification work.

W3. The performance gain under larger batches and alternative resource allocations is unclear. Appendix A.8 reports utilization and component execution times at batch sizes 1 and 8, but not end-to-end execution times. These measurements do not show whether ALD remains beneficial as the sequential baseline becomes better utilized. For workloads with multiple concurrent requests, larger draft batches, replicated drafters across GPUs, or overlapping drafting and verification are also relevant uses of the same GPU budget. These alternatives need not improve single-request latency, but the paper does not establish when layer-parallel execution is preferable.

W4. The choice and net benefit of the similarity criterion are insufficiently validated. Appendix A.11 states that mean squared error (MSE) is less stable than cosine similarity, but provides no quantitative comparison. The same threshold is used across models, but Figure 3 does not report model-specific acceptance and end-to-end speed across thresholds. Table 7 reports similarity-check latencies of 2.0ms to 2.3 ms per parallel window on a single T4, but there is no controlled comparison showing whether adaptive selection justifies this cost over, for example, fixed advancement. The missing evidence is whether the similarity check improves the acceptance–latency trade-off across models, rather than merely changing the degree of approximation.

Questions for authors

Q1. Please specify the device/resource mapping and scheduling in the four-GPU setup, and include cache calibration in the per-round latency breakdown. Can you provide an execution trace separating computation, communication, and synchronization? Please report CUDA Graph usage for both methods and compare against an optimized sequential baseline, including a graph-enabled baseline where supported. This would help identify the source of the reported gain.

Q2. Can you add representative training-free comparisons covering drafting overhead (PEARL or REST), accepted length (OPT-Tree), and verification efficiency (TETRIS or AdaSpec)? These comparisons are important for assessing practical competitiveness. Initial results on one or two compatible target models would provide a useful starting point. Please match hardware and sampling settings, evaluate batch-oriented methods under matched concurrency, and report drafting time, verification time, accepted length, and end-to-end throughput.

Q3. Can you report end-to-end throughput and per-request latency across increasing batch sizes or request concurrency? Under a fixed GPU and memory budget, please include a straightforward batched or replicated-drafter configuration, and an overlap-based schedule if feasible. Are advancement decisions made per request or per batch? If they are per request, how is batching maintained when requests advance to different layers?

Q4. Can you quantify the cosine-MSE comparison in Appendix A.11 on one representative task for both Qwen and LLaMA? Please sweep each metric's threshold and report acceptance, end-to-end speedup, and decision overhead. Include several fixed advancement lengths using the same proxy computation and cache handling, and compare at similar drafting cost. This would help establish whether the criterion is robust across models and whether its benefit outweighs its cost.

Reviewer_qkBWOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Context and problem. In speculative decoding, the draft model generates candidate tokens autoregressively, so the verifier waits while drafting proceeds. A deeper, more accurate drafter improves acceptance but makes each drafting step slower. The paper argues that this sequential drafting is the dominant bottleneck. It also observes that consecutive layers of a draft model produce highly similar hidden states (Fig. 1b), and aims to exploit this redundancy to reduce drafting latency without retraining.

Method. The paper proposes Adaptive Layer-parallel Drafting (ALD), a training-free change to how the layers of an existing multi-layer drafter are executed. The method keeps a committed hidden state at some depth mm. In each iteration, it feeds this same state as a shared proxy input into the next KK layers (K=10K=10), and evaluates them concurrently via batched inference. It then computes the cosine similarity between each candidate output and the committed state, scanning in order. At the first layer where the similarity drops below a threshold τ\tau (τ=0.95\tau=0.95), the last candidate before it becomes the new committed state, and a new window starts from there. If no candidate falls below τ\tau, the window advances to its end. This repeats until the last layer is reached. A Lipschitz-based bound (Theorem 1) describes how the local approximation errors propagate to the final hidden state. Because ALD produces approximate KV states, it adopts EasySpec's bonus calibration: after each draft–verify iteration, those KV entries are discarded and the accepted tokens are reprocessed by the drafter. Verification is unchanged, so the output distribution of standard speculative decoding is preserved.

Contributions. (i) A plug-and-play, training-free drafting method that adaptively reduces the drafter's effective sequential depth, using an online hidden-state similarity check. (ii) An error-propagation bound for the resulting approximation. (iii) An empirical evaluation against standard speculative decoding, EasySpec, and LayerSkip.

Validation. The paper evaluates two drafter–target pairs, Qwen-4B → Qwen-30B and LLaMA-3B → LLaMA-70B, on 4× RTX A6000 GPUs with a draft length of 6. Benchmarks are WMT, MMLU, MGSM, Math, IF-Eval, and SpecBench.

  • ALD reduces drafting time by 1.40–1.81× relative to standard SD, with slightly lower acceptance (e.g., 0.42 → 0.39 on Math with Qwen).
  • End-to-end speedup over autoregressive decoding is 2.61–3.75× for Qwen (vs. 1.85–2.47× for SD) and 2.21–2.79× for LLaMA (vs. 1.71–2.32×). It is higher than EasySpec and LayerSkip on most settings.
  • A Gemma-3-1B → Gemma-3-27B pair shows similar trends.
  • Ablations cover the threshold τ\tau, the window size KK (gains saturate beyond K=10K=10), the effective depth reduction (10.8–18.5 layers), and cosine vs. MSE similarity.
  • A profiling study on a single T4 GPU reports higher drafter GPU utilization (43.5% → 57.4% at batch size 1), small extra memory (2.7 MB), and a runtime breakdown.
  • Repeated runs across random seeds are also reported.
Critical strengths

Training-free and acceptance-aware drafting acceleration. ALD requires no retraining or architectural change and leaves verification untouched, so the output distribution of standard speculative decoding is preserved. Unlike fixed approximations, it explicitly accounts for the trade-off between draft latency and verifier acceptance. Its online similarity check limits how far approximate execution proceeds, and in the evaluated setting this keeps acceptance closer to standard SD than EasySpec or LayerSkip (e.g., 0.36 vs. 0.31 and 0.23 on MMLU with Qwen). Why it matters: it shows, within the evaluated setting, that adaptively controlling approximate drafting can reduce the acceptance loss that fixed approximations incur.

Critical weaknesses

1. The reported gains are unlikely to carry over to production serving, and they appear tied to slow hardware and an unoptimized execution path.

  • Synchronization and CUDA graphs. Where the next window starts depends on the outcome of the similarity check, so every ALD iteration needs a GPU-to-CPU synchronization. The paper itself notes one synchronization point per iteration (Appendix A.8.3). There are several iterations per draft token and 6 draft tokens per step. This data-dependent control flow is incompatible with CUDA-graph capture and CPU–GPU overlap, both of which are standard in production engines. The paper does not discuss this.
  • Cost versus benefit on current GPUs. A synchronization bubble typically costs on the order of 100–400 µs. The Qwen-4B drafter has about 8 GB of weights (Table 6), so one layer reads roughly 220 MB per step at batch size 1. By a bandwidth estimate, that takes about 30 µs on a B200 and about 70 µs on an H100. One synchronization therefore costs more than the several layers ALD tries to save. More fundamentally, decoding is memory-bound: evaluating KK layers concurrently still reads all KK layers' weights, about the same cost as running them sequentially and exactly. A real speedup is thus possible mainly when per-layer time is dominated by fixed overheads rather than weight reads.
  • The evaluated setting is exactly that regime. The main experiments run on RTX A6000 GPUs, and the profiling runs on a T4. On the T4, one layer takes 5.8 ms (Table 7), far above its bandwidth bound of roughly 0.7 ms. On the A6000, drafting takes about 0.3 s per token (Table 1). These numbers point to an overhead-dominated implementation, where batching KK layers mainly amortizes framework overhead.
  • Plausibility of the numbers. On Math with Qwen, the end-to-end gain of ALD over SD (3.75/2.19 ≈ 1.71×) equals its drafting speedup (1.71×), even though acceptance drops from 0.42 to 0.39. Verification time is non-zero and fewer tokens are accepted per step, so the end-to-end gain should be strictly below the drafting gain. In addition, the abstract reports up to 1.65× drafting speedup, while Table 1 shows up to 1.81×.
  • Minor points. The title claims to eliminate verifier idle time, but the verifier still waits during drafting, and utilization is measured only for the drafter. Across random seeds (Table 9), the standard deviations of speedup (0.08–0.25×) and acceptance (0.07–0.11) are comparable to many of the gaps to EasySpec.

Why it matters: the practical value of a drafting optimization depends on real deployments. On current datacenter GPUs with CUDA graphs, the gains are likely to vanish or reverse.

2. Narrow applicability and limited novelty.

  • Applicability. ALD needs a deep multi-layer drafter. Mainstream drafters such as EAGLE-3, MTP, and DFlash use one or a few layers, so the method does not apply to them, as the authors acknowledge in Appendix A.5. It is therefore limited to standalone small-LM drafters, whose speedup ceiling is lower. ALD reaches 2.21–3.75× end to end, while mainstream methods report roughly 3–6× at batch size 1.
  • Novelty. The method builds directly on EasySpec, reusing its layer-parallel proxy execution and its bonus calibration for the KV cache. The main addition is a cosine-similarity threshold that decides how far to advance. Theorem 1 is a generic Lipschitz bound and is not connected to the cosine criterion, as the authors note in Appendix A.3.
  • Missing baselines. Two natural baselines are missing. One is SWIFT, a training-free adaptive layer-skipping method. The other is simply a smaller drafter from the same model family (e.g., a 1–2B model), which also cuts drafting time. It is also unclear how LayerSkip is applied to Qwen-30B and LLaMA-70B, since it requires models trained with its recipe.

Why it matters: even if the gains held, they would cover a narrow and shrinking part of practical speculative decoding, and the contribution over EasySpec is incremental.

Questions for authors
  1. Results on current GPUs with CUDA graphs. Could the authors report end-to-end results on an H100 or A100 with an optimized inference engine and CUDA-graph capture enabled? Please also report how many GPU-to-CPU synchronizations ALD performs per draft token and per draft–verify step, and their total cost. How would the data-dependent window advancement be made compatible with CUDA graphs and CPU–GPU overlap?

  2. Consistency of the reported numbers. On Math with Qwen, the end-to-end gain over SD (about 1.71×) equals the drafting speedup (1.71×), although acceptance drops from 0.42 to 0.39. How is this possible, given that verification time is non-zero? Please also clarify the exact target models ("Qwen-30B" and "LLaMA-70B"; e.g., is Qwen-30B the MoE Qwen3-30B-A3B?), and how LayerSkip was applied to these targets, since it requires models trained with its recipe.

  3. Missing baselines. How does ALD compare with SWIFT, a training-free adaptive layer-skipping method, and with simply using a smaller drafter from the same model family (e.g., a 1–2B model) under the same setup?

Reviewer_JhKWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Adaptive Layer-parallel Drafting (ALD) to speed up draft generation in speculative decoding. In this method, multiple layers of the draft model compute in parallel starting from the same hidden state, and the forward depth is determined based on cosine similarity. The target model verifies candidate tokens following the standard speculative decoding pipeline. At the end of each step, the draft model reconstructs the KV cache approximated by this execution.

Critical strengths

The authors identify a practical bottleneck: while the target model can verify multiple candidate tokens in one pass, it still waits for the draft model to generate these tokens sequentially.

The paper reports draft latency, acceptance rate, and end-to-end speedup together, avoiding the oversimplification that faster draft generation directly translates to faster overall system performance.

Ablations on threshold values, parallel window sizes, and runtime overhead in the appendix also help clarify the operating regimes where this method yields net benefits.

Critical weaknesses
  • The standalone contribution of the adaptive decision rule is not sufficiently clear. The paper compares overall results against existing layer-parallel methods, but lacks a fixed-forward-depth baseline implemented under the same setup with comparable compute cost. It is currently difficult to isolate how much gain comes from the similarity-based adaptive advancement itself.

  • There is no direct characterization linking the similarity criterion to approximation error. High cosine similarity does not guarantee that the proxy hidden state closely matches the true serial state. The theorem presented describes how local error propagates, but does not quantify how tightly a cosine threshold bounds such local error. The appendix acknowledges this limitation to some extent, yet the criterion remains justified primarily by final empirical results.

  • Some end-to-end gains require finer-grained latency breakdowns. For instance, in the Qwen Math setting, ALD’s acceptance rate drops from 0.42 to 0.39, while end-to-end speedup rises from 2.19× to 3.75×. This is not inherently problematic, but acceptance rate and per-100-draft-token latency alone are insufficient to explain how many extra tokens are produced per step and where time is saved across pipeline stages.

Questions for authors
  1. Could you add a baseline with fixed forward depth, matched to similar draft latency or compute budget, for comparison on accepted sequence length and end-to-end latency?
  2. What is the relationship between cosine similarity and ground-truth hidden-state error, logit difference, and candidate acceptance outcomes? Could you provide an analysis grouped by similarity bins?
  3. For experiments with large improvements such as Math, can you report the average accepted length, together with per-step latency breakdowns on the main evaluation hardware for drafting, verification, synchronization, and KV cache calibration?

I would be open to raising my score if the authors adequately address my concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.