acceptodds
Under review as a conference paper at ICLR 2027

Correct While Verifying: Intermediate Target Predictions for Speculative Draft Correction

Abstract

Speculative decoding accelerates large language model inference, yet its rejection handling remains reactive. Even parallel approaches such as PEARL overlap drafting with verification, but the system still discovers that a draft token is wrong only after final verification and discards the entire suffix after the first rejection. We make a simple observation: target verification is layered, and intermediate target computation already produces useful predictions about the correct continuation before final logits are available. Based on this observation, we correct drafts while they are being verified. A trainable decoder layer maps intermediate target states to preliminary token distributions, whose top- candidates seed correction branches. While the target model completes final verification from cached intermediate states, a frozen draft model expands these branches in parallel under a custom attention mask. Final target verification evaluates both the original draft and the corrected branches and commits the longest valid continuation under the target model's verification rule. Across Llama, Qwen2.5, and Vicuna target-draft pairs, our method consistently outperforms PEARL and achieves up to 4.53 speedup over autoregressive decoding.

Primary area:
generative models
Keywords:
speculative decoding, inference acceleration, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_NDFuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Correct-While-Verifying (CWV), a correction mechanism for parallel speculative decoding built on PEARL. A trainable decoder layer maps intermediate target-model states to preliminary token predictions, and their top-k candidates seed alternative branches, which a frozen draft model expands while the target model completes verification. Final target predictions determine the accepted prefix and the selected branch’s unverified extensions are carried into the next cycle. The paper evaluates CWV on mathematical reasoning, code generation, and dialogue benchmarks across Llama, Qwen2.5, and Vicuna model pairs under greedy decoding.

Critical strengths
  1. The use of intermediate target computation for correction is a contribution. CWV uses information available before final verification to prepare alternative continuations, addressing the wasted drafting that occurs when a parallel speculative pipeline follows an incorrect token. The method goes beyond parallel drafting and verification by introducing an intermediate correction mechanism.
  2. The experiments provide encouraging evidence that this mechanism can improve PEARL across several workloads and model families. The gains are particularly noticeable with the Llama3.2-1B drafter. The reported top-k prediction accuracy and correction-trigger rates provide additional evidence consistent with the proposed mechanism. These results suggest that the idea has practical potential, even though its benefits vary substantially across configurations.
Critical weaknesses
  1. The algorithm description lacks important details about error identification and correction. Section 3.2 and Figure 3 illustrate replacing an erroneous token, but do not clearly explain how CWV selects the correction position before final target verification identifies the actual first rejection. Additionally, Eq. (1) gives the acceptance probability for stochastic speculative sampling, although the surrounding text describes a greedy regime. Under greedy decoding, acceptance should instead be determined by agreement with the target model’s argmax token. The paper should distinguish these verification rules and specify the correction procedure more precisely.
  2. The experimental results do not yet sufficiently establish the magnitude and robustness of CWV’s incremental benefits. The improvements over PEARL are encouraging with the Llama3.2-1B drafter, but Table 1 shows mostly approximately 0.5%–6% gains for the other two Llama pairs relative throughput gains over PEARL. Section 4 reports aggregate throughput across evaluation samples and a standard error of 2%–4%, but does not specify whether timing experiments were repeated independently or averaged across multiple runs. Several improvements are smaller than this reported uncertainty. EAGLE-2/3 use a different target model and are explicitly presented as reference results, rather than controlled comparisons. They therefore do not establish CWV’s competitiveness against learned drafting methods under matched conditions. Since CWV also requires training, focusing primarily on training-free baselines leaves this trade-off incompletely evaluated. The paper also lacks throughput ablations isolating the benefit of correction overlap or examining the effects of branch count and insertion depth. Prediction accuracy and correction-trigger statistics are useful, but do not establish how much throughput benefit comes specifically from early correction overlap or how that benefit compares with the additional computation.
Questions for authors

Section 3.3.2 states that CWV intervenes in PEARL’s Pre-verify scenario. Is the correction position therefore fixed by the scheduling boundary, or is it inferred from intermediate predictions within the current verification window? Please clarify how this rule corresponds to Figure 3 and the branch construction in Algorithm 1, and, if correction can occur within a multi-token window, how CWV handles an actual first rejection that occurs before the selected correction position. Were throughput measurements repeated across independent runs, and is there controlled evidence isolating the benefit of early correction from additional candidate generation?

Reviewer_3jgSOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

In speculative decoding, a small draft model proposes several tokens, and a large target model verifies them in one forward pass, committing the accepted prefix and discarding everything after the first rejected token. Parallel speculative decoding runs the draft and target concurrently (typically on separate GPUs) so the draft keeps drafting ahead during verification instead of waiting. However, when a token is rejected, the tokens drafted after it were built on that rejected token and are discarded, so that effort is wasted and recovery advances only one corrected token. The proposed method (CWV) reduces the cost of that discard-and-restart by exploiting the fact that a target forward is computed layer by layer. The hidden states after the first half of the target's layers already carry a usable prediction of the correct token before the final logits are ready.

Concretely, CWV adds one trainable decoder layer after the target's first N=L/2 layers that emits a preliminary next-token distribution. When a draft token is likely to be rejected, CWV takes the top-k tokens from this preliminary distribution as correction candidates, and a frozen draft model expands each candidate into a short continuation all while the target finishes its remaining layers to produce the authoritative logits. The authoritative logits then confirm which candidate is correct. Only that target-verified token is committed, and its already-drafted continuation is carried into the next step, so the output is identical to standard decoding under greedy sampling. Here, a single decoder layer is trained, with both the draft and target models frozen. Built on the parallel scheduler PEARL, CWV is evaluated on three Llama pairs, Qwen2.5-72B/7B, and Vicuna across GSM8K, MATH500, AIME 2024/2025, HumanEval, MBPP, and MT-bench.

Critical strengths

• Only a single decoder layer is trained while the draft and target models stay frozen, so the training cost is low. • Using the half-depth prediction to seed correction branches and carrying their continuation into the next step is well motivated. • The paper clearly mentions that the cost shifts to the draft side (a CWV step costs about 2.7–3.1× more) and that acceptance length does not map one-to-one to throughput.

Critical weaknesses

• The core idea is not new. Using intermediate/early-exit target predictions for speculation appears in Self-Speculative Decoding methods in the literature. • The trainable decoder layer is very similar to the adapter layer of Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting. • The improvement over PEARL (baseline used in this paper) is marginal. For some model pairs we see only 1-2% improvement in speedup. • The correction only intervenes in the pre-verify case and leaves post-verify unchanged, and the per-correction benefit is bounded by the branch depth that fits in the remaining-layer time, so the attainable gain is structurally limited. • The headline 4.66× is a single MT-bench sub-category on the weakest-drafter pair rather than a main-benchmark result.

Questions for authors
  1. Beyond the "repair vs. propose" framing, what does CWV's decoder layer add over Kangaroo's near-identical adapter?
  2. On well-matched pairs, several CWV-vs-PEARL margins are within the 2–4% throughput noise. Why is the improvement so small?
  3. How does the comparison look under a compute- or hardware-normalized metric (tokens/s per GPU), given the extra GPU and k-branch draft expansion?
  4. How sensitive are the results to N (exit depth), k (seeds), and the branch depth, and what is the added KV-cache memory from branch partitioning?
  5. Did you consider applying the intermediate-prediction correction within post-verify as well, and would it help for well-matched pairs?
Reviewer_Am5vOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper focuses on the verification side of speculative decoding: the draft model is idle when the target model is verifying the drafts. To mitigate this idle time, this paper proposes Correct-While-Verifying (CWV), which predicts the verification results midway. The draft model starts to draft the next tokens with the speculated verification results.

Critical strengths
  1. The proposed CWV method is indeed an early-exit method for the verification. The motivation is reasonable because the idle draft model does create some time waste.
  2. The training is relatively lightweight because only a side layer requires training.
Critical weaknesses
  1. Limited improvement upper bound. The earliest time that the draft model can start to predict is when half of the verification and the side head have been processed. So, in the best case, CWV can reduce the bubble by less than half. If the prediction is wrong, there will be no gain.
  2. Marginal empirical gains. Many improvements in speedups are within noise. For the 8B and 7B drafters, improvements over PEARL are 0.02–0.10× (e.g., 3.08→3.10, 2.60→2.62), inside the authors' own stated 2–4% standard error. Vicuna (1.52→1.54) and Qwen show similarly small gains. No variance across runs or significance test is reported, so the "better in all eighteen combinations" claim is not supported. Only the weak 1B drafter shows a meaningful gain.
  3. The additional deployment cost is not mentioned in comparisons. The side layer requires additional memory space to deploy. This can take up another GPU, which could have been used to deploy another model to increase parallelism. To fairly compare these methods, the paper should make sure to use the same computational resources.
  4. Incompatible with tree drafting. With a tree draft, the rejection point differs for each path. Seeds would then be needed at every node, and the mid-layer head would have to run over the whole tree. The paper does not discuss this.
Questions for authors

Exactly which positions receive the top-k correction seeds? Do you place seeds at every draft position, or only where L_early's top-1 disagrees with the draft? What happens when the true first rejection occurs at a position without seeds, or within the tail inherited from the previous cycle?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.