acceptodds
Under review as a conference paper at ICLR 2027

Give the Drafter a Second Chance to Guess: Parallel Reconsideration for Speculative Decoding

Abstract

Speculative decoding accelerates large language model inference with a small drafter, whose per-guess acceptance rate determines the achievable speedup. Feature-level drafters share a design choice: every token, easy or hard, is drafted under a single forward pass. Many of the tokens the target rejects are ones the drafter is itself unsure about yet the target answers decisively. To address this, we introduce Parallel Reconsideration: guided by its own uncertainty, the drafter reconsiders its hesitant guesses with extra passes that run in parallel with the one-pass draft, adding negligible latency. A co-designed training recipe and serving-time drafting algorithm append these reconsidered guesses to the original draft tree as refined candidates. Across models and benchmarks, on average, Parallel Reconsideration improves over the baselines by 5.3% to 10.2%on acceptance length and by 3.5% to 9.2% on speedup.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Speculative Decoding, LLM Efficient Inference, Parallel Refinement

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8V8POpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Parallel Reconsideration for feature-level speculative drafters. A learned self-query lets the drafter revisit a feature without advancing to the next token, while a learned halting gate controls further reconsideration. Reconsideration rows are batched with ordinary draft steps, and the resulting candidates compete within a fixed verification-tree budget. Training combines the ordinary prediction loss, depth-weighted reconsideration losses, a gate objective, and mixed-depth caches. Evaluation on three target models and seven benchmarks reports average accepted-length improvements of 5.3–10.2% and relative speedup improvements of 3.5–9.2% over EAGLE-3.1. Target-only tree verification is unchanged.

Critical strengths
  1. The method offers a concrete way to spend additional computation on uncertain draft features without adding a separate sequential draft step. The self-query, mixed-cache training, and serving algorithm address the distinction between revising the current prediction and advancing its continuation. Appendix C.4 also explains why changing the candidate tree does not change the target output distribution under the specified verifier.

  2. The experimental controls are stronger than a comparison at equal verified-node counts alone. The paper retrains component ablations, compares against a two-layer drafter, and gives EAGLE-3.1 the same additional rows for wider drafting. Table 4’s PR-without-reconsideration control is particularly useful for separating the effect of the new training recipe from the effect of actually reconsidering at inference. The main results cover multiple models, domains, temperatures, and request batch sizes within the same serving engine.

Critical weaknesses
  1. The latency measurements do not fully support the broad claim of negligible drafting overhead. Appendix H reports a whole-cycle ratio of 1.015–1.022 on Qwen3, combining drafting and target verification; it does not measure the increase in drafting time separately. All wall-clock results use one H200. Table 2 also shows cases where PR is slower than EAGLE-3.1, including Llama LiveCodeBench at request batch sizes 8 and 16. The paper should specify when the additional computation remains cheap and explain these regressions.

  2. The paper does not report variation across independently trained drafters or repeated timing measurements. Several individual benchmark gains are small, and the component ablations cover selected configurations. Since the method changes both training and inference, it is unclear how consistently the gains would appear across training runs.

Questions for authors
  1. Can you separate draft-model execution, reconsideration bookkeeping, and target verification time for the existing batch-size experiments? What explains the Llama LiveCodeBench regressions at batch sizes 8 and 16?

  2. How many independent drafter-training and timing runs underlie the main table and the equal-row controls? If repeated runs are available, please report their variation and the additional training cost relative to EAGLE-3.1.

Reviewer_uodoOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Parallel Reconsideration (PR), a speculative decoding method that aims to improve the average acceptance length of feature-level drafters with little additional decoding latency. The authors observe that many rejected draft tokens occur at positions where the drafter has low confidence, while the target model is relatively confident at some of these positions. Motivated by this observation, PR allows the drafter to perform additional feature refinement at selected uncertain positions instead of relying on a single-pass prediction.

Specifically, the authors introduce a learned self-query to guide feature refinement, a gate to decide whether further refinement is worthwhile, and a bidding strategy to prioritize nodes under a limited computation budget. The resulting candidates are incorporated into the draft tree. To avoid additional sequential drafting steps, the refinement computations are batched with ordinary drafting operations. The authors also design a training procedure involving depth-weighted supervision, marginal-entropy regularization, mixed KV caches, and two on-policy rollout paths.

The method is evaluated on Llama-3.1-8B, Qwen3-8B, and Gemma-3-12B across seven benchmarks. Compared with EAGLE-3.1, PR improves average acceptance length by 5.3%–10.2% and relative decoding speedup by 3.5%–9.2% across the reported configurations. Ablation and controlled experiments compare PR with alternative training choices, a deeper drafter, and wider draft trees.

Critical strengths
  1. An effective parallel feature-refinement strategy. The paper presents a practical way to improve acceptance length without introducing additional sequential drafting steps. By executing reconsideration alongside ordinary drafting, the method makes use of extra parallel computation rather than increasing the critical path. The end-to-end measurements, including the approximately 1.5%–2.2% per-cycle overhead reported for Qwen3-8B in Appendix H, provide evidence that this design can be effective in the evaluated setting.

  2. Meaningful controlled experiments. Beyond evaluating three target models and seven benchmarks, the authors conduct targeted comparisons that help isolate the source of the gains. In particular, comparisons against a two-layer drafter and wider draft trees—including controls that use the additional batch rows for ordinary tree expansion—suggest that the improvements do not merely result from spending more computation. The additional evaluation on HASS also offers preliminary evidence that reconsideration is not specific to EAGLE-3.1.

Critical weaknesses
  1. The independent contributions of the learned routing components are not sufficiently established. PR combines a learned gate, which determines whether and how long to reconsider a node, with a bidding strategy that prioritizes eligible nodes. However, the node-selection ablation in Table 5 compares the proposed bid against highest-confidence and random selection, without including a straightforward lowest-confidence selection baseline. More importantly, it does not directly establish the advantage of the learned gate over a simple confidence-threshold rule. Since the motivation already demonstrates a strong relationship between low drafter confidence and rejection, it remains unclear how much the more complex routing mechanism improves upon simple uncertainty-based heuristics.

  2. The generality of the latency advantage is not sufficiently demonstrated. The low overhead of co-forwarding depends on the hardware and serving workload, especially the extent to which additional batch rows can be absorbed without substantially increasing latency. The authors do explain this argument and provide measurements in Appendix H, but the primary evaluation uses a single H200 GPU. Moreover, Table 2 shows settings in which PR is slower than EAGLE-3.1: for Llama-3.1-8B on LiveCodeBench, the speedups are 1.33× versus 1.40× at batch size 8 and 1.29× versus 1.31× at batch size 16. These results suggest that the benefit is not uniform across serving conditions, and further analysis would help clarify when co-forwarding remains advantageous.

Questions for authors
  1. How much of the improvement comes from the learned gate and the bidding strategy, respectively? Could the authors compare the learned gate against a simple confidence-threshold policy and compare the proposed bid against lowest-confidence selection, under matched computation and verification budgets? This would help establish whether learning recoverability provides a substantial advantage over inexpensive uncertainty heuristics.
  2. How sensitive is PR to the reconsideration budget and depth? Could the authors vary the per-step reconsideration budget mm and the maximum reconsideration depth RR, and report acceptance length, wall-clock speedup, and additional computation or latency? This would help clarify the trade-off between refinement effort and practical acceleration.
Reviewer_NNVUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
3(+1)
Read review
Paper summary

This paper proposes Parallel Reconsideration (PR), a speculative decoding method that selectively allocates additional computation to uncertain draft predictions. A learned gate determines which positions should be reconsidered, while a self-query mechanism enables the drafter to refine its predictions through additional forward passes. These computations are executed in parallel with ordinary drafting to minimize latency overhead. The refined candidates are incorporated into the draft tree without modifying target verification.

Critical strengths
  • The proposed parallel reconsideration mechanism improves draft quality while introducing minimal additional latency. Its integration with existing speculative decoding frameworks makes the approach practically valuable.

  • The paper provides extensive evaluations across multiple models and benchmarks, including matched-budget comparisons and ablation studies. These experiments demonstrate consistent improvements and support the effectiveness of the proposed method.

Critical weaknesses
  • The core idea of allocating additional computation to uncertain predictions has been extensively explored in prior works such as PonderNet(https://arxiv.org/abs/2107.05407) and CALM(https://arxiv.org/abs/2207.07061). The proposed method appears to be an incremental adaptation of existing adaptive computation techniques to speculative decoding, with its main novelty lying in the parallel execution strategy.

  • The paper does not adequately demonstrate why a learned gating mechanism is necessary compared to simpler uncertainty-based heuristics. For example, selecting low-confidence tokens using a fixed threshold or Top-K strategy could potentially achieve similar improvements with less complexity.

  • The method is primarily evaluated on EAGLE-style drafters, while its compatibility with more efficient parallel drafting approaches such as DFlash remains unexplored. This limits the evidence supporting its broader applicability to modern speculative decoding systems.

  • Despite the additional architectural and training complexity, the average speedup improvement over EAGLE-3.1 is marginal. Furthermore, Table 2 shows that the method occasionally underperforms the baseline at larger batch sizes, raising concerns about its practical benefits in high-throughput serving scenarios.

Questions for authors
  • Could the authors compare the learned gating mechanism against simpler confidence-based strategies, such as selecting the Top-K lowest-confidence tokens for reconsideration? This would help clarify whether the additional algorithmic complexity is necessary.

  • As shown in Table 2, the speedup improvement diminishes at larger batch sizes, and PR occasionally underperforms EAGLE-3.1. Could the authors explain the underlying bottlenecks and discuss potential solutions to mitigate this performance degradation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.