acceptodds
Under review as a conference paper at ICLR 2027

Nucleus Speculative Decoding: Plausibility-Aware Verification Beyond Exact Distribution

Abstract

Speculative decoding accelerates autoregressive generation by using a lightweight draft model to propose multiple tokens that are verified by a target model in parallel. However, the standard acceptance rule focuses on exact distribution correction and rejects tokens that remain highly plausible under the target model when the draft model assigns excess probability. This conservative verification limits the number of draft tokens retained after each verification forward pass. We introduce Nucleus Speculative Decoding (NSD), a relaxed verification method that incorporates target-model plausibility into speculative decoding. NSD accepts a draft token if it satisfies the standard acceptance rule or belongs to the target model's nucleus. We theoretically characterize the distributional deviation introduced by our method and show that the single-step error is exactly determined by the draft model's excess probability within the target nucleus. We further derive sequence-level fidelity bounds that quantify how local deviations accumulate over autoregressive decoding. Experiments across multiple target models and proposal mechanisms demonstrate that NSD consistently improves speculative decoding efficiency while maintaining competitive task performance. Our method achieves throughput speedups of over autoregressive decoding and over standard speculative decoding. These improvements coincide with longer accepted lengths, allowing more output tokens to share the cost of each target verification pass. Analysis shows that plausibility-aware verification provides an effective approach for relaxed verification and speculative decoding efficiency.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Speculative decoding, large language models, efficient inference

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_57tTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Nucleus Speculative Decoding (NSD), a relaxed verification rule for stochastic speculative decoding. In addition to the standard acceptance event, NSD accepts a draft token whenever it belongs to the target model's top- pp nucleus set. Standard residual correction is retained after rejection. The method is training-free and is evaluated with several proposal mechanisms, including diffusion-based self-speculation, DSpark, DFlash, EAGLE-3, and MTP.

The main theoretical result shows that the increase in one-step acceptance probability is exactly ESE_S, the excess proposal mass inside the target nucleus, and that the one-step total variation error introduced by NSD is also exactly ESE_S. Empirically, NSD consistently increases accepted lengths and reduces target-model calls. The paper reports an average speedup of approximately 4.23×4.23 \times over autoregressive decoding and 1.85×1.85 \times over standard speculative decoding.

The idea is simple and the efficiency gains are promising. However, the current evidence does not yet justify the paper's broader fidelity claims. The exact theoretical result relies on assumptions that are not verified for several experimental backends. The paper does not measure ESE_S in practice or establish the uniform conditional bounds needed for its sequence-level argument. In addition, the main quality and throughput comparisons lack repeated trials, uncertainty estimates, and sufficiently detailed measurement protocols.

Critical strengths
  • NSD changes only the verification rule and does not require training a new draft or target model. It appears straightforward to incorporate into existing speculative decoding pipelines.

  • The experiments cover multiple speculative decoding backends rather than demonstrating the method on only one draft architecture.

Critical weaknesses
  • The proofs require the theoretical qq to match the actual proposal law and assume specific rejection, bonus-token, and cache-transition behavior. The appendix acknowledges that historical DSpark confidence settings, external Nemotron routines, and the MTP rejection kernel cannot be fully recovered or verified. Numerical safeguards may also violate the exact assumptions. Consequently, the theorem does not currently certify all reported systems.

  • Although ESE_S is exactly the one-step TV error, the paper reports neither its mean nor its tail behavior. The sequence-level result assumes a uniform conditional bound over histories, but no such bound is established. The example using ϵ=10−3\epsilon=10^{-3} is illustrative rather than empirically supported.

  • Relative to standard speculative decoding, NSD records 15 wins, one tie, and 14 losses, with only a small average improvement. These results are compatible with sampling variation and do not establish consistent quality preservation.

  • The main results omit raw tokens/s, timing variance, batch size, warm-up, numerical precision, software versions, and prompt/output-length statistics. The appendix states that the DFlash timing was obtained from a single seed under potentially contended conditions, making that result unreliable.

  • Relaxed baselines also have tunable tolerance parameters, but the paper does not explain how these parameters were selected. Comparisons should be made at matched accuracy degradation or matched divergence, with complete quality–throughput curves.

  • Accepting tokens inside the target top-pp set is intuitive and closely related to prior lossy or relaxed speculative decoding. The paper should discuss and compare against ⊕ Faster Cascades via Speculative Decoding, which already considers lossy speculative sampling in top-pp settings.

Questions for authors

Refer to weaknesses

Reviewer_RGBqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Nucleus Speculative Decoding (NSD), which accepts a draft token if it passes standard speculative acceptance or belongs to the target distribution's θ\theta-nucleus (i.e., the minimal set of top-θ\theta tokens). It quantifies the method's lossiness; for the stated single-step proposal kernel, the paper proves that the increase in acceptance probability equals the total variation distance from the target distribution, namely, the draft's excess probability mass inside the nucleus. It also derives a sequence-level bound conditional on uniform error bounds for the decoder's actual next-token distributions. Furthermore, in experiments on real-world LLM decoding benchmarks, it claims that NSD exhibits higher throughput than standard speculative decoding and other lossy speculative decoding methods while maintaining decoding accuracy.

Critical strengths
  • The presented experimental results (in particular, in Table 1) seem to be promising compared to the lossless/lossy baselines.
Critical weaknesses

W1: The motivation for nucleus-based verification and the evidence for generation quality are insufficient.

  • The paper does not sufficiently justify why the target nucleus is the appropriate additional acceptance set. Fig. 1 shows that many rejected proposals remain highly ranked or belong to the target nucleus, which motivates relaxing verification but does not establish why nucleus membership must determine acceptance. Despite its central role, the abstract uses the term "nucleus" without defining it, and the introduction provides only a brief cumulative-probability description. I believe that a clearer explanation of what the target nucleus represents and why every proposal within it should be accepted would strengthen the motivation of NSD. Moreover, Theorem A.1 seems to hold for any fixed acceptance set (instead only for a nucleus), and the experiments do not isolate the benefit of nucleus selection over alternative plausibility criteria at comparable task performance. Meanwhile, Sec. 3.1 repeats the standard acceptance rule and the rejection of highly ranked proposals, both already discussed in Sec. 1-2: this space could instead explain and justify the central design choice.

  • The paper discusses little about the quality of the generated outputs due to NSD. Theoretically, although Eq. (4) identifies the single-step TV error as the draft excess probability mass ESE_S within the target nucleus, the paper does not certify a small value of ESE_S in practice. For example, the paper applies ϵ=10−3\epsilon=10^{-3} to explain the coupling agreement lower bound in Equation (5) is merely (0.999)32≈0.9685(0.999)^{32}\approx0.9685 when T=32T=32; but when T=1024T=1024, it plunges to (0.999)1024≈0.359(0.999)^{1024}\approx0.359. Indeed, the lower bound decreases exponentially with output length, so the short-horizon illustration does not guarantee small sequence-level distributional deviation for long generations. Empirically, Sec. 4.4 analyzes additional acceptances and verification efficiency, while the evidence for generation quality remains largely mathematical accuracy and functional pass@1. Although similar accuracy/pass@1 supports competitive performance on the evaluated benchmarks, it cannot be used to argue for comparable generation quality in a broader sense. Qualitative examples and an analysis of failure modes in long generations would help connect plausibility-aware verification to output quality.

W2: Comparisons beyond AR and standard SD do not sufficiently justify sacrificing exact distribution preservation.

  • The additional verification baselines focus on relaxed methods, leaving stronger lossless verification untested. A previously studied method, 'Block Verification' (Sun et al., 2025), can increase the mean accepted length while preserving the target output distribution; in fact, Sun et al. prove the optimality in accepted lengths among lossless verificaiton rules along a single-chain draft path. Measuring throughput and mean accepted length in the same experimental setting of this paper would establish whether relaxation provides an additional speed benefit over a stronger lossless verification.

  • In addition, Appendix C of Sun et al. (2025) introduces 'Greedy Block Verification', which can retain more draft tokens in one iteration for fixed draft and target distributions but requires subsequent target distribution modification to preserve exact sequence sampling. A proposed lossy baseline (without the distribution modification) could retain its greedy acceptance and residual recovery while breaking the losslessness and does not guarantee an end-to-end speedup. Comparing NSD against this explicitly defined variant at comparable accuracy and pass@1 would help establish the practical benefit of nucleus-based verification.

Questions for authors
  1. Clarification of the evaluated implementation: Which checkpoints, DSpark confidence settings, and Nemotron generation paths produced Table 1? Was proposal selection disabled, or did the verifier include its conditioning in the proposal probabilities? Can you establish that the SD reference is equivalent to the NSD implementation with only the additional nucleus condition disabled, using the same target sampling transformations and stopping rules?

  2. Clarification of the quality evidence: How many repetitions underlie the main AR, SD, and NSD results, and what uncertainty accompanies the paired quality differences? This would help me judge the quality reference against which NSD is evaluated.

  3. Clarification of the Figure 3 experimental setup: It seems to use Qwen3-4B with DSpark for panels (a,b,c) and Nemotron-14B with self-speculation for panel (d), but the benchmarks, sample sizes, and aggregation procedure are unclear. Could you specify the experimental details (benchmark(s), numbers of prompts and runs, and aggregation procedure) for each panel, including whether statistics are pooled over tokens or averaged per prompt or benchmark? Please also clarify whether the block lengths, temperature, and nucleus threshold match the settings in Section 4.1.

Reviewer_Sk3gOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Nucleus Speculative Decoding (NSD), a relaxed verification method that improves token acceptance while allowing distributional deviation. Theoretical analysis and experiments demonstrate improved decoding efficiency with competitive task performance.

Critical strengths

The paper proposes a novel relaxed verification method for speculative decoding.

Critical weaknesses
  1. The theoretical analysis only provides a very loose upper bound on distributional deviation, offering limited practical guidance. It also fails to explain why the proposed method is preferable to existing approaches, for example, whether it achieves a smaller total variation distance (TV distance) under comparable conditions. Moreover, the assumption that ϵ≤10−3\epsilon \leq 10^{-3} is not theoretically justified or empirically validated.

  2. The mathematical reasoning accuracy and code generation pass@1 reported in the experiments do not directly measure the distributional deviation introduced by relaxed verification. Therefore, these metrics are insufficient to assess how closely the proposed method preserves the target model's output distribution.

Questions for authors

Could the authors clarify the advantages of NSD over other relaxed verification methods discussed in the Related Work section?

Reviewer_cpQrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Nucleus Speculative Decoding (NSD), which additionally accepts draft tokens within the target model’s nucleus to improve decoding efficiency. It analyzes the resulting distributional deviation and provides conditional sequence-level bounds. Experiments across Nemotron and Qwen models on math and coding benchmarks report 4.23× speedup over autoregressive decoding and 1.85× over standard speculative decoding, with broadly comparable task performance.

Critical strengths
  1. Simple and broadly applicable design. NSD modifies only the acceptance rule and requires no additional training. Reported speedups across multiple models and proposal mechanisms suggest practical value as an easily integrated decoding improvement.
  2. Exact single-step characterization. Under the stated assumptions, the paper proves that additional acceptance probability equals the resulting total variation deviation. This provides a precise account of the distributional cost of relaxing verification.
Critical weaknesses
  1. Limited demonstrated novelty. Plausibility-aware verification, top-k verification, and lenient acceptance have been explored previously. There is currently a lack of direct comparisons with these alternatives, and this method appears to involve only minor modifications to existing methods. Direct comparisons with these alternatives are missing, leaving the benefit of nucleus-based acceptance insufficiently established.
  2. Unclear benefits on resource-constrained devices. Experiments are conducted on NVIDIA Pro5000 GPUs, leaving the gains on edge devices unquantified. Since speculative decoding speedups depend on memory bandwidth, available parallelism, and drafting overhead, comparable benefits cannot be assumed on constrained hardware. Reporting end-to-end latency and memory consumption on representative edge devices would better establish the method’s deployment value.

Reference: Cai, Tianle, et al. "Medusa: Simple llm inference acceleration framework with multiple decoding heads." arXiv preprint arXiv:2401.10774 (2024).

Bachmann, Gregor, et al. "Judge decoding: Faster speculative sampling requires going beyond model alignment." International Conference on Learning Representations. Vol. 2025. 2025.

Leviathan, Yaniv, Matan Kalman, and Yossi Matias. "Fast inference from transformers via speculative decoding." International conference on machine learning. PMLR, 2023.

Questions for authors
  1. Comparison with simpler acceptance rules: How does NSD compare with top-k acceptance and other simple threshold rules under the same backend and matched task performance? This is my main concern. Convincing evidence that nucleus-based acceptance consistently offers a better speed–quality trade-off would materially strengthen the contribution and could raise my rating.
  2. Deployment clarification: What batch size, numerical precision, and memory footprint underlie the reported speedups? Is there existing evidence that the gains extend to resource-constrained edge devices, or should the conclusions be limited to the tested GPU configuration?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.