acceptodds
Withdrawn from ICLR 2027

Lossless, Yet Not Identical: Execution-Level Numerical Bifurcations in Speculative Decoding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Speculative decoding is lossless at the level of the decoding rule, but this guarantee does not require different finite-precision executions to produce identical decisions. We study this distinction in greedy speculative decoding and focus on an attribution problem: when speculative and autoregressive generation disagree, does the disagreement arise specifically under speculative verification, or is the same state already sensitive to the autoregressive execution path? We address this question with a three-way diagnostic that compares incremental autoregressive execution (), full-prefix autoregressive execution (), and speculative verification () at the same token prefix. We treat as verifier-specific disagreement relative to these two matched autoregressive controls. Among 555 reconstructed first-divergence events in Qwen3, 166 satisfy this criterion, while 389 are already autoregressive-path-sensitive. Across Qwen3, SmolLM2, and Pythia, all 1,850 reconstructed BF16 divergence events collapse to under backend-matched FP32 replay, which we use only as a diagnostic control. The observed bifurcations are strongly enriched near low-margin decisions, but their prevalence also varies with execution shape and backend. The results therefore do not support a universal margin threshold or block-size law. A targeted replication on A100/Ampere again identifies the same event class at both tested block sizes, and all 145 reconstructed events collapse under matched FP32 replay. The verifier-specific fraction is nevertheless lower than on RTX4090/Ada, showing that the event class persists across the two tested GPU architectures while its observed magnitude can differ. We further use Selective Canonical Re-verification as a mechanism-grounded intervention. Changing only the execution realization recovers most lost execution identity in most tested settings, while high-margin residuals expose the limits of margin-only selection and direct GPU timing quantifies the cost of the present implementation. These results separate two notions of exactness: losslessness constrains the decoder, whereas execution identity also depends on how that decoder is realized under finite precision.

Primary area:
foundation or frontier models, including LLMs
Keywords:
speculative decoding, large language model inference, numerical precision, deterministic inference, execution-level equivalence

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_bsxtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

In principle, running an LLM with greedy decoding always produces the same output. This paper raises the issue that greedy Speculative Decoding (SD) can produce different outputs in low-precision settings (FP16/BF16) due to floating-point errors. To determine whether these discrepancies originate from SD, the authors propose a Three-way Diagnostic and Selective Canonical Re-verification (SCR).

The Three-way Diagnostic compares two autoregressive (AR) execution variants and SD using the same prefix. Agreement between the two AR controls helps determine whether a discrepancy can be attributed to SD. The paper observes such discrepancies across multiple models and two GPU environments, with patterns varying by precision, decision margin, block size, and backend.

SCR recomputes selected verification decisions through the AR execution path and can restore nearly identical outputs. However, its effectiveness varies substantially across models and backends, limiting its general applicability. Nevertheless, SCR serves as an experimental intervention for examining the possibility and limits of restoring identical outputs across execution paths.

Critical strengths

S1. Differences caused by floating-point errors are difficult to disentangle and attribute to specific causes. By introducing an additional AR execution variant, the authors make SD–AR discrepancies easier to diagnose. Although SCR has limited practical applicability, it demonstrates the possibility of correcting generation outputs.

Critical weaknesses

W1. Limited contribution

It is already known that SD may fail to preserve output consistency in low-precision settings. The main contribution of this work is a more detailed classification of discrepancies using two AR execution paths.

To strengthen this contribution, the paper should explain why SD selects a different token even when the two AR paths agree. Potential causes include differences in computation shape, cache construction, kernel dispatch, and operation order.

Controlling these execution factors and identifying which changes alter the selected token would make the contribution clearer.

W2. Insufficient investigation of execution factors

Although the paper studies a phenomenon that is highly sensitive to execution details, its experimental settings are limited. What happens with large batches, long generations, or sampling with a fixed seed? Why was a setting without KV cache reuse chosen? Do the same patterns appear in SD implementations that retain the KV cache?

For example, AR processes one token at a time, whereas SD performs prefill operations. An additional analysis could measure how this difference affects numerical errors in the KV cache and the hidden states at each layer, then investigate whether error accumulation is driven by batch size or other computational characteristics.

W3. Effectiveness of SCR

The SCR results need a deeper analysis of why the observed behavior occurs. For example, examining whether AR and SD have small or large decision margins—including cases where both have large margins—could help identify how to improve the selector.

W4. Presentation

The presentation needs improvement independently of the issues above. Instead of repeating statements, the paper should clearly distinguish what the proposed method reveals, what it still cannot explain, and the empirical findings it establishes. Some symbols and terms are used before they are defined. Citation formatting should also be consistent.

Questions for authors

Q1. In a representative case where A = B ≠ C, which numerical differences in intermediate computations cause the final token choice to change?

Q2. For the high-margin residual cases in Pythia/eager, how do A and C differ in their decision margins and top-token rankings? Can this analysis explain when a margin-only selector fails?

Q3. Does SCR maintain its recovery performance with KV cache reuse and long generations?

Reviewer_FFYKOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper argues that even though speculative decoding can guarantee theoretical mathematical equivalence during decoding in practice because of grouped operations realized on real hardware floating point inconsistencies might lead the model to incorrect paths all-together. This is an interesting hypothesis and the authors have validated that such divergence does occur. The authors define 3 scenarios of decoding as follows: A - Incremental autoregressive generation using a KV cache B - Autoregressive evaluation that recomputes the full prefix C - Speculative block verification The authors look at the first differing token, before different histories would affect the comparison. They identify 2 possible conditions when this might happen: A = B ≠ C: both ordinary executions agree, but speculative verification differs. They call this verifier-specific disagreement. A ≠ B: ordinary generation is already sensitive to execution path. The disagreement cannot be attributed specifically to speculative verification using these controls.

In the Qwen3 analysis, the authors identify the approximately 30% are verifier specific disagreements. Next the authors introduce SCR to identify candidates where speculative verification produces a small-margin decision, and reverify ordinary AR verifier forward.

This approach basically identifies failure modes of Speculative Decoding and uses extra compute to solve the mismatches by running the main model in regular AR mode. Overall, identification of the problem is good but under very constrained settings. The reported results as of now are bare minimum and we need more circumstantial evidence to show that the problem affects accuracy.

Critical strengths
  1. Good problem identification
  2. Good demonstration of failure modes across smaller size models
Critical weaknesses
  1. Proposed solution doesn't seem practical. Conducting verifier decode using SCR for the 13% cases identified already needs around 70% extra compute. Additionally, the decode time is a function of block size, gemm kernels etc used which are typically highly optimized for number of verification tokens passed to the verifier at each step. Running an extra step using the same kernels might regress GPU performance significantly.
  2. The experiments use greedy decoding, relatively small models, batch size one, and short generations.
  3. They do not establish effects on accuracy or generalization to stochastic sampling.
Questions for authors
  1. Have you tested it on more recent models?
  2. You need to verify the method using stochastic sampling and report actual difference in accuracy.
  3. What decoding kernels have been used for the verifier?
Reviewer_Rni8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates an important research question of execution-level numerical bifurcations in speculative decoding. This means different execution paths can produce different token decisions despite algorithmic equivalence. It proposes a three-way attribution analysis to distinguish verifier-specific divergences from those arising across autoregressive execution paths. The experiment show such divergences across three model families (Qwen3, SmolLM2, and Pythia) and two GPU architectures (A100/Ampere, RTX4090/Ada), and find that the identified disagreements disappear under matched FP32 replay, and show that selective canonical re-verification can recover agreement with varying effectiveness but incurs substantial runtime overhead.

However, the connection to prior work on batch invariance and execution-dependent numerical differences is insufficiently discussed, making the paper’s novelty and contribution unclear or incremental. The empirical evaluation is narrow, focusing on batch size 1 and Hugging Face Transformers without KV-cache reuse across speculative blocks, limiting the evidence that the findings generalize to optimized inference engines such as vLLM and SGLang. Moreover, the paper lacks kernel-level analysis to explain the root causes of token divergences and provides insufficient evidence about their impact on downstream task performance or the effectiveness of the proposed mitigation. Overall, despite the importance of the research question, the limited technical insight, narrow evaluation, and unclear practical implications make the contribution insufficiently compelling for acceptance.

Critical strengths

The paper investigates an important and useful question: numerical bifurcations in speculative decoding. Connecting the analysis to real workloads in vLLM or SGLang could help developers build and debug production inference systems.

Critical weaknesses

The relation to prior batch-invariance work could be clearer. The paper does not cite He and Thinking Machines Lab (2025), which explains numerical differences across cached, full-prefix, and chunked execution. Discussing the connection would help clarify what the paper's attribution adds.

Limited evaluation on optimized inference engines. The experiments use Hugging Face Transformers at batch size 1, without KV-cache reuse across speculative blocks. Since the observed effects depend on execution shape and backend, evaluation on vLLM or SGLang with cache reuse and larger batches would help establish how well the findings transfer to real world inference engines.

Limited kernel-level analysis. The SDPA/eager comparison does not identify which operations change the token decision. vLLM traced batch-dependent outputs to cuBLAS matmul dispatch, while SGLang pins FA3's num_splits=1 for determinism. Tracing one or two token flips to specific kernel choices and checking whether controlling those choices restores agreement would help explain the root cause.

I find the research question important, but the limited explanation of the bifurcations and the narrow BS=1 Transformers evaluation provide too little new insight or evidence of practical relevance to support acceptance.

Questions for authors

Do the token divergences affect GSM8K answer accuracy or HumanEval pass@1, and does SCR change these scores?

Do you have evidence or hypotheses about which execution differences account for the RTX 4090/A100 gap in verifier-specific event fractions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.