- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
In principle, running an LLM with greedy decoding always produces the same output. This paper raises the issue that greedy Speculative Decoding (SD) can produce different outputs in low-precision settings (FP16/BF16) due to floating-point errors. To determine whether these discrepancies originate from SD, the authors propose a Three-way Diagnostic and Selective Canonical Re-verification (SCR).
The Three-way Diagnostic compares two autoregressive (AR) execution variants and SD using the same prefix. Agreement between the two AR controls helps determine whether a discrepancy can be attributed to SD. The paper observes such discrepancies across multiple models and two GPU environments, with patterns varying by precision, decision margin, block size, and backend.
SCR recomputes selected verification decisions through the AR execution path and can restore nearly identical outputs. However, its effectiveness varies substantially across models and backends, limiting its general applicability. Nevertheless, SCR serves as an experimental intervention for examining the possibility and limits of restoring identical outputs across execution paths.
Critical strengths
S1. Differences caused by floating-point errors are difficult to disentangle and attribute to specific causes. By introducing an additional AR execution variant, the authors make SD–AR discrepancies easier to diagnose. Although SCR has limited practical applicability, it demonstrates the possibility of correcting generation outputs.
Critical weaknesses
W1. Limited contribution
It is already known that SD may fail to preserve output consistency in low-precision settings. The main contribution of this work is a more detailed classification of discrepancies using two AR execution paths.
To strengthen this contribution, the paper should explain why SD selects a different token even when the two AR paths agree. Potential causes include differences in computation shape, cache construction, kernel dispatch, and operation order.
Controlling these execution factors and identifying which changes alter the selected token would make the contribution clearer.
W2. Insufficient investigation of execution factors
Although the paper studies a phenomenon that is highly sensitive to execution details, its experimental settings are limited. What happens with large batches, long generations, or sampling with a fixed seed? Why was a setting without KV cache reuse chosen? Do the same patterns appear in SD implementations that retain the KV cache?
For example, AR processes one token at a time, whereas SD performs prefill operations. An additional analysis could measure how this difference affects numerical errors in the KV cache and the hidden states at each layer, then investigate whether error accumulation is driven by batch size or other computational characteristics.
W3. Effectiveness of SCR
The SCR results need a deeper analysis of why the observed behavior occurs. For example, examining whether AR and SD have small or large decision margins—including cases where both have large margins—could help identify how to improve the selector.
W4. Presentation
The presentation needs improvement independently of the issues above. Instead of repeating statements, the paper should clearly distinguish what the proposed method reveals, what it still cannot explain, and the empirical findings it establishes. Some symbols and terms are used before they are defined. Citation formatting should also be consistent.
Questions for authors
Q1. In a representative case where A = B ≠ C, which numerical differences in intermediate computations cause the final token choice to change?
Q2. For the high-margin residual cases in Pythia/eager, how do A and C differ in their decision margins and top-token rankings? Can this analysis explain when a margin-only selector fails?
Q3. Does SCR maintain its recovery performance with KV cache reuse and long generations?