- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper asks whether quantization changes batch-context non-determinism in greedy LLM inference. Prompts, checkpoints and decoding are held fixed. Only the vLLM batch cap (8 or 32) and a seeded request order (three seeds per cap) are varied, and answer disagreement is measured over all 15 pairs of the resulting six serving contexts.
On Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct, across seven benchmarks on L40S GPUs, FP8 W8A8 raises disagreement from 8.7% to 17.2% for Qwen and from 13.9% to 25.3% for Llama. GPTQ W8A16 stays within about one point of BF16, and an FP8 KV cache alone raises Llama to 18.2%. Example-clustered bootstrap intervals are reported throughout.
For the mechanism, the authors show three things: first divergences occur at near-tied decoding steps; vLLM's batch-invariant mode gives exact alone-vs-batch agreement for BF16 and FP8 but not GPTQ; and the FP8 excess disappears on an A10G, where the checkpoint runs without FP8 activations. Extensions cover five further models, an H100 run, selective BF16-attention mitigations and calibration sensitivity
Critical strengths
[S1] The central finding with FP8 quantization is carefully controlled and robust. The FP8-over-BF16 gap has tight intervals. The regime ordering holds in each pair category, under three parse-failure treatments, on every benchmark, and in four extension models. It also holds on parser-free measures: token-sequence divergence and alone-vs-batch exact matches.
[S2] The batch-invariant intervention gives the paper a causal handle rather than only a correlation. With the mode enabled, BF16 and FP8 reach exact matches while GPTQ stays only at partial matches. The throughput cost is measured under a matched control, which makes this a concrete, actionable result for FP8 deployments.
Critical weaknesses
[W1] The paper establishes that FP8 W8A8 increases batch-context sensitivity, but not why, and it generalizes the GPTQ result beyond what was tested.
On the FP8 quantization side, the evidence is consistent with activation quantization amplifying small batch-dependent perturbations that originate in generic operations: the batch-invariant mode reportedly contains no quantization-specific override yet makes FP8 exact, and on the A10G the same checkpoint without FP8 activations loses the excess. However, the paper never separates a larger source perturbation from stronger amplification. Typical margins are not systematically smaller under FP8, the size of the batch-induced logit perturbation is never measured per regime, and the A10G control changes the kernel and activation quantization together.
For the expreiments with GPTQ, the "close to BF16" and "not fixed by batch invariance" results come from one checkpoint family (llmcompressor, per-channel) on one kernel. App. N calls the latter "a statement about the AllSpark path" and notes that the authors did not inspect its source. Mistral-Nemo GPTQ is an exception (19.8% vs. 13.8%).
The Section 8 recommendations therefore overreach. "Take GPTQ W8A16, which is the more stable and the faster" rests on Table 4 medians over uncontrolled runs pooled across benchmarks: for Llama, GPTQ runs at 475 tok/s vs. 465 for FP8, and BF16 at 624 is faster than both. "GPTQ currently has no batch-invariant solution" was tested on a single kernel.
[W2] The experimental environment and the answer-level metric are not documented tightly enough to verify the reported magnitudes.
App. R states that the primary matrix mixes vLLM 0.9.2 / PyTorch 2.7 / CUDA 12.6 with vLLM 0.20.1 / PyTorch 2.11 / CUDA 13.0, without saying which cells ran where. This matters because App. D shows that a change of campaign alone moves single-request answers by 13–25 points, comparable to the effect under study. The homogeneity across pair categories in Table 7 suggests each cell's six contexts shared one stack, but the cross-regime comparison needs this stated.
The released parser compares the raw "Final answer:" line as a string, with no lower-casing or numeric/algebraic extraction, contrary to App. E. Since 54.3% of changed answers also flip correctness, some of the remaining free-form "disagreements" may be formatting variants. The direction is not at risk given the multiple-choice rows of Table 12 and Table 13.
The released code covers the main pipeline but not the batch-invariant, padding, throughput or margin experiments. There is also no accuracy-scoring code and no accuracy table, so the score-drift claims and the McNemar statements cannot be checked.
Questions for authors
Clarifications:
- For each (model, regime, benchmark), which vLLM/PyTorch/CUDA stack produced the six contexts? Were BF16 and the quantized regimes of a model always run on the same stack? Could you report Table 1 restricted to one stack? Please also state whether chunked prefill and prefix caching were enabled in the primary matrix: the script uses vLLM defaults, while the throughput control explicitly disabled prefix caching.
- Under the batch-invariant mode, are FP8 linear layers routed through an overridden operation, or is the FP8 scaled matmul already batch-invariant? A teacher-forced, per-regime comparison of logit differences between alone and batched runs on identical prefixes would show whether FP8 enlarges the perturbation or amplifies the same upstream perturbation.
- Llama FP8 retains 3/15 BoolQ pairs and 10/15 DROP pairs. Do these reflect entire contexts with no parsable answers, or runs that were missing or excluded? What exclusion criterion was applied (App. L)?
- Could you report answer disagreement on the multiple-choice benchmarks only, or with equivalence normalization for MATH500/GSM8K/DROP, and reconcile the released parser with App. E?
- Could you provide per-regime, per-context accuracy tables and the scoring code behind Sec. 4.2 and App. Q?
Suggestions (these would strengthen the paper but are not required for my assessment): 6. Evaluate GPTQ W8A16 on a second kernel or checkpoint family, with and without the batch-invariant mode, before recommending it as the stable eight-bit choice. 7. Discuss He and Thinking Machines Lab (2025), "Defeating Nondeterminism in LLM Inference", which introduced batch-invariant kernels, in the related work. The paper currently cites only the vLLM documentation for batch invariance. This does not affect the novelty of the quantization question.