acceptodds
Under review as a conference paper at ICLR 2027

Quantization Modulates Batch-Context Non-Determinism in Greedy LLM Inference

Abstract

Quantization reduces the memory and compute cost of LLM inference, and requests are usually batched to serve many of them at once. However, recent studies have shown that batching causes non-determinism in LLM inference despite the use of greedy decoding. We study whether quantization changes this batch-context non-determinism, holding prompts, decoding settings and checkpoints fixed. For that we evaluate primarily Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct across seven benchmarks and six serving contexts using BF16, GPTQ, FP8 weight-and-activation quantization, and FP8 KV-cache quantization. We observe answer disagreement across every pair of our serving contexts. FP8 weight-and-activation quantization roughly doubles how often an example changes its answer, from 8.7% to 17.2% for Qwen2.5-7B-Instruct and from 13.9% to 25.3% for Llama-3.1-8B-Instruct, while weight-only GPTQ stays close to BF16. Quantizing only the KV cache also raises the rate, and smaller evaluations on five further models show the same pattern. We trace the cause to the order in which floating-point reductions are carried out, which depends on batch composition. When we use batch-invariant kernels, the non-determinism effect vanishes for BF16 and FP8 at a throughput loss, but it remains for GPTQ.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
quantization, LLM inference, reproducibility, non-determinism, FP8, GPTQ, batch invariance, evaluation, decoding

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_yRRyOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether quantization changes batch-context non-determinism in greedy LLM inference. Prompts, checkpoints and decoding are held fixed. Only the vLLM batch cap (8 or 32) and a seeded request order (three seeds per cap) are varied, and answer disagreement is measured over all 15 pairs of the resulting six serving contexts.

On Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct, across seven benchmarks on L40S GPUs, FP8 W8A8 raises disagreement from 8.7% to 17.2% for Qwen and from 13.9% to 25.3% for Llama. GPTQ W8A16 stays within about one point of BF16, and an FP8 KV cache alone raises Llama to 18.2%. Example-clustered bootstrap intervals are reported throughout.

For the mechanism, the authors show three things: first divergences occur at near-tied decoding steps; vLLM's batch-invariant mode gives exact alone-vs-batch agreement for BF16 and FP8 but not GPTQ; and the FP8 excess disappears on an A10G, where the checkpoint runs without FP8 activations. Extensions cover five further models, an H100 run, selective BF16-attention mitigations and calibration sensitivity

Critical strengths

[S1] The central finding with FP8 quantization is carefully controlled and robust. The FP8-over-BF16 gap has tight intervals. The regime ordering holds in each pair category, under three parse-failure treatments, on every benchmark, and in four extension models. It also holds on parser-free measures: token-sequence divergence and alone-vs-batch exact matches.

[S2] The batch-invariant intervention gives the paper a causal handle rather than only a correlation. With the mode enabled, BF16 and FP8 reach exact matches while GPTQ stays only at partial matches. The throughput cost is measured under a matched control, which makes this a concrete, actionable result for FP8 deployments.

Critical weaknesses

[W1] The paper establishes that FP8 W8A8 increases batch-context sensitivity, but not why, and it generalizes the GPTQ result beyond what was tested.

On the FP8 quantization side, the evidence is consistent with activation quantization amplifying small batch-dependent perturbations that originate in generic operations: the batch-invariant mode reportedly contains no quantization-specific override yet makes FP8 exact, and on the A10G the same checkpoint without FP8 activations loses the excess. However, the paper never separates a larger source perturbation from stronger amplification. Typical margins are not systematically smaller under FP8, the size of the batch-induced logit perturbation is never measured per regime, and the A10G control changes the kernel and activation quantization together.

For the expreiments with GPTQ, the "close to BF16" and "not fixed by batch invariance" results come from one checkpoint family (llmcompressor, per-channel) on one kernel. App. N calls the latter "a statement about the AllSpark path" and notes that the authors did not inspect its source. Mistral-Nemo GPTQ is an exception (19.8% vs. 13.8%).

The Section 8 recommendations therefore overreach. "Take GPTQ W8A16, which is the more stable and the faster" rests on Table 4 medians over uncontrolled runs pooled across benchmarks: for Llama, GPTQ runs at 475 tok/s vs. 465 for FP8, and BF16 at 624 is faster than both. "GPTQ currently has no batch-invariant solution" was tested on a single kernel.

[W2] The experimental environment and the answer-level metric are not documented tightly enough to verify the reported magnitudes.

App. R states that the primary matrix mixes vLLM 0.9.2 / PyTorch 2.7 / CUDA 12.6 with vLLM 0.20.1 / PyTorch 2.11 / CUDA 13.0, without saying which cells ran where. This matters because App. D shows that a change of campaign alone moves single-request answers by 13–25 points, comparable to the effect under study. The homogeneity across pair categories in Table 7 suggests each cell's six contexts shared one stack, but the cross-regime comparison needs this stated.

The released parser compares the raw "Final answer:" line as a string, with no lower-casing or numeric/algebraic extraction, contrary to App. E. Since 54.3% of changed answers also flip correctness, some of the remaining free-form "disagreements" may be formatting variants. The direction is not at risk given the multiple-choice rows of Table 12 and Table 13.

The released code covers the main pipeline but not the batch-invariant, padding, throughput or margin experiments. There is also no accuracy-scoring code and no accuracy table, so the score-drift claims and the McNemar statements cannot be checked.

Questions for authors

Clarifications:

  1. For each (model, regime, benchmark), which vLLM/PyTorch/CUDA stack produced the six contexts? Were BF16 and the quantized regimes of a model always run on the same stack? Could you report Table 1 restricted to one stack? Please also state whether chunked prefill and prefix caching were enabled in the primary matrix: the script uses vLLM defaults, while the throughput control explicitly disabled prefix caching.
  2. Under the batch-invariant mode, are FP8 linear layers routed through an overridden operation, or is the FP8 scaled matmul already batch-invariant? A teacher-forced, per-regime comparison of logit differences between alone and batched runs on identical prefixes would show whether FP8 enlarges the perturbation or amplifies the same upstream perturbation.
  3. Llama FP8 retains 3/15 BoolQ pairs and 10/15 DROP pairs. Do these reflect entire contexts with no parsable answers, or runs that were missing or excluded? What exclusion criterion was applied (App. L)?
  4. Could you report answer disagreement on the multiple-choice benchmarks only, or with equivalence normalization for MATH500/GSM8K/DROP, and reconcile the released parser with App. E?
  5. Could you provide per-regime, per-context accuracy tables and the scoring code behind Sec. 4.2 and App. Q?

Suggestions (these would strengthen the paper but are not required for my assessment): 6. Evaluate GPTQ W8A16 on a second kernel or checkpoint family, with and without the batch-invariant mode, before recommending it as the stable eight-bit choice. 7. Discuss He and Thinking Machines Lab (2025), "Defeating Nondeterminism in LLM Inference", which introduced batch-invariant kernels, in the related work. The paper currently cites only the vLLM documentation for batch invariance. This does not affect the novelty of the quantization question.

Reviewer_Kwc6OpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the the LLM inference determinism under different quant approaches and investigate the relationship between the non-determinism and batching context. Authors identify the general trend the designed ablations to isolate different confounders, which provides a guide towards deterministic LLM inference.

Critical strengths
  1. This work targets on a problem sounds minor but very critical to LLM development and iteration. I understand sometimes users do not care about this too much in most cases because users' query would anyway vary a lot. However, for scientific development/iteration, the deterministic eval is very critical to ensure the experimental conclusion is accurate and reproducible.
  2. The ablation design in this work is also rigid and sound. For example, in Section 5.3, authors tried to design ablations to verify the root cause is padding tokens or not.
Critical weaknesses
  1. I think one major thing can be improved is, authors shall consider run one single model family but at different scales to see how is the findings in this work change with dynamic. For example, on one or multiple benchmarks, does larger Qwen models more resistant to quant or batching factors? Does the negative effect diverge? And for your solutions, does it still work well at scale? Is the gain diminishing or increasing?
Questions for authors

See weakness

Reviewer_gxaXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper evaluates how quantization affects batch-dependent variability in greedy LLM inference across models, benchmarks, and serving contexts. FP8 W8A8 generally increases answer disagreement relative to BF16, while weight-only GPTQ remains closer to BF16. The authors examine token divergences and find that existing batch-invariant kernels eliminate observed variability for BF16 and FP8, but not the tested GPTQ path.

Critical strengths
  1. Practical relevance. The study highlights reproducibility differences between quantization schemes that average benchmark accuracy can obscure, with measurements across multiple models and GPUs.

  2. Useful controls. Fixed-engine batch-invariance tests, alternative parse-failure treatments, and calibration experiments strengthen the empirical observations and reveal limitations of the tested GPTQ execution path.

Critical weaknesses
  1. Limited novelty, mechanistic insight, and mitigation. The paper reads primarily as an experimental benchmarking report. Batch-dependent numerical nondeterminism and batch-invariant kernels are already established. The analysis does not isolate why FP8 amplifies variability, since activation precision, scaling, and kernel dispatch change together. The mitigations rely on an existing vLLM mode or selective mixed precision with inconsistent or statistically inconclusive benefits. The contribution therefore remains largely comparative measurements, without sufficiently new explanatory insight or a broadly effective new solution.

  2. Unresolved experimental confounding. Despite Section 3's claim that only batch cap and request order vary, Appendix R reports two software stacks in the primary matrix without mapping them to individual comparisons. Appendix D also reports substantial, unexplained disagreement across single-request campaigns. Without within-stack results and identical-context repeat controls, it is unclear how much of the headline disagreement is attributable specifically to batch context. The separate fixed-engine tests establish batch sensitivity, but do not resolve the provenance of the main effect sizes.

Questions for authors
  1. Experimental controls: Can you clarify how software stacks and experimental campaigns are assigned across the primary comparisons, and show whether the main findings persist under a fixed stack with identical-context repeat controls?

  2. Mechanistic insights and mitigation: Beyond reporting disagreement rates, can you explain why FP8 is more sensitive than weight-only GPTQ, distinguishing activation quantization, scaling, and kernel effects? Can these insights motivate a targeted mitigation with consistent benefits beyond the existing batch-invariant mode and exploratory mixed-precision results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.