acceptodds
Under review as a conference paper at ICLR 2027

The Arithmetic Error Bar: A First-Passage Law for the Numerical Fragility of Language Models

Abstract

A benchmark score carries an uncertainty nobody reports: change the arithmetic that produced it — a numerical precision, a machine — and the number moves. On a 1000-question set the aggregate accuracy is unchanged, 66.2% against 66.1%, while 53 answers differ and 29 items change verdict; the cancellation that keeps the total still is an accident, and on a second machine of a different architecture the same kind of flips cost a full point. We call the resulting band the arithmetic error bar — here ±1.06 points — and we show that it is predictable in advance, item by item, from a single run, and that it can be paid down at a stated compute price. The mechanism is not the one usually assumed. Chaos predicts that reducing a pertur- bation makes the divergence arrive later; measurement shows that it removes it, and a state perturbation decays by an order of magnitude within ten tokens and never grows. Generation is a hybrid system — a contracting flow interrupted by a quantisation — and all of its sensitivity sits in the quantisation. The distance of a decision from its boundary is the geometric margin ρj = (ℓ1−ℓj)/∥w1−wj∥; normalised by the state norm it defines a threshold landscape along a generation, and the probability that the text diverges is a first-passage problem whose only parameter is the amplitude of the numerical noise, measured rather than fitted. It predicts which answers change (AUROC 0.960 against 0.797 for entropy, on held- out questions), places all 29 observed correctness flips in the half of the items it had flagged, and predicts 43 changed answers against the 53 observed; car- ried unchanged to the second machine it predicts exact agreement in float32, confirmed on all 1500 questions, and 71 divergences in bfloat16 against 66 observed. Run on two further models the law ranks well on one and not on the other. Rather than record this as a limitation we bound it: if the law holds, many questions are genuine coin tosses that no score can separate, so achievable rank- ing accuracy has a ceiling computable from the landscape and the amplitude alone, before any comparison of precisions is run. It depends on essentially one number, how often the arithmetic decides the answer, and against it all three models land at or near the best any score could achieve. Finally, re-running in float32 only the tenth of a corpus the law flags recovers 72.4% of the correctness flips and halves the error bar, with the residual predicted at every budget.

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic Interpretability, First-Passage Problem, Stability Radius, Unembedding Geometry, Autoregressive Generation, Numerical Precision Sensitivity

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_gpgNOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors study what they call the "Arithmetic Error Bar", which to my understanding is supposed to measure a degree of variance that is usually ignored in reporting accuracy on benchmarks. They argue that simple accuracy scores obscure potential flipped answers, and that their approach is a means to account for this variation. That running the same model with greedy decoding on different computational architectures has variation is an important fact, and it is interesting to note that some of the uncertainty in language models are likely due to the hardware, and not the model per se. They study three small language models and find that their measurement holds for 2 out of 3 models. They claim that the mechanism of this numerical variation is determined by a discrete threshold.

Critical strengths

The object of study is of interest with respect to the reliability of models. AI systems are often discussed without taking the underlying hardware into account, and the paper thus points to an interesting question. However, it could be clearer that this is the actual focus of the paper. I find the paper very hard to parse, and therefore struggle to single out the actual contribution and potential implications of the findings. This could potentially be alleviated by a general re-writing of the paper and general clarifications that clearly explain the methodology and implications / contributions of the work.

Critical weaknesses
  1. The paper is generally very hard to parse.
  2. It is not exactly clear what the contribution is, and as a non-expert reader there is no background on the theory of which the work is based.
  3. While this is also mentioned in the limitations the approach is tested on three models wherein it applies to two of them (that are only tested on CPUs).
Questions for authors
  1. Can you clarify the key implications of your work?
  2. How would this apply if models were run on GPUs? Almost all inference is done on GPUs, which begs the question of whether the finding also holds there.
  3. What questions are the models being evaluated on, and how is the evaluation performed?
Reviewer_ZGEZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper uses geometric margins and a measured noise amplitude to predict output changes under different numerical precisions or hardware. It further discusses score uncertainty, prediction ceilings, and selective high-precision rerunning.

Critical strengths
  1. The topic is novel, and the focus on individual answers is an interesting angle. Stable aggregate accuracy can hide changes in the underlying predictions. Identifying these changes in advance has practical value.

  2. The geometric interpretation is intuitive. The reference-model and cross-machine results are promising.

Critical weaknesses
  1. The scope is broad, and some claims extend beyond the main experimental evidence. The paper addresses risk prediction, generation dynamics, error bars, prediction ceilings, and rerun policies. These questions are connected, but they require different forms of validation. The strongest evidence concerns item-level risk prediction in the tested settings. The experiments use small models on CPU and mainly short answers. They provide a limited basis for general claims about generation dynamics. The error-bar interpretation also needs a clearer link between predicted output changes and changes in correctness. A more focused presentation would make the contribution easier to assess.

  2. The added value over a simple logit-gap measure remains unclear. In the rerun experiment, the minimum logit gap achieves the same recovery rate as fragility. The geometric model may offer better probability estimates, but this needs a direct comparison. The practical result also relies on float32 reference scores. Selection from the bfloat16 run is assessed through simulated perturbations rather than a direct evaluation. These gaps make it difficult to judge the method’s practical advantage.

Questions for authors
  1. The main experiments focus on small models and short answers on CPU. How do these results support the broader claim that numerical fragility comes from threshold crossings rather than chaotic amplification?
  2. In what concrete evaluation or deployment scenario would the proposed method offer a practical advantage over simple logit-gap selection?
  3. The results vary across models. Is there evidence from longer generations or GPU inference that the method’s predictive performance remains reliable?
Reviewer_19qLOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the numerical stability of language model inference. It shows that individual predictions can depend on numerical precision or hardware, despite similar aggregate benchmark accuracy.

Critical strengths

None

Critical weaknesses
  1. Very poorly written and difficult to follow.

  2. Limited evaluation with 0.5B and 1.7B models on CPU, making it unclear how the findings can be generalized to other models.

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.