acceptodds
Under review as a conference paper at ICLR 2027

SNAP: Error Bars on Benchmark Averages Can Miss Run Covariance

Abstract

Error bars on a language model's benchmark average usually sum per-benchmark variances and ignore the covariance between benchmarks across training runs. On 375 DataDecide runs, the standard deviation of the ten-benchmark average in per-byte margin is 1.244 [1.143, 1.338] times its value under independence, while accuracy gives 1.078 with an interval that includes one. A margin comparison that assumes independence at a nominal 0.05 therefore has a true size near 0.11. That covariance sits within scoring formats, where BoolQ, alone in its format, carries 68% of the margin covariance trace. SNAP, a cross-half estimator, also gives per-benchmark run variances net of item noise, which the replicate standard deviation can't separate when item noise is large. A correction for the covariance changes few of DataDecide's recipe orderings. A pre-specified test on four held-out tasks fails at 1.217, with a lower limit below one. A second pre-specified rule also fails, since nine seeds at five PolyPythias sizes give margin inflation of 1.302 with an interval [0.921, 1.595] that includes one. A disjoint item bank leaves the source of that inflation undecided. All other analyses in the paper are exploratory. We recommend that authors report the replicate standard deviation of the battery average with a benchmark-removal check, and that standard deviation needs no item split and gives an inflation of 1.237 here.

Primary area:
datasets and benchmarks
Keywords:
benchmark evaluation, uncertainty quantification, error bars, run-to-run variance, random seeds, covariance estimation, split-half reliability, language models, reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_1YaFOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates how covariance between benchmark scores across training runs affects uncertainty. It proposes a method, SNAP, that estimates cross-benchmark covariance across various training runs to separate item-specific variation from covariance that generalizes across questions. It shows, through various experiments, that: (1) the benchmarks included in the estimation can have significant impact, and (2) independence assumptions can change the standard deviations of the average margin scores by 24%. The authors recommend to report standard deviation of benchmark averages across training replicas and repeat the calculation after removing each benchmark.

Critical strengths

None

Critical weaknesses

This paper has limited novelty, weak demonstrated benefit, and its presentation is difficult to follow because many paper-specific terms remain undefined at the moment of their introduction. I list the two critical weaknesses below.

1. Poor writing
By far the weakest point of the paper is its poor writing. The paper contains many undefined terms that are not explained at the time of their introduction and includes many numbers that are very hard to interpret, but still left to the reader to interpret. For instance, these are all undefined terms in the abstract:

  • DataDecide/PolyPythias: niche training runs, appears sudden and are unexplained.
  • per-byte margin: author-defined term that is only explained in section 2, but carries no meaning for a reader at this point.
  • margin comparison that assumes independence: not explained what kind of independence is assumed, or what a margin comparison is
  • 0.11: not explained what practical meaning this value has
  • alone in its format: not explained what it means for BoolQ to be "alone in its format", a reader has to wait until L273 where it is mentioned in brackets that it refers to the yes/no nature of this benchmark.
  • cross-half estimator: cross-half has no meaning at this point. Also, from the text alone it is unclear that the authors propose SNAP.
  • net of item noise: not sure what it means to be net of item noise
  • fails at 1.217: not specified why the arbitrary looking number 1.217 indicates that this experiment failed.
  • gives an inflation of 1.237 here: what is "here", what does it mean to have an inflation of 1.237?

These writing errors and mistakes appear everywhere in the paper. Additionally, the goal of the paper is insufficiently motivated and explained.

2. Incremental Nature
The authors do not convincingly show the practical benefit of their problem and solution: it remains unclear who would use the method and why. Overall, the paper is also very incremental: no ground-breaking experiments or interesting observations are done. While subjective, at no point while reading the paper I felt surprised or interested in the conclusions being presented.

Overall, the proposed method does not address a real, motivated problem. The authors should clarify:
(1) Who would use their method and who the recommendation is aimed at
(2) Find practical and interesting scenarios where ignoring cross-covariance between benchmarks would lead to wrong conclusions
(3) What value their work adds in those cases, and show explicitly that their method would avoid those wrong conclusions
Additionally, the authors themselves indicate several times that most analyses were exploratory and at least sometimes inconclusive, further decreasing the value of the work.

Questions for authors

See above

Reviewer_8dpiOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how correlations in benchmark performance across independent training runs affect uncertainty estimation in aggregated LLM evaluations. The authors argue that conventional variance estimates based on benchmark independence can underestimate the variability of average benchmark scores, potentially leading to overconfident statistical comparisons. To address this issue, they introduce SNAP, a split-half covariance estimator designed to isolate run-level covariance from item-specific evaluation noise. The analysis reveals that cross-benchmark covariance can meaningfully inflate aggregate uncertainty, with its magnitude depending on the evaluation metric and benchmark composition.

Critical strengths
  • The paper addresses a relevant problem in evaluating the reliability of aggregated language model benchmark scores.

  • The statistical analysis is detailed, including uncertainty estimation, sensitivity analyses, and independent validation attempts. The authors also transparently report negative findings.

  • The paper is well-written and easy to follow.

Critical weaknesses
  • Standard multi-run evaluation already accounts for cross-benchmark covariance when computing the standard deviation of aggregated scores. Although SNAP additionally separates run-level variation from item-level noise, the paper does not convincingly demonstrate when this distinction is necessary in practice. The proposed estimator therefore offers limited demonstrated advantages over a simpler evaluation protocol.

  • SNAP largely adapts established split-half covariance estimation techniques to LLM benchmark evaluation. While the application is relevant, the underlying statistical principles are well established. The paper would benefit from a clearer explanation of what fundamentally new methodological capabilities SNAP introduces beyond existing approaches.

  • Neither pre-specified validation test establishes statistically significant margin inflation, although their estimates remain compatible with the primary finding. Moreover, the experiments are restricted to models of at most 1B parameters and likelihood-based multiple-choice evaluations.

  • The downstream impact is limited. Covariance correction changes relatively few data-recipe comparisons and does not demonstrate statistically significant improvements in evaluation decisions.

Questions for authors
  • In what practical evaluation scenarios does SNAP offer a meaningful advantage over directly computing the standard deviation of benchmark averages across training runs?

  • Can the authors demonstrate that SNAP's separation of run-level and item-level variability improves downstream evaluation decisions or uncertainty estimation on newly sampled benchmark items?

Reviewer_9A8XOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates how correlations in performance fluctuations across benchmarks and training runs can lead to underestimated uncertainty in aggregated LLM evaluation scores. The authors propose SNAP, a split-half estimator that measures cross-benchmark run covariance while separating run-level variation from item-specific noise. Experiments on DataDecide show that ignoring covariance underestimates the standard deviation of benchmark averages, particularly for log-likelihood margins. However, the effect is not statistically confirmed in the pre-specified held-out evaluations, and covariance correction has limited impact on practical model selection decisions.

Critical strengths
  1. The paper highlights that benchmark scores may exhibit correlated fluctuations across training runs, challenging the common independence assumption used when estimating uncertainty in aggregated benchmark scores. The empirical analysis provides useful evidence of this effect and its relationship with scoring formats.

  2. SNAP uses cross-half estimation to separate item-general run covariance from item-specific noise. This provides a statistical framework for understanding benchmark dependencies and estimating effective benchmark counts.

Critical weaknesses
  1. The authors repeatedly use expressions such as "the reported analysis," "the source calculation," "the analysis record," "the shipped estimate," and "the planning document." Rather than presenting the research in a conventional scientific narrative, the paper often reads like a rewritten audit log of experimental procedures. This unusually repetitive and procedural writing style raises concerns about the extent to which the manuscript may have been generated using LLMs, beyond the language polishing disclosed in the AI Use Statement. While this is not conclusive evidence of AI-generated content, it raises questions about the manuscript's clarity and presentation.
Questions for authors

See Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.