acceptodds
Under review as a conference paper at ICLR 2027

Measuring all the noises of LLM evals

Abstract

Separating signal from noise is central to experiments. Applying well-established statistical methods effectively to LLM evals requires consideration of their unique noise characteristics. We clearly define and measure three types of noise: prediction noise from generating different answers on a given question, data noise from sampling questions, and their combined total noise following the law of total variance. To emphasize relative comparisons and gain statistical power, we propose the all-pairs paired method, which applies the paired analysis to all pairs of LLMs and measures all the noise components based on millions of question-level predictions across many evals and settings, revealing clear patterns. First, each eval exhibits a characteristic and highly predictable total noise level across all model pairs. Second, paired prediction noise typically exceeds paired data noise, which means reducing prediction noise by averaging can significantly increase statistical power. By measuring all the noises together, we can assess eval results in context, lowering the barrier of using the best analysis to make sound empirical decisions.

Primary area:
datasets and benchmarks
Keywords:
evals, noise

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aYkJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Paper decomposes llm eval noise into prediction, data, and total noise using the law of total variance.

They suggest an all pairs method where they apply paired analysis to every pair of models.

They find that total noise is predictable from accuracy (and can be estimated from a Beta model) and that prediciton noise > data noise, so averaging multiple samples per task increases power.

Critical strengths
  • Addresses real issue with evals/comparisons within evals
  • Pretty simple and accessible
  • Small-K correction is a practical contribution, along with other rules of thumbs
  • empirical study on a lot of data
  • releases code and data for reproducibility
Critical weaknesses
  • I'd prefer the work to be more self-contained rather than on a website. Nit, but I'd prefer it if some aspects of the website were also viewable in the appendix.
  • One contribution is that noise is estimateable from accuracy. However, much of that relies on Beta(p, 1-p), so with a+b = 1. However, the appendix seems to suggest that a+b varies. Does the analysis still stand?
Questions for authors

The derivation assumes a+b =1, but appendix b say sthat the fitted a+b varies. Which model was used in Figure 1? Does predictability sitll hold empirically without the beta explanation if a+b !=1?

1 comment

Thanks for the review and the questions.

I'd prefer the work to be more self-contained rather than on a website. Nit, but I'd prefer it if some aspects of the website were also viewable in the appendix.

Thanks for this feedback. This is a deliberate design decision where the paper is about the concepts and high-level results and the website can be much more data rich, interactive, and up-to-date. This is unavailable and inappropriate for a paper in pdf format. The advantage of this design decision is that the paper can be more durable and does not need to be kept in sync with the latest date. This is actually common: In related work, the "Adding error bars to LLM eval" paper only contains some concepts and examples, and does not contain any real results. Their result is not released by anthropic and many of examples are not self-contained in their realism. Popular benchmark papers often have a separate leaderboard. Even regular experimental paper needs code/data to be fully specified, so the paper is rarely very self-contained in empirical ML.

One contribution is that noise is estimateable from accuracy. However, much of that relies on Beta(p, 1-p), so with a+b = 1. However, the appendix seems to suggest that a+b varies. Does the analysis still stand?

The more important point is you can judge graphically how well the data is fitting, and they are fitting reasonably well regardless of what the explanation is. The beta theory is just a bit of extra data modeling. It is not judged to be important to the main point and thus is in the appendix (Getting this analysis should be possible now by prompting for a beta analysis of the data points).

In our data, a+b tend to start high and then decreases. Note that once the answers become fairly deterministic, a+b tends to 0 (bi-modal), whereas guessing on every multiple choice question would result in large a+b, so that explains the trend in a+b. However, in the intermediate range where the model is neither acing the benchmark nor just guessing, a+b tend to be pretty close to 1.

The derivation assumes a+b =1, but appendix b say sthat the fitted a+b varies. Which model was used in Figure 1? Does predictability sitll hold empirically without the beta explanation if a+b !=1?

As the paper describes in appendix B, Beta(p, 1-p) is used. This is why the line is not quite in the middle of the all the data points, and why I consider this model/finding worth presenting. If I fit Beta(a,b) for each accuracy bucket, then presumably the fit line will be near the mean of the real data points but does not explain much.

The curve on the main plot is just the code below, which is equivalent to the Beta(p, 1-p) model with another parameter max_acc to account for questions that no model answered in all attempts. The last sentence of appendix B talks about another parameter to model questions that were never answered correctly. This is why the semi-circles end early (rather than at 1).

x = np.linspace(0, max_acc, 100)
y = np.sqrt(x/max_acc*(1-x/max_acc) * (max_acc) / (data_sz))

This should not be confused with the study on beta fit, where we actually try to fit the beta distribution parameters.

Reviewer_Y1niOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper reiterates prediction noise, data noise, and their combined total noise to establish or verify that each measured evaluation has a predictable noise level across model pairs.

Critical strengths
  • The intentions behind this work are vaguely correct, trying to establish bounds on current LLM evaluations.
  • Sound mathematical (albeit foundational) understanding
Critical weaknesses

Note: I would be more than happy to revisit my assessment if the following are addressed promptly and to a thorough degree.

  • I don't find a clear narrative to this paper, it is hard to have a concrete takeaway for the novelty and contribution presented in this work. While I agree that identifying signal from noise is important, what does this paper specifically aim to present?
  • The experimental results of this paper have been delegated to an anonymous website, however from the discussions I infer that the main finding is "total noise is approximately the prediction noise when prediction noise is greater than the data noise". Is this a finding?
  • What does the heatmap represent? Why should I care about Figure 3 or 4 (not a leading question, generally curious)? Overall, would recommend that each figure have a clear takeaway that supports the story. Also, in writing - please do not start your subsubsections with Data Heatmap, etc.
Questions for authors
  • If I was a researcher, what would be the main takeaway from your work for me? Sell me this work.
  • Could you extend your analyses to other benchmarks that investigate general trivia (MCQs, rare fruit classification, etc.)?
Reviewer_UT7BOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies three sources of variance in LLM evaluations such as prediction noise, data noise, and total noise. It proposes an all pairs paired analysis to estimate these components across model comparisons. Using millions of predictions across benchmarks including SWE-bench Verified, MATH500, and CRUXEval, the authors find that total variance is relatively predictable and prediction noise often exceeds data noise. They demonstrate that paired comparisons and repeated sampling can substantially improve statistical power.

Critical strengths
  1. Practical statistical contribution. The all pairs analysis provides an effective way to quantify uncertainty in model comparisons. The empirical results demonstrate meaningful improvements in statistical power.

  2. Extensive empirical validation. Experiments across multiple benchmarks, models, and sampling temperatures provide evidence that prediction noise can dominate data noise. The estimator validation in Appendix F further supports the methodology.

Critical weaknesses
  1. Limited generalization. All empirical results concern correctness-based evaluations. It remains unclear whether the observed variance patterns extend to preference-based evaluations, continuous metrics, or more complex agent evaluations.

  2. Statistical assumptions. The analysis largely assumes independent evaluation questions and relies on normal approximations. Correlated questions and small evaluation sets could affect the reported significance estimates. The paper acknowledges these limitations but does not quantify their impact.

Questions for authors
  1. How accurately does the proposed Beta-based variance prediction generalize to unseen benchmarks or model families? Can you report quantitative prediction errors on held-out evaluations?
  2. How sensitive are the reported statistical power improvements to clustering or dependence between evaluation questions?
2 comments

Thanks for the review and feedback. The first two issues are addressed in the limitation section, here we elaborate more. This paper tries to be as simple as possible and still it has too many complications to be more adopted, so any further complications with clustered questions, more evaluation types is probably not advisable until the basics is better adopted.

Limited generalization. All empirical results concern correctness-based evaluations. It remains unclear whether the observed variance patterns extend to preference-based evaluations, continuous metrics, or more complex agent evaluations.

Preference-based evaluation in the form should one pick A or B, is a form of correctness evaluation. SWE-bench and terminal bench are two complex agent evaluation. Indeed it does not address continuous metrics. Still, the most important evals currently are still correctness evals.

Statistical assumptions. The analysis largely assumes independent evaluation questions and relies on normal approximations. Correlated questions and small evaluation sets could affect the reported significance estimates. The paper acknowledges these limitations but does not quantify their impact.

We did quantify and survey some works on the normal approximation. This is textbook standard and we verified that indeed not much is gained by doing it exactly on this dataset. For clustered questions, the data is not quite there to address this satisfactorily.

It is also a solved problem addressed by previous work, at least conceptually.

How accurately does the proposed Beta-based variance prediction generalize to unseen benchmarks or model families? Can you report quantitative prediction errors on held-out evaluations?

We do not consider generalization to unseen benchmarks or model families. All benchmarks are analyzed independently, so there is no sense of transferring a prediction to unseen benchmarks. For model and model families, we are fitting one parameter to thousands of data points, the result is noisy but there is no way we have overfit. The prediction holds for the 20+ evals we tested on over 100+ models.

How sensitive are the reported statistical power improvements to clustering or dependence between evaluation questions?

Clustering can matter a lot, but it does not apply in many evals where there is no clustering. In other evals, we do not tend to have the data to study it. Still, clustering should only make all the intervals wider.

Another sense of dependency where the model's chance of answering questions correctly depends on the question is the key issue addressed by pairing.

Dear Authors

Thank you so much for your thoughtful response. I really appreciated your concise yet transparent and insightful reply. I very much agree with "This paper tries to be as simple as possible and still it has too many complications to be more adopted, so any further complications with clustered questions, more evaluation types is probably not advisable until the basics is better adopted." While I plan to keep my rating, my internal estimation of the value of this paper did increase. thank you, authors!!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall response

We thank the reviewers for their time and feedback. Below we summarize the main points raised and our responses.

What is the point of the paper (Y1ni)

The goal is to have meaningful standard errors for LLM evals. However, this basic goal has complications due to the prediction noise of LLMs. So we develop the methods and concepts to refer to particular standard errors taking into account both pairing and prediction noise. It is likely that well-trained statisticians understanding the key properties of LLMs will also reach the same conclusions, and that's desirable in our view where we tried hard to avoid anything non-standard. However, landmark public evals and leaderboards still use wrong or underspecified error bars or punt on them. So we thought it might be valuable to make the basic goal of getting reasonable error bars much easier and global using the all-pairs analysis. We accomplished the goal of giving the eval community correct error bars by summarizing how they tend to behave in the paper, and with more details on the interactive website.

This should all be in the abstract / intro. The reviewer is asking what the paper is about, rather than a review. For example, they point to the total variation decomposition, which is just the tool we picked for the task of getting clear error bars. If the reviewer claims it is unnecessary, or if there is a better way, then they should make a concrete argument for that instead of pointing to key words of the paper.

Results on the website vs. in the paper (Y1ni, aYkJ)

Putting most experimental results on an accompanying website was a deliberate choice. The paper focuses on concepts and general findings, while the website is data-rich, interactive, and can be kept up to date. In a way the paper is the documentation of the website and the website is the result of the paper. The method description in the paper is self-contained, but verifying the some of the observational results across evals requires the website. The website also has much more details on any exceptions due to its interactive nature.

Scope of evaluations (UT7B, Y1ni)

anticipated in limitations. We have to start with some particular evals, and to my knowledge this is more comprehensive than any similar work. Further more, whenever the question level eval result is available, then they can be added semi-automatically due to the implementation.

Other points are side issues that should not affect the main message, but the reviewer and AC can decide. They are addressed in more details separately.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.