- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Paper decomposes llm eval noise into prediction, data, and total noise using the law of total variance.
They suggest an all pairs method where they apply paired analysis to every pair of models.
They find that total noise is predictable from accuracy (and can be estimated from a Beta model) and that prediciton noise > data noise, so averaging multiple samples per task increases power.
Critical strengths
- Addresses real issue with evals/comparisons within evals
- Pretty simple and accessible
- Small-K correction is a practical contribution, along with other rules of thumbs
- empirical study on a lot of data
- releases code and data for reproducibility
Critical weaknesses
- I'd prefer the work to be more self-contained rather than on a website. Nit, but I'd prefer it if some aspects of the website were also viewable in the appendix.
- One contribution is that noise is estimateable from accuracy. However, much of that relies on Beta(p, 1-p), so with a+b = 1. However, the appendix seems to suggest that a+b varies. Does the analysis still stand?
Questions for authors
The derivation assumes a+b =1, but appendix b say sthat the fitted a+b varies. Which model was used in Figure 1? Does predictability sitll hold empirically without the beta explanation if a+b !=1?
Thanks for the review and the questions.
Thanks for this feedback. This is a deliberate design decision where the paper is about the concepts and high-level results and the website can be much more data rich, interactive, and up-to-date. This is unavailable and inappropriate for a paper in pdf format. The advantage of this design decision is that the paper can be more durable and does not need to be kept in sync with the latest date. This is actually common: In related work, the "Adding error bars to LLM eval" paper only contains some concepts and examples, and does not contain any real results. Their result is not released by anthropic and many of examples are not self-contained in their realism. Popular benchmark papers often have a separate leaderboard. Even regular experimental paper needs code/data to be fully specified, so the paper is rarely very self-contained in empirical ML.
The more important point is you can judge graphically how well the data is fitting, and they are fitting reasonably well regardless of what the explanation is. The beta theory is just a bit of extra data modeling. It is not judged to be important to the main point and thus is in the appendix (Getting this analysis should be possible now by prompting for a beta analysis of the data points).
In our data, a+b tend to start high and then decreases. Note that once the answers become fairly deterministic, a+b tends to 0 (bi-modal), whereas guessing on every multiple choice question would result in large a+b, so that explains the trend in a+b. However, in the intermediate range where the model is neither acing the benchmark nor just guessing, a+b tend to be pretty close to 1.
As the paper describes in appendix B, Beta(p, 1-p) is used. This is why the line is not quite in the middle of the all the data points, and why I consider this model/finding worth presenting. If I fit Beta(a,b) for each accuracy bucket, then presumably the fit line will be near the mean of the real data points but does not explain much.
The curve on the main plot is just the code below, which is equivalent to the Beta(p, 1-p) model with another parameter max_acc to account for questions that no model answered in all attempts. The last sentence of appendix B talks about another parameter to model questions that were never answered correctly. This is why the semi-circles end early (rather than at 1).
This should not be confused with the study on beta fit, where we actually try to fit the beta distribution parameters.