- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper is an empirical investigation of resolution across evaluation families, its association with ranking stability, and the extent to which the disagreement rates used to estimate resolution transfer across settings.
Critical strengths
- The problem of evaluating ranking stability is interesting and the authors do a fairly good job of motivating the problem in Section 1.
Critical weaknesses
-
Poor presentation - the paper has about 56 propositions, and a similar number of Lemmata and Theorems. There is no discussion why they are needed, which of them are relevant. The appendix of this paper is, in many ways, a dumping ground for the propositions and lemmata. I found it very hard to follow despite spending significant time.
-
Writing - the main paper is mainly empirical, which is fine. But the prose in sections 3, 4 and 5 is simply loaded with numbers. I could not gather any useful information despite reading it several times. Important information are scattered and not presented in a coherent manner.
-
Main conclusion - the authors make broad conclusions which are only based on limited empirical analyses (202 leaderboards from eight families). There is no theoretical support. But again the appendix has multitude of theorems which have not been connected to the main text in any way.
Questions for authors
-
Can you write the paper in a way that it reveals the main points to your reader?
-
I do not mind having an appendix with 60 odd theory statements and proofs. But at the very least, they need to be connected to your main points in the paper.
-
Your empirical analyses is limited. So can you provide any theoretical support towards your claims?
5 comments
Response to Reviewer FfGA (2/3)
Changes to the presentation. The revised abstract and introduction now state these three contributions consistently. We have added Table 1 to define the terminology before use, reorganized Sections 3–5 so that each begins with its main finding, and consolidated the detailed numerical comparisons into Tables 2–4. We have also revised the theory summary and added an appendix roadmap connecting the mathematical results to the main-text claims.
Q2. How do the theoretical statements connect to the main contributions?
We appreciate this concern. The original appendix contained many mathematical statements without sufficiently explaining which results were necessary for the main conclusions.
We have revised Appendix D to make these connections explicit. A new roadmap in Appendix D.0 maps each main-text claim to the theoretical results supporting it. The 56 numbered statements are organized into 15 core, 23 supporting, and 18 peripheral results. We have also grouped standard mathematical results separately and replaced unnecessary proofs of established results with appropriate citations.
The theoretical support for the three contributions can be summarized as follows.
Contribution 1 — Leaderboard resolution.
Theorems 30 and 38 provide the main theoretical foundation.
For a fixed set of scores and a common threshold L, Theorem 30 establishes that the greedy tier count equals the largest number of systems whose scores are pairwise separated by at least L.
Theorem 38 then establishes the bound
k ≤ 1 + ⌊R/L⌋,
where R denotes the score range.
This result applies regardless of how many systems are ranked. At a fixed threshold and score range, adding more systems cannot reduce the greedy tier count or increase it beyond this bound.
Importantly, this is a statement about the common-threshold greedy construction. It is not an upper bound on significance groupings: at one common rate, the significance threshold permits at least as many tiers (Proposition 37), and pair-specific thresholds may give more or fewer distinctions (Proposition 40). We have clarified this distinction in the revised manuscript.
We also added Corollary 58 to explain the influence of leaderboard crowding. It establishes that the proportion of adjacent score gaps below MDD tends toward one as more systems occupy a fixed score range. This motivates our decision to emphasize greedy tiers and score range in MDD units rather than treating the adjacent-gap percentage as the primary measure of resolution.
Contribution 2 — Ranking stability.
Theorems 2, 17, and 18 establish the exact power calculation and characterize the existence and uniqueness of MDD whenever a finite threshold exists under the stated assumptions.
These results provide a mathematically defined power threshold for the paired comparison. However, they do not establish that rankings above MDD must remain stable on different evaluation items.
The relationship between MDD and ranking reversals is therefore evaluated empirically in Section 4. We have made this distinction explicit: the theory establishes the threshold and its properties, while the held-out experiments measure its association with ranking stability.
Contribution 3 — Practical estimation and reporting.
The theoretical analysis characterizes how disagreement assumptions affect MDD. In particular, Corollary 24 establishes a directional property of using an overestimated disagreement rate over the surveyed sample sizes and stated parameter range. This helps explain when borrowing an inaccurate rate can lead to misleading resolution classifications.
However, the theory does not establish whether disagreement rates transfer between different benchmarks. That finding comes from the empirical transfer experiments in Section 5.
We have clarified these distinctions and added direct references between the main-text claims and the corresponding theoretical and empirical results.
Response to Reviewer FfGA (3/3)
Q3. Can the paper provide theoretical support beyond the empirical analysis?
The revised manuscript now distinguishes more explicitly between conclusions that follow from mathematical results and those that require empirical validation.
The central theoretical claim concerns the resolution of a ranking under a specified threshold. For a common MDD L, Theorem 38 bounds the number of greedy tiers by the score range in MDD units. The bound holds for any fixed collection of scores satisfying its assumptions; it is not fitted to, or restricted to, the 202 leaderboards in our survey.
For example, the median HELM leaderboard spans approximately 5.3 MDDs at the reference disagreement rate. Under the common-threshold construction, this permits at most six greedy tiers. The observed median is five.
The theoretical analysis also establishes properties of the exact paired-test power calculation, the conditions under which MDD exists, and the influence of disagreement-rate assumptions.
These results provide the mathematical basis for estimating and interpreting benchmark resolution. They do not, however, predict the reversal frequencies observed on held-out items or guarantee the transferability of disagreement rates across benchmarks.
We therefore provide separate empirical evidence for those claims.
For ranking stability, we evaluate model orderings on held-out MMLU subjects, random item partitions across additional benchmarks, and independently collected test sets. The main held-out MMLU result shows 21.5% reversals below MDD compared with 0.27% above it.
We also compare MDD with established statistical approaches. MDD achieves an AUC of 0.919 on the held-out MMLU evaluation, compared with 0.920 for paired bootstrap intervals. A calibrated gap rule scaled by 1/√n also performs comparably. We therefore present MDD as a power-based operating threshold, rather than claiming that it is a superior predictor of reversal risk.
Finally, we investigate the effect of assumptions not represented by the i.i.d. item-sampling model. Across 71 HELM Classic boards, variation in prompt selection exceeds the item-sampling component on 39 boards. In five of eight MathArena evaluations, whole-run variance exceeds the item-wise prediction by a factor of 1.37–1.76, indicating shared run-level effects.
These results clarify the scope of the theoretical model. MDD characterizes statistical power under the specified evaluation assumptions; prompt variation, shared run-level effects, and distribution shifts may require additional uncertainty modeling.
Taken together, the theoretical results establish how leaderboard resolution can be calculated and bounded under a defined comparison criterion, while the empirical studies evaluate how this framework behaves across real benchmarks and different sources of uncertainty.
We thank the reviewer again for the detailed comments. The revised manuscript now presents the three contributions more directly, connects the theoretical statements to the claims they support, and separates mathematical guarantees from empirical findings. We hope these changes make the central results and their limitations easier to assess.
Response to Reviewer FfGA (1/3)
We thank the reviewer for the careful reading of our manuscript and for highlighting the difficulties in following the presentation and connecting the theoretical analysis to the empirical findings. We appreciate these concerns and have substantially revised the manuscript to make the main contributions clearer, organize the results around them, and explain precisely what the theory establishes.
We address the reviewer's three questions below.
Q1. Can the manuscript be rewritten to communicate its main points more clearly?
We agree that the original presentation contained too many numerical results without sufficiently emphasizing their significance. We have reorganized the manuscript around three main contributions, each corresponding to a central research question.
1. Quantifying leaderboard resolution. How many distinctions can a leaderboard support, given its evaluation size and the statistical power of the underlying comparison?
We build on the minimum detectable difference (MDD) framework of Card et al. (2020). MDD describes the smallest true accuracy difference that a paired significance test can detect with a specified probability. It is a measure of statistical power, not a test of whether an observed difference is significant.
We use MDD to characterize the resolution of entire leaderboards. Under a common threshold, we group ranked systems into greedy tiers and establish a bound on the number of tiers based on the leaderboard's score range.
Across 202 leaderboards from eight evaluation families, the median HELM leaderboard ranks 69 systems but contains only five greedy tiers at the reference disagreement rate. The median among the 20 non-HELM boards is 17 tiers, while the median among the 59 boards that are not sub-scenario slices is eight. These figures describe different populations, and we now distinguish them explicitly.
We also examine the observed rankings through paired significance tests. Across 182 HELM boards, the leader significantly outperforms the runner-up on only 15. This provides a complementary assessment of the statistical evidence supporting top-ranked systems.
2. Evaluating ranking stability. Are comparisons below a benchmark's resolution more likely to reverse when evaluated on different items?
We calculate MDD from the evaluation size, disagreement rate, and target statistical power, and compare each pair's observed accuracy gap against this threshold. We then evaluate whether the resulting ordering changes on items excluded from the original comparison. The threshold is not fitted to the observed reversal outcomes.
On 52 held-out MMLU subjects, 21.5% of orderings below MDD reverse, compared with 0.27% above it. We extend this analysis to seven additional boards, independently collected test sets including ImageNet-V2, CIFAR-10.1, and GSM1k, and a further held-out evaluation of Open LLM v2 and GAIA.
The additional experiments also identify limitations. The held-out criterion is supported on GAIA and four of six Open LLM v2 evaluations, while one fails and another is indeterminate. We report these outcomes to distinguish the observed relationship between MDD and ranking stability from a universal guarantee.
3. Estimating and reporting resolution. How should benchmark developers report resolution when the required disagreement rate varies between systems and evaluations?
We find that disagreement rates are not reliably transferable between evaluation settings. Across 42 ordered transfers, eleven retain at most one-quarter of the fit obtained by fitting the target leaderboard directly, with nine of these cases involving Winogrande.
We therefore recommend estimating disagreement rates from per-item outcomes whenever they are available. For leaderboards publishing only aggregate scores, we develop a sensitivity-based verdict interval that uses feasible disagreement-rate assumptions and permits an undecided result when the available information is insufficient.
MDD also supports evaluation planning by estimating the number of items needed to detect a specified improvement. For example, at a disagreement rate of 0.17, resolving the observed difference between the top two systems requires a median of 51 times the current item count across the 202 surveyed boards, counting 34 tied boards as having an infinite requirement. Among the 168 boards with nonzero top-two gaps, the median is 33 times.