acceptodds
Under review as a conference paper at ICLR 2027

Leaderboards rank finer than they resolve

Abstract

Machine learning adjudicates progress by ranking methods on shared benchmarks. But a benchmark is an instrument, with a resolution below which it returns noise. That resolution is almost never reported, so a reader cannot tell which leaderboard gaps are evidence and which are chance. Power analysis names it the minimum detectable difference: the smallest true accuracy gap a benchmark's own significance test detects at least eight times in ten. We ask three questions of it. First, how much of what the field publishes does its data support? Across 202 leaderboards in eight families, the median board ranks dozens of systems from first to last, yet its data can separate them into only two to five groups, depending on how strictly a group is drawn. Second, does a benchmark's resolution predict which of its published orderings hold up when measured again? No resolution measure has been checked against a second measurement of the same ordering. We rescore each ordering on items disjoint from those that assigned its label. Orderings the benchmark cannot resolve reverse in 21.5% of cases, against 0.27% of those it can, and the same test separates on four further boards, one in an unrelated field. A benchmark's resolution tells a reader, before any replication exists, which orderings to trust. Third, is resolution a property of the benchmark alone, so that its authors could compute it once and print it on the benchmark's card? It is not: it also depends on the rate at which the two compared systems disagree, which belongs to the comparison rather than to the benchmark, and a rate borrowed from a neighbouring board keeps its shape but loses its scale. Resolution has to be measured on the board itself, from the per-item outcomes its leaderboard already holds. Doing so costs no experiment, and leaderboards should report it.

Primary area:
datasets and benchmarks
Keywords:
benchmark evaluation, leaderboards, statistical power, minimum detectable difference, evaluation methodology

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FfGAOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper is an empirical investigation of resolution across evaluation families, its association with ranking stability, and the extent to which the disagreement rates used to estimate resolution transfer across settings.

Critical strengths
  1. The problem of evaluating ranking stability is interesting and the authors do a fairly good job of motivating the problem in Section 1.
Critical weaknesses
  1. Poor presentation - the paper has about 56 propositions, and a similar number of Lemmata and Theorems. There is no discussion why they are needed, which of them are relevant. The appendix of this paper is, in many ways, a dumping ground for the propositions and lemmata. I found it very hard to follow despite spending significant time.

  2. Writing - the main paper is mainly empirical, which is fine. But the prose in sections 3, 4 and 5 is simply loaded with numbers. I could not gather any useful information despite reading it several times. Important information are scattered and not presented in a coherent manner.

  3. Main conclusion - the authors make broad conclusions which are only based on limited empirical analyses (202 leaderboards from eight families). There is no theoretical support. But again the appendix has multitude of theorems which have not been connected to the main text in any way.

Questions for authors
  1. Can you write the paper in a way that it reveals the main points to your reader?

  2. I do not mind having an appendix with 60 odd theory statements and proofs. But at the very least, they need to be connected to your main points in the paper.

  3. Your empirical analyses is limited. So can you provide any theoretical support towards your claims?

5 comments
AuthorsOpenReview ↗

Response to Reviewer FfGA (1/3)

We thank the reviewer for the careful reading of our manuscript and for highlighting the difficulties in following the presentation and connecting the theoretical analysis to the empirical findings. We appreciate these concerns and have substantially revised the manuscript to make the main contributions clearer, organize the results around them, and explain precisely what the theory establishes.

We address the reviewer's three questions below.

Q1. Can the manuscript be rewritten to communicate its main points more clearly?

We agree that the original presentation contained too many numerical results without sufficiently emphasizing their significance. We have reorganized the manuscript around three main contributions, each corresponding to a central research question.

1. Quantifying leaderboard resolution. How many distinctions can a leaderboard support, given its evaluation size and the statistical power of the underlying comparison?

We build on the minimum detectable difference (MDD) framework of Card et al. (2020). MDD describes the smallest true accuracy difference that a paired significance test can detect with a specified probability. It is a measure of statistical power, not a test of whether an observed difference is significant.

We use MDD to characterize the resolution of entire leaderboards. Under a common threshold, we group ranked systems into greedy tiers and establish a bound on the number of tiers based on the leaderboard's score range.

Across 202 leaderboards from eight evaluation families, the median HELM leaderboard ranks 69 systems but contains only five greedy tiers at the reference disagreement rate. The median among the 20 non-HELM boards is 17 tiers, while the median among the 59 boards that are not sub-scenario slices is eight. These figures describe different populations, and we now distinguish them explicitly.

We also examine the observed rankings through paired significance tests. Across 182 HELM boards, the leader significantly outperforms the runner-up on only 15. This provides a complementary assessment of the statistical evidence supporting top-ranked systems.

2. Evaluating ranking stability. Are comparisons below a benchmark's resolution more likely to reverse when evaluated on different items?

We calculate MDD from the evaluation size, disagreement rate, and target statistical power, and compare each pair's observed accuracy gap against this threshold. We then evaluate whether the resulting ordering changes on items excluded from the original comparison. The threshold is not fitted to the observed reversal outcomes.

On 52 held-out MMLU subjects, 21.5% of orderings below MDD reverse, compared with 0.27% above it. We extend this analysis to seven additional boards, independently collected test sets including ImageNet-V2, CIFAR-10.1, and GSM1k, and a further held-out evaluation of Open LLM v2 and GAIA.

The additional experiments also identify limitations. The held-out criterion is supported on GAIA and four of six Open LLM v2 evaluations, while one fails and another is indeterminate. We report these outcomes to distinguish the observed relationship between MDD and ranking stability from a universal guarantee.

3. Estimating and reporting resolution. How should benchmark developers report resolution when the required disagreement rate varies between systems and evaluations?

We find that disagreement rates are not reliably transferable between evaluation settings. Across 42 ordered transfers, eleven retain at most one-quarter of the fit obtained by fitting the target leaderboard directly, with nine of these cases involving Winogrande.

We therefore recommend estimating disagreement rates from per-item outcomes whenever they are available. For leaderboards publishing only aggregate scores, we develop a sensitivity-based verdict interval that uses feasible disagreement-rate assumptions and permits an undecided result when the available information is insufficient.

MDD also supports evaluation planning by estimating the number of items needed to detect a specified improvement. For example, at a disagreement rate of 0.17, resolving the observed difference between the top two systems requires a median of 51 times the current item count across the 202 surveyed boards, counting 34 tied boards as having an infinite requirement. Among the 168 boards with nonzero top-two gaps, the median is 33 times.

AuthorsOpenReview ↗

Response to Reviewer FfGA (2/3)

Changes to the presentation. The revised abstract and introduction now state these three contributions consistently. We have added Table 1 to define the terminology before use, reorganized Sections 3–5 so that each begins with its main finding, and consolidated the detailed numerical comparisons into Tables 2–4. We have also revised the theory summary and added an appendix roadmap connecting the mathematical results to the main-text claims.

Q2. How do the theoretical statements connect to the main contributions?

We appreciate this concern. The original appendix contained many mathematical statements without sufficiently explaining which results were necessary for the main conclusions.

We have revised Appendix D to make these connections explicit. A new roadmap in Appendix D.0 maps each main-text claim to the theoretical results supporting it. The 56 numbered statements are organized into 15 core, 23 supporting, and 18 peripheral results. We have also grouped standard mathematical results separately and replaced unnecessary proofs of established results with appropriate citations.

The theoretical support for the three contributions can be summarized as follows.

Contribution 1 — Leaderboard resolution.

Theorems 30 and 38 provide the main theoretical foundation.

For a fixed set of scores and a common threshold L, Theorem 30 establishes that the greedy tier count equals the largest number of systems whose scores are pairwise separated by at least L.

Theorem 38 then establishes the bound

k ≤ 1 + ⌊R/L⌋,

where R denotes the score range.

This result applies regardless of how many systems are ranked. At a fixed threshold and score range, adding more systems cannot reduce the greedy tier count or increase it beyond this bound.

Importantly, this is a statement about the common-threshold greedy construction. It is not an upper bound on significance groupings: at one common rate, the significance threshold permits at least as many tiers (Proposition 37), and pair-specific thresholds may give more or fewer distinctions (Proposition 40). We have clarified this distinction in the revised manuscript.

We also added Corollary 58 to explain the influence of leaderboard crowding. It establishes that the proportion of adjacent score gaps below MDD tends toward one as more systems occupy a fixed score range. This motivates our decision to emphasize greedy tiers and score range in MDD units rather than treating the adjacent-gap percentage as the primary measure of resolution.

Contribution 2 — Ranking stability.

Theorems 2, 17, and 18 establish the exact power calculation and characterize the existence and uniqueness of MDD whenever a finite threshold exists under the stated assumptions.

These results provide a mathematically defined power threshold for the paired comparison. However, they do not establish that rankings above MDD must remain stable on different evaluation items.

The relationship between MDD and ranking reversals is therefore evaluated empirically in Section 4. We have made this distinction explicit: the theory establishes the threshold and its properties, while the held-out experiments measure its association with ranking stability.

Contribution 3 — Practical estimation and reporting.

The theoretical analysis characterizes how disagreement assumptions affect MDD. In particular, Corollary 24 establishes a directional property of using an overestimated disagreement rate over the surveyed sample sizes and stated parameter range. This helps explain when borrowing an inaccurate rate can lead to misleading resolution classifications.

However, the theory does not establish whether disagreement rates transfer between different benchmarks. That finding comes from the empirical transfer experiments in Section 5.

We have clarified these distinctions and added direct references between the main-text claims and the corresponding theoretical and empirical results.

AuthorsOpenReview ↗

Response to Reviewer FfGA (3/3)

Q3. Can the paper provide theoretical support beyond the empirical analysis?

The revised manuscript now distinguishes more explicitly between conclusions that follow from mathematical results and those that require empirical validation.

The central theoretical claim concerns the resolution of a ranking under a specified threshold. For a common MDD L, Theorem 38 bounds the number of greedy tiers by the score range in MDD units. The bound holds for any fixed collection of scores satisfying its assumptions; it is not fitted to, or restricted to, the 202 leaderboards in our survey.

For example, the median HELM leaderboard spans approximately 5.3 MDDs at the reference disagreement rate. Under the common-threshold construction, this permits at most six greedy tiers. The observed median is five.

The theoretical analysis also establishes properties of the exact paired-test power calculation, the conditions under which MDD exists, and the influence of disagreement-rate assumptions.

These results provide the mathematical basis for estimating and interpreting benchmark resolution. They do not, however, predict the reversal frequencies observed on held-out items or guarantee the transferability of disagreement rates across benchmarks.

We therefore provide separate empirical evidence for those claims.

For ranking stability, we evaluate model orderings on held-out MMLU subjects, random item partitions across additional benchmarks, and independently collected test sets. The main held-out MMLU result shows 21.5% reversals below MDD compared with 0.27% above it.

We also compare MDD with established statistical approaches. MDD achieves an AUC of 0.919 on the held-out MMLU evaluation, compared with 0.920 for paired bootstrap intervals. A calibrated gap rule scaled by 1/√n also performs comparably. We therefore present MDD as a power-based operating threshold, rather than claiming that it is a superior predictor of reversal risk.

Finally, we investigate the effect of assumptions not represented by the i.i.d. item-sampling model. Across 71 HELM Classic boards, variation in prompt selection exceeds the item-sampling component on 39 boards. In five of eight MathArena evaluations, whole-run variance exceeds the item-wise prediction by a factor of 1.37–1.76, indicating shared run-level effects.

These results clarify the scope of the theoretical model. MDD characterizes statistical power under the specified evaluation assumptions; prompt variation, shared run-level effects, and distribution shifts may require additional uncertainty modeling.

Taken together, the theoretical results establish how leaderboard resolution can be calculated and bounded under a defined comparison criterion, while the empirical studies evaluate how this framework behaves across real benchmarks and different sources of uncertainty.

We thank the reviewer again for the detailed comments. The revised manuscript now presents the three contributions more directly, connects the theoretical statements to the claims they support, and separates mathematical guarantees from empirical findings. We hope these changes make the central results and their limitations easier to assess.

Reviewer_N1u3OpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3(+1)
Soundness
3
Read review
Paper summary

This work investigates what leaderboards really do since ML leaderboards seem to rank systems more finely than their test sets can really support. The authors use the minimum detectable diff metric (e.g. the smallest true accuracy gap a benchmark's significance test catches 80% of the time) to survey around 202 leaderboards and find that the median board ranks dozen of systems but can only separate them into around 2-5 groups and 99% of adjacent gaps fall below this limit. Specifically, on held out MMLU, ordering below the limit reverse 21.5% of the time against the 0.27% above it. Since the limit depends pretty heavily on how often the two compared models/systems disagree, it can't be computed once per benchmark and be reused so the authors recommend that leaderboards report resolution measured from the per item outcomes they already store.

Critical strengths
  • The resolution threshold is strong and well validated. Each pair of models/systems is labelled resolvable or not on one set of items and then the label is tested on items it never saw (e.g. 52 MMLU subjects held out and random item splits). The uncertainty intervals account for each model appearing in many pairs, and the contrast replicates on Open LLM v1 and 5 more boards. This is interesting since it turns a theory-side power argument into direct evidence that below-threshold leaderboard orderings are unstable.
  • There is an actionable reporting recommendation for benchmark hosts/makers, and it’s nicely backed up by the transfer analysis. The paper shows that disagreement rates vary across benchmarks and system populations, and the rates borrowed from another board predict poorly (e.g., 11 of 42 transfers keep a quarter or less of the fit), so this shows a single resolution number on a benchmark card would be misleading. So the authors propose computing resolution from per-item outcomes leaderboards already have, which needs no new runs of experiments.
Critical weaknesses
  • There should be some stronger baselines to compare MDD with, since the paper doesn’t show that MDD beats simpler alternatives. The method is adopted from previous work, i.e., Card et al. (2020). The paper does compare against significance-based labels and a simple √n-scaled gap rule, but the authors admit the √n rule predicts reversals about as well, and even slightly better where the two rules disagree. This is somewhat expected by construction, since pairs labelled unresolvable are by definition those with small observed gaps, and smaller gaps flip more often. Comparisons against something like bootstrap confidence intervals could be a good additional baseline; otherwise, it’s unclear why MDD specifically should be the standard.
  • The survey numbers are conditional and unevenly sourced; the 2–5 groups figure relies mostly on one reference disagreement rate (0.2549), and this is below the median measured rate (0.35). Also, 182 of the 202 boards are from HELM, and 143 of those are subsets of other boards. The authors do run some sensitivity checks, but the five families without per-item outcomes are not validated against reversals. So the direction of the title claim is supported, but the magnitude is not clear.
  • It would be good if the authors estimated at least one other noise source on a board where data exist. Currently, MDD treats the benchmark’s items as independent random draws and models only which items happened to be sampled, ignoring run-to-run variation from seeds, prompts, decoding settings, etc. It also treats disagreement rates as known exactly, even though the appendix shows uncertainty in an estimated rate exceeds the approximation errors the theory bounds. Curated benchmarks are also deduplicated, difficulty-screened, etc., so they are not random samples. Ignoring run-to-run variation means the reported limits likely understate how coarse leaderboards really are, and the other assumptions add uncertainty in an unknown direction.
  • It would be nice to see demonstrations or evidence of the practical stakes, i.e., how unresolvable orderings actually mislead the field. The paper’s one attempt (model likes, downloads and declared base models) didn’t pass the authors’ own controls. The scope is also restricted to binary per-item accuracy, which excludes preference-based arenas like Chatbot Arena and graded generation metrics, both of which are a big part of evals as well.
Questions for authors

See weaknesses

7 comments
AuthorsOpenReview ↗

Response to Reviewer N1u3 (1/5)

We thank the reviewer for the careful evaluation of our manuscript and for recognizing the value of the ranking-stability experiments and the practical reporting recommendations. We appreciate the suggestions regarding stronger baselines, the representativeness of our survey, and additional sources of evaluation uncertainty. We have revised the manuscript and conducted several additional analyses to address these points.

W2. The reference disagreement rate, survey composition, and additional validation

W2.1. Reference disagreement rate. We appreciate the reviewer’s observation that the original reference rate appeared lower than the measured median. We revisited the disagreement-rate estimation procedure and reconstructed the aligned per-item outcomes for all 182 distinct HELM boards, rather than the 80 boards included in our original analysis.

The corrected median disagreement rate is 0.2379, compared with approximately 0.35 previously. The reference rate of 0.2549 therefore lies at the 54th percentile of the revised distribution.

HELM Classic cells average three train trials; the primary estimates read trial 0, with sensitivity checks on the other trials. The median rate over all 182 HELM boards is 0.2379, 0.2391 and 0.2457 at trials 0, 1 and 2.

At each board's measured primary disagreement rate, the median HELM leaderboard has five greedy tiers. Across the corrected 5th–95th percentile range of board-level rates, the estimated number of tiers varies, but the central result remains consistent.

The ranking-stability results also remain robust to the choice of reference rate. Across the corrected band, below-MDD orderings reverse substantially more frequently than above-MDD orderings on HELM and Open LLM v1. The reversal rates are at least 15.3% below MDD, compared with at most 1.51% above it, with non-overlapping model-clustered confidence intervals (Table 3a).

We have updated the manuscript and supplementary results to reflect these corrections and withdrawn the original interpretation that the reference rate necessarily provides optimistic resolution estimates.

W2.2. Representation of HELM in the survey. We agree that the overall survey statistics are strongly influenced by HELM, which accounts for 182 of the 202 boards, including 143 sub-scenario slices.

To make the scope of the findings clearer, Table 2 and the revised abstract now report results separately for different leaderboard populations.

At the reference disagreement rate, the median number of greedy tiers is:

  • 5 across the 182 HELM boards;
  • 17 across the 20 boards outside HELM;
  • 8 across the 59 boards that are not sub-scenario slices;
  • 14 when taking the median of the eight family-level medians.

Thus, the five-tier estimate primarily characterizes HELM and should not be interpreted as the typical resolution of every leaderboard family.

The revised survey makes these population differences explicit. Although the precise tier counts vary substantially across evaluation settings, the number of distinguishable tiers remains considerably smaller than the number of systems ranked.

W2.3. Additional validation across leaderboard families. We have expanded the ranking-reversal analysis to evaluation families not directly validated in the original submission. These experiments use public per-item predictions, newly collected evaluation sets, or independently rerun models.

Vision benchmarks. We examine ranking stability between ImageNet validation and ImageNet-V2, as well as between CIFAR-10 and CIFAR-10.1.

Orderings below MDD reverse in 23.7% of cases on ImageNet-V2, compared with 0.51% above MDD. For CIFAR-10.1, the corresponding rates are 39.1% and 1.35%.

We also estimate disagreement rates directly using public per-image predictions from 205 models. At the measured rates, the greedy tier counts for ImageNet validation, ImageNet-V2, ImageNet-A, and ImageNet-R are 68, 38, 62, and 96, respectively. These replace the earlier estimates of 88, 58, 105, and 149 obtained using an assumed disagreement rate.

BigCodeBench. We reconstructed the publicly available per-task outcomes and performed random split-half evaluations. Below-MDD orderings reverse in 19.2% of cases for BigCodeBench-Instruct and 17.7% for BigCodeBench-Complete, compared with 0.21% and 0.15% above MDD, respectively.

MTEB. We reran 25 open embedding models to obtain per-document predictions and estimate disagreement rates directly. The measured rates range from approximately 0.13 to 0.28 across the four tasks (0.13–0.26 on the three originally curated tasks), rather than the previously assumed 0.05.

AuthorsOpenReview ↗

Response to Reviewer N1u3 (2/5)

Across the evaluated tasks, below-MDD reversal rates range from 5.1% to 21.0%, compared with 0.06–0.28% above MDD. The confidence intervals separate on three of the four tasks, while the result on Imdb is inconclusive.

Open LLM v2 and GAIA. We also conducted a held-out evaluation using a protocol registered in advance, with one amendment frozen after an outcome-free data consistency check.

The criterion is supported on IFEval, MuSR, and MMLU-PRO, as well as BBH in the exploratory analysis, and on GAIA.

The additional experiments also identify limitations. GPQA-extended does not satisfy the criterion because orderings just beyond MDD reverse more frequently than the item-sampling model predicts. MATH Level 5 is indeterminate because the available item records do not pass the required consistency checks.

Furthermore, when entire subtasks are held out, comparisons just beyond MDD on MMLU-PRO and BBH reverse at substantially higher rates than predicted by the item-sampling model.

We have retained these findings, including the negative and indeterminate results, to clarify the conditions under which the framework provides reliable estimates and where additional sources of evaluation variation become important.

W1. Stronger baselines, the scaled-gap rule, and the motivation for MDD

W1.1. Contributions beyond existing power analysis. We agree that MDD and its dependence on pairwise agreement were introduced by Card et al. (2020). Our contribution is not a new power statistic, but an investigation of how power-based resolution relates to leaderboard rankings, their stability, and practical evaluation decisions.

The revised paper presents three main contributions.

First, we quantify the resolution of entire rankings using greedy tiers under a common MDD threshold. Theorem 38 establishes that the tier count is bounded by the score range measured in MDD units. We apply this framework to 202 leaderboards across eight families.

Second, we investigate whether MDD-based classifications are associated with the stability of model orderings on held-out evaluation items. We calculate the threshold from the evaluation size, disagreement rate, and target power, compare each pair's observed gap against it, and then evaluate the resulting labels on new items without fitting the threshold to the observed reversals.

Third, we investigate the transferability of disagreement-rate estimates and develop a reporting procedure for leaderboards with and without per-item outcomes. Our transfer experiments show that estimates fitted to one leaderboard do not consistently generalize to others, motivating direct estimation and sensitivity-based reporting.

These contributions are now organized explicitly in the revised abstract and introduction.

W1.2. Comparison with stronger baselines. Following the reviewer's suggestion, we expanded the baseline comparisons to include paired bootstrap confidence intervals, simultaneous confidence intervals, rank-based methods, and multiple-comparison corrections (Table 3b).

On the held-out MMLU evaluation, MDD achieves an AUC of 0.919, compared with 0.920 for the paired bootstrap interval and 0.919 for McNemar's test.

MDD classifies 64.9% of comparisons as resolvable, compared with 76.6% for the paired bootstrap interval. Among the comparisons classified as resolvable, 0.27% reverse under MDD and 1.00% under bootstrap intervals. However, their model-clustered confidence intervals overlap: 0.00–1.09% for MDD and 0.15–2.43% for bootstrap intervals.

We therefore do not claim that MDD outperforms these methods in predicting ranking reversals.

We also agree with the reviewer that a simple gap-based rule scaled by 1/√n can achieve comparable performance when its constant is appropriately calibrated.

The distinction is that MDD provides a threshold derived from the statistical test, evaluation size, disagreement rate, and target statistical power. This gives the threshold an explicit interpretation in accuracy points and allows it to be used for evaluation planning without fitting a cutoff to observed reversal outcomes.

We further examined how calibrated thresholds transfer across evaluation settings. Across twelve transfers, a 1/√n rule calibrated to a 1% reversal target on one design produces reversal rates of 0.06–2.38% when applied to other designs, whereas MDD produces 0.02–0.82%.

AuthorsOpenReview ↗

Response to Reviewer N1u3 (3/5)

An additional evaluation on Open LLM v2 and GAIA finds no statistically detectable improvement from using each pair's own disagreement rate to rank reversal risk relative to a matched scaled-gap rule. The estimated difference in reversal rate among resolved comparisons is −0.42 percentage points, with an interval of [−0.86, 0.04]. The interval includes zero, and the estimated advantage is largely driven by one exploratory evaluation.

These results help clarify the practical role of MDD. Its main value is providing a power-based operating threshold and supporting sample-size planning, rather than outperforming existing statistical methods in classifying ranking reversals.

W1.3. Whether the reversal contrast is expected. We appreciate this observation and agree that part of the observed relationship is expected.

MDD-based labels separate comparisons according to their observed accuracy gaps, and smaller gaps are generally more likely to reverse. Therefore, some difference between the below-MDD and above-MDD groups is unsurprising.

However, the magnitude of the reversal rates, their calibration, and the consistency of the relationship across evaluation settings are empirical questions.

Within one item pool, the predicted and observed reversal rates show close agreement, with a calibration slope of 1.03. When entire MMLU subjects are held out, the item-sampling model underpredicts reversal rates, with a slope of 0.61. Nevertheless, only 0.27% of comparisons above MDD reverse in that evaluation.

We also find that reversal rates decrease gradually as the accuracy gap increases, rather than changing abruptly at MDD.

We have therefore revised Section 4 to present MDD as a statistically motivated operating threshold whose relationship with ranking stability is evaluated empirically, not as a uniquely optimal classifier or a guarantee that rankings will remain unchanged.

W3. Additional evaluation uncertainty, disagreement-rate estimation, and curated test items

W3.1. Other sources of evaluation uncertainty. We agree that item sampling is not the only source of variability in machine learning evaluations. Following the reviewer's suggestion, we added experiments investigating decoding variability, repeated runs, and prompt selection (Appendix E.6).

Prompt variation. On 71 HELM Classic boards that evaluate models using multiple draws of in-context examples, prompt-draw variance exceeds the item-sampling component on 39 boards.

Approximately 8% of adjacent orderings classified as resolvable under the original MDD change direction across prompt draws. When this additional variance is incorporated, the median HELM Classic board falls from five greedy tiers to four.

This demonstrates that prompt variation can introduce substantial additional uncertainty not represented by the item-sampling calculation.

Repeated runs. We examine eight MathArena boards with four evaluation attempts per model. On five boards, whole-run variability exceeds the item-wise prediction by a factor of 1.37–1.76, indicating the presence of shared run-level effects.

Nevertheless, MDD-based classifications remain informative in these experiments. Between 0% and 0.5% of comparisons classified as resolvable reverse across repeated runs, compared with 11.8–15.7% of comparisons classified as unresolvable.

Decoding variability. We evaluate seven open-weight instruction models on the 514 HELM MMLU items, using greedy decoding and five sampled runs at temperature 0.7.

Decoding variability accounts for a median of approximately 53% and 67% of measured gap variance across the two experimental settings. In these experiments, much of the variability occurs independently at the item level.

An MDD calculated from a single run's observed disagreement rate is within approximately 4–5% of the corresponding calculation that explicitly incorporates the measured run variance.

These results indicate that the observed disagreement rate can reflect some forms of per-item decoding variability under a fixed evaluation protocol. However, prompt variation, shared run-level effects, and evaluation configuration changes require separate consideration.

We have revised the manuscript to clarify these assumptions and distinguish the sources of uncertainty covered by the theoretical MDD framework from those examined through supplementary experiments.

W3.2. Uncertainty in disagreement-rate estimates. We agree that the disagreement rate is estimated from finite data and should not generally be treated as known without uncertainty.

Reviewer_gRzBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors study whether published leaderboard differences exceed the minimum detectable difference (MDD) of a paired accuracy test, which is calculated with the test set size and the disagreement rate when at least one model is correct. They apply this analysis to 202 leaderboards across eight families, finding that almost all gaps for adjacent models fall below the threshold, and that reversals in model rankings are common among these comparisons.

Critical strengths
  • The paper tackles an important problem in LLM benchmark design.

  • The paper provides an extensive empirical audit, running many analyses for a large set of leaderboards.

Critical weaknesses
  1. I feel like the novelty of the paper is rather limited. The MDD concept, its dependence on pairwise agreement, and the regression of agreement on accuracy all come from Card et al. (2020), and the authors have just applied it to more recent LLM benchmarks. Further, I feel that many recent benchmarks already report confidence intervals and statistical tests, so the message that small leaderboard gaps are often not statistically meaningful is unlikely to surprise the community.

  2. The paper is hard to parse, and there are many terms that are not defined before they are used, for example "greedy tiers", “single-linkage groups”, and "ordered transfers".

  3. The authors have only studied variance from i.i.d. item sampling, but not other sources of variance like from decoding with temperature. The authors should at least run a small test to quantify these effects and show that the item sampling is the more dominant form of variance that one should be concerned about.

  4. I feel the main numbers mostly reflect how crowded a leaderboard is rather than the benchmark’s quality. Packing more models into the same score range shrinks the gaps between neighboring models, which naturally pushes the number of gaps below MDD toward 100% and the single-linkage count toward one regardless of the benchmark.

  5. It is unclear to me what additional practical value MDD provides for interpreting an existing leaderboard. MDD has a clear role in planning evaluation size for benchmark design, but a below-MDD observed difference between two models can still be statistically significant, so it is not useful for model selection. This is especially true given the authors' result that a simple sqrt-n-scaled gap rule performs similarly as MDD when predicting which orderings reverse on held-out items.

Questions for authors
  1. How does the MDD calculation compare with other ways to get confidence intervals, perhaps when per-item responses are available, such as bootstrapping, or rank intervals?

  2. Given the transfer results, what disagreement rate, or range of rates, should practitioners use for benchmarks without per-item outputs, and how should they choose it?

4 comments
AuthorsOpenReview ↗

Response to Reviewer gRzB (1/4)

We thank the reviewer for carefully evaluating our manuscript and for recognizing the importance of the problem and the scope of our empirical analysis. We appreciate the concerns regarding novelty, presentation, additional sources of evaluation uncertainty, and the practical value of MDD. We have revised the manuscript and conducted additional experiments to address these points.

W1. What the paper adds beyond Card et al. (2020)

We agree that the minimum detectable difference (MDD), its dependence on pairwise agreement, and the agreement regression were introduced by Card et al. (2020). Our contribution is not the introduction of MDD itself, but an investigation of what this framework tells us about the resolution of published leaderboards, the stability of their rankings, and the interpretation of reported improvements.

Our paper makes three main contributions.

1. Quantifying leaderboard resolution. We characterize the resolution of an entire ranking using the number of greedy tiers under a common MDD threshold. We establish a theoretical bound showing that this count cannot exceed one plus the score range measured in MDDs (Theorem 38). The bound depends on the score range and threshold, rather than directly on the number of ranked systems.

We apply this framework to 202 leaderboards across eight evaluation families, compared with the nine benchmarks examined by Card et al. At the reference disagreement rate, the median HELM leaderboard ranks 69 systems but separates them into only five greedy tiers. Furthermore, exact paired tests establish that the leading system significantly outperforms the runner-up on only 15 of 182 HELM boards.

These results quantify how much distinction a leaderboard's evaluation data can support under a specified statistical-power threshold, rather than simply observing that small score differences may be uncertain.

2. Evaluating ranking stability. We investigate whether MDD can help identify model orderings that remain stable on evaluation items not used to establish the original comparison.

We calculate MDD using the evaluation size, disagreement rate, and target statistical power, and then compare each pair's observed accuracy gap against it. We evaluate the resulting labels on held-out items without fitting the threshold to the observed reversals. This design builds on earlier information-retrieval studies of ranking stability across disjoint topic sets.

On 52 held-out MMLU subjects, 21.5% of orderings below MDD reverse, compared with 0.27% above it. We observe similar results on seven additional boards and independently collected test sets, including ImageNet-V2, CIFAR-10.1, and GSM1k.

These experiments extend the power analysis of Card et al., who compared development-set power labels against test-set significance, by directly measuring how the original orderings behave on held-out evaluation items.

3. Practical estimation and reporting. We investigate whether the disagreement rates required to calculate MDD can be reliably estimated from other leaderboards. We find substantial variation across evaluation settings and system populations. Across 42 ordered transfers, eleven retain at most one-quarter of the fit obtained by fitting the target leaderboard directly. Nine of these eleven involve Winogrande, indicating that the transfer failures are concentrated in particular evaluation settings rather than uniformly distributed.

These results suggest that disagreement rates should be measured for the relevant benchmark and model population whenever per-item outcomes are available. For leaderboards publishing only aggregate scores, we introduce a sensitivity-based verdict interval that uses the disagreement rates compatible with the reported accuracies and allows an undecided outcome when the available information is insufficient.

We also examine the historical progression of SWE-bench Verified. Among 32 successive record increases in the archived submissions, exact paired tests fail to distinguish 24 from the previous record, while 29 fall below their pair-specific MDDs. Importantly, the overall record progression provides evidence of substantial cumulative improvement, even when individual increments cannot be reliably distinguished.

We have revised the Introduction and Related Work sections to acknowledge the established contributions of Card et al., clarify how our work extends them, and discuss other studies of leaderboard uncertainty.

AuthorsOpenReview ↗

Response to Reviewer gRzB (2/4)

Regarding existing uncertainty reporting, we also examined the reviewer's observation directly. None of the eight surveyed leaderboard families displays an uncertainty interval or significance test alongside its ranking, while five of thirteen newer boards do. Three of those five continue to order models by their point estimates. These findings motivate our recommendation that leaderboards report paired uncertainty intervals alongside their rankings.

W5 and Q1. Practical value of MDD and comparisons with existing methods

We appreciate this concern. We agree that MDD should not replace statistical significance testing when deciding whether one model outperforms another. We have revised the manuscript to make this distinction explicit.

MDD and statistical significance. MDD describes the smallest true difference an evaluation is powered to detect at a specified significance level and target power. It does not, by itself, establish whether an observed accuracy difference is statistically significant. In particular, an observed difference below MDD can still be significant.

Under the normal approximation, the 80%-power MDD is approximately 1.43 times the significance boundary. We also checked this relationship numerically across 14,796 evaluated configurations. When MDD is calculated using the pair's own disagreement rate on the same items, we found no configuration in which an observed gap at least as large as MDD failed the exact McNemar test at α = 0.05. This is a numerical verification over the evaluated configurations, not a general theoretical proof. The relationship does not necessarily hold when using a borrowed disagreement rate.

For a prespecified comparison between two systems with per-item outcomes, we recommend an appropriate paired significance test. When interpreting multiple comparisons simultaneously, the testing procedure should also account for multiplicity.

Comparison with stronger baselines. Following the reviewer's suggestion, we expanded our comparison to include paired bootstrap confidence intervals, marginal rank intervals, simultaneous rank sets, and multiple-comparison methods (Table 3b).

The following results are from 1,696 model-pair orderings labelled using 514 HELM MMLU items and evaluated on 52 held-out subjects.

MethodPairs classified resolvableResolvable pairs that reverseAUC
MDD64.9%0.27%0.919
McNemar, α = 0.0575.8%0.86%0.919
Paired bootstrap, 95% CI76.6%1.00%0.920
Holm correction47.4%0.00%0.919
Simultaneous max-t48.5%0.00%0.920
Marginal rank intervals67.0%0.35%0.914
Simultaneous rank sets19.0%0.00%0.860
Arena-style rank upper bound92.9%5.71%0.849

The results show that MDD does not outperform the paired bootstrap, the other pairwise interval rules, or a tuned 1/√n rule in predicting ranking reversals. It classifies fewer comparisons as resolvable than a conventional paired significance test or bootstrap interval, and fewer of those comparisons reverse. However, the differences in reversal rates among resolved pairs are not statistically distinguishable from those of the bootstrap.

We also agree that a simple gap-based rule scaled by 1/√n can perform comparably once its constant is appropriately calibrated.

What MDD adds is a statistical-power interpretation and an evaluation-design criterion. The threshold is determined by the evaluation size, disagreement rate, significance level, and target power rather than by fitting a cutoff to reversal outcomes. This allows benchmark developers to determine the sample size required to detect an improvement of a specified magnitude.

For example, detecting a two-percentage-point improvement at 80% power and d = 0.17 requires 3,424 items under the exact paired-test calculation.

Across our 202 surveyed leaderboards, using d = 0.17, the median estimated requirement for resolving the observed top-two gap is 51 times the current item count, with the 34 boards having tied top scores assigned an infinite requirement. Among the remaining 168 boards with nonzero gaps, the median is 33 times the current size.

We also examine the transferability of calibrated thresholds. A 1/√n rule calibrated to a 1% reversal target on one design produces reversal rates of 0.06–2.38% when transferred to other designs, compared with 0.02–0.82% for MDD across the twelve evaluated transfers.

We therefore present MDD as a power-based operating threshold useful for interpreting benchmark resolution and planning evaluations, rather than as a superior method for determining which observed model differences are statistically significant.

AuthorsOpenReview ↗

Response to Reviewer gRzB (3/4)

W4. Effect of leaderboard crowding

We agree with the reviewer that the proportion of adjacent gaps below MDD is strongly influenced by how many systems are ranked within a fixed score range.

We investigated this effect explicitly. Corollary 58 establishes that, for a fixed score range measured in MDD units, increasing the number of ranked systems pushes the fraction of adjacent gaps below MDD toward one. Across the 202 leaderboards, the median lower bound arising from this effect is approximately 92.3%, compared with the observed median of 98.9%.

When we randomly reduce every leaderboard to ten systems, the median fraction of adjacent gaps below MDD decreases from 98.9% to 92.2%. This confirms that the adjacent-gap percentage is substantially influenced by leaderboard crowding and should not be interpreted independently as a measure of benchmark quality.

We have therefore revised Section 3 to emphasize the greedy tier count and the score range measured in MDD units, with the adjacent-gap percentage presented as a supporting statistic.

Under a common threshold L, Theorem 38 establishes that the greedy tier count satisfies

k ≤ 1 + ⌊R/L⌋,

where R is the leaderboard's score range. This bound does not depend directly on the number of systems ranked.

For example, the median HELM leaderboard spans 5.3 MDDs, allowing at most six greedy tiers under the common-threshold construction. The observed median is five.

Across the surveyed boards, the log score range in MDD units explains approximately 90% of the variance in log tier counts, while adding the number of systems contributes a partial R² of 0.03.

At a fixed threshold, adding systems cannot lower the tier count (Theorem 30), and within a fixed score range it cannot raise it beyond 1 + ⌊R/L⌋. If the disagreement rate and threshold are recalculated for a changing model population, the first property need not hold.

We have also clarified that greedy tiers summarize separation under a specified power threshold. They are not equivalent to simultaneous statistical significance groups or confidence sets for model ranks.

Q2. Choosing the disagreement rate without per-item outcomes

This is an important practical question, particularly because our transfer experiments show that disagreement rates are not fixed properties of a benchmark.

Where per-item outcomes are available, we recommend calculating the disagreement rate directly for the systems being compared.

Where only aggregate accuracies are published, the disagreement rate cannot be uniquely determined. However, the two accuracies impose the following mathematical bounds:

|a₁ − a₂| ≤ d ≤ min(a₁ + a₂, 2 − a₁ − a₂).

A universal fixed disagreement rate can violate these bounds. For example, the reference rate d = 0.2549 is infeasible for approximately 23% of HELM adjacent pairs.

We therefore propose a verdict interval that combines the accuracy-implied constraints with empirically calibrated disagreement-rate assumptions.

A pair is classified as resolvable only when its observed gap exceeds MDD calculated using the independence disagreement rate,

d_ind = a₁(1 − a₂) + a₂(1 − a₁).

It is classified as unresolvable only when the gap is below MDD at a smaller calibrated fraction, approximately 0.40–0.49 of the independence rate. Comparisons between these thresholds remain undecided. The resulting interval is a sensitivity-based classification procedure, rather than a confidence interval with guaranteed statistical coverage.

We evaluate this procedure on 12,269 adjacent system pairs, leaving each evaluation family out during calibration.

The method makes three incorrect resolvable classifications and twelve incorrect unresolvable classifications, while leaving 144 pairs (1.2%) undecided. Its overall misclassification rate is 0.12%, compared with 0.45–0.81% for the evaluated single-rate alternatives.

Because most adjacent pairs are unresolvable, we also report performance specifically on the 248 pairs classified as resolvable using their measured disagreement rates. Of these, the method correctly identifies 179 (72%), leaves 57 undecided (23%), and incorrectly classifies twelve as unresolvable (5%).

Where a single estimate is required, we additionally evaluate a rate approximately equal to 0.6 times the independence rate, clipped to the accuracy-implied bounds. This reduces the median MDD estimation error on HELM from 1.67 to 0.82 accuracy points relative to the common reference-rate approach, although it does not improve performance on every leaderboard family.

We therefore recommend using measured disagreement rates whenever possible and reporting sensitivity to plausible rates when per-item outcomes are unavailable.

AuthorsOpenReview ↗

Response to Reviewer gRzB (4/4)

W3. Uncertainty beyond item sampling

We appreciate the suggestion to examine additional sources of evaluation variability. Following the reviewer's recommendation, we added experiments involving decoding, prompt variation, repeated runs, and changes in evaluation configuration (Appendix E.6).

Decoding variability. We evaluated seven open-weight instruction models, ranging from 1.5B to 8B parameters, on the 514 HELM MMLU items. Each model was evaluated using greedy decoding and five sampled runs at temperature 0.7.

Decoding variability accounts for a median of approximately 53% and 67% of the measured gap variance across the two experimental settings. In these experiments, the decoding variation was independent from item to item, with no shared run-level shocks. The MDD calculated using a single run's disagreement rate is within a median of 4–5% of an estimate that explicitly includes the measured run variance.

Furthermore, all 13 and 14 pairs classified as resolvable under greedy decoding retain their ordering when compared with the five-run mean in the respective experimental settings.

These findings indicate that MDD based on the observed disagreement rate can remain informative when decoding variation occurs independently across items under a fixed evaluation protocol. However, this does not imply that MDD captures every form of decoding or run-level variability.

Prompt variation. Across 71 HELM Classic boards, variation from different draws of in-context examples exceeds the item-sampling component on 39 boards. Approximately 8% of adjacent orderings above the original MDD change direction across prompt draws. When this additional variance is included, the median number of greedy tiers on these boards decreases from five to four.

Repeated runs. We also examined eight MathArena boards with four attempts per model. On five boards, the observed whole-run variability exceeds the item-wise prediction by a factor of 1.37–1.76. Nevertheless, the MDD-based categories remain informative: between 0% and 0.5% of resolvable orderings reverse across repeated runs, compared with 11.8–15.7% of unresolvable orderings.

These experiments demonstrate that prompt selection and shared run-level effects can introduce uncertainty comparable to, or larger than, item sampling. We have therefore clarified that the theoretical MDD framework describes item-sampling uncertainty under its stated assumptions, and that additional sources of variability may require separate estimation.

We also identify changes in evaluation configuration or decoding pipelines as a separate concern. Such changes can introduce systematic differences, and comparisons should therefore use consistent evaluation protocols.

W2. Definitions and presentation

We agree that several terms were not sufficiently explained before their use in the original manuscript.

To address this, we have added Table 1, at the top of page 2, defining the main terms, including MDD, disagreement rate, greedy tiers, single-linkage groups, and ordered transfers.

In particular, greedy grouping starts a new tier when a system's score is at least one MDD below the leader of the current tier. Single-linkage grouping separates consecutive groups only where adjacent scores differ by at least one MDD. An ordered transfer refers to a disagreement-rate regression fitted on one leaderboard and evaluated on another; seven boards provide 42 ordered transfers.

We have also reorganized Sections 3–5 to introduce the main finding at the beginning of each section and consolidated the supporting numerical comparisons into Tables 2–4.

A new roadmap in Appendix D.0 connects the theoretical results to the corresponding claims in the main text. We have further clarified which claims are theoretically established and which are supported by empirical observations.

We thank the reviewer again for the constructive comments. The additional experiments and revisions have helped us clarify the distinction between statistical power and significance, address the effect of leaderboard crowding, and explain where MDD provides practical value alongside established statistical methods. We hope these changes address the concerns raised in the review.

Comments on the paper

10 comments
AuthorsOpenReview ↗

General Response (1/6)

We thank all three reviewers for their careful evaluation of our manuscript and constructive suggestions. We appreciate the concerns regarding the presentation, the distinction between MDD and statistical significance, and the scope of our empirical findings. We have revised the manuscript to clarify the main contributions, strengthen the experimental comparisons, and connect the theoretical results to the corresponding findings. We summarize the main changes below and address each reviewer's comments separately.

1.1 Three main contributions (gRzB W1, N1u3 W1, FfGA Q1)

Our work builds on the minimum detectable difference (MDD) framework of Card et al. (2020). MDD represents the smallest true accuracy difference that a paired significance test can detect with a specified statistical power. It depends on the number of evaluation items, the disagreement rate between the compared systems, and the chosen significance and power levels. Importantly, MDD describes what an evaluation is powered to detect; it does not establish whether an observed performance difference is statistically significant.

Building on this framework, our paper makes three main contributions.

Contribution 1. Quantifying leaderboard resolution. We characterize the resolution of an entire leaderboard using the number of greedy tiers under a common MDD threshold. Theorem 38 establishes that this count cannot exceed one plus the score range measured in MDDs, regardless of how many systems are ranked. Although adding systems can increase the number of observed tiers, it cannot increase the count beyond this bound when the threshold and score range remain fixed.

We apply this analysis to 202 leaderboards across eight evaluation families. At the reference disagreement rate, the median HELM leaderboard ranks 69 systems but separates them into only five greedy tiers. The median leaderboard outside HELM has 17 tiers, while the median among boards that are not sub-scenario slices is eight. We also examine the top-ranked systems using exact paired significance tests. Across 182 HELM boards, the leader is significantly ahead of the runner-up on only 15.

These findings quantify the difference between the number of systems ranked and the distinctions supported by a specified evaluation threshold.

Contribution 2. Evaluating ranking stability. We investigate whether MDD can help identify model orderings that remain stable when evaluated on new items. We calculate the threshold using the evaluation size, disagreement rate, and target statistical power, then compare each pair's observed accuracy gap against it. The resulting labels are evaluated on held-out items that were not used to establish the original comparisons. The threshold is not calibrated using the observed reversal outcomes.

On 52 held-out MMLU subjects, 21.5% of orderings below MDD reverse, compared with 0.27% of those above it. We observe a similar separation on seven additional boards and independently collected test sets, including ImageNet-V2, CIFAR-10.1, and GSM1k. An additional held-out evaluation supports the criterion on GAIA and four of six Open LLM v2 boards, while identifying one failure and one indeterminate result (Section 3.1).

We also compare MDD with established statistical approaches. MDD does not outperform paired bootstrap intervals or a tuned 1/√n gap rule in predicting reversals. Its contribution is a threshold with a statistical-power interpretation that can be determined without fitting to reversal outcomes and can be used to plan evaluation size (Section 2.1).

Contribution 3. Practical estimation and reporting. We investigate how leaderboard resolution should be estimated and communicated in practice. The disagreement rate depends on both the evaluation data and the systems being compared. Our experiments show that rates estimated on one leaderboard do not consistently transfer to another: in 11 of 42 transfers, the transferred regression retains at most one-quarter of the fit obtained by fitting the target leaderboard directly.

We therefore recommend measuring disagreement rates from the per-item outcomes already available to leaderboard maintainers. For leaderboards that publish only aggregate scores, we develop a verdict interval based on feasible disagreement rates, with an undecided category when the available information is insufficient (Section 4.1).

We also examine the practical implications for interpreting leaderboard improvements. On SWE-bench Verified, exact tests fail to distinguish 24 of 32 successive record increases from the records they displaced, while 29 increases fall below their pair-specific MDDs. These findings do not imply an absence of progress, as the overall record sequence provides evidence of substantial cumulative improvement.

AuthorsOpenReview ↗

General Response (2/6)

Finally, MDD can inform future evaluation design by estimating the number of test items required to detect a specified difference. At a disagreement rate of d = 0.17, resolving the observed top-two gap requires a median of 51 times the current number of items across the 202 surveyed boards, counting 34 boards with tied top scores as having an infinite requirement. Among the 168 boards with nonzero top-two gaps, the median is 33 times the current size.

1.2 Where the shared concerns are addressed

  • Section 2.1: MDD, statistical significance, and comparisons with existing baselines (gRzB W5, Q1; N1u3 W1).
  • Section 2.2: Theoretical contributions, their assumptions, and improvements to presentation (gRzB W2; FfGA W1–W3, Q1–Q3).
  • Section 2.3: Leaderboard crowding and its influence on resolution summaries (gRzB W4; N1u3 W2).
  • Section 3.1: Expanded reversal validation, including negative and indeterminate results (N1u3 W2; FfGA W3).
  • Section 3.2: Additional sources of evaluation uncertainty (gRzB W3; N1u3 W3).
  • Section 3.3: Corrections and updated estimates.
  • Section 4.1: Estimating resolution when per-item outcomes are unavailable (gRzB Q2; N1u3 W2).
  • Section 4.2: Related work and practical consequences (gRzB W1; N1u3 W1, W4).

2.1 MDD, statistical significance, and baselines (gRzB W5, Q1; N1u3 W1)

What MDD measures. MDD characterizes the statistical power of an evaluation, rather than the significance of an observed comparison. At a fixed evaluation size, disagreement rate, significance level, and target power, it specifies the smallest true difference the paired test is designed to detect with the target probability.

Comparing an observed accuracy gap with MDD produces a power-based label, not a significance-test result. An observed gap below MDD can still be statistically significant.

Under the normal approximation, the 80%-power threshold is approximately 1.43 times the corresponding significance boundary. We also checked the relationship using the exact McNemar test across 14,796 evaluated (n, m) configurations. We found no cases in which an observed gap at least as large as MDD, calculated using the pair's own disagreement rate on the same items, failed the exact test at α = 0.05. This is a numerical verification over the evaluated configurations, rather than a general theoretical proof. The exact ratio varies between 1.00 and 1.48, with a median of 1.40. The implication does not necessarily hold when a disagreement rate is borrowed from another evaluation.

For a prespecified comparison between two systems with per-item outcomes, we recommend an appropriate paired significance test. When interpreting multiple comparisons or an entire leaderboard, multiplicity correction should also be considered.

Comparison with existing baselines. Following the reviewers' suggestions, we expanded our evaluation to include paired bootstrap confidence intervals, significance tests, simultaneous confidence intervals, rank-based methods, and multiple-comparison corrections (Table 3b).

On the held-out MMLU evaluation, MDD achieves an AUC of 0.919, compared with 0.920 for paired bootstrap intervals and 0.919 for McNemar's test. It classifies 64.9% of comparisons as resolvable, compared with 76.6% for paired bootstrap intervals. Among these resolved comparisons, 0.27% reverse under MDD and 1.00% under bootstrap intervals, although the corresponding uncertainty intervals overlap.

A tuned 1/√n gap rule also performs comparably. We therefore do not claim that MDD provides a better ranking of reversal risk than these methods. Part of the separation between below-MDD and above-MDD comparisons is expected because smaller observed gaps are generally more likely to reverse. The magnitude of the observed reversal rates and their behavior across evaluation settings are established empirically.

What MDD adds. The main practical value of MDD is its statistical-power interpretation and its application to evaluation design. It provides a threshold in accuracy points based on the test, evaluation size, disagreement rate, and desired power. Unlike a cutoff fitted to observed reversal rates, MDD does not require calibration using those outcomes.

For example, detecting a true improvement of two accuracy points with 80% power at d = 0.17 requires 3,424 evaluation items under the exact paired-test calculation.

AuthorsOpenReview ↗

General Response (3/6)

We also examine how well a threshold calibrated on one evaluation transfers to another. Across twelve transfers, a 1/√n rule calibrated to a 1% reversal target produces reversal rates of 0.06–2.38% among resolved comparisons, while MDD produces 0.02–0.82%. These findings support the usefulness of a power-based operating threshold without implying superior reversal prediction in every setting.

2.2 Theory and presentation (gRzB W2; FfGA W1–W3, Q1–Q3)

We agree that the original manuscript did not adequately explain how the theoretical results supported the main findings.

To improve the presentation, we added Table 1 to define the terminology before its use. We also reorganized Sections 3–5 so that each begins with the main finding and then presents the supporting evidence. Much of the numerical detail has been moved into Tables 2–4.

We added a roadmap in Appendix D.0 connecting the theoretical statements to their corresponding claims in the main text. The 56 statements are organized into 15 core results, 23 supporting results, and 18 peripheral results. Standard theoretical results are separated from those directly supporting the contributions, and the main text now references the relevant theorems alongside their applications.

The central theoretical result concerning leaderboard resolution is Theorem 38. For a fixed collection of scores and one common threshold L, Theorem 30 characterizes the greedy tier count, and Theorem 38 bounds it by

k ≤ 1 + ⌊R/L⌋,

where R is the score range.

This is a deterministic bound under the common-threshold greedy grouping rule. It does not bound significance groupings: at one common rate these can contain at least as many tiers (Proposition 37), and with pair-specific thresholds more or fewer (Proposition 40). Under a fixed threshold and score range, adding systems cannot reduce the greedy tier count or increase it beyond the bound. If the threshold is recalculated using a changing model population, this monotonicity need not hold.

The theoretical analysis also establishes the calculation and conditional uniqueness of MDD under its stated assumptions. The reversal results and disagreement-rate transfer findings, however, are empirical and are not consequences of Theorem 38.

We have revised the manuscript to make these distinctions explicit and to clarify which conclusions are theoretically established and which depend on the evaluated data.

2.3 Leaderboard crowding (gRzB W4; N1u3 W2)

We appreciate the observation that the fraction of adjacent gaps below MDD is influenced by the number of systems on a leaderboard.

Our additional analysis confirms this concern. Corollary 58 establishes that increasing the number of systems within a fixed score range pushes the fraction of adjacent gaps below MDD toward one. Across the surveyed boards, the median lower bound arising from this effect is approximately 92.3%, compared with the observed median of 98.9%. When each board is randomly reduced to ten systems, the median observed fraction decreases to 92.2%.

We have therefore revised the manuscript to emphasize the number of greedy tiers and the score range measured in MDD units, rather than relying primarily on the adjacent-gap percentage.

For the median HELM leaderboard, the score range spans 5.3 MDDs, allowing at most six greedy tiers under the common-threshold construction. The observed median is five. Across the surveyed boards, a regression using the log score range in MDD units explains approximately 90% of the variation in log tier counts, while adding the number of ranked systems contributes a partial R² of 0.03.

These results distinguish the crowding effect on adjacent gaps from the resolution summarized by the greedy tier count. We now report the two measures separately and explain their different interpretations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.