acceptodds
Under review as a conference paper at ICLR 2027

Leaderboards Display Ranks, Benchmarks Certify Tiers: Simultaneous Inference and a Capacity Law for LLM Evaluation

25 views

Abstract

Public LLM leaderboards display a total order over models, but the finite item sample behind each leaderboard supports only a partial order. We make the certified objects explicit: simultaneous rank sets, the certified pairwise partial order, and certified tiers: ordered groups whose every cross-tier comparison holds jointly with familywise confidence . Auditing five public leaderboards across three data modalities shows that the data support a great deal (57–77% of pairwise orderings certified) while the displayed total order is almost entirely unsupported: AlpacaEval 2.0 shows 65 ranks but certifies 2 tiers; LiveBench 94 and 1; a 57k-battle Chatbot Arena release 63 and 2; SWE-bench Verified 134 and 1; the Open LLM Leaderboard's 4,576 ranks reduce to at most 2 tiers per benchmark (5 on its top-100 slice); and top-1 plausible sets range from a unique champion to 24 models (87 under task-clustered inference). A tier capacity law, an approximate scaling relation in the sense of empirical scaling laws, predicts how certifiable tiers grow with items as and collapse with model density (slope constants correct to within ; the crowding decline reproduced within 35–46% at the smallest subsets), and inverts into item budgets for target resolution. On two refreshing benchmarks, certified tiers did not invert on fresh items in the one refresh where the check is non-vacuous, and rank-flip rates match the closed-form prediction over 130 pooled adjacent pairs. We release tiercert, a CPU-only toolkit that turns any score matrix or battle log into rank sets, certified tiers, and item-budget prescriptions; code, data, and result files are provided as supplementary material.

Primary area:
datasets and benchmarks
Keywords:
llm evaluation, leaderboards, simultaneous inference, multiple comparisons, family-wise error rate, benchmark design, ranking uncertainty

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_4LyhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a statistical framework for evaluating the reliability of LLM leaderboards. Instead of reporting only individual model rankings, it introduces certified tiers and simultaneous rank sets to identify which model comparisons are statistically supported. The paper also develops an approximate capacity law to estimate how benchmark size affects ranking resolution. Experiments across five public leaderboards show that many displayed rankings cannot be reliably distinguished, highlighting the importance of uncertainty-aware model evaluation.

Critical strengths
  • The paper addresses an important and practical problem in LLM evaluation. The proposed certified tiers provide a more reliable and interpretable way to compare models than conventional leaderboard rankings.
  • The experiments cover five major leaderboards with different evaluation settings. The combination of statistical analysis, theoretical insights, and a publicly available toolkit makes the work useful for both researchers and benchmark developers.
Critical weaknesses
  • The proposed capacity law relies on simplifying assumptions about model score distributions and adjacent score gaps. Its predictions can be inaccurate when these assumptions are violated, as seen on MMLU-Pro. This limits its reliability for estimating benchmark requirements across different leaderboard settings.
  • The certified-tier framework builds largely on existing simultaneous inference and multiple-testing methods, making its methodological novelty somewhat limited. Moreover, the out-of-sample validation covers only a small number of benchmark updates, leaving its practical generalizability less established.
Questions for authors
  • How sensitive are the capacity-law predictions to violations of the assumed score-spacing distribution?
  • Can the authors further clarify how certified tiers differ from existing simultaneous rank inference and multiple-testing approaches?
  • Can the authors provide additional out-of-sample evaluations to demonstrate that the certified tier structure remains reliable across benchmark updates?
1 comment

Thank you again for the positive assessment and your three questions. As the discussion period closes on 18 November, we wanted to check whether our replies address them. In brief:

  • Spacing assumption. A controlled study measures how sensitive the closed form is to it (Table 9): it stays within ±5% under exponential spacings and errs by tens of percent under strong departures. The new pilot simulation makes no spacing assumption and has a mean error of 4.3% on held-out items (Table 8).
  • Relation to existing methods. Proposition 2 (§4) shows that certified tiers are the finest partition the simultaneous certificates imply, and places them relative to rank confidence sets.
  • Out-of-sample evidence. Across 15 refreshes of three benchmarks, 6 of 3,344 certified orderings reversed on fresh items, and none of 81 cross-tier pairs did (§6.3, Table 10).

If anything remains unclear, we would be glad to clarify before the deadline. We would also be grateful if you would consider whether the revision changes your assessment.

Reviewer_n3ehOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper examines whether the LLM leaderboards have enough evidence to support the ranks. It proposes grouping models into tiers whose differences are supported by the data, and estimates how many evaluation questions are needed to distinguish more tiers.

Critical strengths
  • The research question is relevant: how much confidence should users place in the ranks shown on LLM leaderboards?
  • The analysis covers five public leaderboards and gives concrete examples of model comparisons that the available data cannot resolve.
Critical weaknesses
  • The main issue is that the claim is subjective. I found no justification for why this is the right target for leaderboard users or evidence showing that it helps them choose better models. LLM ranking can be seen from different perspectives, as discussed in "Ranking Reasoning LLMs under Test-Time Scaling" (ACL'26) and "A Statistical Framework for Ranking LLM-Based Chatbots" (ICLR'25).

  • The ranking methods are very limited. There are many more methods that are missing (see https://scorio.readthedocs.io/en/latest/api/rank.html).

  • The capacity law has clear limitations. On MMLU-Pro, it predicts 1.4 tiers against five observed tiers. The item-budget recommendations are not validated by collecting additional items and checking whether the target resolution is achieved.

Questions for authors
  • How do you justify your formalization? Why is the proposed tier definition the right target for leaderboard users, and what evidence shows that it helps them choose better models?

  • Compare the main conclusions against ranking methods other than mean scores and BT.

  • Can the item-budget recommendations be validated on fresh data to check whether the predicted number of tiers is achieved? (There are several studies on this topic that you can find that are worth mentioning.)

1 comment

Thank you again for your review. As the discussion period closes on 18 November, we wanted to check whether our replies address your three questions.

Q1 (does certification help users choose models?): We ran 15 refreshes of three benchmarks. In all 15, the top-1 plausible set contained the model that was best on the fresh items; the displayed #1 was that model in only 8 of 15. On the fresh items, 6 of 3,344 certified orderings reversed, against 39% of displayed-adjacent pairs (§6.3, Table 10).

Q2 (other ranking methods): We compared 36 alternatives to the mean from the scorio library you pointed to. 31 of them reproduce every certified ordering wherever they converge. They still disagree on 5–34% of uncertified pairs and on which model is #1 (§6.5, Table 12).

Q3 (validating item budgets): We check budgets computed from a pilot of 100–300 items on disjoint held-out items. The new pilot simulation has a mean error of 4.3%. On MMLU-Pro it predicts 4.67 tiers against 4.55 observed, where the closed form gave 1.24 (Table 8).

If any of this is unclear or unconvincing, we would be glad to clarify before the deadline. We would also be grateful if you would consider whether the revision changes your assessment.

Comments on the paper

1 comment

Thank you again for three precise questions. As the discussion period closes on 18 November, we wanted to check whether our replies address them. In brief:

  • The rank-set rule you proposed is now part of Proposition 2 (§4). For our single-step procedures it gives the same tiers as Definition 1 on every board (Table 7).
  • A pilot of 100–300 items predicts the mean tier count on disjoint held-out items up to 39× the pilot size. The new pilot simulation has a mean absolute error of 4.3% (Table 8).
  • The full MMLU-Pro board is now computed exactly, with no assumption on how model errors pair: it certifies 4 tiers (§6.2, App. E).

If anything remains unclear, we would be glad to clarify before the deadline. If our replies do address your questions, we would be grateful if you would take the revision into account in your final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.