acceptodds
Under review as a conference paper at ICLR 2027

Practical Certification of Binary-Score Leaderboards

Abstract

Leaderboards report orderings between models, rarely with a guarantee that the reported directions are right. We study certification of the population ranking of a fixed set of models scored correct or incorrect on shared, independently sampled items. We derive instance-specific lower bounds on the number of items any procedure needs to certify the ranking: one for a single comparison, given by an exact finite sum, and a mixture bound that prices several comparisons at once without assumptions on their dependence. We then show that testing all pairs of models with McNemar's test at in place of , and releasing the order only when all pass, certifies a complete ranking with error at most . On eighteen configurations from five benchmark families, resampling each benchmark's own items, this certificate needs to times the larger lower bound to certify the complete ranking correctly of the time, and McNemar's test on the hardest single comparison alone is within of its bound.

Primary area:
learning theory
Keywords:
LLM evaluation, benchmark evaluation, leaderboards, ranking, certification, multiple testing, McNemar test, instance-specific lower bounds, change of measure, learning theory

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_r7WGOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how to statistically certify the correct ranking of models evaluated on benchmarks with binary scores. The specific goal is to use as few samples as possible subject to constraints on the error rate. The authors first establish instance-specific lower bounds that quantify the sample complexity of the problem. A ranking certification procedure is then developed based on McNemar's exact test with a refined significance threshold. Experiments on various data sets are conducted to illustrate the effectiveness of new bounds.

Critical strengths

-The paper provides instance-specific lower bounds for leaderboard certification. The construction of alternative distributions preserves the conditional distribution of the full score vector. The method can accommodate arbitrary dependence between pairwise comparisons.

-The proposed certification procedure uses McNemar's exact test with an improved significance threshold. Relaxing the threshold from δ/K\delta/K to δ/(M−1)\delta/(M-1) with finite-sample error control is useful. The connection to Holm procedure shows the merit of this result.

Critical weaknesses
  1. The methodological novelty is limited: the certification procedure relies largely on classical McNemar's exact test, and its main algorithmic advance is a refined multiple-testing threshold. No general theoretical upper bound on its sample complexity is provided. Moreover, it remains unclear whether McNemar's exact test is the most suitable choice for complete-ranking certification. A more systematic theoretical analysis of the gap between the proposed method and the optimal certification procedure would strengthen the contribution.

  2. Although leaderboard certification is the central problem of the paper, its motivation and formal mathematical definition are weak. The core issue is uncertainty quantification for rankings, but this perspective is not thoroughly investigated. Actual evaluation problems depend on the difficulty of separating similar models and the specific application setting. I am not convinced that providing abstract bounds alone adequately addresses these practical considerations.

  3. Although the paper invokes multiple comparisons, it does not sufficiently explain the construction of the p-values, the treatment of dependence in the testing procedure, or the background to Holm's procedure. The connection to closed testing is also not discussed. Without a more systematic account of these statistical foundations, the proposed approach appears ad hoc.

  4. The writing is not good. For example, Section 2.1 has a title that does not directly convey the section's main content. Section 2.3 lists several criteria without sufficiently developing the discussion, and the opening of Section 3 is convoluted and lacks a concise, professional exposition. The manuscript also sometimes refers to algorithms by theorem labels, such as "Theorem 4" in Figure 2 and Table 1, making it difficult to distinguish theoretical guarantees from concrete procedures. Some notation is introduced late or lacks an explicit definition, including KK and the sample mean θ^\hat{\theta}. More concise and technically informative headings, clearer notation, and a better distinction between procedures and their guarantees would improve readability.

Questions for authors

Q1. Can the authors establish a general sample-complexity upper bound for the proposed certification procedure and quantify its gap to the lower bounds? Furthermore, since the proposal–validation approach in Appendix J achieves better sample efficiency in some settings, could alternative certification procedures improve further on the McNemar-based approach?

Q2. The proposed method focuses primarily on binary correctness scores from a fixed set of models evaluated on shared benchmark items. However, real-world LLM evaluations often involve non-binary scores, including evaluations based on LLM-as-a-judge. Can the authors clarify whether the proposed certification framework can handle such settings while maintaining comparable statistical guarantees and sample efficiency? Moreover, practical leaderboards may continuously incorporate new models and evaluation items. Can the proposed method still provide valid certification under such dynamic updates, particularly when models are added adaptively or the ranking is repeatedly updated as new evaluation results become available?

Reviewer_7hWNOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper explores - in a leaderboard with binary scores - how many independently sampled, shared evaluation entries are needed to certify a ranking.

There are telltale signs of LLM-generated text throughout this paper, e.g., awkward phrasings.

Critical strengths

The general question of "when can we trust a leaderboard" is interesting.

Critical weaknesses

I skimmed the math - not carefully - but it seems reasonable.

My overall reaction to the paper is: Who cares? It's a matter of taste, but I just don't find this an interesting research question. Why is complete cerification even a worthwhile goal? When I'm looking at a leaderboard, I really only care about the top few entries, and of those, all I really care about is: what's the "top group". In other words, I want to know the top performing systems and the equivalence class they are in - in other words, error bars.

I suppose the paper provides a mathematically interesting answer, but to a very narrow statistical question that has limited relevance.

Questions for authors

Why is this an interesting research question to tackle?

Reviewer_Rn13OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how many shared binary-scored items are needed to certify a fixed model ranking. It develops instance-specific lower bounds and a less conservative complete-order certificate, evaluated through benchmark replay and held-out experiments.

Critical strengths
  1. Leaderboard reliability is a practical problem, especially when models have very similar scores. Helping benchmark organizers determine which ranking claims are statistically justified is valuable.
  2. The theoretical treatment appears careful. The instance-specific lower bounds and the ranking-level error control built on classical McNemar tests form a coherent contribution, connecting pairwise disagreements to certification of the whole leaderboard.
Critical weaknesses
  1. The exposition is precise but gives limited intuition for the main results. Despite the schematic and running MMLU example, the connection between the lower bound, the all-or-nothing ranking certificate, and what an evaluator can report at a fixed budget remains difficult to follow.
  2. The main replay experiments use selected 10- or 20-model sets with a minimum adjacent gap. Appendix R provides useful sensitivity checks, but the empirical efficiency conclusions have not been established for larger leaderboards with naturally occurring near-ties.
Questions for authors
  1. Could you provide a worked example showing how to interpret the lower bound alongside the complete-order and partial-order certificates at a fixed item budget?
  2. How does certification coverage change for larger model sets with naturally occurring small gaps, beyond the configurations tested in Appendix R?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.