acceptodds
Under review as a conference paper at ICLR 2027

How Many Votes Make a Valid Leaderboard?

Abstract

Pairwise-vote leaderboards rank language models, but their displayed distinctions may exceed what the votes establish. We ask which leaderboard claims are statistically supported and which comparisons would resolve the remaining uncertainty. Our framework treats the best model, a top-k set, and a quality–cost frontier as separate claims. Under the Bradley–Terry model, each claim is tested against parameter regions supporting a different answer. We measure evidence by the smallest likelihood loss required to reach any such region. Tractable convex decompositions and conservative covers make these tests computationally feasible. We derive an anytime-valid stopping rule controlling false certification under adaptive sampling and stopping. To select new comparisons, we approximate the likelihood loss through the comparison graph’s information geometry. Our algorithm, CERTIFY, targets comparisons that strengthen evidence against the hardest remaining alternatives. An audit of 1.16 million Chatbot Arena votes across 129 models reveals a gap between stability and certification. Split-half rankings remain highly correlated, yet none of the 63 adjacent top-64 positions clears our proved threshold. Even 355,507 comparisons among the top 20 provide only 84% of the evidence required to certify the leader. Thus, a reproducible ordering need not establish its individual positions. On held-out recorded votes for a 16-model subset, CERTIFY identifies the leader in a median 12,480 comparisons. Across 15 runs at a nominal 5% error level, every returned answer is correct. Under the same stopping rule, competing sampling strategies require approximately 5.6–7.5x as many votes. Experiments also examine quality–cost and two-attribute frontiers, extending the approach beyond identifying a single winner. Together, these results connect the statistical support for leaderboard claims to the comparisons needed to establish them.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
LLM Evaluation, Pure Exploration, Ranking

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_TXf9OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2(+1)
Read review
Paper summary

This paper studies whether pairwise-vote leaderboards provide sufficient statistical evidence to support claims such as the best model, a top-k set, or a Pareto frontier. Under the Bradley–Terry model, it measures evidence through the minimum likelihood loss required to reach parameters that contradict the current claim and develops an anytime-valid certification rule. The proposed algorithm, CERTIFY, uses a quadratic approximation to guide comparisons toward the alternatives that are hardest to rule out. Experiments on Chatbot Arena votes and simulated data examine the distinction between ranking stability and statistical certification, and evaluate whether targeted sampling reduces the number of comparisons needed.

Critical strengths
  1. The paper considers an important and practically relevant question: whether the distinctions reported by a leaderboard are statistically supported by the available votes. Distinguishing ranking stability from statistical certification provides a useful perspective on the reliability of model comparisons.

  2. The framework connects statistical certification with adaptive comparison selection, using the remaining uncertainty to guide which model pairs should receive additional votes. This provides a practical strategy for collecting evidence relevant to the leaderboard claim of interest.

Critical weaknesses

Unclear presentation and notation. This paper, especially Sections 2 and 3, is difficult to follow because the motivation for the definitions and theoretical results, and the connections between them, are not clearly explained. They often read as a sequence of statements rather than a coherent development of the method. Several symbols are insufficiently explained when introduced, requiring readers to repeatedly search for their meanings. For example, lines 164–165 use an unexplained Σ\Sigma in the definition of Ft−1\mathcal{F}_{t-1}; The notation W(θ)\mathcal{W}(\theta) is used in lines 183–184 before the concept of a witness set is formally introduced in Definition 2.3. The experimental presentation has similar issues: labels such as GLR, GR-1, and ELO-CI appear in figures without clear introductions in the main text. Figure captions also do not consistently explain the plotted summary statistics, error bars, or certification criteria, making the results difficult to interpret. Also, RELAXED appears in the experimental comparisons without a clear operational definition in the main text. The brief reference to a “weakest-link relaxation” does not adequately explain how it differs from CERTIFY; readers must consult Appendix G.1 for this information. Some labels overlap in Figures 4 and 5, making them difficult to read.

Questions for authors
  1. Theorem 3.2 claims almost-sure termination under any sampling rule, whereas Appendix F.6 only establishes control of false certification. What additional sampling conditions are needed to guarantee termination, and should the theorem statement be revised accordingly?

  2. Appendix G.9 reports that SEEKS-v2 succeeds in only 47 of 60 runs despite a nominal 99% correctness guarantee and acknowledges a possible implementation error. Have the authors identified the cause, and how does this issue affect the reliability of the sample-efficiency comparisons?

  3. Could the authors reconcile the top-20 certification results in the main text with Appendix G.6? The main text reports successful certification using 13,394 votes, but the appendix withdraws this result and states that no method certifies under the proved threshold. Which reported gains remain valid, and which apply only to the reduced K=16K=16 subset?

  4. Could the authors clarify how to interpret Figure 1, including the certification thresholds and denominators in panel (a), the summary statistics and error bars, and the treatment of runs that reach the budget cap? Panel (b) also shows uncertainty-directed sampling requiring the most votes, while the caption describes it as the strongest baseline; which is correct?

2 comments

Thank you for the detailed review. Your comments helped us clarify the termination guarantee, investigate the SEEKS-v2 failures, and bring the main text into line with the appendix.

W1. Presentation and notation. Section 3 now explains how the witness set W(theta), the family of convex pieces C(theta), and the cover C_t fit together. We define Z_t at first use, distinguish the pair set from witness regions, and use separate notation for summation and the generated σ-algebra. We also corrected the likelihood contrast to theta_i - theta_j and fixed a stray brace in the interior notation. Section 5 and Appendix G.1 define the method names. We clarified the captions and moved overlapping labels in Figures 4 and 5.

Q1. Can the rule run forever? Yes, under arbitrary sampling. Proposition F.8 gives an example where the sampling rule leaves a decisive contrast unidentified, so the witness price stays at zero. Proposition F.9 proves that CERTIFY terminates almost surely with forced exploration, bounded scores, a strict claim gap, and unlimited votes. Theorem 4.2 states the anytime error guarantee separately.

Q2. Why did SEEKS-v2 recover the correct set in only 47 of 60 runs? The failed runs returned fewer than k items after the elimination step discarded candidates below the pivot. All 30 instrumented runs exited through the published stopping condition before reaching our iteration cap. In a separate check, our EQS implementation returned exactly min(k, n) items on all 400 test instances.

The pivot choice may explain why SEEKS-v2 failed more often than SEEKS, which recovered 58 of 60 sets. The two implementations share the DI subroutine and elimination logic, but SEEKS-v2 uses EQS and SEEKS uses TKS. Across 12 replications per method, the EQS pivot ranked closer to the top on average (1.48 versus 1.92), consistent with more true top-k items being eliminated. That small comparison is suggestive, and differences in the accuracy schedule could still matter. See §G.9 for the reproduction check and §G.8 for the direct CERTIFY–GR-1 comparison.

Q3. Which routing gains still hold? On the 16-model field, CERTIFY used 15.7% of the median votes required by uniform + GLR. The four baselines using the same stopping boundary needed 5.5–6.9× as many votes, and every rule returned the correct leader in all 15 runs (Table 5). We corrected eight passages in the main text to match the appendix. The full top-20 archive reached only 84% of the proved boundary, and all five shared-boundary rules exhausted the replay pool without certifying. See §§G.2 and G.6.

Q4. What does Figure 1 show? In panel (a), “all ranked” counts all K(K−1)/2 pairs and “consecutive” counts the K−1 adjacent pairs. At K = 64, the simultaneous-control series gives 90.6% and 4.8%, respectively. Under the proved boundary, zero of the 63 adjacent pairs certify (Table 4). The error bars are 95% Wilson score intervals for the plotted proportions. Panel (b) shows medians over 15 seeds, with ratios of medians as labels. Panel (c) uses 25 seeds. The hatched Elo-CI bars are lower bounds because all 25 runs reached the three-million-vote cap. If these address your concerns, we would be grateful if you would consider revising your score.

Thank you for the authors' response. I appreciate these clarifications, and I will increase my score.

Reviewer_SwGgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CERTIFY, a framework for statistically certifying leaderboard claims and adaptively allocating pairwise comparisons. It formulates incorrect ranking claims as witness regions, develops likelihood-ratio-based stopping rules, and uses an information-geometric approximation to guide comparison selection. The framework covers top-1 identification, top-k selection, and Pareto-frontier certification. Experiments on Chatbot Arena votes and synthetic benchmarks demonstrate the potential of adaptive sampling to reduce the number of comparisons required for certification.

Critical strengths
  1. Important problem formulation. The paper distinguishes ranking stability from statistical evidence supporting individual leaderboard claims. This distinction is practically relevant and provides a clear motivation for certification beyond conventional ranking estimation.

  2. A coherent technical framework. The combination of witness regions, likelihood-ratio-based certification, and adaptive comparison selection is technically interesting. The framework also extends beyond top-1 identification to top-k sets and Pareto frontiers, with empirical evaluation on recorded votes and synthetic settings.

Critical weaknesses
  1. Theorem 3.2 claims that the procedure stops almost surely under any sampling rule, but Appendix F.6 establishes the error-control bound without proving almost-sure termination. In particular, a sampling rule that repeatedly samples only a subset of model pairs may leave some witness regions insufficiently explored, preventing certification.

  2. The formal error-control guarantee relies on the Bradley–Terry model, while the empirical analysis reports discrepancies between predicted and observed win rates on Chatbot Arena data. The stated guarantee therefore does not automatically transfer to the real-data setting. As a secondary presentation concern, the manuscript lacks dedicated Related Work and Conclusion sections, making the positioning against prior work and the summary of the paper's limitations less explicit.

Questions for authors
  1. As noted in Weakness 1, why does Theorem 3.2 in the main text claim that (\mathbb{P}\theta(\tau < \infty)=1), whereas the proof in Appendix F.6 (around line 1222) establishes only that (\mathbb{P}\theta(\tau < \infty \text{ and } \operatorname{ans}(\hat{\theta}_\tau)\neq \operatorname{ans}(\theta))\leq\delta)? Could the authors clarify this discrepancy and provide a proof of the almost-sure termination claim?

  2. Given the observed deviations from the Bradley–Terry model on Chatbot Arena data, what guarantees remain under model misspecification? In particular, why should misspecification affect sample efficiency rather than the probability of false certification?

  3. The main text (Lines 103–106 and 145–151) reports that CERTIFY identifies the top model in the full top-20 setting using 31% of the votes required by uniform routing, with 13,394 votes for CERTIFY versus 43,634 for uniform sampling. However, Appendix G.6 (Lines 1638–1645) explicitly states: “We report it ... they are withdrawn.” Given that these results are explicitly withdrawn and no method successfully certifies the leader in the full top-20 replay under the proved threshold, why do the claims and figures in the main text remain inconsistent with the corrected experimental results, and why are the full top-20 and 16-model subset results not clearly distinguished?

3 comments

Thank you for detailed feedback on our results. Your question about misspecification also prompted us to work out how it affects the stopping threshold.

W1 / Q1. Does Theorem 3.2 guarantee termination for any sampling rule? No. A sampling rule can leave a decisive contrast unidentified, keeping the corresponding witness price at zero indefinitely (Proposition F.8). With forced exploration, bounded scores, a strict claim gap, and unlimited votes, CERTIFY terminates almost surely (Proposition F.9). The revision states anytime validity separately. Under Bradley–Terry, Theorem 4.2 bounds the probability of stopping and returning an incorrect answer by δ for any sampling rule with a complete witness cover.

W2 / Q2. Can misspecification increase false certification? Yes. It can introduce drift into the likelihood ratio. At the Bradley–Terry projection, the first-order term cancels when averaged under the design weights. Under its stated conditions, Theorem G.3 gives a uniform threshold multiplier of 3.55× at K = 16. The 1.43× value applies to the binding witness (Remark G.4). In simulations, we observed zero errors against the Condorcet winner in 40 runs per setting, up to seven times the measured Arena deviation.

Under a balanced design, the Bradley–Terry projection ranks models by Borda score, which can be computed directly from the pairwise win rates. Under adaptive sampling, the projected winner can depend on the design. It stayed the same across all 33 designs we tested. A finite-time adaptive guarantee remains open and would require quantitative control of how quickly the sampling design converges. See §G.6.

Related Work and Conclusion. We added a Related Work section (§2) covering benchmark stability, pairwise ranking, and anytime inference. Section 6 explains the conditions for termination, how the projection depends on the sampling design, and why the full top-20 archive falls short of certification.

Q3. Which numbers should readers use? Table 5 reports the 16-model comparison. CERTIFY used a median of 12,480 votes versus 79,460 for uniform + GLR. Uncertainty used the most votes among the four baselines sharing CERTIFY’s stopping rule. We found eight passages that still used superseded results, corrected them to match the appendix, and marked the changes in blue. On the full top-20 field, the archive fell short of the proved boundary and the shared-boundary rules exhausted the replay pool. We withdrew the earlier comparison and its 31% figure. See §§G.2 and G.6 for the audit and §G.4 for the corrected frontier counts.

If these address your concerns, we would be grateful if you would consider revising your score.

All three of your three issues have been addressed in the revision.

W1. Proposition F.9 proves almost-sure termination for CERTIFY from the forced-exploration floor alone, with explicit constants. Proposition F.8 proves the matching converse: a rule that starves a contrast never certifies, so the exploration floor in Algorithm 1 is necessary and not merely prudent. The validity guarantee the paper rests on, Theorem 4.2, never referenced termination and is unchanged.

W2. Theorem G.3 bounds misspecification as a constant factor on the threshold rather than a term growing with t, and Remark G.4 says which figure is the guarantee. Separately, under a balanced design the projection provably orders models by Borda score, a model-free functional.

Related Work is restored as Section 2 and Conclusion and Limitations as Section 6.

We would be grateful if you would consider whether this supports a higher score. Otherwise, please let us know if there is a point you would like us to take further.

Thank you for your reply. I believe it has resolved most of my doubts, and I will reconsider my score

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.