- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2(+1)
Read review
Paper summary
This paper studies whether pairwise-vote leaderboards provide sufficient statistical evidence to support claims such as the best model, a top-k set, or a Pareto frontier. Under the Bradley–Terry model, it measures evidence through the minimum likelihood loss required to reach parameters that contradict the current claim and develops an anytime-valid certification rule. The proposed algorithm, CERTIFY, uses a quadratic approximation to guide comparisons toward the alternatives that are hardest to rule out. Experiments on Chatbot Arena votes and simulated data examine the distinction between ranking stability and statistical certification, and evaluate whether targeted sampling reduces the number of comparisons needed.
Critical strengths
-
The paper considers an important and practically relevant question: whether the distinctions reported by a leaderboard are statistically supported by the available votes. Distinguishing ranking stability from statistical certification provides a useful perspective on the reliability of model comparisons.
-
The framework connects statistical certification with adaptive comparison selection, using the remaining uncertainty to guide which model pairs should receive additional votes. This provides a practical strategy for collecting evidence relevant to the leaderboard claim of interest.
Critical weaknesses
Unclear presentation and notation. This paper, especially Sections 2 and 3, is difficult to follow because the motivation for the definitions and theoretical results, and the connections between them, are not clearly explained. They often read as a sequence of statements rather than a coherent development of the method. Several symbols are insufficiently explained when introduced, requiring readers to repeatedly search for their meanings. For example, lines 164–165 use an unexplained in the definition of ; The notation is used in lines 183–184 before the concept of a witness set is formally introduced in Definition 2.3. The experimental presentation has similar issues: labels such as GLR, GR-1, and ELO-CI appear in figures without clear introductions in the main text. Figure captions also do not consistently explain the plotted summary statistics, error bars, or certification criteria, making the results difficult to interpret. Also, RELAXED appears in the experimental comparisons without a clear operational definition in the main text. The brief reference to a “weakest-link relaxation” does not adequately explain how it differs from CERTIFY; readers must consult Appendix G.1 for this information. Some labels overlap in Figures 4 and 5, making them difficult to read.
Questions for authors
-
Theorem 3.2 claims almost-sure termination under any sampling rule, whereas Appendix F.6 only establishes control of false certification. What additional sampling conditions are needed to guarantee termination, and should the theorem statement be revised accordingly?
-
Appendix G.9 reports that SEEKS-v2 succeeds in only 47 of 60 runs despite a nominal 99% correctness guarantee and acknowledges a possible implementation error. Have the authors identified the cause, and how does this issue affect the reliability of the sample-efficiency comparisons?
-
Could the authors reconcile the top-20 certification results in the main text with Appendix G.6? The main text reports successful certification using 13,394 votes, but the appendix withdraws this result and states that no method certifies under the proved threshold. Which reported gains remain valid, and which apply only to the reduced subset?
-
Could the authors clarify how to interpret Figure 1, including the certification thresholds and denominators in panel (a), the summary statistics and error bars, and the treatment of runs that reach the budget cap? Panel (b) also shows uncertainty-directed sampling requiring the most votes, while the caption describes it as the strongest baseline; which is correct?
2 comments
Thank you for the authors' response. I appreciate these clarifications, and I will increase my score.
Thank you for the detailed review. Your comments helped us clarify the termination guarantee, investigate the SEEKS-v2 failures, and bring the main text into line with the appendix.
W1. Presentation and notation. Section 3 now explains how the witness set
W(theta), the family of convex piecesC(theta), and the coverC_tfit together. We defineZ_tat first use, distinguish the pair set from witness regions, and use separate notation for summation and the generated σ-algebra. We also corrected the likelihood contrast totheta_i - theta_jand fixed a stray brace in the interior notation. Section 5 and Appendix G.1 define the method names. We clarified the captions and moved overlapping labels in Figures 4 and 5.Q1. Can the rule run forever? Yes, under arbitrary sampling. Proposition F.8 gives an example where the sampling rule leaves a decisive contrast unidentified, so the witness price stays at zero. Proposition F.9 proves that CERTIFY terminates almost surely with forced exploration, bounded scores, a strict claim gap, and unlimited votes. Theorem 4.2 states the anytime error guarantee separately.
Q2. Why did SEEKS-v2 recover the correct set in only 47 of 60 runs? The failed runs returned fewer than
kitems after the elimination step discarded candidates below the pivot. All 30 instrumented runs exited through the published stopping condition before reaching our iteration cap. In a separate check, our EQS implementation returned exactlymin(k, n)items on all 400 test instances.The pivot choice may explain why SEEKS-v2 failed more often than SEEKS, which recovered 58 of 60 sets. The two implementations share the DI subroutine and elimination logic, but SEEKS-v2 uses EQS and SEEKS uses TKS. Across 12 replications per method, the EQS pivot ranked closer to the top on average (1.48 versus 1.92), consistent with more true top-
kitems being eliminated. That small comparison is suggestive, and differences in the accuracy schedule could still matter. See §G.9 for the reproduction check and §G.8 for the direct CERTIFY–GR-1 comparison.Q3. Which routing gains still hold? On the 16-model field, CERTIFY used 15.7% of the median votes required by uniform + GLR. The four baselines using the same stopping boundary needed 5.5–6.9× as many votes, and every rule returned the correct leader in all 15 runs (Table 5). We corrected eight passages in the main text to match the appendix. The full top-20 archive reached only 84% of the proved boundary, and all five shared-boundary rules exhausted the replay pool without certifying. See §§G.2 and G.6.
Q4. What does Figure 1 show? In panel (a), “all ranked” counts all
K(K−1)/2pairs and “consecutive” counts theK−1adjacent pairs. AtK = 64, the simultaneous-control series gives 90.6% and 4.8%, respectively. Under the proved boundary, zero of the 63 adjacent pairs certify (Table 4). The error bars are 95% Wilson score intervals for the plotted proportions. Panel (b) shows medians over 15 seeds, with ratios of medians as labels. Panel (c) uses 25 seeds. The hatched Elo-CI bars are lower bounds because all 25 runs reached the three-million-vote cap. If these address your concerns, we would be grateful if you would consider revising your score.