- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies how to statistically certify the correct ranking of models evaluated on benchmarks with binary scores. The specific goal is to use as few samples as possible subject to constraints on the error rate. The authors first establish instance-specific lower bounds that quantify the sample complexity of the problem. A ranking certification procedure is then developed based on McNemar's exact test with a refined significance threshold. Experiments on various data sets are conducted to illustrate the effectiveness of new bounds.
Critical strengths
-The paper provides instance-specific lower bounds for leaderboard certification. The construction of alternative distributions preserves the conditional distribution of the full score vector. The method can accommodate arbitrary dependence between pairwise comparisons.
-The proposed certification procedure uses McNemar's exact test with an improved significance threshold. Relaxing the threshold from to with finite-sample error control is useful. The connection to Holm procedure shows the merit of this result.
Critical weaknesses
-
The methodological novelty is limited: the certification procedure relies largely on classical McNemar's exact test, and its main algorithmic advance is a refined multiple-testing threshold. No general theoretical upper bound on its sample complexity is provided. Moreover, it remains unclear whether McNemar's exact test is the most suitable choice for complete-ranking certification. A more systematic theoretical analysis of the gap between the proposed method and the optimal certification procedure would strengthen the contribution.
-
Although leaderboard certification is the central problem of the paper, its motivation and formal mathematical definition are weak. The core issue is uncertainty quantification for rankings, but this perspective is not thoroughly investigated. Actual evaluation problems depend on the difficulty of separating similar models and the specific application setting. I am not convinced that providing abstract bounds alone adequately addresses these practical considerations.
-
Although the paper invokes multiple comparisons, it does not sufficiently explain the construction of the p-values, the treatment of dependence in the testing procedure, or the background to Holm's procedure. The connection to closed testing is also not discussed. Without a more systematic account of these statistical foundations, the proposed approach appears ad hoc.
-
The writing is not good. For example, Section 2.1 has a title that does not directly convey the section's main content. Section 2.3 lists several criteria without sufficiently developing the discussion, and the opening of Section 3 is convoluted and lacks a concise, professional exposition. The manuscript also sometimes refers to algorithms by theorem labels, such as "Theorem 4" in Figure 2 and Table 1, making it difficult to distinguish theoretical guarantees from concrete procedures. Some notation is introduced late or lacks an explicit definition, including and the sample mean . More concise and technically informative headings, clearer notation, and a better distinction between procedures and their guarantees would improve readability.
Questions for authors
Q1. Can the authors establish a general sample-complexity upper bound for the proposed certification procedure and quantify its gap to the lower bounds? Furthermore, since the proposal–validation approach in Appendix J achieves better sample efficiency in some settings, could alternative certification procedures improve further on the McNemar-based approach?
Q2. The proposed method focuses primarily on binary correctness scores from a fixed set of models evaluated on shared benchmark items. However, real-world LLM evaluations often involve non-binary scores, including evaluations based on LLM-as-a-judge. Can the authors clarify whether the proposed certification framework can handle such settings while maintaining comparable statistical guarantees and sample efficiency? Moreover, practical leaderboards may continuously incorporate new models and evaluation items. Can the proposed method still provide valid certification under such dynamic updates, particularly when models are added adaptively or the ranking is repeatedly updated as new evaluation results become available?