- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a statistical framework for evaluating the reliability of LLM leaderboards. Instead of reporting only individual model rankings, it introduces certified tiers and simultaneous rank sets to identify which model comparisons are statistically supported. The paper also develops an approximate capacity law to estimate how benchmark size affects ranking resolution. Experiments across five public leaderboards show that many displayed rankings cannot be reliably distinguished, highlighting the importance of uncertainty-aware model evaluation.
Critical strengths
- The paper addresses an important and practical problem in LLM evaluation. The proposed certified tiers provide a more reliable and interpretable way to compare models than conventional leaderboard rankings.
- The experiments cover five major leaderboards with different evaluation settings. The combination of statistical analysis, theoretical insights, and a publicly available toolkit makes the work useful for both researchers and benchmark developers.
Critical weaknesses
- The proposed capacity law relies on simplifying assumptions about model score distributions and adjacent score gaps. Its predictions can be inaccurate when these assumptions are violated, as seen on MMLU-Pro. This limits its reliability for estimating benchmark requirements across different leaderboard settings.
- The certified-tier framework builds largely on existing simultaneous inference and multiple-testing methods, making its methodological novelty somewhat limited. Moreover, the out-of-sample validation covers only a small number of benchmark updates, leaving its practical generalizability less established.
Questions for authors
- How sensitive are the capacity-law predictions to violations of the assumed score-spacing distribution?
- Can the authors further clarify how certified tiers differ from existing simultaneous rank inference and multiple-testing approaches?
- Can the authors provide additional out-of-sample evaluations to demonstrate that the certified tier structure remains reliable across benchmark updates?
Follow-up
Thank you again for the positive assessment and your three questions. As the discussion period closes on 18 November, we wanted to check whether our replies address them. In brief:
If anything remains unclear, we would be glad to clarify before the deadline. We would also be grateful if you would consider whether the revision changes your assessment.