- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates uncertainty in current ML leaderboards and argues that reported performance is specific to the evaluated datasets and may not generalize to unseen datasets. It first conducts an audit by comparing the reported advantage of the selected leaderboard winner against its performance on held-out datasets, and then proposes SCB (selection-corrected bootstrap), which uses bootstrap resampling to account for selection bias.
Critical strengths
The paper focuses on an important issue where existing benchmarks may overestimate the performance advantage of higher ranked models due to selection bias and proposes a reasonable bootstrap approach to account for this bias.
Critical weaknesses
-
My main concern is the novelty of the work. The audit experiment showing that performance measured on a small set of datasets may not generalize to unseen datasets is expected and not particularly surprising. The proposed SCB method mainly combines existing statistical ideas, including bootstrap resampling, variance estimation, etc which limits the methodological novelty of the work.
-
Existing literature has also studied over-optimism in benchmark and leaderboard evaluation. E.g, Charlie, Junior. "The Winner’s Curse in LLM Pipeline Selection: How Correlated Candidate Pools Break Best-of-N Reporting, and How to Correct It."; Xu, Yang, et al. "Towards Reliable LLM Evaluation: Correcting the Winner's Curse in Adaptive Benchmarking."; This further limits the novelty of the work.
Questions for authors
Please see my comments in the Weaknesses section above. It would be helpful if the authors could further clarify the novelty of this work relative to existing literature, and the methodological contributions.