acceptodds
Under review as a conference paper at ICLR 2027

What Is the Top of a Leaderboard Worth? Measuring and Correcting Selection Bias in Tabular and Time-Series Benchmarks

Abstract

A leaderboard's top row is a selected quantity: the winner minimizes average error over the very datasets on which its margin is then reported. We measure what this costs on four method dataset matrices (TabArena, GIFT-Eval, fev-bench, and a long-horizon grid of our own) with a subsampling audit whose ground truth comes from disjoint held-out datasets. At , the suite size typical of long-horizon forecasting tables, TabArena's top entry keeps only % of its reported advantage over a tuned gradient-boosting baseline out of sample on our primary scale (% to % across three aggregations, so is the defensible ceiling); the reported number one is the true number one % of the time; on GIFT-Eval the leader's margin over the runner-up reverses sign; and drawing whole source families rather than i.i.d. units costs more still. Textbook remedies fail: leave-one-dataset-out returns exactly the naive value in % of folds on all four matrices, and the classical envelope overpredicts measured optimism by -, mostly because entries differ in true effect rather than because they correlate (- against -). We therefore introduce SCB, a selection-corrected bootstrap that resamples datasets jointly across methods around a variance-matched plug-in, needs only the published matrix, and has a cluster-blocked variant for families. It removes bias where there is bias and does not cry wolf where there is none: at full suite size TabArena's margin gives up about a third, while two champions and a published ICLR 2025 table keep % or more. Code, data and corrected leaderboards are in the anonymized supplement.

Primary area:
datasets and benchmarks
Keywords:
benchmark selection bias, winner's curse, leaderboards, tabular learning, time series forecasting, bootstrap, bias correction

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_mxdJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates uncertainty in current ML leaderboards and argues that reported performance is specific to the evaluated datasets and may not generalize to unseen datasets. It first conducts an audit by comparing the reported advantage of the selected leaderboard winner against its performance on held-out datasets, and then proposes SCB (selection-corrected bootstrap), which uses bootstrap resampling to account for selection bias.

Critical strengths

The paper focuses on an important issue where existing benchmarks may overestimate the performance advantage of higher ranked models due to selection bias and proposes a reasonable bootstrap approach to account for this bias.

Critical weaknesses
  1. My main concern is the novelty of the work. The audit experiment showing that performance measured on a small set of datasets may not generalize to unseen datasets is expected and not particularly surprising. The proposed SCB method mainly combines existing statistical ideas, including bootstrap resampling, variance estimation, etc which limits the methodological novelty of the work.

  2. Existing literature has also studied over-optimism in benchmark and leaderboard evaluation. E.g, Charlie, Junior. "The Winner’s Curse in LLM Pipeline Selection: How Correlated Candidate Pools Break Best-of-N Reporting, and How to Correct It."; Xu, Yang, et al. "Towards Reliable LLM Evaluation: Correcting the Winner's Curse in Adaptive Benchmarking."; This further limits the novelty of the work.

Questions for authors

Please see my comments in the Weaknesses section above. It would be helpful if the authors could further clarify the novelty of this work relative to existing literature, and the methodological contributions.

Reviewer_Dv91OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes SCB estimator to correct the overestimated advantage of the top-ranked method on a benchmark.

Critical strengths

None

Critical weaknesses

The paper lacks a clear logic flow. I felt that the majority of this paper is written without human logic, at least for me it is hard to read. The authors are supposed use AI in a healthy way, instead of simply generating the whole paper and submit irresponsibly!

Even with the absence of readability issues, I didn't find enough novelty and contributions in this paper. The proposed SCB estimator is simply a bootstrap-based correction that estimates the advantage of the best model. It cannot be used to identify a better model, and it's not clear how this benchmark can be useful in practice.

Questions for authors

(1) How do you justify the practical effectiveness of the SCB estimator? If the top-ranked model is not the best model, how can we use this estimator to identify a better model?

(2) Can you provide theoretical guarantee to support the effectiveness of this estimator?

Reviewer_7MTTOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies selection bias in tabular and time-series leaderboards. It audits four benchmark result matrices and proposes a selection-corrected bootstrap (SCB), combining variance-matched shrinkage with joint resampling to correct optimism in the selected winner’s advantage over a fixed baseline.

Critical strengths

1.The paper addresses an important practical problem: selection bias in benchmark-based model comparisons. 2.The audits span tabular and time-series benchmarks and provide useful evidence on leaderboard instability.

Critical weaknesses
  1. W1. SCB's variance calibration contains two substantive mismatches. Section 5 omits cross-method covariance when estimating the variance of true effects. Appendix A.5 uses the variance of an equally weighted family average for a unit-weighted mean, despite unequal family sizes. Both issues directly affect shrinkage and bias correction, requiring a corrected derivation and re-evaluation of the results.
  2. W2. SCB combines established variance-matching and bootstrap bias-correction ideas, but its methodological advance remains insufficiently established. The paper omits a direct comparison with BBC-CV adapted to the dataset level (Tsamardinos et al., 2018). Without this comparison, it is unclear whether the proposed construction improves bias or estimation error over closely related existing approaches.
Questions for authors

See Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.