What Is the Top of a Leaderboard Worth? Measuring and Correcting Selection Bias in Tabular and Time-Series Benchmarks

A leaderboard's top row is a selected quantity: the winner minimizes average error over the very datasets on which its margin is then reported. We measure what this costs on four method $\times$ dataset matrices (TabArena, GIFT-Eval, fev-bench, and a long-horizon grid of our own) with a subsampling audit whose ground truth comes from disjoint held-out datasets. At $D^{\prime }=8$, the suite size typical of long-horizon forecasting tables, TabArena's top entry keeps only $8.0\pm 0.3$% of its reported advantage over a tuned gradient-boosting baseline out of sample on our primary scale ($-8.5$% to $+38$% across three aggregations, so $0.4$ is the defensible ceiling); the reported number one is the true number one $38$% of the time; on GIFT-Eval the leader's margin over the runner-up reverses sign; and drawing whole source families rather than i.i.d. units costs more still. Textbook remedies fail: leave-one-dataset-out returns exactly the naive value in $100$% of folds on all four matrices, and the classical $\sqrt{2\log M}$ envelope overpredicts measured optimism by $4.8$-$6.0\times$, mostly because entries differ in true effect rather than because they correlate ($4.8$-$4.9\times$ against $1.8$-$2.1\times$). We therefore introduce SCB, a selection-corrected bootstrap that resamples datasets jointly across methods around a variance-matched plug-in, needs only the published matrix, and has a cluster-blocked variant for families. It removes bias where there is bias and does not cry wolf where there is none: at full suite size TabArena's margin gives up about a third, while two champions and a published ICLR 2025 table keep $97.7$% or more. Code, data and corrected leaderboards are in the anonymized supplement.