acceptodds
Under review as a conference paper at ICLR 2027

The Shortlist Illusion: Why Leaderboard Separation Fails Candidate Models

Abstract

Public LLM leaderboards rank hundreds of models across macro benchmark frames, and practitioners routinely rely on these aggregate scores to guide high-stakes model selection and enterprise deployment among a small shortlist of candidate models. We show that macro-level benchmark performance fundamentally fails to transfer to candidate shortlists, creating a pervasive shortlist illusion. Using cohort-conditional separation—the fraction of within-cohort model pairs distinguishable beyond an operational performance margin—we audit 9,103 model–date records for 4,108 models across 133 snapshots of the Open LLM Leaderboard v2. We isolate three distinct failure mechanisms that misinform deployment decisions: (1) Non-identification: separation is fundamentally undefined without naming the comparison cohort; on the same MUSR snapshot, a 35-model frame separates 69.6% of pairs, whereas a 5-model candidate shortlist separates only 20.0%. (2) Selection-induced compression: filtering top- models restricts the observed score range, collapsing discriminative power; on MATH-Hard, separation plummets from 91.1% across the full frame to 0% among the top 5 models. (3) Small-sample noise: while random sub-cohorts are unbiased in expectation, small candidate sets suffer severe dispersion, reducing top-model selection to near-chance agreement. We translate these findings into an actionable, zero-cost six-field reporting contract and recommend cohort conditionality as an indispensable auditing requirement to curb misleading model cards and prevent wasteful deployment choices.

Primary area:
datasets and benchmarks
Keywords:
LLM evaluation, leaderboard reliability, model selection, cohort-conditional evaluation, shortlist illusion

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_HehrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper show that public LLM leaderboards may not useful for practically selection.

Critical strengths

The problem setup is clear and easy to follow, with C1/C2/C3 corresponding to full cohorts, score-selected top-K cohorts, and target cohorts. The empirical results are easy to understand.

Critical weaknesses
  1. The main conclusion is relatively intuitive, and several theoretical results follow almost directly from the definition of cohort-conditional separation, limiting the theoretical novelty of this paper. I do not get much new intuition after reading the theortical analysis.
  2. The paper analyzes the stability of a custom separation statistic rather than actual model-selection accuracy, so the connection to real deployment decisions is not fully established and it requires more evidences.
Questions for authors
  1. Can the authors directly evaluate whether cohort-dependent separation actually leads to incorrect model-selection decisions, rather than only measuring the stability of the separation statistic? Please provide more details.
  2. I find that most of the propositions follow almost directly from the definition of cohort-conditional separation. I am curious that what insight does the formalization provide beyond the intuitive observation that "score gaps depend on the compared model set"?
5 comments

We thank the Reviewer for the constructive feedback, which strengthened our theoretical presentation and model-selection analysis.

W1: Theoretical Novelty of the Propositions

Stating Propositions 1–3 briefly in §3.2 while placing our U-statistic covariance derivation in App. A.4 made the theory appear closer to definitional statements than intended. We will integrate the closed-form degree-2 U-statistic variance decomposition directly into Proposition 3 (§3.2): Var[d(B,S,t,θ)]=1K2[π(1−π)∑pyp+∑p≠qypyq(π∣p∩q∣−π2)]\mathrm{Var}\big[d(B,S,t,\theta)\big] = \frac{1}{K^2}\Big[\pi(1-\pi)\sum_p y_p + \sum_{p\neq q}y_py_q\big(\pi_{|p\cap q|}-\pi^2\big)\Big] where K=(m2)K=\binom{m}{2}, N=(n2)N=\binom{n}{2}, π=K/N\pi=K/N, π1=m(m−1)(m−2)n(n−1)(n−2)\pi_1=\frac{m(m-1)(m-2)}{n(n-1)(n-2)} (∣p∩q∣=1|p\cap q|=1), and π0=m(m−1)(m−2)(m−3)n(n−1)(n−2)(n−3)\pi_0=\frac{m(m-1)(m-2)(m-3)}{n(n-1)(n-2)(n-3)} (p∩q=∅p\cap q=\emptyset). For 3≤m<n3 \le m < n ((n,m)≠(4,3)(n,m)\neq(4,3)), pairs sharing a model carry positive covariance (π1−π2>0\pi_1 - \pi^2 > 0). Thus, a 5-model shortlist does not yield 10 independent tests; shared-model dependence shrinks effective sample size below KK and analytically explains the 53.1%→95.4%53.1\% \to 95.4\% agreement recovery across m∈{5,10,20,50}m \in \{5, 10, 20, 50\} (Fig. 2).

W2: Connection to Real Deployment Decisions

We will expand §3.1 to show that d(B,C,t,θ)d(B,C,t,\theta) directly bounds binary and top-1 model-selection validity on shortlist CC:

  1. Pairwise Decision-Ambiguity Rate: Choosing between i,j∈Ci, j \in C under margin θ\theta is resolvable only if ∣ai−aj∣>θ|a_i - a_j| > \theta. Thus, d(B,C,t,θ)d(B,C,t,\theta) is the pairwise decision-resolution rate, and 1−d(B,C,t,θ)1 - d(B,C,t,\theta) is the exact probability that a binary choice in CC falls in the ≤θ\le \theta tie zone where raw ranks cannot support a reliable deployment choice.
  2. Bound on Top-1 Winner Margin: Whenever d(B,C,t,θ)=0d(B,C,t,\theta) = 0 (e.g., MATH-Hard Top-5, 2025-03-13), max⁡i,j∈C∣ai−aj∣≤θ\max_{i,j \in C}|a_i - a_j| \le \theta, guaranteeing Δ1,2=a(1)−a(2)≤θ\Delta_{1,2} = a_{(1)} - a_{(2)} \le \theta and rendering top-1 selection unresolvable beyond θ\theta.

Q1: Empirical Evaluation of Model-Selection Failures

In §5.1–§5.3, our audit of 133 OLLB v2 snapshots (4,108 models, 26,600 size-5 shortlists) directly quantifies three deployment selection failures:

  1. 74.3%–76.1% Pairwise Indecidability & Top-1 Tie Blindness (§5.2, Table 2): In Top-5 frontiers, pairwise ambiguity (1−d31 - d_3) surges from 21.6%–47.7%21.6\%\text{--}47.7\% (full frame) to 74.3% on GPQA (d3=25.7%d_3=25.7\%) and 76.1% on MUSR (d3=23.9%d_3=23.9\%) across 133 snapshots, and 100% (10/10 pairs) on MATH-Hard (2025-03-13, d3=0%d_3=0\%). In our 5-model MUSR shortlist ({42.06,42.20,43.52,43.65,45.37}\{42.06, 42.20, 43.52, 43.65, 45.37\}, §3.2), trusting the macro report (d3=69.6%d_3=69.6\%) commits to #1 (45.37%45.37\%) over #2 (43.65%43.65\%) on a 1.72 pp margin (#2 over #3 on 0.13 pp), where 80% (8/10) of pairs fail the 3 pp rule.
  2. 64.1%–66.5% Unresolved Choices Under Noise (§5.1): Propagating archived per-model standard errors (95% CI lower bound >3 pp> 3\text{ pp}) shows that 64.1% of separated pairs on GPQA (52.3%→20.2%52.3\% \to 20.2\%, retaining 35.9%35.9\%) and 66.5% on MUSR (60.6%→21.1%60.6\% \to 21.1\%, retaining 33.5%33.5\%) fail statistical resolution. Because Top-KK filtering compresses gaps into ≤3 pp\le 3\text{ pp} (Table 2), naive top-1 selection predominantly picks winners indistinguishable from noise.
  3. 46.9% Shortlist Benchmark-Selection Error (§5.3.1, Fig. 2b–c): Across 26,600 size-5 shortlists, choosing the most discriminative benchmark via full-frame d3d_3 matches the shortlist-optimal benchmark only 53.1% [49.9, 56.4] of the time—a 46.9% error rate (indistinguishable from 50%) incurring a 3.92 pp [3.54, 4.30] separation loss.

Q2: Insights Beyond "Score Gaps Depend on the Compared Set"

As clarified in §6.1, naive intuition cannot explain why a macro report fails to transfer or how to fix it. Our formalization yields three counter-intuitive insights:

  1. Refuting "Small-Shortlist Bias" (Prop. 3): For uniform shortlists, expected bias is zero (E[d(S)]=d(F)\mathbb{E}[d(S)] = d(F), ≤0.11 pp\le 0.11\text{ pp} vs. 13.9–18.5 pp13.9\text{--}18.5\text{ pp} SD, Table 3; bias/SD <1:170< 1:170). Transport error (10.8–14.8 pp10.8\text{--}14.8\text{ pp} at m=5m=5) is pure variance from shared-model covariance (π1−π2>0\pi_1 - \pi^2 > 0) and (m2)−1\binom{m}{2}^{-1} lattice steps.
  2. Sampling Noise vs. Selection Compression (Prop. 2 vs. 3): Uniform sampling error vanishes as m→nm \to n, whereas top-KK filtering breaks exchangeability and induces range compression (14.1–36.6 pp14.1\text{--}36.6\text{ pp}, Table 2) that need not vanish at any mm (d=0d=0 remains attainable while full-frame d→24/25d \to 24/25).
  3. Non-Monotonicity & Why "Better Benchmarks" Fail (Prop. 1): Since C′⊂CC' \subset C implies no ordering between d(C′)d(C') and d(C)d(C), expanding a shortlist can inflate or collapse separation. Because these failures stem from cohort projection rather than task quality, new benchmarks reproduce the illusion unless reports disclose our six-field contract (Table 7).

Thank you for the detailed response. The theoretical clarifications are helpful, but my main concern remains: the results quantify pairwise ambiguity and benchmark-selection disagreement, rather than directly evaluating incorrect model choices or downstream performance loss. So I think the connection to practical model selection is still not established and I decide to maintain my score.

Thank you again for the follow-up. To follow up on your remaining concern regarding incorrect model choices and downstream performance loss, we conducted a direct evaluation across the 26,600 size-5 candidate shortlists in OLLB v2, and will incorporate these results into §5.3.3 of the revised manuscript.

Core Conclusion: Even when holding the exact 5 candidate models in a shortlist fixed, a collapse in shortlist separation d3d_3 directly causes practitioners who pick the #1 model to select the wrong model in the majority of cases and suffer nearly twice the downstream accuracy loss:

  1. Incorrect Model Choices: For the exact same 5 candidate models, selecting the top-1 model via a low-separation benchmark with d3≤30%d_3 \le 30\% rather than a high-separation benchmark with d3≥70%d_3 \ge 70\% increases the false-winner rate under evaluation standard errors by 5.4×5.4\times, from 9.0% to 48.6%, and raises the wrong model choice rate on held-out downstream tasks IFEval and MMLU-PRO from 47.7% to 66.1%.
  2. Downstream Performance Loss: On the exact same 5 candidate models, relying on a low-separation benchmark nearly doubles downstream accuracy loss on IFEval and MMLU-PRO from 4.69 pp to 9.12 pp. Likewise, in Top-5 frontier selection, heavily compressed benchmarks such as GPQA and MUSR inflict 9.18–11.66 pp of downstream loss, nearly double the 5.39–5.82 pp loss on BBH and MATH-Hard.

We sincerely hope these direct results on incorrect model choices and downstream performance loss address your remaining concern, and we would be deeply grateful if you would kindly consider raising your score. Thank you again for your valuable guidance in helping us strengthen the paper!

Reviewer_auVxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the reliability of LLM leaderboards for practical model selection. The authors argue that a benchmark's ability to distinguish models depends on the cohort being compared. They define cohort-conditional separation, analyze the effects of score-based selection and random sampling, and propose a six-field reporting contract. Experiments on 4,108 models across 133 snapshots reveal substantial differences in separation rates across candidate cohorts.

My main concerns are: (i) the paper overlaps with prior work, while its benchmark coverage and direct validation of practical model selection remain limited (W1); and (ii) the proposed separation metric does not adequately capture the identifiability of the best-performing model or account for multi-metric deployment decisions (W2). Addressing these issues would strengthen the paper, and I would consider increasing my rating to 3. Otherwise, I may maintain my current rating or consider lowering it to 1 only if the authors' responses fail to resolve any substantive aspect of either W1 or W2.

Critical strengths
  • The motivation for evaluating benchmark discriminative power is reasonable and valuable.

    Practical model selection requires not only comparing absolute scores but also determining whether meaningful performance differences exist among candidate models.

  • The proposed approach incurs almost no additional computational cost and improves evaluation transparency.

    The authors systematically compare full eligible cohorts, score-selected Top-K cohorts, and randomly sampled target cohorts, and propose a six-field reporting contract.

Critical weaknesses

W1. Overlap with Prior Work and Limited Experimental Coverage

  • The novelty relative to existing studies needs clearer articulation. 📍R1, §3.2, Eq. (2) introduces a Discriminability Score (DS) that includes the following metric: ∑j<k1[∣sj−sk∣>ϵ](m2),\frac{\sum_{j<k}\mathbf{1}[|s_j-s_k|>\epsilon]}{\binom{m}{2}}, which is mathematically almost identical to the proposed cohort-conditional separation metric (differing only in notation) defined in 📍§3.1, Eq. (1): ∑i<j,  i,j∈C1[∣ai(B,t)−aj(B,t)∣>θ](∣C∣2)\frac{\sum_{i<j,\;i,j\in C}\mathbf{1}[|a_i(B,t)-a_j(B,t)|>\theta]}{\binom{|C|}{2}}. R1, §3.2 further combines it with score variability to measure discriminative power. Similarly, R2, §3.2 examines benchmark agreement across different model cohorts (Top-5, Top-10, and Top-15), while R3, §5.5 investigates how changes in model cohorts affect leaderboard results. The authors should clarify their contributions beyond these studies, particularly in terms of theoretical analysis and practical implications.
  • The experimental coverage is relatively limited compared with related work. Although the paper includes 4,108 models, it evaluates only four benchmarks from Open LLM Leaderboard v2. In comparison, R2, §2 covers over 40 benchmarks, R1, §4.1 covers 15, and R4, Introduction analyzes 11 benchmarks with 41,871 evaluation items. Thus, while the paper has strong model coverage, its benchmark diversity is more limited than that of these related studies. More importantly, the paper does not directly validate whether changes in separation rates lead to incorrect model selection. I suggest expanding the benchmark coverage and evaluating the relationship between separation rates and actual model selection outcomes to strengthen the paper's practical motivation.

W2. Limitations of the Separation Metric for Practical Model Selection

  • The proposed separation rate does not necessarily reflect the identifiability of the best-performing model. For instance, consider two benchmarks with model scores,

    • A = {5, 30, 79.9, 80, 80.1}. Under the authors' 3-percentage-point threshold, Benchmark A achieves a 70% separation rate, although its top two models differ by only 0.1 points.

    • B = {75, 75, 75, 75, 80}. In contrast, Benchmark B achieves only 40%, despite its best-performing model leading all others by 5 points.

    Thus, a higher separation rate does not necessarily indicate better identification of the best-performing model. This distinction need clarify because Section 6.3 connects separation rates to practical model selection reliability.

  • The framework also overlooks multi-metric deployment decisions. Real-world model selection often involves multiple benchmarks and constraints, such as F1 ≥ 0.90, p95 latency ≤ 40 ms, and peak memory ≤ 100 MB, rather than individual benchmark scores.

To strengthen the paper's novelty and practical impact, I encourage the authors to extend the analysis to more realistic model selection scenarios involving combinations of multiple benchmarks, evaluation metrics, and deployment constraints. I believe this idea could help address my main concerns in both W1 and W2, and making the paper's contributions more distinctive and relevant to real-world deployment.

References

Questions for authors
  1. Discrepancy between Figure 2(b) and the reported results.

    Section 5.3.1 reports a winner agreement of 53.1% (95% CI: [49.9%, 56.4%]) for five-model target cohorts, whereas Figure 2(b) appears to show approximately 67% with a different confidence interval. Could the authors clarify which result is correct and whether the figure and text use the same experimental settings?

  2. Interpretation of winner agreement and the chance-level baseline.

    The paper defines the winner as the benchmark with the highest separation rate, rather than the best-performing model. Could the authors clarify how this metric supports the conclusion that model selection approaches chance-level agreement? In particular, why is 50% used as the chance-level reference in Section 5.3.1, while Figure 2(b) shows a 25% uniform-guess baseline?

3 comments

We thank the Reviewer for the feedback and references [R1–R4].

W1.1: Novelty Beyond R1–R3

We will add [R1–R3] (§2.2, §3.1). While R1's DS shares Eq. (1)'s kernel, our estimand and theory differ:

  1. Static Score (R1) vs. Cohort U-Statistic: R1 scores benchmarks on a fixed pool. Holding BB fixed, we analyze Eq. (1) as a cohort-indexed degree-2 U-statistic over CC: (i) Prop. 1: d(B,C′,t,θ)d(B,C',t,\theta) is neither monotone in nor fixed by d(B,C,t,θ)d(B,C,t,\theta) (C′⊂CC'\subset C); (ii) Prop. 2: top-mm filtering (L(B,t,m)L(B,t,m)) compresses ranges at any mm (dθ→0d_\theta\to 0 locally vs. 24/2524/25 globally); (iii) Prop. 3 & App. A.4: uniform draws are unbiased (E[d(S)]=d(F)\mathbb{E}[d(S)]=d(F); bias ≤0.11\le 0.11 pp vs. 13.9–18.513.9\text{--}18.5 pp SD, Table 3), while pair covariance (π1−π2>0\pi_1-\pi^2>0) shrinks effective sample size below K=(m2)K=\binom{m}{2}.
  2. Rank Correlation (R2, R3) vs. Transport: R2/R3 study cross-benchmark rank correlation; we formalize within-benchmark transport (TG,TEm\mathrm{TG},\mathrm{TE}_m, Eqs. 3–4) and a 6-field contract (Table 7).

W1.2: Benchmark Coverage vs. R1, R2, R4 & Model Selection

  1. Longitudinal Depth: We will add R4 (§2.3). Unlike static studies (R1, R2, R4), isolating temporal variance (block bootstrap) and score noise required a dated archive with SEs (133 snapshots, 4,108 models, 26,600 shortlists); d(B,C,t,θ)d(B,C,t,\theta) is metric-agnostic (§3.1, §6.4).
  2. Selection Errors: (i) Reversal (Fig. 1, Table 6): On 2025-03-13 (Geedorah, NQLSG, Top-T, Noxis, TEMP), NQLSG beats Noxis by +1.45+1.45 pp on MUSR (43.65%43.65\% vs. 42.20%42.20\%), while Noxis leads by >3>3 pp on GPQA. Macro d3d_3 favors MUSR (69.6%69.6\% vs. 67.2%67.2\%), steering teams to pick NQLSG. Locally, d3d_3 reverses by 50 pp: MUSR drops to 20.0% (2/10 pairs; #1–#2 1.721.72 pp, #2–#3 0.130.13 pp) vs. 70.0% on GPQA (7/10 pairs >3>3 pp)—trading a >3>3 pp GPQA lead for MUSR noise. (ii) Top-5 Ties (§5.2, Table 2) & Noise (§5.1): Top-5 ambiguity (1−d31-d_3) hits 74.3% (GPQA), 76.1% (MUSR), and 100% (0/10, MATH-Hard 2025-03-13); SEs eliminate 64.1% (GPQA, 52.3%→20.2%52.3\%\to 20.2\%) and 66.5% (MUSR, 60.6%→21.1%60.6\%\to 21.1\%) of separated pairs.

W2.1: All-Pairs Separation (dθd_\theta) vs. Top-1 Margin (AA vs. BB)

We will add A={5,30,79.9,80,80.1}A=\{5,30,79.9,80,80.1\} (d3=70%,Δ1,2=0.1d_3=70\%,\Delta_{1,2}=0.1 pp) vs. B={75,75,75,75,80}B=\{75,75,75,75,80\} (d3=40%,Δ1,2=5d_3=40\%,\Delta_{1,2}=5 pp) (§3.1, §6.3): (1) dθ(C)d_\theta(C) measures all-pairs resolution (ranking, routing, cascading); for 0<dθ(C)<10<d_\theta(C)<1, high dθd_\theta does not guarantee Δ1,2=a(1)−a(2)>θ\Delta_{1,2}=a_{(1)}-a_{(2)}>\theta if lower-tier gaps dominate (AA). (2) At dθ(C)=0d_\theta(C)=0 (MATH-Hard Top-5, 2025-03-13), max⁡i,j∈C∣ai−aj∣≤θ\max_{i,j\in C}|a_i-a_j|\le\theta (Prop. 2), guaranteeing Δ1,2≤θ\Delta_{1,2}\le\theta. (3) For 0<dθ(C)<10<d_\theta(C)<1, §6.4 & Table 7 require the pair denominator and gap matrix (≤45\le 45 pairs, m≤10m\le 10) with dθd_\theta (Fig. 1 right).

W2.2: Multi-Metric and Constrained Deployment

Stratifying OLLB v2 metadata (§6.3) by scale (≤8\le 8B, 8–328\text{--}32B, >32>32B), license, and score floors isolates two regimes: (1) Metadata Alone: size/license filtering shifts d3d_3 by −2.1-2.1 to −8.9-8.9 pp, with C3 dispersion (∣TE∣5=10.4–14.5|\mathrm{TE}|_5=10.4\text{--}14.5 pp) at m=5m=5. (2) Capability Floors (ai(Bk)≥τka_i(B_k)\ge\tau_k) & Top-KK: joint floors or Top-5 in a tier restrict to the upper tail (Prop. 2, C2), re-inducing compression (−13.8-13.8 to −34.2-34.2 pp vs. −14.1-14.1 to −36.6-36.6 pp, Table 2) and rank trade-offs (Table 6).

Q1 & Q2: Fig. 2(b) (~66%) vs. §5.3.1 (53.1%), Winner Definition & Baselines

  1. Winner Definition (Q2): The 53.1% metric (C3) measures benchmark separation-winner agreement (arg⁡max⁡Bd3\arg\max_B d_3), not model choice; we will recalibrate the Abstract/Intro/Conclusion.
  2. 53.1% vs. 66.3% in Fig. 2(b) (Q1): At m=5m=5 (1010 pairs), d3d_3 takes 1010 pp steps (Prop. 3), causing ties. §5.3.1 reported strict agreement (53.1% [49.9, 56.4]), uncrediting ties; Fig. 2(b) plotted canonical tie-breaking (BBH ≻\succ GPQA ≻\succ MATH-Hard ≻\succ MUSR), giving 66.3% [62.4, 70.1] at m=5m=5 as ties default to BBH. At m∈{10,20,50}m\in\{10,20,50\}, ties vanish and both converge (78.2%/80.2%, 85.8%/86.3%, 95.4%/95.8%); §5.3.1 & Fig. 2(b) report both.
  3. Baselines (Q2): 25% (grey dotted, Fig. 2b) is 4-benchmark uniform guessing (1/41/4). 50% (§5.3.1) is a coin flip between the two benchmarks winning >96%>96\% of frames (BBH 78.4%78.4\%, MATH-Hard 74.7%74.7\%), spanned by [49.9,56.4][49.9, 56.4]. 65.0% (red dashed, Fig. 2b) is the modal baseline (predicting BBH: 65.0%, 63.4%, 59.5%, 63.2% at m∈{5,10,20,50}m\in\{5,10,20,50\}). At m=5m=5, canonical agreement (66.3% [62.4, 70.1]) beats 65.0%65.0\% by +1.3+1.3 pp (in 95% CI): under either rule, macro winners at m=5m=5 offer no significant lift over a trivial baseline.

Thank you for the detailed rebuttal. I appreciate the clarifications and additional experimental results, particularly regarding the theoretical contributions and the discrepancy in Figure 2(b).

However, my main concerns remain partially unresolved. For W1, while the cohort-based analysis provides useful insights, the novelty remains somewhat limited given the overlap with prior work, and the benchmark coverage is still relatively narrow.

For W2, the authors clarify the distinction between all-pairs separation and top-1 identifiability, but the additional examples do not directly demonstrate improved model-selection outcomes.

Overall, the rebuttal addresses some of my concerns, but the evidence is not sufficient to change my assessment. I will maintain my rating of 2.

Dear Reviewer auVx,

Thank you for the follow-up. You named two remaining issues: (1) novelty and benchmark coverage, and (2) the lack of direct model-selection evidence in realistic settings. The revision addresses the second directly and states the limits on the first.

Direct model-selection outcomes (new §5.3.3). We now evaluate the top-1 choice itself, not the stability of the separation statistic. The analysis covers 26,600 uniform five-model shortlists. The held-out target is the mean of IFEval and MMLU-PRO, which are never used for selection, so the target itself combines two benchmarks. On the same five candidates, selecting with a low-separation benchmark (d3 ≤ 30%) rather than a high-separation one (d3 ≥ 70%):

  • raises the wrong-model rate from 47.7% [44.2, 51.3] to 66.1% [63.2, 68.9];
  • raises held-out loss from 4.69 to 9.12 pp.

This covers 5,047 shortlists. On score-selected top-5 frontiers, the wrong-model rate is 63.9% for GPQA and 73.7% for MUSR, against 46.6–48.1% for BBH and MATH-Hard. These are observational contrasts, and the paper says so.

This also bears on your W2 example. §3.1 now includes your cohorts A and B and states that d_θ measures all-pairs resolution, not top-1 identifiability. Empirically, low shortlist d3 still goes with more wrong top-1 picks.

Novelty. R1–R4 are now cited in §2.2, §2.3 and §3.1, and the paper states how our work differs. R1 scores benchmarks on a fixed model pool. R2 studies cross-benchmark rank agreement, and R3 shows that removing a single model can change the leader. We instead fix the benchmark and analyze how separation transports from the full frame to a shortlist. This includes the exact finite-population variance (Prop. 3) and the split of transport failure into identification failure, selection compression and sampling variance.

Coverage. We did not add benchmarks. The C1–C3 analyses need dated snapshots with per-model SEs, which we have for four OLLB benchmarks; §5.3.3 adds IFEval and MMLU-PRO as held-out targets. We agree that broader coverage would strengthen the paper.

If the direct selection results and the corrected constrained analysis change your view of W2, we would be grateful if you would reconsider your rating.

Reviewer_R8i4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Paper studies whether benchmark that separates models well across a large leaderboard also does when the choice of pool of model is very small. 4 benchmarks and 133 dates from Open LLM Leaderboard v2 are used by author to compare the full eligible set with the top-K models and randomly sampled smaller groups. It uses a score gaps of above 3 percentage to measure separation. Experiments show that selecting the top 5 lowers separation by 14.1–36.6 point on average across benchmars and random 5 model group give almost the same result as the full set on average but individual group can differ quite a lot. Also accounting for the reported score uncertainty also lowers separation substantially on GPQA and MUSR under the assumptions used in their sensitivity analysis. The paper provides theoretical explanations for why separation depends on the models compared, how selecting top-scoring models can reduce it and why uniform random sampling preserves it on average.Authors also interpret changes in which benchmark separates best as evidence of near chance model selection and propose 6 reporting fields to make these comparisons easier to check.

Critical strengths
  • The strength of the paper is that it separates the effect of selecting top scoring models from the variation caused by random sampling. Theoretical results help to interpret by showing how top K selection can reduce separation and why random sampling can produce variation across shortlists. The experiments result show drop of 14.1–36.6 point on average after top 5 selection whereas random 5 model groups stay close to the full set value on average but vary substantially between groups.

  • the authors did a control comparisons by keeping the benchmark, date, scores and threshold fixed and this helped understand and isolate how changing the model group affected separation. Results on 4 benchmarks across 133 dates makes the findings more convincing with different benchmarks and dates.Author also did some experiments to check how the results change with different thresholds and when score uncertainty is taken into account.

Critical weaknesses
  • The major concern here is claim that model selection becomes close to chance. The experiment checks whether the same benchmark separates models best on the full set and shortlist but it do not check if the correct model is selected. 53.1% result also comes from random groups rather than group of top models so this claim needs more support or should be limited to what the experiment actually showed.

  • Could the authors clarify which agreement value is correct for 5 model groups? because author reported 53.1% but the figure shows around 66% and the interval shown does not include 50% as stated. Why is 50% used as the chance baseline when the figure also shows baselines at 25% and around 65%? this would affects how the near chance claim to be interpreted.

  • It is claimed that agreement improved with larger shortlists but the results are based on different dates. 5 model shortlists was tested across 133 dates whereas only 25 dates have enough models to test shortlists of 50 model. Could both size be compared on those same 25 dates to check how much of the improvement comes from shortlist size? and reporting how many models were available on each date would also help.

  • The authors extend the findings to shortlists formed using license, latency and memory constraints but they do not test these filters. Reduction of about 36.6 point came from choosing the top 5 models by benchmark score so choosing models based on practical constraints may give a different result. These filters should therefore be tested before claiming that they lead to the same reduction in separation.

Questions for authors
  • The example on 2025-03-13 includes only 35 model so does the full set include all eligible models available on the leaderboard at that time or only records dated that day?

  • Could you give a concrete example of how relying on the full set separation rate leads to a different or worse model choice when the shortlist scores are already available?

  • The 5 model agreement is reported as 53.1% but the figure shows around 66% so please clarify the correct value and interval, and how the model winner baseline is calculated and provide more details why 50% is treated as chance.

  • Could the same score uncertainty adjustment be applied to the top 5 and random shortlist results to check whether the main findings will still hold?

2 comments
AuthorsOpenReview ↗

Response to Reviewer

We thank the Reviewer for the valuable comment.

W1: Winner Agreement & Random vs. Top-5

  1. Winner vs. Model Choice: 53.1% [49.9, 56.4] (§5.3.1) tests benchmark winner agreement (arg⁡max⁡Bd3(S)=arg⁡max⁡Bd3(F)\arg\max_B d_3(S)=\arg\max_B d_3(F)), not model choice.
  2. Random (C3) vs. Top-5 (C2): 53.1% uses 26,600 uniform 5-model draws (bias −0.01-0.01 to +0.11+0.11 pp, SD 13.9–18.5 pp, Table 3; ∣TE∣5|\text{TE}|_5=10.8–14.8 pp, Table 5). Top-5 sets are tested in C2 (§5.2, Table 2): score selection cuts d3d_3 by 35.1, 26.6, 14.1, 36.6 pp (91.1%$\to$0% MATH-Hard, 2025-03-13; 74.3% GPQA & 76.1% MUSR pairs ≤3\le 3 pp).

W2 & Q3: 53.1% vs. ~66%, Intervals & Baselines

  1. 53.1% (Strict) vs. ~66% (Canonical): At m=5m=5 (10 pairs, 10 pp steps, Prop. 3), ties are frequent. Strict = 53.1% [49.9, 56.4] (§5.3.1) requires a unique winner matching Bt∗B^\ast_t, rising to 78.2% [74.1, 82.1], 85.8% [82.7, 89.1], 95.4% [92.8, 98.0] (m=10,20,50m=10,20,50). Canonical = ~66% [~62%, ~70%] (Fig. 2b) breaks ties via BBH$>GPQAGPQA>MATH−HardMATH-Hard>$MUSR (§3.3), defaulting ties to modal winner BBH (78.4%, Table 1; +13 pp at m=5m=5); at m≥10m\ge 10, ties vanish (Fig. 2b).
  2. Baselines: 25% (grey dotted) is 4-way uniform guessing; ~65% (red dashed) is the modal baseline (always BBH); 50% is a coin flip between top two winners (BBH 78.4%, MATH-Hard 74.7%, Table 1). Under either rule, m=5m=5 is near-chance: strict (53.1% [49.9, 56.4]) covers 50%, and canonical (~66%) sits on the ~65% modal baseline (in its 95% CI).

W3: Shortlist Sizes & Model Counts

  1. Same-Date Control (m=5m=5 vs. 1010): m=5,10m=5,10 share all 133 dates, driving >60%>60\% of recovery: agreement jumps +25.1 pp (53.1%$\to$78.2%) and mismatch gap drops 71% (3.92$\to$1.14 pp).
  2. Date-Set Stability: In Fig. 2a, mean MUSR d3d_3 is 60.5%, 60.5%, 61.8%, 61.9% at m=5,10,20,50m=5,10,20,50 (shift ≤1.4\le 1.4 pp). In Fig. 2b, the modal baseline is slightly lower at m=50m=50 (~63%) than m=5m=5 (~65%), ruling out "easier" dates. Recovery follows U-statistic contraction (Prop. 3, App. A.2): within-date SD drops 16.1$\to$1.9 pp (m=5→50m=5\to 50, §5.3.2).
  3. Counts: 4,108 models (9,103 records) across 133 dates (≥10\ge 10 models/date; n=35n=35 on 2025-03-13), with 133, 133, 86, 25 dates at m=5,10,20,50m=5,10,20,50 (§4.2, Fig. 4).

W4: Practical Filters vs. Top-5 Selection

OLLB v2 lacks latency/memory/license metadata, so we did not test them. We will revise §6.3–6.4 to drop the claim that they cause a 36.6 pp drop: score-uncorrelated filters follow Prop. 3 (unbiased, high variance), while 14.1–36.6 pp compression (Table 2) holds only for score-selected top-KK sets (C2) (or top-KK within a constrained pool, Prop. 2).

Q1: 2025-03-13 Snapshot (35 Models)

Includes only records dated 2025-03-13 (n=35n=35 models, 595 pairs; §4.2, Table 6), not the cumulative union (4,108 models, 223 dates). Same-day batches avoid serial correlation; Props. 1–3 hold on cumulative frames (§3.1).

Q2: How Global d3d_3 Misleads Choice

  1. Overpaying in Tie Zones: On 2025-03-13 5-model MUSR ({42.06,42.20,43.52,43.65,45.37}%\{42.06, 42.20, 43.52, 43.65, 45.37\}\%, Fig. 1, §3.2), #1 beats #2 by 1.72 pp (#2 beats #3 by 0.13 pp). Trusting global d3d_3 (69.6%) misleads teams to overpay for #1 when local d3d_3 is 20.0% (2/10 pairs >3>3 pp; 0/6 among lower 4). On MATH-Hard (2025-03-13), global d3d_3 is 91.1%, yet Top-5 d3d_3 is 0% (0/10 pairs >3>3 pp, §5.2)—picking #1 over #5 overpays in a ≤3\le 3 pp tie zone.
  2. Weighting Reversal (Table 6): On 2025-03-13, MUSR beats GPQA in global d3d_3 (69.6% vs. 67.2%). If models differ by ≤1.5\le 1.5 pp on MUSR (42.06–43.65%) but >3>3 pp on GPQA, global-d3d_3 weighting favors MUSR. Locally, d3d_3 reverses by 50 pp: MUSR drops to 20.0% (2/10) while GPQA hits 70.0% (7/10)—favoring local noise (8/10 ties) over a decisive test (7/10 resolved).

Q4: Score-Uncertainty Adjustment on C2 & C3

  1. C3 Invariance (Prop. 3): Under yijSE=1[∣ai−aj∣−1.96SEi2+SEj2>3]y_{ij}^{\text{SE}} = \mathbb{1}[|a_i-a_j|-1.96\sqrt{\text{SE}_i^2+\text{SE}_j^2}>3] (§4.3, §5.1), Prop. 3 & App. A.2 hold for any binary {yp}\{y_p\}, so uniform draws stay unbiased (E[d3SE(S)]=d3SE(F)\mathbb{E}[d_3^{\text{SE}}(S)] = d_3^{\text{SE}}(F)) with 10 pp steps at m=5m=5 and π1−π2>0\pi_1-\pi^2>0.
  2. C2 & C3 (Table 5): Since yijSE≤yijy_{ij}^{\text{SE}} \le y_{ij}, SE adjustment acts as θeff>3\theta_{\text{eff}} > 3 pp (cutting C1 d3d_3: 52.3%$\to$20.2% GPQA, 60.6%$\to$21.1% MUSR, −8.0,−9.8-8.0,-9.8 pp BBH, MATH-Hard, §5.1). Testing θ:3→5\theta: 3\to 5 pp (Table 5), C2 top-5 Δ\Delta stays negative (−42.5,−21.1,−17.6,−32.2-42.5, -21.1, -17.6, -32.2 pp at θ=5\theta=5 vs. −35.1,−26.6,−14.1,−36.6-35.1, -26.6, -14.1, -36.6 pp at θ=3\theta=3), C3 ∣TE∣5|\text{TE}|_5 rises (10.8–14.8$\to$12.5–14.8 pp), and d3=0%d_3=0\% sets (MATH-Hard Top-5, 2025-03-13) stay 0%0\%.

Dear Reviewer R8i4,

Thank you for the careful review. We have uploaded a revision with every change highlighted in yellow. Below is how each of your points is now handled in the paper.

W1 (the "near-chance" claim). We removed "near chance" from the paper. The 53.1% figure measures benchmark-winner agreement, not model choice, and the abstract, introduction and conclusion now say so. The new §5.3.3 tests model choice directly. The comparison holds the same five candidates fixed and picks the top-1 model with either a low-separation benchmark (d3 ≤ 30%) or a high-separation one (d3 ≥ 70%). Using the low-separation benchmark:

  • raises the wrong-model rate on held-out IFEval/MMLU-PRO from 47.7% [44.2, 51.3] to 66.1% [63.2, 68.9];
  • raises held-out loss from 4.69 to 9.12 pp.

This covers 5,047 shortlists. For score-selected top-5 sets, GPQA and MUSR (the benchmarks with the lowest top-5 d3) give wrong-model rates of 63.9% and 73.7%, against 46.6–48.1% for BBH and MATH-Hard. We present these as observational contrasts, not causal effects.

W2/Q3 (53.1% vs. ~66%, and the baseline). §5.3.1 now reports both values: strict agreement is 53.1% [49.9, 56.4], and canonical tie-broken agreement, the value plotted in Fig. 2b, is 66.3% [62.4, 70.1]. We dropped 50% as a reference. The comparison is now the modal baseline that always predicts BBH (65.0%). Canonical agreement beats it by only 1.3 points at m=5 and by 32.6 points at m=50.

W3 (different date sets). We ran the comparison you asked for (§5.3.1, new App. B.3, Tables 6–7). Every size is evaluated on the same 25 dates that support m=50, using the original draws. Canonical winner agreement rises from 61.4% [54.7, 67.6] at m=5 to 95.6% [91.5, 99.1] at m=50, and the mismatch gap falls from 4.83 to 0.03 points. The three steps (5→10, 10→20, 20→50) contribute 35%, 28% and 37% of the gain; in the unmatched ladder the shares are 48%, 20% and 32%. Date composition lowers the unmatched contrast by 5.0 points, so it hides part of the size effect rather than creating it.

At m=5, canonical agreement shows no detectable advantage over the modal always-BBH baseline (61.4% vs. 64.4%, −3.0 pp [−8.3, +1.3]); from m=10 onward the advantage is clear and grows (+10.6, +18.7, +32.6 pp, all intervals excluding zero). The full-frame winner thus fails exactly where the paper says small cohorts fail.

One concern remains: on most of these 25 dates m=50 is close to the whole frame (median n=61). Yet excluding the 13 dates with 50/n > 0.8 leaves the agreement at 99.4% (m=50) vs. 66.9% (m=5), a 32.5 pp recovery, so the gain is not a mechanical artifact of sampling fraction.

The same pattern holds for score-selected top-K sets (C2, Table 6): on matched dates the top-5 minus top-50 gap is −24.8 to −42.8 pp, against −9.3 to −25.4 pp unmatched. Date composition again widens the gap, so the unmatched profile understated the compression rather than creating it.

W4 (practical filters). OLLB records both license and parameter count, so §6.3 now tests them.

  • The filter mainly moves the frame: −18.6 to +0.9 pp.
  • Uniform shortlists inside a stratum still disperse: |TE|₅ = 9.9–14.6 pp.
  • Taking the top 5 within a stratum compresses d3 by 16.7–25.0 pp on BBH, GPQA and MUSR; MATH-Hard ranges from −8.0 to +7.1 pp.

Q1/Q2. App. B.4 states that the 2025-03-13 frame contains only the 35 records dated that day (595 pairs). It also gives two concrete cases where macro d3 misleads a choice even though the shortlist scores are visible: overpaying inside a tie zone, and a 50-point reversal in benchmark weighting.

Q4 (uncertainty adjustment for C2/C3). §5.1 uses θ=5 as a uniform proxy for the SE-adjusted threshold and labels it as a proxy, not an exact recomputation. Under it, C2 top-5 compression stays negative and C3 |TE|₅ rises to 12.5–14.8 pp. In §5.3.3, score uncertainty enters directly: under binomial SEs, the false-winner rate for the same five models is 9.0% with high-separation benchmarks and 48.6% with low-separation ones.

The paper now answers each of your questions in the text and limits its claims to what the experiments show. If you agree, we would be grateful if you would consider raising your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.