- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper show that public LLM leaderboards may not useful for practically selection.
Critical strengths
The problem setup is clear and easy to follow, with C1/C2/C3 corresponding to full cohorts, score-selected top-K cohorts, and target cohorts. The empirical results are easy to understand.
Critical weaknesses
- The main conclusion is relatively intuitive, and several theoretical results follow almost directly from the definition of cohort-conditional separation, limiting the theoretical novelty of this paper. I do not get much new intuition after reading the theortical analysis.
- The paper analyzes the stability of a custom separation statistic rather than actual model-selection accuracy, so the connection to real deployment decisions is not fully established and it requires more evidences.
Questions for authors
- Can the authors directly evaluate whether cohort-dependent separation actually leads to incorrect model-selection decisions, rather than only measuring the stability of the separation statistic? Please provide more details.
- I find that most of the propositions follow almost directly from the definition of cohort-conditional separation. I am curious that what insight does the formalization provide beyond the intuitive observation that "score gaps depend on the compared model set"?
5 comments
Thank you for the detailed response. The theoretical clarifications are helpful, but my main concern remains: the results quantify pairwise ambiguity and benchmark-selection disagreement, rather than directly evaluating incorrect model choices or downstream performance loss. So I think the connection to practical model selection is still not established and I decide to maintain my score.
Thank you again for the follow-up. To follow up on your remaining concern regarding incorrect model choices and downstream performance loss, we conducted a direct evaluation across the 26,600 size-5 candidate shortlists in OLLB v2, and will incorporate these results into §5.3.3 of the revised manuscript.
Core Conclusion: Even when holding the exact 5 candidate models in a shortlist fixed, a collapse in shortlist separation directly causes practitioners who pick the #1 model to select the wrong model in the majority of cases and suffer nearly twice the downstream accuracy loss:
- Incorrect Model Choices: For the exact same 5 candidate models, selecting the top-1 model via a low-separation benchmark with rather than a high-separation benchmark with increases the false-winner rate under evaluation standard errors by , from 9.0% to 48.6%, and raises the wrong model choice rate on held-out downstream tasks
IFEvalandMMLU-PROfrom 47.7% to 66.1%. - Downstream Performance Loss: On the exact same 5 candidate models, relying on a low-separation benchmark nearly doubles downstream accuracy loss on
IFEvalandMMLU-PROfrom 4.69 pp to 9.12 pp. Likewise, in Top-5 frontier selection, heavily compressed benchmarks such as GPQA and MUSR inflict 9.18–11.66 pp of downstream loss, nearly double the 5.39–5.82 pp loss on BBH and MATH-Hard.
We sincerely hope these direct results on incorrect model choices and downstream performance loss address your remaining concern, and we would be deeply grateful if you would kindly consider raising your score. Thank you again for your valuable guidance in helping us strengthen the paper!
We thank the Reviewer for the constructive feedback, which strengthened our theoretical presentation and model-selection analysis.
W1: Theoretical Novelty of the Propositions
Stating Propositions 1–3 briefly in §3.2 while placing our U-statistic covariance derivation in App. A.4 made the theory appear closer to definitional statements than intended. We will integrate the closed-form degree-2 U-statistic variance decomposition directly into Proposition 3 (§3.2): Var[d(B,S,t,θ)]=K21[π(1−π)∑pyp+∑p=qypyq(π∣p∩q∣−π2)] where K=(2m), N=(2n), π=K/N, π1=n(n−1)(n−2)m(m−1)(m−2) (∣p∩q∣=1), and π0=n(n−1)(n−2)(n−3)m(m−1)(m−2)(m−3) (p∩q=∅). For 3≤m<n ((n,m)=(4,3)), pairs sharing a model carry positive covariance (π1−π2>0). Thus, a 5-model shortlist does not yield 10 independent tests; shared-model dependence shrinks effective sample size below K and analytically explains the 53.1%→95.4% agreement recovery across m∈{5,10,20,50} (Fig. 2).
W2: Connection to Real Deployment Decisions
We will expand §3.1 to show that d(B,C,t,θ) directly bounds binary and top-1 model-selection validity on shortlist C:
Q1: Empirical Evaluation of Model-Selection Failures
In §5.1–§5.3, our audit of 133 OLLB v2 snapshots (4,108 models, 26,600 size-5 shortlists) directly quantifies three deployment selection failures:
Q2: Insights Beyond "Score Gaps Depend on the Compared Set"
As clarified in §6.1, naive intuition cannot explain why a macro report fails to transfer or how to fix it. Our formalization yields three counter-intuitive insights: