- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
It is about the fact that small differences on leaderboards are mostly noise, so we shouldn't over-interpret rankings, and should use proper statistical testing instead.
Critical strengths
They suggest reporting the minimum detectable difference for each comparison.
Critical weaknesses
Impossible to read (cf. examples below) and not novel (seems to me like it's mainly about using statistics?).
Poor writing:
Clear AI signatures:
- “The gap matters to anyone who promotes a system because its score rose.”
- Contributions. Finding. These are two different properties of the same data: a leaderboard record can be nearly exact while the comparisons read off it stay unresolved. […]. Evidence. Three audits support the finding […].
- “ We acquired the per-instance records on 2026-09-11 from the official SWE-bench experiments repository at upstream commit 40f164d5b8f1; the local manifest verifies every raw file by SHA-256.”
Sentences that are difficult to understand:
- “Power inversion is standard; our contribution is an executable certificate generator that binds each result to the full comparison contract and provenance. “
- “The accompanying ledger cannot establish temporal priority without an independently timestamped manifest, and the certificate covers one fixed pair rather than run-level error control for an adaptive search.”
- “Exact McNemar and rank intervals then apply a declared iid discordant-task model as an operational diagnostic; entries differ in base model, scaffold, and budget, so neither output attributes a gap to one component”
- “Exclusion is also not a second scoring rule — dropping uncontested instances scores each entry on its own denominator, and comparison-specific denominators do not define one common leaderboard score — so we quote it as a sensitivity only.”
- “The last row is a more hostile analysis that also frees every absent row — treating sparse summaries as if their complements were unknown; we report it as an upper envelope, but what it relaxes is the grader’s semantics, not the records.“
Sentences in the intro that have nothing to do in introduction
- "We call that missing quantity the evaluation’s resolution: the minimum detectable effect (MDE) of a declared comparison contract, formalized as R(C; ρd) in Equation 2, where C is the comparison contract and ρd the declared discordance input."
- "Three further questions remain separate, and Table 1 assigns each to a distinct certificate output: […]"
Sentence that don’t fit in a paper:
- “Its exact floor, 0.000488, exceeds the initial cutoff 0.000053. Thus no comparison in the declared all-pairs family can be rejected on any data; the observed 0/946 count is design-implied rather than evidence for equality. Adding tasks inside the same repositories cannot refine the grid.”
Questions for authors
Is this submission part of an eval? Was it fully AI generated?
I will keep my score as a clear reject.