acceptodds
Under review as a conference paper at ICLR 2027

Precise Records, Unresolved Ranks: Auditing the Resolution of Agent Leaderboards

10 views

Abstract

We audit the SWE-bench Verified leaderboard and find that its ranking means two different things. Read as a record of what happened under the leaderboard's own scoring rule, it is nearly exact: the median compatible-rank width — the number of rank positions still possible for an entry — is 1, and it rises only to 3 when every genuinely unrecoverable outcome is allowed to vary. Read as a measurement of relative performance, it resolves little: simultaneous rank intervals have a median width of 56 positions on a 175-entry leaderboard, and no adjacent pair survives multiple-testing correction. The gap matters to anyone who promotes a system because its score rose. When an acceptance rule keeps the best of several seeds, it promotes all 14 harness variants whose rendered prompts are byte-identical to their baseline — every such promotion is noise. At a budget of 500 tasks, the one-sided fixed-look rule requires a gap of roughly 5.2-6.2 percentage points to reach target power under the discordance envelope measured for one worked pair — several times the gap between adjacent rows. We therefore define the resolution of an evaluation — the smallest effect a declared comparison rule reliably detects at a target power — and release an executable tool that produces certificates binding each reported resolution to its data, rule, assumptions, and provenance.

Primary area:
datasets and benchmarks
Keywords:
agent evaluation, leaderboard auditing, statistical power, rank uncertainty

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_NCdzOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

It is about the fact that small differences on leaderboards are mostly noise, so we shouldn't over-interpret rankings, and should use proper statistical testing instead.

Critical strengths

They suggest reporting the minimum detectable difference for each comparison.

Critical weaknesses

Impossible to read (cf. examples below) and not novel (seems to me like it's mainly about using statistics?).

Poor writing:

Clear AI signatures:

  • “The gap matters to anyone who promotes a system because its score rose.”
  • Contributions. Finding. These are two different properties of the same data: a leaderboard record can be nearly exact while the comparisons read off it stay unresolved. […]. Evidence. Three audits support the finding […].
  • “ We acquired the per-instance records on 2026-09-11 from the official SWE-bench experiments repository at upstream commit 40f164d5b8f1; the local manifest verifies every raw file by SHA-256.”

Sentences that are difficult to understand:

  • “Power inversion is standard; our contribution is an executable certificate generator that binds each result to the full comparison contract and provenance. “
  • “The accompanying ledger cannot establish temporal priority without an independently timestamped manifest, and the certificate covers one fixed pair rather than run-level error control for an adaptive search.”
  • “Exact McNemar and rank intervals then apply a declared iid discordant-task model as an operational diagnostic; entries differ in base model, scaffold, and budget, so neither output attributes a gap to one component”
  • “Exclusion is also not a second scoring rule — dropping uncontested instances scores each entry on its own denominator, and comparison-specific denominators do not define one common leaderboard score — so we quote it as a sensitivity only.”
  • “The last row is a more hostile analysis that also frees every absent row — treating sparse summaries as if their complements were unknown; we report it as an upper envelope, but what it relaxes is the grader’s semantics, not the records.“

Sentences in the intro that have nothing to do in introduction

  • "We call that missing quantity the evaluation’s resolution: the minimum detectable effect (MDE) of a declared comparison contract, formalized as R(C; ρd) in Equation 2, where C is the comparison contract and ρd the declared discordance input."
  • "Three further questions remain separate, and Table 1 assigns each to a distinct certificate output: […]"

Sentence that don’t fit in a paper:

  • “Its exact floor, 0.000488, exceeds the initial cutoff 0.000053. Thus no comparison in the declared all-pairs family can be rejected on any data; the observed 0/946 count is design-implied rather than evidence for equality. Adding tasks inside the same repositories cannot refine the grid.”
Questions for authors

Is this submission part of an eval? Was it fully AI generated?

Reviewer_YwPcOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper audits the SWE-bench Verified leaderboard (175 entries) and argues that its ranks are exact as a record but unresolved as a measurement. Under the leaderboard's own scoring rule the median compatible-rank width is 1. Task-iid simultaneous rank intervals have a median width of 56, and no adjacent pair survives Holm-corrected exact McNemar. The authors call the minimum detectable effect of a declared comparison rule its "resolution" and release a CLI that emits a hash-bound report. On 14 harness variants whose rendered prompts are byte-identical to the baseline, a best-of-3-seeds acceptance rule promotes all 14 (Qwen), while an anytime-valid paired gate promotes none. One frozen prompt comparison (200 tasks, one model pair) serves as the worked example.

Critical strengths
  1. Reproducible audit with careful handling of missing records.
  2. Honest scoping of the evidence. The frozen comparison is labeled descriptive, and the authors report that the discordance input fails to transfer across splits, and they state that repository-level sign-flip inference can reject nothing at 12 clusters.
Critical weaknesses
  1. Almost nothing new.
  2. Two headline results hold by construction. Rank width 1 follows from ranks being a deterministic function of the published file, so only exact ties remain.
  3. The best-of-seeds experiment does not support the abstract. The 14/14 result is Qwen only; DeepSeek gives 1/3 (all-pairs 2/6). All 14 comparisons reuse one set of base outcomes. The main text never says how the baseline is scored under best-of-k, and Table 12 shows variants with a pooled rate below base (0.523, 0.540) being promoted, which suggests the variant takes a max over seeds and the base does not. The authors say these are not error-rate estimates, what remains is a demonstration that an asymmetric rule favors the variant.
  4. The writing is hard to follow. The AI-use statement says this paper was drafted with assistance and revised by the authors but it needs a rewrite for a human reader.
Questions for authors
  1. How is the baseline scored under best-of-3-seeds? What is the promotion rate when both arms take the best of 3 seeds?
  2. What does the paper establish beyond Liu et al. (arXiv 2609.17394)? Does any finding hold on a second leaderboard that publishes per-instance outcomes?
1 comment

I will keep my score as a clear reject.

Reviewer_ZrgKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper audits the SWE-bench Verified leaderboard and distinguishes between recorded ordering and the measurement of relative performance, further investigating uncertainty in performance rankings. It applies established statistical analysis such as paired testing and power analysis to quantify the resolution of evaluation and provides this as a tool. The analysis of the unreliability of the performance rankings includes behaviorally unchanged harness variants, repeated execution, and leaderboard analysis.

Critical strengths

The paper formulates auditable empirical evidence of the ranking unreliability through multiple analyses to show how the score differences can be overinterpreted. These provide quantifiable evidence on the current limitations of evaluation noise in SWE-bench Verified and a reusable tool to replicate the analysis. The assumptions behind each claim are explicitly stated supporting transparency.

Critical weaknesses

The demonstrated contribution beyond established methodology is limited. The central claims target known - albeit important - problems in evaluation and ranking, while the main added value appears to be a SWE-bench-specific audit and the resulting tool with limited actionable insights to address the identified limitations in SWE-bench verified ranking. Although the authors acknowledge the limited scope of using a single benchmark family, since evaluation noise is inherently tied to how evaluation and tasks are designed, I believe it’s critical to cover diverse benchmarks and demonstrate generalizability to sufficiently support the claims.

The connection between conditional statistical analysis and meaningful evaluation decisions has insufficient validation. The repository and task independence analysis can be a plausible setting, but the paper doesn’t establish other relevant characteristics such as task types or domains. The eight-point cut-off is an author-chosen operational threshold, but there is no validation that the resulting decision accurately reflects the underlying capability difference. Practical thresholds such as 5% error level and 80% power target are reasonable conventions, but the paper doesn’t connect them or explore the hyperparameter decisions to evaluation cost and benefits.

Questions for authors

The paper acknowledges that its empirical evidence is limited to the SWE-bench family. Within this scope, which major conclusions about ranking reliability remain robust to the choice of comparison family and dependence model?

Beyond the reporting and sample-sizing recommendations mentioned in the paper, what evidence demonstrates that the proposed certificate improves evaluation or promotion decisions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.