acceptodds
Under review as a conference paper at ICLR 2027

What SWE-bench Cannot Tell Apart: The Resolution Limit of Agentic Leaderboards

Abstract

Public leaderboards rank coding agents by resolve rate on a few hundred tasks, and the small gaps at the top drive model selection. We assemble the per-task outcomes of 175 public SWE-bench Verified submissions into a 175x500 matrix that reproduces every official score. The top of the board is compressed: the leading ten systems span 3.4 points, adjacent ranks differ by a median of a single task, and among the top thirty, 243 of the 500 tasks are solved by all of them or by none. Under task subsampling, the 0-1 and 1-2 point gap bands need 497 and 421 of the 500 tasks to reach 95% mean ordering preservation. We then examine sensitivity to run-to-run variation: fifteen pairs of same-agent, same-model resubmissions put the per-run flip rate at 5.6% of tasks (95% CI 3.9 to 7.2). Carrying that rate through the resampling, under an independent equal-probability flip model, leaves the bands within two points below 95% mean ordering accuracy at any subset size, the full benchmark included; at the point estimate 293 of 435 top-thirty pairs (67%) fall short. That result is conditional on the noise model, and a band average does not determine the reliability of any individual pair inside it. A comparison with HELM MMLU shows similar task-sampling behaviour once pool size and score range are matched. Tasks chosen by a held-out panel of comparable systems are worth two to three times their number, but the panel has to be contemporaneous: freezing the chosen set and reusing it on later submissions gives the advantage back. Finally, among the 39 submissions publishing per-task dollars, resolve-rate order and cost-per-resolved-issue order are close to unrelated here (Spearman rho = -0.08).

Primary area:
datasets and benchmarks
Keywords:
Evaluation Methodology, Benchmark Reliability, Coding Agents, SWE-Bench, Leaderboards, Measurement Error, Item Response, Cost-Aware Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3SoPOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper highlights the limited ability of the public SWE-bench Verified leaderboard to distinguish among top coding agents: all agents in the analyzed frontier group solve 243 of the 500 tasks. This finding motivates developing new SWE benchmarks with greater discriminatory power among strong systems.

Critical strengths
  1. The analysis shown in this paper is inexpensive to reproduce
  2. The coding ability is the most important ability for agentic systems
  3. The authors conduct an extensive empirical experiments
Critical weaknesses
  1. This paper does not propose any novel ideas, it give us some insights via large-scale empirical experiments. Actually the observations shown in this paper are common among frontier labs, in other words, SWE-bench Verified has been solved by frontier model.
  2. Among 175 submission, 64 of them are anthropic-backed, what is the model used for other submissions?
  3. One open question here is that SWE-bench Verified has been solved by frontier model, how to design the next coding benchmarks?
Questions for authors

Please see critical weaknesses

2 comments
AuthorsOpenReview ↗

Response to 3SoP

Models behind the submissions can be seen at Appendix A, Table 5. By vendor of the underlying model as stated in metadata: Anthropic 64, OpenAI 33, Qwen 9, Google DeepMind 7, Moonshot AI 6, Meta 5, Mistral 4, Z.ai 3, DeepSeek 2, MiniMax 2, ByteDance 1, not stated 38 (8 of which can be assigned from the submission name). In the top thirty (one submission assigned from its name): Anthropic 19, OpenAI 5, Google DeepMind 3, one each for Z.ai, MiniMax, ByteDance. Because of this concentration we repeated the main calculation with one submission per model, with Anthropic models only, and with other vendors' models only (Table 1): the task budgets in the two closest bands change by at most 9 tasks.

SWE-bench Verified has been solved. In our snapshot, the best submission resolves 79.2% and fails 104 tasks; 35 tasks are resolved by none of the top thirty and 26 by no submission. Some unsolved tasks may be defective rather than hard (audits of Verified report inadequate tests), so this is an upper bound on remaining headroom. We agree that the top of the leaderboard no longer separates systems, and we now cite the concurrent work that reports this. Our question is quantitative: how large a published gap must be before its order is stable under stated sources of variation.

Section 5.5 and the Discussion. Here q denotes the assumed per-entry perturbation rate. The continuity-corrected normal approximation to the preservation probability (Equation 6) differs from the exact calculation by at most 5e-4 over the evaluated pairs at the two calibrated rates. Equation 7 gives the corresponding approximate gap threshold. Under the independent equal-rate model at the reference rate p = 0.058, keeping the published order of systems one point apart with 95% probability needs about 3,800 task-runs per system (about 8,700 at d = 0.110); on 500 tasks, three runs resolve 1.8 points, five 1.4 and ten 1.0. These are preservation thresholds for the published order under the model and inherit the proxy; they are not estimates of how well real repeated runs recover the true order. More runs reduce only run-to-run variation; the variation from the choice of tasks is reduced only by more tasks, and dependence within repositories only by more repositories. Repeated runs would also give the change rate itself, which no public record provides for these leaderboard systems.

For novelty, we agree the descriptive observations are known and have narrowed the claims; please kindly see the last bullet of our response to Reviewer mT5v.

Thanks for your response! However, I still want to keep the score. The main reason why I keep the score is that the paper is not novel enough to be accepted by ICLR 2027. Perhaps, you can submit the paper to evaluation track of other conferences. Thanks!

Reviewer_mT5vOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates the problem of leaderboard saturation on SWE-bench. In particular, the paper presents an analysis of which tasks still hold discriminative power, whether a panel of models can identify these tasks, and what the role of run-to-run variation is.

Critical strengths
  • Benchmark saturation is a timely problem. Investigating the issues that come with such saturation is important.
Critical weaknesses
  • It is blatantly clear that the paper is completely comprised of AI generated text. Many sentences are overly formulaic, there is plenty of jargon, many terms are used without definitions, and most paragraphs do not have an understandable flow. This leads to substantive clarity issues throughout the whole paper. I am not fully opposed to using AI to help with polishing. However, the usage of AI writing in this paper clearly goes beyond simple editing and appears to be substantial and fundamental. I find this very irresponsible.

  • Aside from the clarity issues, I do not think that the proposed analysis is all that interesting. As pointed out in the paper, run-to-run variation is already understood to have a meaningful influence on benchmark performance. Similarly, prior work has likewise shown that different models might succeed / fail consistently on some tasks and only a few task in the benchmark actually discriminate performance between models [1,2].

  • Overall, I think this paper seems to be more appropriate for a workshop rather than a top tier conference. I think the empirical analysis does have some value but arguably the novelty bar for ICLR is not met.

References:

[1]: Wang, Zhuo, et al. "MetaEval: Measuring the Discrimination of Benchmarks for Efficient LLM Evaluation." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 40. No. 40. 2026.

[2]: Ge, Chris, et al. "Agent psychometrics: Task-level performance prediction in agentic coding benchmarks." arXiv preprint arXiv:2604.00594 (2026).

Questions for authors
  • I don't think the selection of tasks that discriminate is all that well explained. Can you restate the method?
1 comment
AuthorsOpenReview ↗

Response to mT5v

For AI Use, please see our separate comment correcting the disclosure.You were right that it was incomplete.

For clearity, the paper has been rewritten. Section 3 defines each quantity before it is used (outcome matrix, pair counts, lead and gap, subset preservation, task budget, re-run preservation, the two readings of the change rate, unresolved pair, resolution), with numbered equations for the main ones, and each has one name throughout. Methods precede results.

The selection method is restated. Given the submissions already on the leaderboard: (1) rank them by score and take ranks 31-45 as the panel; (2) for each task compute the fraction of panel members that resolved it; (3) keep the tasks for which this fraction lies strictly between 0.15 and 0.85; (4) evaluate on 100 tasks drawn uniformly from those kept. The window and interval were chosen on submissions dated before 1 July 2025 and not changed. This is Procedure 1 in Section 4.4.

For novelty, it's our mistake that parts of the submission were not new and have rewritten the contribution statement.

  • Ge et al. (Agent psychometrics) assemble a matrix from the same repository and fit a Rasch model to predict per-task success (AUC). They also select tasks adaptively for a new agent and relate the number of tasks to the error of its ability estimate. They do not analyse pairwise order preservation by score gap or run-to-run variation, and do not quantify task budgets for pairwise order preservation at specified score gaps.
  • Wang et al. (MetaEval) estimate item discrimination on five multiple-choice benchmarks with 213 models and select subsets that predict full-benchmark accuracy and preserve rank correlation. They use no agentic data, no pairwise analysis by gap and no re-run variation. Their conclusion that a few percent of items suffice concerns rank correlation over models of widely differing ability and is compatible with ours, which concerns pairs one or two points apart.
  • Closer work that we had missed and now discuss: Liu et al. (arXiv:2609.17394) and Huang (arXiv:2609.07785), which reach our descriptive conclusion about the top of this leaderboard; Ndzomga (arXiv:2603.23749), which uses a selection rule of the same kind as ours; Kotawala (arXiv:2605.30315) and S. I. Wang (arXiv:2512.21326) on pairwise resolution and noise decomposition; Kapoor et al. (arXiv:2510.11977) on cost.
  • Yauney et al. introduced pairwise subset preservation as a function of score gap on multiple-choice benchmarks; we compute it exactly on agentic data and say so.
  • What remains ours: exact per-pair subset preservation and task budgets on this leaderboard; an outcome-perturbation sensitivity analysis with its range over five families of noise model and a resolution formula in tasks and runs; a chronological replay of panel-based selection with its dependence on panel strength and on the age of the selected set; the matched comparison with a multiple-choice leaderboard as a scope check.
Reviewer_Z4xFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper reconstructs a 175-by-500 matrix of public SWE-bench Verified outcomes and quantifies how subsets preserve the recorded full-benchmark ordering. It studies sensitivity to rerun noise under a homogeneous independent-flip model calibrated by a resubmission proxy. A HELM MMLU comparison, retrospective held-out-panel task selection, and cost-reporting submissions provide additional evidence about compression, subset transfer, and the relationship between accuracy and cost rankings.

Critical strengths
  1. The reconstruction describes upstream schema inconsistencies and aggregate-only exclusions, checks retained row totals, and uses an appropriate pairwise hypergeometric formulation. It separates observed-order preservation from expected-performance inference (pp.2-4, Secs.3-4.2).
  2. The panel experiment reports that historical fixed task sets lose their advantage while contemporaneous held-out panels help, and that different-family panels can fail. The HELM and cost analyses add useful qualifications to agent-specific and accuracy-only narratives (pp.6-9, Tables2-4; AppendixA).
Critical weaknesses
  1. The 67% unresolvable-pair result depends on homogeneous independent flips beyond the uncertain scalar proxy rate. Aggregate same-system disagreement does not identify which tasks are unstable or how model outcomes co-vary. The 15 resubmission pairs are confounded, and the selected 13-task check cannot establish that structure (pp. 5-6; Appendix B). The explicit conditional language in the abstract limits this concern, so it does not outweigh the descriptive contribution, but the operational interpretation needs sensitivity analysis or consistently narrow wording.
Questions for authors
  1. How do the Table1 pair counts and the specific frontier-pair conclusion change under task-dependent instability or dependence, rather than just a different homogeneous p? A small sensitivity analysis using the existing matrix would be sufficient; a new large rerun is not requested. Please maintain the model-conditional interpretation in the discussion and conclusion.
  2. How are held-out test targets and contemporaneous comparable reference-panel members selected in Table3, what full-score information is needed, and how sensitive are the gains to the panel choice? Please distinguish a deployable procedure for a new submission from the retrospective experiment already acknowledged in AppendixA.
1 comment
AuthorsOpenReview ↗

Response to Z4xF

Q1. Sensitivity to task-dependent instability or dependence. Done with the existing matrix, exactly or by quadrature (Section 5.4, Table 3, Appendix B). At equal mean change rate we compared: rates proportional to u(1-u) of a task's solve rate (over all systems, or over the top thirty); rates following the empirical relation between solve rate and disagreement in the resubmission pairs; rates that rise with an outcome's departure from the consensus of the other systems; a different rate per run (Beta-distributed); correlation between the two systems' changes on the same task; and repository-level dependence within a run.

  • Stable across every model at both calibrated rates: all pairs less than 2 points apart are below 95%; the example pair (76.8 vs 75.8, lead of 5 tasks) is never resolved, with preservation between 56% and 80%.
  • Not stable: the total count, which ranges from 189 to 422 of the 422 untied pairs at p = 0.058 and from 279 to 422 at d = 0.110 (equal-rate model: 280 and 354). Empirically shaped task-dependent rates move it by at most 4 pairs. These models were run at the two calibrated rates; at the low end of the interval (p = 0.033) with between-system correlation 0.75, 40 of the 185 pairs under 2 points become resolved, so the statement about close pairs is made at the calibrated rates. Run-level and repository-level dependence raise it; positive correlation between the two systems gives the largest reductions, and two task-dependent models lower it slightly at p.
  • A caveat we state in the paper: all models change entries symmetrically, which lowers the expected score of a re-run for strong systems, and part of the run-level effect comes from this.

We also now state in Section 3 that the change rate does not identify the distribution of a real re-run given the published outcomes, and present these results as an outcome-perturbation sensitivity analysis. The discussion and conclusion state every such result as conditional on the rate and the model, and limit the statement that holds across models to pairs less than 2 points apart at the calibrated rates.

Q2. Selection of compared systems and panel; information needed; sensitivity; deployable procedure. Section 4.4 now gives the procedure and separates two evaluations.

  • Retrospective (the submitted Table 3). Compared systems are the thirty highest-scoring submissions dated on or after the cutoff; the panel is ranks 31-45 of the same period. This uses the full scores of the compared systems to define both sets, which is not available in advance. It is also sensitive to a four-way tie at rank 30: the gain in the closest band ranges from 10.3 to 13.8 points depending on which two of the four tied submissions are treated as compared systems.
  • Chronological replay (new). Each submission from the cutoff onward is treated as an arrival; the panel is ranks 31-45 of the submissions dated strictly before it, and the arrival is compared with the thirty highest-scoring earlier submissions. Only earlier submissions' outcomes are needed; the code asserts that the arrival's outcomes do not enter the selection. The study is still retrospective, since the settings were fixed after the full leaderboard had been seen. Gains over 100 random tasks: +5.5 [0.2, 11.0], +12.0 [7.8, 15.6], +10.6 [4.7, 15.0], +9.9 [8.1, 11.3] in the four bands below 5 points (74 arrivals, bootstrap over arrivals). These are about half the retrospective gains.
  • Sensitivity to the panel. Ranks 31-60: +4.4 to +11.8. Ranks 46-60: about zero. Ranks 61-74: -4 to -16. Fifteen random systems from the same period: median within 2.3 points of zero in every band. The gain requires a panel close in strength to the systems compared; being from the same period is not sufficient.
  • Age of the selected set. Gain in the 1-2 point band: +12.0 at 0 days, +10.0 at 30, +7.3 at 60, +4.5 at 90, -2.1 at 180; negative in every band after a year (over the 32 arrivals for which a selection a year earlier exists).
  • Arrivals that enter the top thirty (39): +4.5 [-0.5, 9.6], +12.4, +12.7, +9.9. The panel-strength result above comes from the retrospective design.
  • The panel settings were chosen by a sweep over three rank windows and four intervals on submissions before 1 July 2025; the full leaderboard had been examined before that split was fixed.
  • A variance-based selector gives similar gains above one point; the data do not establish a clear advantage of either, and we no longer claim one.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Change statement

We thank the three reviewers. The title was narrowed to what the evidence supports: "Pairwise Order Stability on SWE-bench Verified". Summary of changes:

  1. Corrections we found ourselves while preparing this response. (a) One submission in our matrix had an empty per-task record and entered as a row of zeros; it was one of the 15 resubmission pairs used to calibrate the re-run rate. It is now excluded (174 submissions, 14 pairs; the calibrated disagreement moves from 10.5% to 11.0%). (b) The submitted text described the re-run result as the probability that "a re-run reproduces the reported ordering", but computed it with the per-run deviation rate p rather than the between-run disagreement d that this reading requires. The revision defines both reference rates (Section 3), states that neither identifies the distribution of a real re-run given the published outcomes, and presents the calculation throughout as an outcome-perturbation sensitivity analysis. At d = 0.110, 354 of the 422 untied top-thirty pairs are below 95%; at p = 0.058, 280. The submitted figure, 293 of 435, also counted the 13 tied pairs, and belonged to the p reading while being described as replication. (c) The task-selection result in the submitted Table 3 depended on how a four-way tie at rank 30 was resolved, and the "two to three times" figure came from an exploratory table. The revision reports a chronological replay instead, in which each arriving submission's outcomes are withheld from the selection; the gains are smaller.
  2. Writing structure: conventional section order; a Problem Setup section with numbered equations in which every term is defined once; the selection method as a step-by-step procedure; neutral headings; a Limitations paragraph.
  3. New analyses requested by the reviewers: sensitivity of the re-run results to five families of noise model; a deployable version of the selection procedure, evaluated in a chronological replay, with sensitivity to the panel and to the age of the selected set; composition of the leaderboard and robustness to it; a closed-form resolution formula for tasks and repeated runs.
  4. Related work. We now discuss the two papers cited by Reviewer mT5v and six further recent papers we had missed, two of which reach conclusions close to our descriptive findings. The contribution statement has been narrowed accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.