- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper highlights the limited ability of the public SWE-bench Verified leaderboard to distinguish among top coding agents: all agents in the analyzed frontier group solve 243 of the 500 tasks. This finding motivates developing new SWE benchmarks with greater discriminatory power among strong systems.
Critical strengths
- The analysis shown in this paper is inexpensive to reproduce
- The coding ability is the most important ability for agentic systems
- The authors conduct an extensive empirical experiments
Critical weaknesses
- This paper does not propose any novel ideas, it give us some insights via large-scale empirical experiments. Actually the observations shown in this paper are common among frontier labs, in other words, SWE-bench Verified has been solved by frontier model.
- Among 175 submission, 64 of them are anthropic-backed, what is the model used for other submissions?
- One open question here is that SWE-bench Verified has been solved by frontier model, how to design the next coding benchmarks?
Questions for authors
Please see critical weaknesses
2 comments
Thanks for your response! However, I still want to keep the score. The main reason why I keep the score is that the paper is not novel enough to be accepted by ICLR 2027. Perhaps, you can submit the paper to evaluation track of other conferences. Thanks!
Response to 3SoP
Models behind the submissions can be seen at Appendix A, Table 5. By vendor of the underlying model as stated in metadata: Anthropic 64, OpenAI 33, Qwen 9, Google DeepMind 7, Moonshot AI 6, Meta 5, Mistral 4, Z.ai 3, DeepSeek 2, MiniMax 2, ByteDance 1, not stated 38 (8 of which can be assigned from the submission name). In the top thirty (one submission assigned from its name): Anthropic 19, OpenAI 5, Google DeepMind 3, one each for Z.ai, MiniMax, ByteDance. Because of this concentration we repeated the main calculation with one submission per model, with Anthropic models only, and with other vendors' models only (Table 1): the task budgets in the two closest bands change by at most 9 tasks.
SWE-bench Verified has been solved. In our snapshot, the best submission resolves 79.2% and fails 104 tasks; 35 tasks are resolved by none of the top thirty and 26 by no submission. Some unsolved tasks may be defective rather than hard (audits of Verified report inadequate tests), so this is an upper bound on remaining headroom. We agree that the top of the leaderboard no longer separates systems, and we now cite the concurrent work that reports this. Our question is quantitative: how large a published gap must be before its order is stable under stated sources of variation.
Section 5.5 and the Discussion. Here q denotes the assumed per-entry perturbation rate. The continuity-corrected normal approximation to the preservation probability (Equation 6) differs from the exact calculation by at most 5e-4 over the evaluated pairs at the two calibrated rates. Equation 7 gives the corresponding approximate gap threshold. Under the independent equal-rate model at the reference rate p = 0.058, keeping the published order of systems one point apart with 95% probability needs about 3,800 task-runs per system (about 8,700 at d = 0.110); on 500 tasks, three runs resolve 1.8 points, five 1.4 and ten 1.0. These are preservation thresholds for the published order under the model and inherit the proxy; they are not estimates of how well real repeated runs recover the true order. More runs reduce only run-to-run variation; the variation from the choice of tasks is reduced only by more tasks, and dependence within repositories only by more repositories. Repeated runs would also give the change rate itself, which no public record provides for these leaderboard systems.
For novelty, we agree the descriptive observations are known and have narrowed the claims; please kindly see the last bullet of our response to Reviewer mT5v.