acceptodds
Under review as a conference paper at ICLR 2027

SWE-Sweep: Can Agents Autonomously Find and Fix Bugs?

Abstract

Most existing software engineering benchmarks evaluate coding agents on concrete, well-specified tasks, commonly by providing a codebase together with a user-reported issue to resolve. However, as users delegate increasingly broad outcomes to coding agents, the natural next step is for agents to determine not only how to perform useful work, but also what useful work needs to be done. An agent entrusted with a repository should be able to decide what is broken, which problems matter, and how to solve them before they are reported. We introduce SWE-sweep, a benchmark for this open-ended setting. Given a codebase containing many concurrent bugs and no information about their nature or location, an agent must autonomously discover and fix as many bugs as possible. For each repository, we collect real GitHub issue–pull request pairs, then identify a single commit where the maximum number of bugs are present at the same time. Each repair is evaluated against hidden tests from the corresponding pull request, along with the existing test suite to check for regressions. SWE-sweep encompasses 4,068 bugs across 22 programming languages and 100 repositories, including NumPy (Python), OpenCV (C++), and Lean 4 (Lean). We evaluate 7 LMs and find that the best agent resolves only 4.7% of all bugs, with the challenge generally lying in discovering bugs rather than repairing them. SWE-sweep exposes a substantial gap between resolving reported issues and autonomously maintaining software.

Primary area:
datasets and benchmarks
Keywords:
Language Models, Benchmark, Software Engineering

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_cAa5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SWE-sweep, which evaluates whether coding agents can autonomously discover and repair multiple bugs in an entire repository without issue descriptions or localization hints. The benchmark contains 4,068 historical bugs across 100 repositories and 22 programming languages. Its construction pipeline selects commits with coexisting bugs, adapts and validates upstream patches, and filters candidates for discoverability and test appropriateness. Repairs are evaluated using hidden bug-specific tests and the retained regression suite. Across seven language models in ten configurations, the strongest achieves 4.7% bug resolution. Information ablations indicate that discovering defects and inferring their intended behavior account for substantial difficulty.

Critical strengths
  1. The paper provides a substantial benchmark with meaningful construction and validation work. Selecting historical snapshots with many coexisting defects creates a useful evaluation of repository-wide discovery and repair. Executable reproduction checks, ten-run test-stability checks, regression testing, and audits of patch adaptation materially strengthen confidence in the resource. The unrestricted search scope is a meaningful extension over Active-SWE, which derives the supplied inspection scope from reference-patch filenames.

  2. The information ablation convincingly establishes an additional source of difficulty. On the same 20 repositories containing 684 bugs, Sol’s resolution rate increases from 4.8% without hints to 11.0% with filenames, 19.2% with line ranges, and 75.3% with issue descriptions (Table 7). This strongly supports evaluating discovery and specification inference alongside implementation. The additional harness and stopping-behavior checks also reduce the plausibility that the low baseline score is simply an artifact of one scaffold or forced budget exhaustion.

Critical weaknesses
  1. The discoverability validation does not fully match the documented inference criteria. The inference and construction prompts restrict accepted evidence to user-facing documentation, including types and incorporated standards and universal failure expectations. The subsequent validation explicitly also accepts existing tests, callers, identifiers, and internal consistency. Consequently, its high validation rate does not directly establish discoverability under the instructions given to evaluated agents. Separately, the stricter audit flags 12.4% of cases as potentially lacking support for some test expectations), without reported adjudication or score sensitivity. These flags are not established dataset errors, but they leave uncertainty about grading fairness. The human study only partly addresses this concern because reviewers saw issue descriptions and gold fixes.

  2. Evidence for broader maintenance quality is weaker than evidence for recovering reference bugs. The historical reference set is a reasonable, explicitly acknowledged evaluation choice. However, the supplementary analysis counts attempted fixes outside that set without checking their correctness, completeness, or execution. Agreement in the ordering of two models therefore cannot establish the ordering of their total useful repairs or the precision of their changes. Existing regression tests provide a valuable safeguard, but leave this broader claim unresolved. Active-SWE’s separate evaluation using generated reproducer tests offers a relevant methodological comparison, although it also relies partly on model judgments.

Questions for authors
  1. Are existing tests and internal consistency intended to be admissible evidence for evaluated agents? How many retained bugs are supported exclusively by sources added in E.3, and does explicitly aligning the inference prompt with that broader definition materially change performance?

  2. Have the cases flagged in A.4.3 been adjudicated by reviewers blinded to issue descriptions and gold fixes? How stable are scores and model rankings after excluding confirmed unsupported cases?

  3. Is there an executable or independently reviewed correctness audit of fixes outside the reference set? If unavailable, can the conclusions be confined to reference-bug recovery without treating attempted-fix counts as validation of overall maintenance quality?

1 comment

Thank you for your thorough and constructive review. We appreciate that you highlight our careful construction and validation work, including our ablations, as well as the unrestricted search scope of this benchmark.

We are happy to provide additional evidence that this benchmark exposes a missing capability in frontier models, rather than having low model scores being perceived as benchmark methodology issues.

W1/Q1

The inference and construction prompts restrict accepted evidence to user-facing documentation (...) The subsequent validation explicitly also accepts existing tests, callers, identifiers, and internal consistency.

Thank you for pointing this out. The criteria for bugs were meant to be identical between the inference prompt and the prompts used for filtering. We will fix this. However, the exact prompt has no significant effect on model scores:

We reran three models on the 20 ablation repositories (684 bugs) with two prompt variants: one adds the missing evidence sources, and one removes all bug criteria.

Bugs fixedSolLuna (xhigh)Terra (xhigh)
Original prompt (mean of 3 runs)29.717.318.3
Aligned with validation criteria (change, 95% CI)-0.7 [-10.3, +9.7]+3.7 [-0.3, +7.3]+2.7 [-5.7, +11.7]
No bug criteria (change, 95% CI)-3.7 [-11.3, +4.7]+0.7 [-3.0, +5.0]+2.7 [-4.3, +9.0]

The affected bugs are also few: 389 of 4,068 bugs (9.6%) are supported only by the sources the validation added. Without them, Sol scores 4.73% instead of 4.70%, and no model changes rank.

W1/Q2

Separately, the stricter audit flags 12.4% of cases as potentially lacking support for some test expectations

We appreciate the attention to detail. As we write in the paper, we treat the 12.4% number as an upper bound: (1) the validation task itself is challenging for an agent, (2) a single test assertion without support in the repository is enough for a flag, and (3) we empirically observe that every model fixes 8-14 of these bugs.

When we exclude all flagged bugs (12.4%), no model changes rank, and every score changes by less than 0.25 pp and not in a uniform direction.

Bugs scoredSolLuna (xhigh)Terra (xhigh)Luna (high)Opus
All4.70%2.53%1.55%1.40%1.28%
Without the 12.4%4.93%2.59%1.53%1.36%1.23%

W2/Q3

Evidence for broader maintenance quality is weaker than evidence for recovering reference bugs (...) [The] supplementary analysis counts attempted fixes outside that set without checking their correctness, completeness, or execution

We are happy to improve on this supplementary analysis from the appendix.

We audited all bug-fix attempts of the top 3 models Astra (newly added), Sol and Luna (xhigh) on the 20 ablation repositories. For each attempt, an agent from a different vendor (Claude Opus 5.5) decided whether it fixes a real bug ("verified fix"). Checks included generating F2P tests for every verified fix, adopting the methodology from Active-SWE.

AstraSolLuna (xhigh)
Score (20 repos, reference bugs resolved)8.8%4.8%2.0%
Verified out-of-reference fixes778522294
Precision (verified fixes / attempts)87%73%63%

The order of Astra > Sol > Luna holds for all three metrics (also across 10k bootstrapped samples).

Separately, we have also studied the sensitivity of our scores to the selection of the (reference) fixes. When we keep a random 50% of the bugs in every repository (10k draws) and rescore existing agent submissions, we observe a standard deviation of at most 0.4 pp for each score. The top 3 models Astra, Sol and Luna (xhigh) keep the same rank in every draw.

Together, these analyses support both the broader relevance of our score and its robustness to subsampling the reference set. We agree with the reviewer that assessing out-of-reference fixes introduces subjectivity through reliance on model judgments. We therefore treat the above results as complementary evidence only, while retaining our main metric built on clear maintainer ground truth.


Thank you again for reading the paper and appendices so closely. We hope these new analyses address your concerns, and we are happy to answer any further questions.

Reviewer_BJwLOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SWE-sweep, a benchmark for evaluating the ability of coding agents to autonomously find and solve multiple bugs in code repositories without any issue descriptions or bug localization hints. The authors built the dataset from real GitHub issue-PR histories. Through bug lifetime evaluation, choosing the commit with the most existing bugs, test validation, and quality filtering, they finally obtained a dataset comprising 100 repositories, 22 languages, and 4,068 bugs. The experiments show that current mainstream frontier models can only solve 4.7% of the bugs, and an ablation study further shows that the bottleneck is bug discovery.

Critical strengths
  1. The benchmark setting is valuable and meets real-world needs. SWE-sweep requires the agent to autonomously explore the entire repository without any guidance or localization, solving multiple bugs simultaneously. This setting is closer to real-world software engineering development work compared to SWE-bench tasks.

  2. A large-scale dataset across multiple repositories and languages. The paper constructs a large-scale, cross-lingual, and cross-repository benchmark. The data construction process incorporates rigorous checks for reproducibility and quality, ensuring that the experimental results hold significant reference value.

Critical weaknesses
  1. The motivation and high-level idea have already been demonstrated in previous work. TIDE and Active-SWE have investigated issue-free and autonomous multi-bug discovery. In particular, TIDE has already adopted the method of finding multiple concurrent bugs in historical repository snapshots. However, this paper lacks a detailed and direct comparison with these closely related works.

  2. The filtering is very strict and may introduce distribution bias. The paper filters the original 704,386 bugs down to 4,068 bugs, and this selection is made by an LLM. This raises the concern that the filtering may only retain bugs that are easy and can be clearly evaluated by LLMs, while filtering out hard or complex ones. The human review of a 50-bug sample also cannot address the issue, because it only reviewed the accepted bugs.

[1] Jeong, Soyeong, et al. "TIDE: Proactive Multi-Problem Discovery via Template-Guided Iteration." arXiv preprint arXiv:2606.04743 (2026).

[2] Li, Haobin, et al. "Active-swe: Benchmarking coding agents for proactive bug fixing without issue reports." arXiv preprint arXiv:2608.04682 (2026).

Questions for authors
  1. Could the authors provide a systematic evaluation of out-of-reference bug discovery and repair? In particular, please report the bug discovery precision and false positive rates, along with concrete validation of whether these out-of-reference fixes are correct or introduce secondary defects.
3 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

Thank you to reviewer BJwL for your feedback on the paper. We appreciate how you recognized SWE-sweep’s diversity with respect to the spread of languages and repositories captured, along with how the incorporation of bug discovery into the task makes it more faithful to real-world software engineering. We hope to address your concerns as follows:


W1: Missing comparisons against closely related work, namely TIDE and Active-SWE

We appreciate the reviewer surfacing these recent two works; they are closely related, and in the current revision of the paper, we mention them in the Related Works’s Bug Detection subsection, albeit briefly.

Both works certainly share the concept of challenging models to discover bugs / localize faults, but differ meaningfully in (a) how much information the model is given, (b) the number of coexisting bugs per task, and (c) how fixes are graded:

  • In Active-SWE, the model is also asked to find and fix bugs without an issue description. However:

    • [Fewer bugs] Each task instance only features 1-2 PRs (far lower than 4 to 239 bugs per SWE-sweep task).
    • [Extra Information] In Active-SWE, the model is informed of the files edited by the reference patch. In our Table 7 ablation, giving Sol the gold-fix filenames (roughly what Active-SWE provides) raises its score from 4.8% to 11.0%. Active-SWE's own appendix (v2, Table 11) shows the same effect from the other side: widening the scope to all files drops GLM-5.2 from 15.0% to 2.5%.
  • TIDE is mainly a method paper, which includes a small benchmark of 146 bugs in 20 instances from 11 Python projects.

    • [Fewer bugs] Like SWE-sweep, TIDE groups bugs that coexisted at one historical commit, but with 2-41 bugs per instance, versus 4-239 execution-verified bugs per SWE-sweep snapshot.
    • [Not open-ended / repo-level] The model is given a list of 6 to ~600 Python functions, containing both buggy and “distractor” (not-buggy) functions. The model is never asked to explore the repository itself.
    • [Not agentic] The TIDE method gives models at most 3 long context calls (no tool calls or code execution).
    • [LM-as-a-judge grading] The model’s proposed code patch is graded by having an LM judge compare it against a reference solution, rather than executing tests.

We believe the density of bugs and comparative lack of information in SWE-sweep meaningfully distinguish our benchmark in difficulty and tested model capabilities.

We'll add a comparison table to Section 6, incorporate these points as a more extended discussion in the appendix of the paper, and make our description of TIDE more precise.


W2: Task filtering + selection process may introduce a bias towards easier tasks

To clarify, most of the filtered-out tasks are not removed due to LMs, and if they are, no gate is based upon gauging bug difficulty.

From Table 1:

  • 704k ⇒ 147k, Pre-filter [Light LM use]:
    • Deterministic checks of PR (adds tests, has a source fix, size limits, no duplicates)
    • The main LM check performed here is to assess whether the PR fixes a bug (as opposed to adding a feature or writing documentation). This is done based on GitHub PR/issue tags, titles, descriptions.
  • 147k ⇒ 9k, Patches apply + Reproduces [No LM filtering]:
    • The fix must apply at the base commit, and its tests must fail before and pass after it.
    • These are the canonical SWE-bench filtering checks.
  • 9k ⇒ 4k, Quality control [LM filtering]:
    • This is the heaviest use of LMs: We use a “panel” of 3 models that need to unanimously agree that a bug is discoverable, self-contained, and fairly tested. This is what we also cross-check against human agreement.

All in all, the LM checks ensure (a) each task can be graded fairly, e.g., its tests do not require one specific implementation, and (b) the bug itself is discoverable from the repository.

  • At no stage do we filter on difficulty; as stated in Section 2, "our checks do not assess the importance or difficulty of a bug."
  • We understand you may be concerned some gates accidentally correlate with some bias towards easier bugs; however, the gates reject bugs that would be unfair to grade rather than hard to solve (e.g., when the expected behavior is only stated in the issue), and the retained bugs remain very difficult, with the best model (Astra 6, newly added) resolving under 9%. We are open to suggestions for additional analyses the reviewer would find convincing.

In terms of the percentage of PRs that we successfully retain and convert into bug instances, our retention percentage is quite ordinary compared to past precedent: SWE-bench retains about 2.5% of crawled PRs and SWE-bench Multimodal 0.46%, comparable to our 0.6%.

AuthorsOpenReview ↗

Rebuttal (2/2)

Q1: Precision, false-positive rate, and correctness of out-of-reference fixes

We are happy to provide additional information on this.

We audited all bug-fix attempts of the top 3 models Astra (newly added), Sol and Luna (xhigh) on the 20 ablation repositories. For each attempt, an agent from a different vendor (Claude Opus 5.5) decided whether it fixes a real bug ("verified fix"). Checks included generating executable F2P tests for every verified fix, similar to Active-SWE, but with tests written by an independent model.

AstraSolLuna (xhigh)
Score (20 repos, reference bugs resolved)8.8%4.8%2.0%
Verified out-of-reference fixes778522294
Precision (verified fixes / attempts)87%73%63%
Secondary defects (attempts that break working behavior)3.7%7.9%9.6%

The order of Astra > Sol > Luna holds for all four metrics (also verified across 10k bootstrapped samples to account for statistical effects).

These results provide additional evidence that our reference-based score tracks genuine bug-fixing performance beyond the reference set. We are happy to include these numbers in the paper. However, extending the main metric to out-of-reference fixes that do not have maintainer-written executable tests would make our metric more subjective and risk reward hacking with speculative or minor fixes. PROBE (arXiv 2610.35987) shows a distribution shift between bugs that maintainers fix and those that agents find. This means that even technically correct fixes can be difficult to assess for relevance without evidence from maintainers.

For the same reasons, we consider precision a complementary measure rather than our main focus. When reporting precision against the reference set, the false discovery rate would only be an upper bound (as also remarked by IRIS, ICLR 2025, arXiv 2405.17238). On the other hand, once we take out-of-reference bugs into consideration, we hit the previously mentioned caveats. In addition, precision could also be addressed through downstream prioritization/filtering, but a bug that is never discovered cannot be recovered.

Our main metric provides a clear, reproducible, and unsaturated measure of this bug discovery capability. The above findings further support that our score tracks capabilities that generalize beyond the reference set.


Thank you again for your encouraging review! We really appreciate your time and feedback. Please let us know if there’s anything else we can do to address remaining concerns.

Thank you to the authors for the detailed responses and additional experiments. I will maintain my positive score.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

Thank you so much for your detailed review and for mentioning the careful construction and validation, as well as the ablations as critical strengths.

Given that SWE-sweep identifies a significant gap in model capability, we appreciate your close attention towards establishing the reliability of the results and model ranking.

W1/Q1: Main metric and out-of-reference edits

The metric measures only recall against historically fixed bugs, not precision, so it is unclear that it measures discovery ability. The reference set undercounts real bugs. Whether the remaining edits are genuine fixes or spurious changes is unknown.

We agree that the reference set may undercount real bugs: it contains only defects that maintainers later fixed and that we could reproduce. This is a deliberate trade-off in order to use maintainer-written tests for objective grading. Most benchmarks such as SWE-bench Verified make the same choice. Our score answers a concrete discovery question: what fraction of known, reproducible defects can an agent fix without bug-specific guidance?

An incomplete reference set matters only if the ranking depends on which bugs it happens to contain. When we keep a random 50% of the bugs in every repository (10k draws) and rescore existing agent submissions, we observe a standard deviation of at most 0.4 pp for each score. The top 3 models Astra (newly added, 8.9%), Sol and Luna (xhigh) keep the same rank in every draw. The remaining tiers match those from the repository bootstrap (Q2).

We also tested directly whether the score tracks real bugs outside the reference set. We audited all bug-fix attempts of the top 3 models Astra, Sol and Luna (xhigh) on the 20 ablation repositories. For each attempt, an agent from a different vendor (Claude Opus 5.5) decided whether it fixes a real bug ("verified fix"). Checks included generating F2P tests for every verified fix. An author manually reviewed and confirmed 30 randomly sampled model judgments.

AstraSolLuna (xhigh)
Score (20 repos, reference bugs resolved)8.8%4.8%2.0%
Verified out-of-reference fixes778522294
Precision (verified real fixes / attempts)87%73%63%

The order of Astra > Sol > Luna holds for all three metrics (also across 10k bootstrapped samples). These results provide additional evidence that the reference-based score tracks genuine bug-fixing performance beyond the reference set.

We agree that precision is a useful complementary measure and are happy to add the above numbers to our paper. However, we maintain our choice of main metric: it provides objective grading and directly addresses our main research question. While precision is important in a product setting where users have limited capacity to review suggested fixes, it can be improved through downstream filtering and prioritization. Bugs that are never discovered cannot be recovered.

W2/Q2: Statistical robustness

Can you report run-to-run variance and repository-level bootstrap confidence intervals? ... Many repositories score zero, and differences hinge on single-digit bug counts.

SWE-sweep's main goal is to show a capability that current models lack, which is reflected in the low overall scores. Our leaderboard nonetheless allows us to clearly identify the top 3 models in stable order.

Repository-level cluster bootstrap (10k resamples): Astra 8.92% [6.85, 11.67], Sol 4.70% [3.53, 6.27], Luna (xhigh) 2.53% [1.85, 3.44], Terra (xhigh) 1.55% [0.99, 2.33], Opus 1.28% [0.83, 1.87]. Astra ranks first, Sol second, and Luna (xhigh) third in 100% of resamples. Below them, the leaderboard forms three tiers: {Terra (xhigh), Luna (high), Opus}, {Kimi K3, Luna (default), GPT-5.4 Mini (high)}, and {GPT-5.4 Mini (default), Gemini 3.5 Flash Lite}. Every model in a tier outscores every model in the next tier in at least 99.5% of resamples. We will report the tiers in Table 4.

Run-to-run variance: We ran Sol, Luna (xhigh), and Terra three times on the 20 ablation repositories. The resulting standard deviation, scaled to the full benchmark, is between 0.11 pp and 0.26 pp.

Zero-score repository counts are statistically compatible with a uniform solve rate. Our main score and ranking use the pooled bug count, and Astra, Sol, Luna (xhigh), Terra, and Opus are separated from each other by more than 10 bugs each.

Micro and macro averages differ markedly (4.7% vs. 10.3%).

We use the micro-average so that every bug counts equally. A macro-average would weigh a bug in our smallest repository 60 times as much as a bug in the largest. Both averages rank the models almost identically (Spearman 0.95 across all 10 configurations). The macro-average is higher because smaller repositories are easier.

AuthorsOpenReview ↗

Rebuttal (2/2)

"Discovery is harder than repair" relies on issue texts, which also supply symptoms and reproduction details. The claim that repair difficulty matches SWE-bench lacks direct evidence.

Thank you, we will state our argument more clearly. In SWE-bench, the issue text identifies the bug, often with reproduction steps, and the agent only has to fix it. In SWE-sweep, the agent must find what goes wrong and how to trigger it (discovery), then fix it (repair). The symptoms and reproduction details that the issue texts supply are exactly what we consider the discovery elements. With issue texts, Sol rises from 4.8% to 75.3% on the same 20 repositories (Table 7).

Can minimal hints (module or function only) separate localization from defect understanding?

Good suggestion. Table 7 already includes file and line-range hints. We added module and function hints (same 20 repos):

HintSolLuna (xhigh)
None4.8%2.0%
Module (new)4.5%2.5%
File11.0%4.8%
Function (new)23.2%6.6%
Line ranges19.2%10.7%
Issue texts75.3%36.8%

W3/Q5 Additional scaffolds

Results rely mainly on one lightweight scaffold, and the native harness (Codex) scores notably lower. Agents stop with most of their budget unused, which suggests a strategy issue rather than a capability ceiling.

We use mini-swe-agent to compare models apples-to-apples in a historically stable scaffold that removes confounding design decisions. Agents receive clear instructions about their scope and are rewarded for iterating on the task without giving up early.

This setup has precedent, e.g., in DeepSWE, ProgramBench (NeurIPS '26), and CodeClash (ICML '26). DeepSWE also reports that mini-swe-agent outperforms native harnesses.

More compute gives diminishing returns: 10x compute (10 sequential or independent agents) raises scores by only 1.9-2.5 pp (Fig. 7).

There are no discovery-oriented baselines (fuzzing/test generation, static analysis, scan-then-fix, multi-agent).

In addition to Fig. 7, we tested a work-queue-based multi-agent setup on the 5 largest repositories and Luna (xhigh). Scaling from 5 parallel agents to 40 saw less than 2x improvement at 6x increased cost.

Static analysis: We ran ruff, pylint, mypy, cppcheck, clang-tidy, and the native Nim and V checkers on five large repositories (926 bugs). Out of 430,049 warnings, only 3 bugs (0.3%) were directly flagged, since SWE-sweep bugs are mostly semantic (wrong values, branches, or edge cases) and these projects already run linters in CI.

We also tested agents with a full discovery tool chain and strategy hints. On the 20 ablation repositories, we installed static analyzers (semgrep, cppcheck, clang-tidy), fuzzers (libFuzzer, AFL++, Jazzer), test generators (Pynguin, Randoop, EvoSuite), property-based testing, and sanitizers, and listed them in the prompt together with common strategies (scan then fix, fuzzing, property-based testing, test generation). Every run used at least one tool, but neither model improved significantly: Sol resolved 24 bugs (26-33 in 3 default runs) and Luna (xhigh) 22 (14-20), and all paired 95% CIs include zero.

W4/Q3, Q4

Filtering uses LMs similar to those evaluated.

We employed models from three vendors for filtering and required unanimous agreement for acceptance. We see no advantage for models used as judges: Opus 5 is a judge, yet it scores significantly below Luna (xhigh), which is not.

Human validation (50 bugs, by the authors, after seeing LM reasoning) is small and not independent.

We’d like to highlight that LM judges are only one layer of our filtering pipeline. For the human validation, we clarify that reviewers cast their first vote before seeing the LM judges' reasoning. On these first votes, 47 of 50 bugs were accepted by a 3/4 majority (one-sided 95% lower bound: 85%).

The "any broken test zeroes your score" wording may induce conservatism, with no prompt-sensitivity test.

The score is zeroed only if the patch breaks tests that already pass in the repository given to the agent, which it can run before submitting.

Regression prompt ablation: We removed the regression rule and its zero-score note from the prompt, keeping scoring unchanged. On the 20 repos, scores stay within run-to-run noise (Sol 26 bugs vs. 26-33 in 3 default runs, Luna (xhigh) 19 vs. 14-20), and regressions do not occur more often.

The prompt discloses discoverability criteria, so the task is not strictly zero-information.

We will replace "zero information" with "no bug-specific information".

Criteria prompt ablation: Removing the criteria does not change scores significantly (Sol -3.7 [-11.3, +4.7], Luna (xhigh) +0.7 [-3.0, +5.0], Terra (xhigh) +2.7 [-4.3, +9.0] bugs on the 20 repos).


Thank you again for engaging so closely with our work. We hope these results address your concerns, and we would welcome any further feedback.

Thank you everyone for your constructive reviews. We are hoping that our rebuttal addressed your questions and concerns. Please let us know if there are any other points that we could address.

In the meantime, we wanted to share some updates to the leaderboard:

ModelBugs resolvedTotal costNotes
Gemini 4 Hydrogen13.5%Independent evaluation with Antigravity from the Gemini 4 Hydrogen model card
Fable 5.212.3%Independent evaluation with mini-swe-agent.
Gemini 4 Carbon9.6%Independent evaluation with mini-swe-agent from the Gemini 4 Carbon model card.
Astra 68.9%$3,255
Sol 5.6 (xhigh)4.7%$7,230
Opus 5.5 (xhigh)3.6%$2,198
Fable 5.12.9%$3,904
Kimi K3.1 Pro (max)2.5%$1,245
Luna 5.6 (xhigh)2.5%$224
Kimi K3.1 Pro (high)2.2%$658
DeepSeek V4.1 Flash (max)1.6%$174
Terra 5.6 (xhigh)1.5%$357
Luna 5.6 (high)1.4%$28
Opus 5 (xhigh)1.3%$5,363
GLM-5.3 (max)1.0%$2,948
DeepSeek V4.1 Flash (high)0.9%$95
GLM 5.3 Flash (max)0.7%$283
MiniMax M3.1 Flash (max)0.7%$153
GLM 5.3 Flash (high)0.6%$177
Kimi K30.6%$2,451
Luna 5.60.5%$4
GPT-5.4 Mini (high)0.5%$122
GPT-5.4 Mini0.2%$5
Gemini 3.5 Flash Lite0.1%$6

We make the following empirical observations:

  • Recently released models make substantial progress on the benchmark, but it still remains extremely challenging
  • Scores with mini-swe-agent range from 0.1% to 12.3% for models all released this year. Costs per resolved bug vary drastically.
  • Performance increases with reasoning/effort level (holds for all evaluated models)
  • Open-weight models are lagging behind in performance (the best score being Kimi K3.1 Pro (max) with only 2.2%). DeepSeek V4.1 Flash (max) is the only model on the Pareto frontier.

We hope this further underlines the usefulness of this benchmark.

Thanks for the response and the added experiments on confidence intervals, localization hints, and prompt ablations. However, two core issues prevent me from raising my score:

1. Contradictory numbers across the manuscript and rebuttal

  • Human validation (50 sampled bugs): Section 2.2 reports 49/50 (98%) agreement (at least 3/4 reviewers). Appendix A.2 claims 49/50 before seeing LM reasoning and 50/50 after. Appendix A.4.1 and the rebuttal claim 47/50 (94%) before and 49/50 (98%) after (with the last bug split 2-1-1). Meanwhile, Appendix B.7 claims 50/50 met the 3/4 majority (46 unanimous, 4 with one dissenting vote), which directly contradicts Table 8's report of a 98% majority and a 2% (2-vs-2) split.

  • Benchmark size and solve rates: Table 4 reports 4,068 bugs with Sol 5.6 and Luna 5.6 (xhigh) at 4.7% and 2.5%. Yet Appendix C.2 (Table 20) evaluates 4,165 uncleaned bugs (164 missing outcomes), giving 5.0% (208/4,120) for Sol and 2.7% (113/4,121) for Luna. Other tables report 4,025 completed annotations (Table 9), 4,121 labeled bugs with 4,076 results (Table 21), and a 4.6% Sol solve rate (Table 11). The language count also shifts between 22 (main text) and 31 (Figure 3, Table 28). Please explain how these denominators and tallies match up.

2. Out-of-reference precision and validation independence For Q1, rather than manually auditing ~100 out-of-reference edits, you used Claude Opus 5.5 to verify fixes and write fail-to-pass tests, with one author checking 30 cases. LM-generated tests on top of a patch only show that the new behavior is testable—they do not prove it fixed a true pre-existing bug rather than altering behavior or refactoring. Precision is also still excluded from the main metric, and both the 50-bug and 30-sample checks rely entirely on unblinded author annotations.

The new ablations are a step in the right direction, but given the unresolved data discrepancies and evaluation concerns, I am maintaining my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.