- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents SWE-sweep, which evaluates whether coding agents can autonomously discover and repair multiple bugs in an entire repository without issue descriptions or localization hints. The benchmark contains 4,068 historical bugs across 100 repositories and 22 programming languages. Its construction pipeline selects commits with coexisting bugs, adapts and validates upstream patches, and filters candidates for discoverability and test appropriateness. Repairs are evaluated using hidden bug-specific tests and the retained regression suite. Across seven language models in ten configurations, the strongest achieves 4.7% bug resolution. Information ablations indicate that discovering defects and inferring their intended behavior account for substantial difficulty.
Critical strengths
-
The paper provides a substantial benchmark with meaningful construction and validation work. Selecting historical snapshots with many coexisting defects creates a useful evaluation of repository-wide discovery and repair. Executable reproduction checks, ten-run test-stability checks, regression testing, and audits of patch adaptation materially strengthen confidence in the resource. The unrestricted search scope is a meaningful extension over Active-SWE, which derives the supplied inspection scope from reference-patch filenames.
-
The information ablation convincingly establishes an additional source of difficulty. On the same 20 repositories containing 684 bugs, Sol’s resolution rate increases from 4.8% without hints to 11.0% with filenames, 19.2% with line ranges, and 75.3% with issue descriptions (Table 7). This strongly supports evaluating discovery and specification inference alongside implementation. The additional harness and stopping-behavior checks also reduce the plausibility that the low baseline score is simply an artifact of one scaffold or forced budget exhaustion.
Critical weaknesses
-
The discoverability validation does not fully match the documented inference criteria. The inference and construction prompts restrict accepted evidence to user-facing documentation, including types and incorporated standards and universal failure expectations. The subsequent validation explicitly also accepts existing tests, callers, identifiers, and internal consistency. Consequently, its high validation rate does not directly establish discoverability under the instructions given to evaluated agents. Separately, the stricter audit flags 12.4% of cases as potentially lacking support for some test expectations), without reported adjudication or score sensitivity. These flags are not established dataset errors, but they leave uncertainty about grading fairness. The human study only partly addresses this concern because reviewers saw issue descriptions and gold fixes.
-
Evidence for broader maintenance quality is weaker than evidence for recovering reference bugs. The historical reference set is a reasonable, explicitly acknowledged evaluation choice. However, the supplementary analysis counts attempted fixes outside that set without checking their correctness, completeness, or execution. Agreement in the ordering of two models therefore cannot establish the ordering of their total useful repairs or the precision of their changes. Existing regression tests provide a valuable safeguard, but leave this broader claim unresolved. Active-SWE’s separate evaluation using generated reproducer tests offers a relevant methodological comparison, although it also relies partly on model judgments.
Questions for authors
-
Are existing tests and internal consistency intended to be admissible evidence for evaluated agents? How many retained bugs are supported exclusively by sources added in E.3, and does explicitly aligning the inference prompt with that broader definition materially change performance?
-
Have the cases flagged in A.4.3 been adjudicated by reviewers blinded to issue descriptions and gold fixes? How stable are scores and model rankings after excluding confirmed unsupported cases?
-
Is there an executable or independently reviewed correctness audit of fixes outside the reference set? If unavailable, can the conclusions be confined to reference-bug recovery without treating attempted-fix counts as validation of overall maintenance quality?
Thank you for your thorough and constructive review. We appreciate that you highlight our careful construction and validation work, including our ablations, as well as the unrestricted search scope of this benchmark.
We are happy to provide additional evidence that this benchmark exposes a missing capability in frontier models, rather than having low model scores being perceived as benchmark methodology issues.
W1/Q1
Thank you for pointing this out. The criteria for bugs were meant to be identical between the inference prompt and the prompts used for filtering. We will fix this. However, the exact prompt has no significant effect on model scores:
We reran three models on the 20 ablation repositories (684 bugs) with two prompt variants: one adds the missing evidence sources, and one removes all bug criteria.
The affected bugs are also few: 389 of 4,068 bugs (9.6%) are supported only by the sources the validation added. Without them, Sol scores 4.73% instead of 4.70%, and no model changes rank.
W1/Q2
We appreciate the attention to detail. As we write in the paper, we treat the 12.4% number as an upper bound: (1) the validation task itself is challenging for an agent, (2) a single test assertion without support in the repository is enough for a flag, and (3) we empirically observe that every model fixes 8-14 of these bugs.
When we exclude all flagged bugs (12.4%), no model changes rank, and every score changes by less than 0.25 pp and not in a uniform direction.
W2/Q3
We are happy to improve on this supplementary analysis from the appendix.
We audited all bug-fix attempts of the top 3 models Astra (newly added), Sol and Luna (xhigh) on the 20 ablation repositories. For each attempt, an agent from a different vendor (Claude Opus 5.5) decided whether it fixes a real bug ("verified fix"). Checks included generating F2P tests for every verified fix, adopting the methodology from Active-SWE.
The order of Astra > Sol > Luna holds for all three metrics (also across 10k bootstrapped samples).
Separately, we have also studied the sensitivity of our scores to the selection of the (reference) fixes. When we keep a random 50% of the bugs in every repository (10k draws) and rescore existing agent submissions, we observe a standard deviation of at most 0.4 pp for each score. The top 3 models Astra, Sol and Luna (xhigh) keep the same rank in every draw.
Together, these analyses support both the broader relevance of our score and its robustness to subsampling the reference set. We agree with the reviewer that assessing out-of-reference fixes introduces subjectivity through reliance on model judgments. We therefore treat the above results as complementary evidence only, while retaining our main metric built on clear maintainer ground truth.
Thank you again for reading the paper and appendices so closely. We hope these new analyses address your concerns, and we are happy to answer any further questions.