- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents SWEeper-Bench, a benchmark of 200 tasks built from real bug-fix pull requests in open-source web applications. In each task, the agent receives the buggy codebase, a running application with a browser, and an open-ended prompt that asks it to find all issues in a given product area and fix them. After the agent finishes, a browser-based agentic verifier runs two tests: a target test, which checks whether the bug is fixed, and a preservation test, which checks whether nearby features still work. The best-performing agent reaches 59.0%, whereas with a detailed bug description agents reach about 96%.
Critical strengths
S1. The tasks are real bugs taken from diverse applications, not synthetic injections. S2. The browser-based testing reliably verifies fixes without the fragile, easily broken nature of hidden unit tests. S3. The finding-versus-fixing ablation, stage-by-stage funnel, and observation that agents rarely use the browser to look for bugs unless instructed are all insightful.
Critical weaknesses
W1. The paper acknowledges (lines 147-152 and the conclusion) that an agent "may fix several issues in the named area, but we evaluate only the target bug", so the benchmark measures recall rather than precision. This should be considered an incompleteness of the work, not a limitation. The prompt asks the agent to "find and fix all issues" in an area, yet any fix outside the one pre-selected bug earns no credit. The agents' real contributions are therefore hidden, and "failed to identify the buggy behavior" conflates "found nothing" with "found a different valid bug". Figure 6 illustrates this: GPT-6 Astra sees the empty sidebar but moves on, then finds and fixes a role-permissions bug in the Share dialog ("make the UI honor the roles"). That bug lies inside the named area ("board sharing and membership"), so the agent followed the prompt, yet it is counted as a failure.
w2. The agent receives a buggy codebase, a running app, and an open-ended prompt to find and fix all issues, but no description of the intended behavior. Without a reference, it is unclear what counts as an issue. This explains the Claude Opus 5 case in Figure 6. It sees the empty sidebar but trusts the existing unit tests, which contain no expectation about the sidebar, and concludes that this is "intended behavior" and "a product decision, not a bug fix". This is not a failure of the model. Nothing in the prompt or the repository says the sidebar should update, so declining to change test-backed behavior is defensible. The agent is penalized only because the hidden target test says otherwise.
W3. Both cases above show that the specification and tests are incomplete, so part of the reported "discovery gap" is an artifact of the specification rather than a measure of agent capability. The paper does not explain how completeness of the tests was ensured. Were the target and preservation tests written or reviewed by multiple annotators, and what is the inter-annotator agreement? None is reported, and the tests appear to have been written and confirmed by the authors alone. Table 1 shows reliability only on the reference fix and the buggy build, not that the tests accept alternative valid fixes or reject plausible wrong ones. The verifier (GPT-5.6 Luna) is also one of the evaluated agents and the judge for the stage analysis. The authors should audit failed patches for valid alternative fixes, give credit for other verified fixes, or report precision.
W4. Section 2.1 says only that each task is built from "a merged pull request that fixes a user-facing bug in an open-source web application", and the appendix covers only the prompts and one example test. The paper does not say how the 200 PRs were found: which sources were used (GitHub search, curated lists of self-hostable apps, other benchmarks), which queries, keywords, or issue/PR labels were used, how many candidate PRs or repositories were screened, or which filters were applied (stars, activity, language, license, PR size, whether the app can be run in a sandbox). It also does not say who screened the candidates, how many were rejected at each step, or how the seven domains and the limit of one task per app were chosen. Without this information, it is hard to judge selection bias (for example, toward popular, easily containerized, or well-documented apps), to reproduce the collection, or to extend the benchmark.
W5 The paper gives no operational definition of this criterion and no human validation of it. It also reports only the median repository stars (4.6k) and not the range. Results are single runs with no confidence intervals (roughly ±7 points at n=200). The time-budget experiment uses only 40 tasks, and the authors admit the differences are not significant. The harness and browser ablations also come without significance tests. GPT-5.6 Luna with Codex scores 42.0 in Table 2 but 42.5 in Table 5.
Questions for authors
Please have a look aT W1-W4, more specifically,
- Why is only the target bug scored? How often do failed runs (e.g., GPT-6 Astra in Figure 6) contain valid fixes for other bugs in the named area, and how many failures stem from under-specification (e.g., Claude Opus 5 trusting the unit tests)?
- How were the 200 PRs collected (sources, queries, filters, number screened, who screened)? How was "hard to find" validated?
- How do you ensure the test cases are complete? Were they written or reviewed by multiple annotators, and what is the inter-annotator agreement?
- Can you add confidence intervals, multiple runs, and significance tests?
3 comments
Response to Reviewer XQLi (2/3)
W3 & Q3. Review of the behavior tests.
Every behavior test is drafted by an agent, revised by one annotator, and reviewed by a second annotator.
- Who wrote the tests? Agents draft each test. One annotator manually uses both the buggy version and the reference version of the application, following the test’s workflow, and revises the test if it fails to reproduce the bug on the buggy version or to reach the expected final outcome on the reference version. A second annotator then reviews it.
- What does a test check? Each test follows a user workflow and checks what users see, not the reference diff. Its pass/fail condition covers only the minimal final outcome that a correct fix must produce.
- Do the tests reject wrong patches? For each task, we generated patches that deliberately break the preserved feature and checked that the preservation test fails on every one. In the main evaluation, preservation tests caught 10 real regressions across 8 tasks and 7 agents.
- Do test verdicts match human judgment? Yes. In a human review of 50 sampled failed runs, annotators agreed with the verifier on all 50 (see the next section).
W3. Human validation and acceptance of alternative fixes.
We conducted an additional human review of 50 sampled failed runs, and the annotators agreed with the verifier’s conclusion on all 50. Each test’s pass/fail condition describes only the minimal final outcome that a correct fix must produce, not how the fix achieves it, so different implementations that produce the same outcome pass.
W3. Validation of the stage judge.
We conducted two additional experiments to validate the stage judge and found that both a second judge and human annotators agree closely with it (88–93%). Because GPT-5.6 Luna is also an evaluated agent, we used Claude Haiku 5.5 (high reasoning), from a different model family, as the second judge. We also had human annotators label 20 randomly selected trajectories.
The first table shows how often each pair of labelers assigns the same stage labels.
| Human | GPT-5.6 Luna | Claude Haiku 5.5 | |
|---|---|---|---|
| Human | — | 91.7% | 93.3% |
| GPT-5.6 Luna | 91.7% | — | 88.3% |
| Claude Haiku 5.5 | 93.3% | 88.3% | — |
The second table shows the share of trajectories that reach each stage under each judge.
| Stage | GPT-5.6 Luna | Claude Haiku 5.5 |
|---|---|---|
| Identified target feature | 75.4% | 64.8% |
| Identified buggy behavior | 50.0% | 47.1% |
| Diagnosed root cause | 43.8% | 42.5% |
| Fixed buggy behavior | 41.0% | 40.7% |
| Maintained existing behaviors | 40.8% | 40.5% |
Both judges support the paper’s conclusion that identifying the buggy behavior is the main bottleneck.
W4 & Q2. Task sourcing.
We selected the 200 tasks from about 500 candidates, nominated from several thousand PRs, through five rounds of screening and verification.
- How were the PRs found? Agents searched several thousand PRs following human-written guidelines and nominated about 500 candidates.
- Which filters were applied? We kept bugs that appear only through specific interactions, are hard to spot by reading code, and seriously disrupt normal use. Each PR also had to (1) fix a problem users see in real use, (2) violate a clear product rule or user expectation rather than a debatable design choice, (3) make one logical fix rather than a refactor, (4) be verifiable in the browser with behavior tests alone, without extra fixtures or scripts, (5) run in a single self-contained container, and (6) come from a repository with no other task. We removed PRs that add new features or change how an existing feature is meant to work, PRs that only change CI, linting, documentation, or performance, and apps that need docker-compose, multiple services, a GPU, or paid cloud services. About 45% of candidates passed. This favors self-contained, self-hostable apps, which we will note as a limitation.
- Who screened the candidates? Human annotators. Agents only nominated candidates; annotators deployed each one and reproduced its bug in the running application. A candidate was dropped if its environment or behavior tests still failed validation after five revision attempts.
- How were the final 200 chosen? From the remaining candidates, we removed redundant bug types and kept one task per application for diversity.
We will add these details to Sec. 2.1.
Response to Reviewer XQLi (3/3)
W5 & Q2. Definition of hard to find.
We call a bug hard to find when it appears only after a sequence of normal user actions: a median of 6 steps, and at least 5 for 80% of tasks. Sequences range from 2 to 14 steps, and many single steps, such as signing in or submitting a form, involve several actions. Each bug also had to occur on a common user path, not through rare timing, unusual focus combinations, race conditions, or contrived gestures, and had to produce a clearly visible symptom with real impact, such as a blocked flow, wrong or lost data, an action applied to the wrong object, or an unusable control.
W5. Repository popularity.
Most repositories are widely used: 150 of the 200 have at least 1k stars, and 57 have at least 10k. Stars range from 30 to 96.9k, with a median of 4.6k (interquartile range 1.0k–11.2k).
W5 & Q4. Variance across repeated runs.
We conducted additional experiments with repeated runs and found that different runs within a model differ by only a few points, with no statistically significant difference between them. We ran GPT-5.6 Luna, GLM 5.3 Flash, and Kimi K3.1 Pro three times each, and Qwen 3.8 Flash twice.
| Model | Run 1 | Run 2 | Run 3 | Mean | 95% CI across runs |
|---|---|---|---|---|---|
| GPT-5.6 Luna | 42.0% | 39.0% | 39.0% | 40.0% | 40.0% ± 4.3 (35.7–44.3%) |
| GLM 5.3 Flash | 39.5% | 37.0% | 42.0% | 39.5% | 39.5% ± 6.2 (33.3–45.7%) |
| Kimi K3.1 Pro | 50.0% | 48.5% | 46.5% | 48.3% | 48.3% ± 4.4 (44.0–52.7%) |
| Qwen 3.8 Flash | 57.5% | 56.0% | — | 56.8% | 56.8% ± 9.5 (47.2–66.3%) |
We compared runs of the same model with exact McNemar tests, plus Cochran’s Q tests for models with three runs. All p-values are above 0.05, and the repeated pass rates are in line with the original results.
| Model | Run 1 vs. 2 | Run 2 vs. 3 | Run 3 vs. 1 | Cochran’s Q |
|---|---|---|---|---|
| GPT-5.6 Luna | 0.31 | 1.00 | 0.38 | 0.45 |
| GLM 5.3 Flash | 0.49 | 0.11 | 0.47 | 0.21 |
| Kimi K3.1 Pro | 0.74 | 0.61 | 0.28 | 0.48 |
| Qwen 3.8 Flash | 0.72 | — | — | — |
W5 & Q4. Significance of model differences.
All model pairs differ significantly except GPT-5.6 Luna and GLM 5.3 Flash. Two-sided permutation tests give p < 0.001 for every other pair and p = 0.87 for that one. Qwen 3.8 Flash beats Kimi K3.1 Pro, and both beat Luna and GLM 5.3 Flash, matching the leaderboard.
W5. Luna’s pass rate in Tables 2 and 5.
The two numbers come from different task sets: Table 2 reports 42.0% on all 200 tasks, while Table 5 reports 42.5% on a 40-task subset. We will make this clear in the revision.
We hope these clarifications address your concerns. Thank you again for your feedback.
Response to Reviewer XQLi (1/3)
Thank you for recognizing that SWEeper-Bench is built from real bugs in diverse applications rather than synthetic injections. We address your remaining concerns below.
W1 & Q1. Scoring only the target bug.
We score only the target bug because a competent sweeper agent should produce a complete set of fixes for the named area, and that set has to include the target bug. The benchmark therefore measures recall, which we will state clearly in the revision.
W2 & Q1. Ambiguity in intended behavior.
Other software components imply the intended behavior: the screens, UI elements, related features, and code paths around a feature show what it should do, and the target bug contradicts them. Agents must work out intended behavior the way users do, by using the application, and this is the skill SWEeper-Bench is designed to measure.
W3. Philosophy of the benchmark.
The discovery gap is not an artifact of missing specifications: finding bugs without a specification is exactly the skill SWEeper-Bench is designed to measure. Every target behavior is a critical, core behavior of the software that other components of the software directly imply. An agent asked to sweep out all bugs in an area should therefore fix it even without a specification. If the agent can find the bug only with a detailed prompt, that is exactly the failure mode we want to capture. Users increasingly expect coding agents to discover and fix bugs on their own without detailed prompting, so this setting reflects an important use case.