acceptodds
Under review as a conference paper at ICLR 2027

SWEeper-Bench: Can Agents Discover Bugs in Interactive Software?

Abstract

Coding agents increasingly interact with users by generating interactive software on the fly rather than replying in text, yet many bugs in such software reveal themselves only after multi-step interactions. Today, users must discover these bugs and wait for fixes, making the development loop slow and expensive. Existing benchmarks typically assume the bugs have already been identified, leaving agents' ability to find interaction-dependent failures largely unevaluated. We introduce SWEeper-Bench, 200 tasks built from real bug fixes in 200 widely used open-source web applications. Each task gives the agent only the codebase and an open-ended request such as "Find and fix all issues in Focalboard's board sharing and membership." Because these bugs are hard to detect from code alone, agents must design interactive tests to uncover them. An agentic verifier then judges each repair by what a user would observe in the browser. The best of 15 frontier agents passes only 59.0% of tasks. Through stage-by-stage analysis of agent trajectories, we show that the biggest bottleneck is identifying the buggy behavior: agents often reach the relevant feature but never trigger the failure or fail to recognize it. SWEeper-Bench measures progress toward agents that close this loop themselves, delivering software that works without waiting for user feedback.

Primary area:
datasets and benchmarks
Keywords:
Software Engineering Agents, Autonomous Bug Discovery, GUI Verification, End-to-End Benchmarking, Web Applications, Browser Interaction, Full-Stack Evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_XQLiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents SWEeper-Bench, a benchmark of 200 tasks built from real bug-fix pull requests in open-source web applications. In each task, the agent receives the buggy codebase, a running application with a browser, and an open-ended prompt that asks it to find all issues in a given product area and fix them. After the agent finishes, a browser-based agentic verifier runs two tests: a target test, which checks whether the bug is fixed, and a preservation test, which checks whether nearby features still work. The best-performing agent reaches 59.0%, whereas with a detailed bug description agents reach about 96%.

Critical strengths

S1. The tasks are real bugs taken from diverse applications, not synthetic injections. S2. The browser-based testing reliably verifies fixes without the fragile, easily broken nature of hidden unit tests. S3. The finding-versus-fixing ablation, stage-by-stage funnel, and observation that agents rarely use the browser to look for bugs unless instructed are all insightful.

Critical weaknesses

W1. The paper acknowledges (lines 147-152 and the conclusion) that an agent "may fix several issues in the named area, but we evaluate only the target bug", so the benchmark measures recall rather than precision. This should be considered an incompleteness of the work, not a limitation. The prompt asks the agent to "find and fix all issues" in an area, yet any fix outside the one pre-selected bug earns no credit. The agents' real contributions are therefore hidden, and "failed to identify the buggy behavior" conflates "found nothing" with "found a different valid bug". Figure 6 illustrates this: GPT-6 Astra sees the empty sidebar but moves on, then finds and fixes a role-permissions bug in the Share dialog ("make the UI honor the roles"). That bug lies inside the named area ("board sharing and membership"), so the agent followed the prompt, yet it is counted as a failure.

w2. The agent receives a buggy codebase, a running app, and an open-ended prompt to find and fix all issues, but no description of the intended behavior. Without a reference, it is unclear what counts as an issue. This explains the Claude Opus 5 case in Figure 6. It sees the empty sidebar but trusts the existing unit tests, which contain no expectation about the sidebar, and concludes that this is "intended behavior" and "a product decision, not a bug fix". This is not a failure of the model. Nothing in the prompt or the repository says the sidebar should update, so declining to change test-backed behavior is defensible. The agent is penalized only because the hidden target test says otherwise.

W3. Both cases above show that the specification and tests are incomplete, so part of the reported "discovery gap" is an artifact of the specification rather than a measure of agent capability. The paper does not explain how completeness of the tests was ensured. Were the target and preservation tests written or reviewed by multiple annotators, and what is the inter-annotator agreement? None is reported, and the tests appear to have been written and confirmed by the authors alone. Table 1 shows reliability only on the reference fix and the buggy build, not that the tests accept alternative valid fixes or reject plausible wrong ones. The verifier (GPT-5.6 Luna) is also one of the evaluated agents and the judge for the stage analysis. The authors should audit failed patches for valid alternative fixes, give credit for other verified fixes, or report precision.

W4. Section 2.1 says only that each task is built from "a merged pull request that fixes a user-facing bug in an open-source web application", and the appendix covers only the prompts and one example test. The paper does not say how the 200 PRs were found: which sources were used (GitHub search, curated lists of self-hostable apps, other benchmarks), which queries, keywords, or issue/PR labels were used, how many candidate PRs or repositories were screened, or which filters were applied (stars, activity, language, license, PR size, whether the app can be run in a sandbox). It also does not say who screened the candidates, how many were rejected at each step, or how the seven domains and the limit of one task per app were chosen. Without this information, it is hard to judge selection bias (for example, toward popular, easily containerized, or well-documented apps), to reproduce the collection, or to extend the benchmark.

W5 The paper gives no operational definition of this criterion and no human validation of it. It also reports only the median repository stars (4.6k) and not the range. Results are single runs with no confidence intervals (roughly ±7 points at n=200). The time-budget experiment uses only 40 tasks, and the authors admit the differences are not significant. The harness and browser ablations also come without significance tests. GPT-5.6 Luna with Codex scores 42.0 in Table 2 but 42.5 in Table 5.

Questions for authors

Please have a look aT W1-W4, more specifically,

  • Why is only the target bug scored? How often do failed runs (e.g., GPT-6 Astra in Figure 6) contain valid fixes for other bugs in the named area, and how many failures stem from under-specification (e.g., Claude Opus 5 trusting the unit tests)?
  • How were the 200 PRs collected (sources, queries, filters, number screened, who screened)? How was "hard to find" validated?
  • How do you ensure the test cases are complete? Were they written or reviewed by multiple annotators, and what is the inter-annotator agreement?
  • Can you add confidence intervals, multiple runs, and significance tests?
3 comments
AuthorsOpenReview ↗

Response to Reviewer XQLi (1/3)

Thank you for recognizing that SWEeper-Bench is built from real bugs in diverse applications rather than synthetic injections. We address your remaining concerns below.

W1 & Q1. Scoring only the target bug.

We score only the target bug because a competent sweeper agent should produce a complete set of fixes for the named area, and that set has to include the target bug. The benchmark therefore measures recall, which we will state clearly in the revision.

  • Why only the target bug? We select target bugs that break the core functionality of the named area, so any complete set of fixes should include them. The target bug is also the one bug with verified intended behavior, so it can be scored reliably; arbitrary extra changes cannot. Extra fixes are not penalized unless they break a nearby feature.
  • The GPT-6 Astra example. Astra saw the target bug directly: the second user’s sidebar read “No boards inside,” and Astra moved on without comment. It then fixed a role-permissions issue in the Share dialog. Astra is not penalized for that fix. It fails because it left unfixed a critical bug it had already seen on screen, and a complete sweep of board sharing and membership should fix both.
  • Is recall informative? We acknowledge that SWEeper-Bench measures recall, not precision. Still, an agent that is poor at discovering bugs is unlikely to find and fix the target bug in task after task by luck, so the pass rate over 200 tasks reflects real discovery ability.

W2 & Q1. Ambiguity in intended behavior.

Other software components imply the intended behavior: the screens, UI elements, related features, and code paths around a feature show what it should do, and the target bug contradicts them. Agents must work out intended behavior the way users do, by using the application, and this is the skill SWEeper-Bench is designed to measure.

  • How surrounding elements imply intended behavior. For every task, human annotators checked that elements on screen clearly imply the intended behavior and that the reference fix restores behavior the rest of the application already implies. We removed PRs that add new features or change how an existing feature is meant to work, and PRs whose old behavior could reasonably be read as a product design choice.
  • What implies it in the Claude Opus 5 example. In Focalboard, the sidebar lists the boards a user belongs to: the owner’s sidebar shows the shared board, and boards a user creates, duplicates, or imports appear there too. When a second user joins the shared board as an editor, their sidebar reads “No boards inside,” which contradicts the rest of the app. The code shows the same pattern: it adds a board to the sidebar when a user creates, duplicates, or imports it, but not when a user joins.
  • The unit test does not represent intended app behavior well and misleads the agent. The AddMemberToBoard unit test only checks that a new membership is saved, with the database mocked out; it never looks at the sidebar. Opus treated this narrowly scoped test as a specification of intended app behavior and ignored the inconsistency outside the test’s scope. Trusting a narrow test over what users see is one of the failure modes we want to measure.

W3. Philosophy of the benchmark.

The discovery gap is not an artifact of missing specifications: finding bugs without a specification is exactly the skill SWEeper-Bench is designed to measure. Every target behavior is a critical, core behavior of the software that other components of the software directly imply. An agent asked to sweep out all bugs in an area should therefore fix it even without a specification. If the agent can find the bug only with a detailed prompt, that is exactly the failure mode we want to capture. Users increasingly expect coding agents to discover and fix bugs on their own without detailed prompting, so this setting reflects an important use case.

AuthorsOpenReview ↗

Response to Reviewer XQLi (2/3)

W3 & Q3. Review of the behavior tests.

Every behavior test is drafted by an agent, revised by one annotator, and reviewed by a second annotator.

  • Who wrote the tests? Agents draft each test. One annotator manually uses both the buggy version and the reference version of the application, following the test’s workflow, and revises the test if it fails to reproduce the bug on the buggy version or to reach the expected final outcome on the reference version. A second annotator then reviews it.
  • What does a test check? Each test follows a user workflow and checks what users see, not the reference diff. Its pass/fail condition covers only the minimal final outcome that a correct fix must produce.
  • Do the tests reject wrong patches? For each task, we generated patches that deliberately break the preserved feature and checked that the preservation test fails on every one. In the main evaluation, preservation tests caught 10 real regressions across 8 tasks and 7 agents.
  • Do test verdicts match human judgment? Yes. In a human review of 50 sampled failed runs, annotators agreed with the verifier on all 50 (see the next section).

W3. Human validation and acceptance of alternative fixes.

We conducted an additional human review of 50 sampled failed runs, and the annotators agreed with the verifier’s conclusion on all 50. Each test’s pass/fail condition describes only the minimal final outcome that a correct fix must produce, not how the fix achieves it, so different implementations that produce the same outcome pass.

W3. Validation of the stage judge.

We conducted two additional experiments to validate the stage judge and found that both a second judge and human annotators agree closely with it (88–93%). Because GPT-5.6 Luna is also an evaluated agent, we used Claude Haiku 5.5 (high reasoning), from a different model family, as the second judge. We also had human annotators label 20 randomly selected trajectories.

The first table shows how often each pair of labelers assigns the same stage labels.

HumanGPT-5.6 LunaClaude Haiku 5.5
Human—91.7%93.3%
GPT-5.6 Luna91.7%—88.3%
Claude Haiku 5.593.3%88.3%—

The second table shows the share of trajectories that reach each stage under each judge.

StageGPT-5.6 LunaClaude Haiku 5.5
Identified target feature75.4%64.8%
Identified buggy behavior50.0%47.1%
Diagnosed root cause43.8%42.5%
Fixed buggy behavior41.0%40.7%
Maintained existing behaviors40.8%40.5%

Both judges support the paper’s conclusion that identifying the buggy behavior is the main bottleneck.

W4 & Q2. Task sourcing.

We selected the 200 tasks from about 500 candidates, nominated from several thousand PRs, through five rounds of screening and verification.

  • How were the PRs found? Agents searched several thousand PRs following human-written guidelines and nominated about 500 candidates.
  • Which filters were applied? We kept bugs that appear only through specific interactions, are hard to spot by reading code, and seriously disrupt normal use. Each PR also had to (1) fix a problem users see in real use, (2) violate a clear product rule or user expectation rather than a debatable design choice, (3) make one logical fix rather than a refactor, (4) be verifiable in the browser with behavior tests alone, without extra fixtures or scripts, (5) run in a single self-contained container, and (6) come from a repository with no other task. We removed PRs that add new features or change how an existing feature is meant to work, PRs that only change CI, linting, documentation, or performance, and apps that need docker-compose, multiple services, a GPU, or paid cloud services. About 45% of candidates passed. This favors self-contained, self-hostable apps, which we will note as a limitation.
  • Who screened the candidates? Human annotators. Agents only nominated candidates; annotators deployed each one and reproduced its bug in the running application. A candidate was dropped if its environment or behavior tests still failed validation after five revision attempts.
  • How were the final 200 chosen? From the remaining candidates, we removed redundant bug types and kept one task per application for diversity.

We will add these details to Sec. 2.1.

AuthorsOpenReview ↗

Response to Reviewer XQLi (3/3)

W5 & Q2. Definition of hard to find.

We call a bug hard to find when it appears only after a sequence of normal user actions: a median of 6 steps, and at least 5 for 80% of tasks. Sequences range from 2 to 14 steps, and many single steps, such as signing in or submitting a form, involve several actions. Each bug also had to occur on a common user path, not through rare timing, unusual focus combinations, race conditions, or contrived gestures, and had to produce a clearly visible symptom with real impact, such as a blocked flow, wrong or lost data, an action applied to the wrong object, or an unusable control.

W5. Repository popularity.

Most repositories are widely used: 150 of the 200 have at least 1k stars, and 57 have at least 10k. Stars range from 30 to 96.9k, with a median of 4.6k (interquartile range 1.0k–11.2k).

W5 & Q4. Variance across repeated runs.

We conducted additional experiments with repeated runs and found that different runs within a model differ by only a few points, with no statistically significant difference between them. We ran GPT-5.6 Luna, GLM 5.3 Flash, and Kimi K3.1 Pro three times each, and Qwen 3.8 Flash twice.

ModelRun 1Run 2Run 3Mean95% CI across runs
GPT-5.6 Luna42.0%39.0%39.0%40.0%40.0% ± 4.3 (35.7–44.3%)
GLM 5.3 Flash39.5%37.0%42.0%39.5%39.5% ± 6.2 (33.3–45.7%)
Kimi K3.1 Pro50.0%48.5%46.5%48.3%48.3% ± 4.4 (44.0–52.7%)
Qwen 3.8 Flash57.5%56.0%—56.8%56.8% ± 9.5 (47.2–66.3%)

We compared runs of the same model with exact McNemar tests, plus Cochran’s Q tests for models with three runs. All p-values are above 0.05, and the repeated pass rates are in line with the original results.

ModelRun 1 vs. 2Run 2 vs. 3Run 3 vs. 1Cochran’s Q
GPT-5.6 Luna0.311.000.380.45
GLM 5.3 Flash0.490.110.470.21
Kimi K3.1 Pro0.740.610.280.48
Qwen 3.8 Flash0.72———

W5 & Q4. Significance of model differences.

All model pairs differ significantly except GPT-5.6 Luna and GLM 5.3 Flash. Two-sided permutation tests give p < 0.001 for every other pair and p = 0.87 for that one. Qwen 3.8 Flash beats Kimi K3.1 Pro, and both beat Luna and GLM 5.3 Flash, matching the leaderboard.

W5. Luna’s pass rate in Tables 2 and 5.

The two numbers come from different task sets: Table 2 reports 42.0% on all 200 tasks, while Table 5 reports 42.5% on a 40-task subset. We will make this clear in the revision.

We hope these clarifications address your concerns. Thank you again for your feedback.

Reviewer_c6n7OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a new benchmark, SWEeper-Bench, which contains 200 tasks collected from popular open-source web applications. The benchmark focuses on evaluating LLM agents' ability to discover and fix interaction-dependent bugs. For each task, the agent receives a buggy codebase and access to a running application through a browser, and is prompted to find and fix bugs with an open-ended instruction (e.g., finding and fixing bugs in a specific feature area). The agent then submits a patch, which is evaluated by an agentic verifier using predefined target and preservation behavior tests. The evaluation shows that even SOTA models and agents still struggle with these tasks, with the best-performing agent achieving a pass rate of 59.0%. The paper also provides extensive analyses of agent behaviors and factors affecting their performance, revealing that identifying buggy behavior is the primary bottleneck.

Critical strengths
  1. This paper introduces and formalizes a new and important task setting that has not been well explored by existing benchmarks. The experimental results also reveal a substantial capability gap in current SOTA agents.

  2. The paper provides extensive analyses and reveals several valuable insights, such as identifying the main bottleneck in the bug discovery process and showing that longer working time and more browser interactions do not necessarily lead to better performance.

Critical weaknesses
  1. In Section 3.2, the stage-wise analysis relies on GPT-5.6 Luna, but the reliability of its judgments is not validated against human annotations or other LLM judges. This raises concerns about the validity of the paper's main bottleneck analysis.

  2. The paper does not clearly explain how Preservation Behavior Tests are selected. This raises concerns about the reliability of the evaluation.

Questions for authors
  1. Have the authors evaluated the agreement between GPT-5.6 Luna's stage-wise judgments and human annotations or other LLM judges, particularly in ambiguous cases where an agent observes a failure but does not explicitly recognize it as a bug? How reliable are the conclusions about the main bottleneck across different judges?

  2. How are the Preservation Behavior Tests selected for each task?

2 comments
AuthorsOpenReview ↗

Response to Reviewer c6n7

Thank you for recognizing that SWEeper-Bench defines a new and important task that existing benchmarks have not explored well. We address your remaining concerns below.

W1 & Q1. Agreement in the stage-by-stage analysis.

We conducted two additional experiments and found that both a second judge and human annotators agree closely with our stage judge (88–93%). We re-ran the stage analysis with Claude Haiku 5.5 (high reasoning) as a second judge, and had human annotators label 20 randomly selected trajectories.

The first table shows how often each pair of labelers assigns the same stage labels.

HumanGPT-5.6 LunaClaude Haiku 5.5
Human—91.7%93.3%
GPT-5.6 Luna91.7%—88.3%
Claude Haiku 5.593.3%88.3%—

The second table shows the share of trajectories that reach each stage under each judge.

StageJudge: GPT-5.6 LunaJudge: Claude Haiku 5.5
Identified target feature75.4%64.8%
Identified buggy behavior50.0%47.1%
Diagnosed root cause43.8%42.5%
Fixed buggy behavior41.0%40.7%
Maintained existing behaviors40.8%40.5%

Both judges support the paper’s conclusion that identifying the buggy behavior is the main bottleneck.

W2 & Q2. Selection of preservation tests.

Each preservation test covers a nearby feature that a fix for the target bug is likely to break. For example, in the Apache Answer task, accepting an answer fails when the question is opened through a short-ID link. The preservation test opens the same question through its usual long-ID link and checks that accepting the answer still works.

Agents draft each test. One annotator manually uses both the buggy version and the reference version of the application, following the test’s workflow, and revises the test if it fails to reach the expected final outcome. A second annotator then reviews it. We also check that each test fails on patches that deliberately break the preserved feature.

We hope these clarifications address your concerns. Thank you again for your feedback.

Thank you for the response. The preservation test explanation addresses my concern. However, GPT-5.6 Luna and Claude Haiku 5.5 have the lowest agreement among the three pairs (88.3%), and they differ substantially on “Identified target feature” (75.4% vs. 64.8%). Please discuss this judge dependence in the paper. The main conclusion holds under both judges, so I will maintain my positive score.

Reviewer_uuV4OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Existing issue-resolution benchmarks give the agent a bug report written by someone who already found the bug. This paper evaluates whether agents can find the bug on their own. SWEeper-Bench contains 200 tasks built from real bug-fix PRs, each from a different open-source web application. The agent receives the pre-fix code, a running instance with a browser, and a short prompt that names only a product area (e.g., "find and fix all issues in Focalboard's board sharing and membership"). Fixes are graded by an LLM browser agent that follows two step-by-step behavior tests per task, one for the bug and one for preserving existing behavior. The best of 15 agents passes 59% of tasks. Five of the agents, which pass 35-51.5% of tasks with the open-ended prompt, reach about 96% when given a detailed bug description. A trajectory analysis suggests that identifying the buggy behavior is the main bottleneck.

Critical strengths
  • The problem setting is well motivated. Asking an agent to check a feature area and fix whatever is broken is a realistic use case, and current issue-driven benchmarks do not test it.
  • The comparison between open-ended and detailed prompts (Table 4) is convincing. The same agents go from 35-51% to about 96% once the bug is described, which indicates that the tasks and environments are sound and that the difficulty comes from discovery.
  • Task construction is careful. Each bug is reproduced manually, the verifier is cheap ($0.16 per task) and consistent across three configurations (Table 1), and the environments and tests will be released.
Critical weaknesses
  • The verifier is only validated on the buggy version and the reference fix, i.e., the two states the behavior tests were written against. Agent patches can look quite different from the reference (Luna's median diff is 49 lines in the open-ended setting vs. 13 with the detailed prompt), and some tests are very specific about the expected UI (Table 6). The claim that any correct implementation passes (L103-104) is therefore not tested. Since the leaderboard depends entirely on the verifier, I see this as the main issue.
  • Overly broad patches are hardly penalized. Each task has a single preservation test, agents pass it 97.5-100% of the time (Table 2), and there is no evidence that it catches regressions, e.g., on deliberately broken patches. The paper acknowledges that unnecessary changes go unpenalized (L483-484), but the benchmark offers no way to measure them.
  • The evidence that browser interaction is necessary is mixed. Without the browser instruction, GLM 5.3 drops from 43.5% to 31.5%, but Luna uses the browser on only 2/200 tasks and still reaches 39.5%, vs. 42.0% with it (Table 3). For some models, many of the bugs can apparently be found from the code alone, and two models are too few to tell which pattern is typical.
  • The stage analysis (Fig. 7) uses GPT-5.6 Luna as the judge without any human validation. Since the stages are cumulative, 41.0% of trajectories reach the target-pass stage (L323), while the mean target pass rate in Table 2 is about 48%. So roughly one in seven trajectories that pass the target test are judged not to have identified or diagnosed the bug. Either the judge misses these cases or some passes do not come from discovery, and the paper does not say which.
  • Each agent is run once and no confidence intervals are reported. With 200 tasks the 95% interval is roughly ±7 points, so the top five agents (56.5 to 59.0) cannot be distinguished.
  • Minor: Luna/Codex is 42.0 in Table 2 but 42.5 in Table 5, and 62.0% is not a possible pass rate on 40 tasks (Sec. 4.4). Sec. 2.1 does not describe how candidate PRs were collected and filtered, or who wrote the prompts and behavior tests.
Questions for authors
  1. Could you have humans check a sample of agent patches and report the verifier's false-pass and false-fail rates? This would address my main concern.
  2. How were the 200 tasks selected, and who wrote the area prompts and behavior tests? Since all reference fixes are public, was contamination considered?
  3. About 7 points of target passes (41.0% at L323 vs. ~48% in Table 2) are judged as not having identified or diagnosed the bug. Are these judge errors, or fixes made without finding the bug? Was the stage judge checked against human labels?
  4. The detailed prompt describes the cause of the bug rather than its symptom. How do agents perform with a symptom-only description?
3 comments
AuthorsOpenReview ↗

Response to Reviewer uuV4 (1/3)

Thank you for recognizing that SWEeper-Bench targets a realistic, well-motivated setting that current issue-driven benchmarks do not test. We address your remaining concerns below.

W1 & Q1. Verifier reliability.

The verifier’s pass/fail condition checks only the minimal final state that a correct fix must produce. For example, in the Focalboard task, the verifier fails a patch only if the second user, kadeRue, is signed in, the Boards category in the left sidebar is visible, and the shared board Q3 Launch Desk is missing from it. It passes the patch only if that same sidebar category lists Q3 Launch Desk.

The intermediate steps of each workflow include specific UI descriptions that help the verifier agent navigate the app reliably. These are hints, not pass/fail conditions: we give the verifier agent the higher-level objective of reaching the final state, and it is free to navigate without following them.

We acknowledge that a weak browser agent might not reliably navigate an app whose UI a patch has changed significantly, since the hints would no longer apply. We therefore audited the agents’ edits and found no significant UI changes. We also conducted an additional human annotator agreement experiment, where we had human annotators judge 50 randomly selected trajectories, and they agreed with the verifier on all 50.

W2. Destructive changes.

Preservation tests catch destructive changes: they fail on deliberately broken patches, and they caught 10 real regressions in our evaluation. When building each task, we generated patches that deliberately break the preserved feature and checked that the preservation test fails on every one.

In the main evaluation, 10 runs across 8 tasks and 7 agents fixed the target bug but broke a nearby feature: they passed the target test and failed the preservation test. We show examples below.

CaseModelTarget featureWhat the agent brokeHow the agent broke it
PluckClaude Opus 5, GPT-5.6 TerraSaving the website module layoutAdmin loginEdited the same-origin check. Login relies on the same check, so valid logins no longer reach the admin panel.
OpenDocManGPT-5.6 Luna, Kimi K3Checking in a new document versionUploading a documentEdited CSRF token generation for all forms. The upload form’s token no longer passes validation, so uploads are silently rejected.
Stirling-PDFGPT-5.6 LunaSaving custom PDF metadataSaving a PDF’s title and authorRewrote the metadata handler. The new handler drops the standard fields, so title and author edits are not saved.
HomarrMuse Spark 1.3Rejecting duplicate board namesCreating a boardEdited board-name validation and the create dialog. Valid new names are now rejected.
FenrusClaude Opus 5Creating a system groupCreating an ordinary groupEdited the Add action shared by the system and ordinary group pages. Saving an ordinary group now raises an application error.

W3. Browser instruction.

We conducted an additional experiment beyond Table 3 and found that the browser instruction raises pass rates for all five models we tested. Table 3 reports only GPT-5.6 Luna and GLM 5.3, so we added three more models and ran all five with and without the instruction on a 40-task subset:

ModelWithout browser instructionWith browser instructionChange (points)
GPT-5.6 Luna37.5%42.5%+5.0
GLM 5.327.5%45.0%+17.5
Kimi K345.0%55.0%+10.0
Claude Opus 542.5%57.5%+15.0
Qwen 3.8 Flash45.0%50.0%+5.0

Stronger models benefit more from the browser: Claude Opus 5 and Kimi K3, the two strongest models here, gain 15.0 and 10.0 points. GPT-5.6 Luna gains only 5.0 points, but not because it can find these bugs from code alone. Even with the browser, it is weak at discovering problems, so the browser helps it less. These results show that browser interaction is important for finding the bugs in SWEeper-Bench.

W4 & Q3. Difference between the stage analysis and Table 2.

The difference comes from excluding DeepSeek V4 Pro. The stage analysis in Fig. 7 excludes it because its trajectories were not saved, so Fig. 7 covers 14 agents while Table 2 covers all 15. We will make this clear in the revision.

AuthorsOpenReview ↗

Response to Reviewer uuV4 (2/3)

W4 & Q3. Validation of the stage judge.

We conducted two additional experiments to validate the stage judge and found that both a second judge and human annotators agree closely with it (88–93%). We re-ran the stage analysis with Claude Haiku 5.5 (high reasoning) as a second judge, and had human annotators label 20 randomly selected trajectories.

The first table shows how often each pair of labelers assigns the same stage labels.

HumanGPT-5.6 LunaClaude Haiku 5.5
Human—91.7%93.3%
GPT-5.6 Luna91.7%—88.3%
Claude Haiku 5.593.3%88.3%—

The second table shows the share of trajectories that reach each stage under each judge.

StageGPT-5.6 LunaClaude Haiku 5.5
Identified target feature75.4%64.8%
Identified buggy behavior50.0%47.1%
Diagnosed root cause43.8%42.5%
Fixed buggy behavior41.0%40.7%
Maintained existing behaviors40.8%40.5%

Both judges support the paper’s conclusion that identifying the buggy behavior is the main bottleneck.

W5. Variance across repeated runs.

We conducted additional experiments with repeated runs and found that different runs within a model differ by only a few points, with no statistically significant difference between them. We ran GPT-5.6 Luna, GLM 5.3 Flash, and Kimi K3.1 Pro three times each, and Qwen 3.8 Flash twice.

ModelRun 1Run 2Run 3Mean95% CI across runs
GPT-5.6 Luna42.0%39.0%39.0%40.0%40.0% ± 4.3 (35.7–44.3%)
GLM 5.3 Flash39.5%37.0%42.0%39.5%39.5% ± 6.2 (33.3–45.7%)
Kimi K3.1 Pro50.0%48.5%46.5%48.3%48.3% ± 4.4 (44.0–52.7%)
Qwen 3.8 Flash57.5%56.0%—56.8%56.8% ± 9.5 (47.2–66.3%)

We compared runs of the same model with exact McNemar tests, plus Cochran’s Q tests for models with three runs. All p-values are above 0.05, and the repeated pass rates are in line with the original results.

ModelRun 1 vs. 2Run 2 vs. 3Run 3 vs. 1Cochran’s Q
GPT-5.6 Luna0.311.000.380.45
GLM 5.3 Flash0.490.110.470.21
Kimi K3.1 Pro0.740.610.280.48
Qwen 3.8 Flash0.72———

W5. Significance of model differences.

All model pairs differ significantly except GPT-5.6 Luna and GLM 5.3 Flash. Two-sided permutation tests give p < 0.001 for every other pair and p = 0.87 for that one. Qwen 3.8 Flash beats Kimi K3.1 Pro, and both beat Luna and GLM 5.3 Flash, matching the leaderboard.

W6. Luna’s pass rate in Tables 2 and 5.

Thank you for catching this. The two numbers come from different task sets: Table 2 reports 42.0% on all 200 tasks, while Table 5 reports 42.5% on a 40-task subset. We will make this clear in the revision.

W6. Time-budget pass rates in Section 4.4.

The 62.0% in Section 4.4 is a typo; the correct peak is 60.0%. GLM 5.3 passes 37.5%, 60.0%, 50.0%, and 55.0% of tasks at 20, 40, 60, and 80 minutes. We will fix the text and figure.

W6 & Q2. Task construction.

Agents nominated candidate PRs and drafted the prompts and tests, while human annotators reproduced every bug and revised and reviewed every prompt and behavior test.

  • How were candidate PRs collected? Agents searched several thousand PRs following human-written guidelines and nominated about 500 candidates.
  • How were they filtered? Annotators deployed each candidate and reproduced its bug. After five rounds of screening and verification, about 45% remained: bugs that appear only through specific interactions, are hard to spot by reading code, and seriously disrupt normal use. Each PR also had to violate a clear product rule rather than a debatable design choice, make one logical fix, be verifiable in the browser with behavior tests alone, and run in a single self-contained container. A candidate was dropped if its environment or behavior tests still failed validation after five revision attempts.
  • How were the 200 tasks selected? From the remaining candidates, we removed redundant bug types and kept one task per application for diversity.
  • Who wrote the area prompts and behavior tests? Agents drafted them. One annotator manually used both the buggy version and the reference version of the application, following each test’s workflow, and revised the test if it failed to reproduce the bug on the buggy version or to reach the expected final outcome on the reference version. A second annotator then reviewed each prompt and test.

We will add these details to Sec. 2.1.

AuthorsOpenReview ↗

Response to Reviewer uuV4 (3/3)

Q2. Data contamination.

Our prompts do not identify the target bug, and agents run without internet access, so they cannot look up the original fix. Each prompt names only a feature area and a broad theme; by rule, prompts contain no reproduction steps, file paths, function names, or PR or issue numbers.

Q4. Symptom-only prompts.

We conducted an additional experiment with symptom-only prompts: pass rates rise from 42.5–57.5% to 80–85%, supporting the paper’s finding that discovering the bug is the hardest part. This setting sits between the open-ended and detailed prompts. For example: “Find and fix issues related to file upload persistence in AbanteCart account edit file upload.” Results on a 40-task subset:

ModelOpen-endedSymptom-onlyDetailed
GPT-5.6 Luna42.5%82.5%95.0%
Claude Opus 557.5%85.0%95.0%
GLM 5.345.0%82.5%95.0%
Qwen 3.8 Flash50.0%80.0%95.0%

With detailed prompts, all four models reach 95.0%.

We hope these clarifications address your concerns. Thank you again for your feedback.

Reviewer_i1srOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a new benchmark called SWEeper-Bench, to help assess if coding agents can resolve various bugs in interactive web apps without explicit bugs provided. In their benchmark there are 200 real-world bugs from 200 open source apps. Coding agents are often given a codebase and asked to look into a piece of functionality within it to find the bugs there and propose remedies.

The authors provide a browser-verifier to assess if patches correctly modify any expected / preserved functionalities. They test 15 frontier coding agents and present that the best agent gets a 59% success rate. They also run a set of ablation studies in the prompts used and analyse the coding agents' trajectories to find that one key failure during this process is identifying the faulty behaviour. Consistently providing a detailed description of the bugs increases the success rate to 96%.

Critical strengths

S1. Useful/motivating evaluation scenario.

The shift from repairing reported bugs to identifying them is an interesting idea. As coding agents become more autonomous, we want them to identify and fix bugs, without relying on user bug reports.

Focalboard (figure 5) demonstrates this well. The fix is pretty small, however to replicate this issue the agents would have to navigate through two accounts relatively specifically.

S2. Important contribution to the problem/trajectories analysis.

Cataloging and replicating 200 real bugs from multiple applications requires considerable effort and this is a valuable resource for the community.

I also appreciate the trajectory analysis in Figures 6-9 which provides insight into where agents struggle, from reaching the relevant functionality to identifying incorrect behavior. The released trajectories could help researchers better understand what agents choose to test, how they interpret the results, and when they decide to stop exploring. I think this makes the work beneficial beyond the reported success rates.

Critical weaknesses

W1. What exactly do we consider a real bug?

For the Focalboard example in Figure 6, Claude Opus 5 reaches the relevant appscreen but concludes that the behavior is intentional, partly based on the existing unit tests. The paper considers this a failure to identify buggy behavior. I'm not sure if that's quite what happened. The agent observed the behavior but wasn't convinced that it was incorrect.

I feel this shows epistemic uncertainty about the intended spec. What evidence should an agent rely on to infer what the intended functionality of an app should be here, especially when the codebase and testcases point to there being no errors?

Could the authors distinguish between agents that never encounter the buggy behavior and those that encounter it but interpret it as intentional? I think this would offer a more informative picture of where agents actually struggle.

W2. We ask the agents to find/fix any bug but only reward it for a specific one?

In this work, against are given an open-ended task of finding and fixing the bug in a feature. However, then they are graded on whether they find and fix a specific bug.

For ex, in AbanteCart the paper notes that the agents try multiple different fixes (like phone/notifs etc.) but it misses the “target” ID photo upload feature. This trajectory gets the same graded score of an agent that no-ops, even though the two trajectories may reflect very different exploration abilities. While the authors acknowledge this limitation, I think the framing of the results should be made more clear under this constraint. Also would be nice to see how often agents fix other genuine bugs.

Questions for authors

Overall this is a really well thoughtout and timely project. Thanks! I just have a couple of questions:

Q1. How did the authors establish that the expected behavior could be inferred without knowing the reference fix? Were there cases where existing tests or documentation supported behavior that the ref fix later changed?

Q2. GPT-5.6 Luna achieves 39.5% success while rarely using the browser. How often did successful agents identify bugs through code inspection or other non-browser methods?

2 comments
AuthorsOpenReview ↗

Response to Reviewer i1sr (1/2)

Thank you for recognizing the shift from repairing reported bugs to discovering them as a well-motivated and timely direction. We address your remaining concerns below.

W1. What counts as a real bug.

Intended behavior comes from what other software components imply, not from tests or documentation. Other screens, UI elements, and related features show what a feature should do, and a bug is behavior that contradicts them. For every task, human annotators checked that elements on screen clearly imply the intended behavior.

In Focalboard, the sidebar lists the boards a user belongs to: the owner’s sidebar shows a shared board, and boards a user creates, duplicates, or imports appear there too. So when a second user joins a shared board as an editor and their sidebar reads “No boards inside,” that contradicts the rest of the app.

W1. Why the unit tests misled Opus.

Claude Opus 5 treated a narrowly scoped unit test as a full specification of expected behavior, so when the app behaved inconsistently in a way the test did not cover, it did not treat that as a bug. The AddMemberToBoard unit test only checks that a new membership is saved, with the database mocked out. In real use, joining a board must also put it in the member’s sidebar, which the test never looks at, so Opus concluded that the empty sidebar was intended.

W1. Agents that see the bug but do not fix it.

We provide a rough taxonomy of runs where the agent saw the buggy behavior but did not go on to fix it:

Where the agent fell shortShareExample
Failed to identify the behavior as inconsistent with other elements on screen40%Speedtest Tracker (network monitor): the settings page accepted an invalid notification URL with no error. Agents saw it save and moved on to other bugs.
Treated incomplete tests or code as a full specification27%Whoogle (search engine): a search ending in “w!” jumped to Wikipedia instead of showing results. Agents kept it because an existing test used the same query.
Wrongly blamed the failure on its own test script or development environment15%TastyIgniter (restaurant app): deleting a reservation crashed with a server error, and the reservation stayed. Agents assumed their own test script caused it.
Ran its own test, which showed the failure, then reported the feature as working13%phpBB (forum): the post-approval page had no Approve button. The agent looked at the page and still reported it as working.
Recognized the bug, then dropped it on weak counter-evidence5%Newsletter signup: resubscribing showed success but was not saved. The agent flagged this, then decided it had checked the wrong record.

Q1. Inferring expected behavior without the reference fix.

The expected behavior can be determined from consistency with other elements on screen, without the reference fix.

  • How was this checked? When building tasks, annotators verified that each reference fix restores behavior that other components of the software already imply. We removed PRs that add new features or change how an existing feature is meant to work, and PRs whose old behavior could reasonably be read as a product design choice.
  • Did documentation support the buggy behavior? No. Because of this filter, the expected behavior in every task follows from other components of the software, not from tests or documentation.
  • Did existing tests support it? In some tasks, yes: some existing unit tests are narrowly scoped or encode the buggy behavior, and they mislead agents. In Focalboard, the AddMemberToBoard test checks only that a membership is saved and never looks at the sidebar. In Whoogle, an existing test uses the same “w!” query that wrongly jumps to Wikipedia. Treating such tests as a full specification accounts for 27% of the runs in the taxonomy above.

W2. Scoring only the target bug.

We score one target bug per task, chosen because it breaks functionality that is critical to the software, so any complete sweep of the area should fix it. We will clarify in the revision that the benchmark measures recall: agents may also make other valid fixes that we do not evaluate.

Even so, recall is informative across many tasks. An agent that is poor at discovering bugs is unlikely to find and fix the target bug in task after task by luck, so the pass rate over 200 tasks reflects real discovery ability.

AuthorsOpenReview ↗

Response to Reviewer i1sr (2/2)

Q2. The role of browser interaction.

Agents rarely use the browser unless explicitly prompted to. We conducted an additional experiment and found that prompting agents to use browsers raises pass rates for all five models we tested. In addition to Table 3, which reports GPT-5.6 Luna and GLM 5.3 with and without the browser instruction on all 200 tasks, we ran five models with and without the instruction on a 40-task subset:

ModelWithout browser instructionWith browser instructionChange (points)
GPT-5.6 Luna37.5%42.5%+5.0
GLM 5.327.5%45.0%+17.5
Kimi K345.0%55.0%+10.0
Claude Opus 542.5%57.5%+15.0
Qwen 3.8 Flash45.0%50.0%+5.0

The browser instruction raises pass rates by 5.0 to 17.5 points, showing that browser interaction is important for success: it helps agents find user-facing bugs that reading code alone can miss.

Thank you again for your positive assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.