- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces a benchmark for discovering and repairing bugs without issue reports. It constructs 1663 tasks from historical pull requests across 87 repositories and 8 language groups, including 252 tasks combining multiple recorded bugs. The main evaluation uses a human-curated 400-task subset. Agents receive a repository snapshot, a bug taxonomy, and a list of files to review. Evaluation combines historical regression tests and patch-localization metrics with a second track that asks agents to generate tests for their proposed fixes and uses an LLM judge to assess the claimed bugs.
Critical strengths
-
The task is practical and challenging. Existing benchmarks evaluates agents with known bugs and (detailed) problem statements. The paper focuses on a more practical scenario: can agents identify and fix bugs by themselves.
-
The multilingual construction and distinction between single-PR and combined-PR tasks provide meaningful breadth. Checking the integrated fail-to-pass tests is a sensible safeguard when combining historical fixes.
Critical weaknesses
-
The main task supplies substantial localisation information, limiting the central claim. The authors should explicitly define the benchmark as reference-file-scoped discovery, or support the broader claim with a realistic file-selection policy evaluated under a comparable total budget. The iterative local-review study does not specify enough budget information to establish the latter.
-
The potential-bug track needs stronger validation of semantic correctness. A test that fails before and passes after an agent's own patch establishes a behavioral change, but does not by itself establish that the new behavior is correct. The current evidence supports promising candidate findings more strongly than it supports a calibrated estimate of valid new-bug discovery.
Questions for authors
- How were the 400 tasks and the subsets for Tables 11-13 selected, and what are the corresponding sample sizes and run counts?
- What are the explicit zero-bug/zero-test scoring rules, and how are unsupported claimed fixes retained or discarded when the judge forms its bug set?
- What is the instance-level human-judge agreement, and how many allegedly additional bugs are independently supported by documented intended behavior?
- How do validated-bug yield and regression failures change under non-oracle file selection and a matched total execution budget?
7 comments
Q2: The potential-bug track needs stronger validation of semantic correctness. A test that fails before and passes after an agent's own patch establishes a behavioral change, but does not by itself establish that the new behavior is correct. The current evidence supports promising candidate findings more strongly than it supports a calibrated estimate of valid new-bug discovery.
A2: Thanks for your comments. We acknowledge that our potential-bug evaluation provides a feasible way rather than perfect evaluation paradigm. Nevertheless, such a paradigm could be one of the first to validate potential bug discovery, and we explain its rationale from both methodological and empirical perspectives.
Methodological perspective. Beyond checking the fail-to-pass behavior of generated tests, we employ LLM-based judge agent to assess the validity of the potential bugs and the corresponding metric is ''Revealed'', as described in Section 3.4.2. Specifically, the Judge Task Template in Appendix E requires the judge to identify real functional bugs and exclude changes without meaningful functional impact. Therefore, our evaluation considers whether the patch fixes a valid bug and whether the tests provide meaningful evidence, rather than treating fail-to-pass behavior alone as sufficient. For your convenience, we reproduce the key definitions below.
Effective Bug: A real functional bug that is fixed by the predicted code patch. Cosmetic edits, formatting changes, pure refactoring, dead-code changes, test-only changes, and undocumented behavior changes are NOT effective bug fixes.
Effective Test: A test that reproduces an effective bug. Tests are NOT effective if they only assert patched outputs, duplicate implementation logic, or check behavior unrelated to the bug.
Empirical perspective. To further address your concerns, we additionally ask seven experts to carefully audit the predicted fixes and generated tests on a subset of 100 tasks, including 75 simple tasks and 25 hard tasks. For instances with Revealed = 1, we report the verification results below.
Table B: Expert verification of potential bugs.
| Verification criterion | Proportion |
|---|---|
| Generated tests exhibit fail-to-pass behavior | 100% |
| Generated tests and corresponding bug fixes are confirmed as valid by experts | 99.0% |
The results show that the judge agent achieves high agreement with experts on individual instances, supporting the reliability of our potential-bug evaluation. The remaining disagreement arises when assessing whether a fix requires domain-specific knowledge, such as determining whether an unusual boundary behavior is intentional.
Q3: How were the 400 tasks and the subsets for Tables 11-13 selected, and what are the corresponding sample sizes and run counts?
A3: Thanks for your comments. Due to the high evaluation cost, we selected a subset of 100 tasks, including 75 simple tasks and 25 hard tasks, preserving the 3:1 ratio of simple to hard tasks in the 400-task benchmark. All evaluation and analytical experiments in this paper were run three times to assess the stability of the results. We will clarify these details in the revised manuscript.
Q4: What are the explicit zero-bug/zero-test scoring rules, and how are unsupported claimed fixes retained or discarded when the judge forms its bug set?
A4: Thanks for your comments. We will answer your questions one by one.
Zero-bug/zero-test scoring. We have carefully designed metrics for potential bug evaluation with explicit scoring rules for zero-bug and zero-test cases. For your convenience, we summarize their motivations and definitions below.
Table C: Metrics for potential-bug evaluation.
| Metric | Motivation | Definition |
|---|---|---|
| Count | Measure the quantity of test evidence | Number of generated tests for potential bugs |
| TV | Check whether tests reproduce behavioral changes | Whether all generated tests exhibit fail-to-pass behavior |
| Revealed | Check whether bug fixes are valid and meaningful | Whether every valid bug fix is supported by valid tests |
Specifically, each task contains at least one recorded bug, which provides a buggy repository for both recorded and potential bug evaluation. For recorded bug evaluation, the predicted patch is evaluated against the reference tests as usual. For potential bug evaluation, if no effective bug fixes are identified, Revealed is set to 0, while Count and TV are computed from the generated tests. If no tests are generated, Count, TV, and Revealed are all set to 0. Clearly, each metric provides meaningful information for scoring zero-bug and zero-test cases.
Details of unsupported bugs. For a given instance, the judge agent may identify claimed bugs that lack valid test support or do not qualify as effective bugs. In such cases, we retain these bugs but assign ''Revealed'' metric as 0, indicating that not all claimed bug fixes are valid or supported by sufficient test evidence.
Thanks for your insightful comments and suggestions. We will answer your questions one by one in the following.
A1: Thanks for your comments. We acknowledge your concern, but we believe that our setting is a practical first step toward proactive bug fixing and could be easily extended to more applications. As discussed in Appendix C, it is nearly impossible for existing coding agents to discover and resolve a specific recorded bug for a given repository. Motivated by this, we consider reviewing code within a specified scope is the step toward proactive bug fixing. Based on the task formulation, both the number of files to review and the extent of the review scope can be flexibly adjusted, demonstrating its extensibility.
To further address your concerns, we conduct additional experiments on the two repositories discussed in Appendix C. Specifically, we compared three file-selection policies: i) Continued global review, which gives the coding agent additional resources, e.g., more turns and execution times, to continue inspecting the repository and fixing bugs; ii) Directory-order review, which sorts source files by directory and path and then reviews them in successive batches; iii) Agent-order review, which allows the coding agent to determine the order in which files are inspected. For directory-order and agent-order review, we inspected 2, 4, or 8 files per batch. Note that the tokens consumed by the agent during file selection are also included in token usage. ''All F2P Pass'' and ''All P2P Pass'' denote whether all F2P and all P2P tests pass, respectively, measuring recorded-bug repair and preservation of existing functionality.
Table A: Comparison of review policies under various total token budgets.
Based on our designs, both directory-order and agent-order review help identify and resolve bugs across the entire repository, even with similar or lower token usage than continued global review. Although global review serves as a single task, it repeatedly consumes tokens while attempting to complete the repository-wide task, resulting in low efficiency. In contrast, each local review consumes relatively few tokens, with the cumulative consumption comparable to continued global review. Such results further support the rationale of proactive bug fixing within specified scopes. Besides, we observe few regression failures, suggesting that current LLMs generally preserve existing functionality when fixing bugs.