acceptodds
Under review as a conference paper at ICLR 2027

Active-SWE: Benchmarking Coding Agents for Proactive Bug Fixing without Issue Reports

Abstract

Coding agents powered by large language models (LLMs) are increasingly adopted in software engineering (SWE) scenarios, capable of fixing a specific bug in large-scale codebase. However, existing SWE benchmarks typically assume that high-quality issue reports with detailed information are always available, which is easily violated in practice due to the complexity of report acquisition and curation. To address this, we introduce Active-SWE, a benchmark for evaluating coding agents on proactively discovering and fixing multiple bugs without report guidance, covering 1,663 tasks across six bug categories and eight languages. Beyond shifting the focus from existing reactive bug fixing to proactive bug fixing, Active-SWE enables a more in-depth evaluation by expanding the scope from fixing a specific recorded bug to multiple-bug fixing and potential bug discovery scenarios. To construct Active-SWE, we propose a novel difficulty-aware task formulation pipeline with a dual-track evaluation framework, facilitating comprehensive evaluation of proactive bug fixing capability. Extensive experiments reveal that most state-of-the-art coding agents struggle with proactive bug fixing tasks, demonstrating limited performance in locating and resolving recorded bugs, handling multiple bug fixing scenarios, and discovering valid potential bugs.

Primary area:
datasets and benchmarks
Keywords:
Coding Agents, Large Language Model, Proactive Bug Fixing, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JBcBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a benchmark for discovering and repairing bugs without issue reports. It constructs 1663 tasks from historical pull requests across 87 repositories and 8 language groups, including 252 tasks combining multiple recorded bugs. The main evaluation uses a human-curated 400-task subset. Agents receive a repository snapshot, a bug taxonomy, and a list of files to review. Evaluation combines historical regression tests and patch-localization metrics with a second track that asks agents to generate tests for their proposed fixes and uses an LLM judge to assess the claimed bugs.

Critical strengths
  1. The task is practical and challenging. Existing benchmarks evaluates agents with known bugs and (detailed) problem statements. The paper focuses on a more practical scenario: can agents identify and fix bugs by themselves.

  2. The multilingual construction and distinction between single-PR and combined-PR tasks provide meaningful breadth. Checking the integrated fail-to-pass tests is a sensible safeguard when combining historical fixes.

Critical weaknesses
  1. The main task supplies substantial localisation information, limiting the central claim. The authors should explicitly define the benchmark as reference-file-scoped discovery, or support the broader claim with a realistic file-selection policy evaluated under a comparable total budget. The iterative local-review study does not specify enough budget information to establish the latter.

  2. The potential-bug track needs stronger validation of semantic correctness. A test that fails before and passes after an agent's own patch establishes a behavioral change, but does not by itself establish that the new behavior is correct. The current evidence supports promising candidate findings more strongly than it supports a calibrated estimate of valid new-bug discovery.

Questions for authors
  1. How were the 400 tasks and the subsets for Tables 11-13 selected, and what are the corresponding sample sizes and run counts?
  2. What are the explicit zero-bug/zero-test scoring rules, and how are unsupported claimed fixes retained or discarded when the judge forms its bug set?
  3. What is the instance-level human-judge agreement, and how many allegedly additional bugs are independently supported by documented intended behavior?
  4. How do validated-bug yield and regression failures change under non-oracle file selection and a matched total execution budget?
7 comments

Thanks for your insightful comments and suggestions. We will answer your questions one by one in the following.

Q1: The main task supplies substantial localisation information, limiting the central claim. The authors should explicitly define the benchmark as reference-file-scoped discovery, or support the broader claim with a realistic file-selection policy evaluated under a comparable total budget. The iterative local-review study does not specify enough budget information to establish the latter. How do validated-bug yield and regression failures change under non-oracle file selection and a matched total execution budget?

A1: Thanks for your comments. We acknowledge your concern, but we believe that our setting is a practical first step toward proactive bug fixing and could be easily extended to more applications. As discussed in Appendix C, it is nearly impossible for existing coding agents to discover and resolve a specific recorded bug for a given repository. Motivated by this, we consider reviewing code within a specified scope is the step toward proactive bug fixing. Based on the task formulation, both the number of files to review and the extent of the review scope can be flexibly adjusted, demonstrating its extensibility.

To further address your concerns, we conduct additional experiments on the two repositories discussed in Appendix C. Specifically, we compared three file-selection policies: i) Continued global review, which gives the coding agent additional resources, e.g., more turns and execution times, to continue inspecting the repository and fixing bugs; ii) Directory-order review, which sorts source files by directory and path and then reviews them in successive batches; iii) Agent-order review, which allows the coding agent to determine the order in which files are inspected. For directory-order and agent-order review, we inspected 2, 4, or 8 files per batch. Note that the tokens consumed by the agent during file selection are also included in token usage. ''All F2P Pass'' and ''All P2P Pass'' denote whether all F2P and all P2P tests pass, respectively, measuring recorded-bug repair and preservation of existing functionality.

Table A: Comparison of review policies under various total token budgets.

Review policyFiles per batchToken UsageLRLPResolvedAll F2P Pass (%)All P2P Pass (%)
Continue Global reviewAll20.8M1.95.72.52.595
Directory-order review241.4M7.628.515.015.0100
Directory-order review422.5M7.229.415.015.0100
Directory-order review811.4M6.525.310.010.0100
Agent-order review235.7M13.538.425.025.0100
Agent-order review419.0M12.938.920.020.0100
Agent-order review810.8M11.833.215.015.0100

Based on our designs, both directory-order and agent-order review help identify and resolve bugs across the entire repository, even with similar or lower token usage than continued global review. Although global review serves as a single task, it repeatedly consumes tokens while attempting to complete the repository-wide task, resulting in low efficiency. In contrast, each local review consumes relatively few tokens, with the cumulative consumption comparable to continued global review. Such results further support the rationale of proactive bug fixing within specified scopes. Besides, we observe few regression failures, suggesting that current LLMs generally preserve existing functionality when fixing bugs.

Q2: The potential-bug track needs stronger validation of semantic correctness. A test that fails before and passes after an agent's own patch establishes a behavioral change, but does not by itself establish that the new behavior is correct. The current evidence supports promising candidate findings more strongly than it supports a calibrated estimate of valid new-bug discovery.

A2: Thanks for your comments. We acknowledge that our potential-bug evaluation provides a feasible way rather than perfect evaluation paradigm. Nevertheless, such a paradigm could be one of the first to validate potential bug discovery, and we explain its rationale from both methodological and empirical perspectives.

Methodological perspective. Beyond checking the fail-to-pass behavior of generated tests, we employ LLM-based judge agent to assess the validity of the potential bugs and the corresponding metric is ''Revealed'', as described in Section 3.4.2. Specifically, the Judge Task Template in Appendix E requires the judge to identify real functional bugs and exclude changes without meaningful functional impact. Therefore, our evaluation considers whether the patch fixes a valid bug and whether the tests provide meaningful evidence, rather than treating fail-to-pass behavior alone as sufficient. For your convenience, we reproduce the key definitions below.

Effective Bug: A real functional bug that is fixed by the predicted code patch. Cosmetic edits, formatting changes, pure refactoring, dead-code changes, test-only changes, and undocumented behavior changes are NOT effective bug fixes.

Effective Test: A test that reproduces an effective bug. Tests are NOT effective if they only assert patched outputs, duplicate implementation logic, or check behavior unrelated to the bug.

Empirical perspective. To further address your concerns, we additionally ask seven experts to carefully audit the predicted fixes and generated tests on a subset of 100 tasks, including 75 simple tasks and 25 hard tasks. For instances with Revealed = 1, we report the verification results below.

Table B: Expert verification of potential bugs.

Verification criterionProportion
Generated tests exhibit fail-to-pass behavior100%
Generated tests and corresponding bug fixes are confirmed as valid by experts99.0%

The results show that the judge agent achieves high agreement with experts on individual instances, supporting the reliability of our potential-bug evaluation. The remaining disagreement arises when assessing whether a fix requires domain-specific knowledge, such as determining whether an unusual boundary behavior is intentional.

Q3: How were the 400 tasks and the subsets for Tables 11-13 selected, and what are the corresponding sample sizes and run counts?

A3: Thanks for your comments. Due to the high evaluation cost, we selected a subset of 100 tasks, including 75 simple tasks and 25 hard tasks, preserving the 3:1 ratio of simple to hard tasks in the 400-task benchmark. All evaluation and analytical experiments in this paper were run three times to assess the stability of the results. We will clarify these details in the revised manuscript.

Q4: What are the explicit zero-bug/zero-test scoring rules, and how are unsupported claimed fixes retained or discarded when the judge forms its bug set?

A4: Thanks for your comments. We will answer your questions one by one.

Zero-bug/zero-test scoring. We have carefully designed metrics for potential bug evaluation with explicit scoring rules for zero-bug and zero-test cases. For your convenience, we summarize their motivations and definitions below.

Table C: Metrics for potential-bug evaluation.

MetricMotivationDefinition
CountMeasure the quantity of test evidenceNumber of generated tests for potential bugs
TVCheck whether tests reproduce behavioral changesWhether all generated tests exhibit fail-to-pass behavior
RevealedCheck whether bug fixes are valid and meaningfulWhether every valid bug fix is supported by valid tests

Specifically, each task contains at least one recorded bug, which provides a buggy repository for both recorded and potential bug evaluation. For recorded bug evaluation, the predicted patch is evaluated against the reference tests as usual. For potential bug evaluation, if no effective bug fixes are identified, Revealed is set to 0, while Count and TV are computed from the generated tests. If no tests are generated, Count, TV, and Revealed are all set to 0. Clearly, each metric provides meaningful information for scoring zero-bug and zero-test cases.

Details of unsupported bugs. For a given instance, the judge agent may identify claimed bugs that lack valid test support or do not qualify as effective bugs. In such cases, we retain these bugs but assign ''Revealed'' metric as 0, indicating that not all claimed bug fixes are valid or supported by sufficient test evidence.

Reviewer_FE6pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Active-SWE, a benchmark for evaluating coding agents on proactive bug fixing without explicit issue reports. The authors construct the benchmark from historical GitHub pull requests, and provide extensive experiments and analyses across different bug types, task difficulties, review scopes, and models.

Critical strengths

The benchmark is carefully constructed, providing a useful testbed for studying proactive software engineering agents

Critical weaknesses

The main contribution is primarily benchmark construction and empirical characterization, while the methodological novelty appears relatively limited.

  1. Since the benchmark is constructed from historical GitHub repositories, including the original buggy code, patches, tests, and pull requests, these data may have appeared in the training corpora of existing LLMs. This raises the possibility that model performance partially reflects memorization rather than proactive bug-fixing ability. The authors should provide a more thorough analysis or discussion of this potential contamination issue.

  2. The review scope is derived from the files modified by the reference patch, which guarantees that the target bug is contained in the provided scope. However, identifying which files or parts of a repository should be inspected is itself an important aspect of proactive bug discovery. This setting therefore seems easier than fully repository-level proactive bug fixing. The authors should clarify this limitation and discuss its impact on the benchmark's realism.

  3. The identification and validation of potential bugs rely substantially on LLM-generated tests and LLM-based judging. Although the authors evaluate multiple judges and include human evaluation, it would be useful to report their agreement more systematically and discuss possible biases introduced by LLM-based evaluation.

Questions for authors

How does Active-SWE fundamentally differ from existing benchmarks for proactive bug discovery, automated code review, and test generation, and what new research questions the benchmark enables.

4 comments

Thanks for your insightful comments and suggestions. We will answer your questions one by one in the following.

Q1: How does Active-SWE fundamentally differ from existing benchmarks for proactive bug discovery, automated code review, and test generation, and what new research questions the benchmark enables.

A1: Thanks for your comments. We will address your question from two perspectives: the most closely related work and other types of benchmarks.

The most closely related work to proactive bug fixing. Among previous works, TestExplora [A] might be the most closely related to proactive bug fixing, where LLMs are instructed to generate executable tests based on curated documents. However, Active-SWE exhibits significant differences against TestExplora in the following aspects:

i) Different tasks. TestExplora generates executable tests to expose bugs, whereas Active-SWE requires agents to proactively discover and fix bugs. In other words, beyond exposing bugs, coding agents in our paper must further generate correct repair patches, which poses higher demands on code exploration and implementation capabilities.

ii) Different guidance. TestExplora relies on designated test interfaces and detailed documents generated by DocAgent, which specify instance-specific expected behaviors, inputs, and outputs ([A], Section 3.2 and Appendix B). In contrast, Active-SWE requires coding agents to complete tasks without detailed instance-specific guidance.

iii) Different evaluation scopes. TestExplora focuses on whether generated tests could expose recorded bugs. In contrast, Active-SWE enables a more in-depth evaluation from two perspectives: i) discovering and resolving multiple bugs rather than a specific recorded bug; ii) revealing potential bugs beyond those recorded in historical fixes.

Other types of benchmarks. We argue that Active-SWE differs from related benchmarks for automated code review, test generation and so on, which is a novel and practical research question. In brief, we summarize the novelty in the following table.

Table A: Comparison of benchmark task formulations and evaluation scopes.

CapabilityAutomated code reviewTest generationActive-SWE
Identify Problems✓✓✓
Require to Resolve Identified Problems——✓
Without Detailed Guidance——✓
Identify Multiple Problems——✓
Validate Potential Problems——✓

In the following, we will elaborate on these differences.

Task Formulation. Automated code review evaluates proposed code changes and produces review comments, while test generation produces executable tests to expose defects. These tasks only focus on revealing problems, whereas Active-SWE requires agents to finally resolve the problems. On the one hand, proactive bug fixing is a more challenging task involving a two-stage process. On the other hand, code review and test generation tasks often provide instance-specific information, e.g., code diff, test interfaces and behavior descriptions, while Active-SWE focus on solving problems without instance-specific issue guidance.

Evaluation. Both automated code review and test generation typically focus on one predefined target, whereas Active-SWE requires coding agents to solve multiple targets. Besides, beyond the historically recorded problems, Active-SWE extends evaluation to potential problems.

Q2: Since the benchmark is constructed from historical GitHub repositories, including the original buggy code, patches, tests, and pull requests, these data may have appeared in the training corpora of existing LLMs. This raises the possibility that model performance partially reflects memorization rather than proactive bug-fixing ability. The authors should provide a more thorough analysis or discussion of this potential contamination issue.

A2: Thanks for your comments. We acknowledge that our benchmark might overlap with the training corpora of existing LLMs. We believe that this problem is a common concern for SWE benchmarks, i.e., nearly all of them are constructed from public GitHub repositories, rather than a specific problem for Active-SWE. Under this background, we argue that Active-SWE is less affected by potential contamination from both task-design and empirical perspectives.

Task design. For potential bugs beyond historical records, the patches and tests are not available in GitHub, so existing LLMs might have no corresponding memory of them.

For recorded bugs, we argue that the evaluation setting of Active-SWE differs substantially from issue-conditioned coding training, where LLMs are typically given issue reports as task prompts to generate solutions. In contrast, proactive bug fixing uses a unified task template instead of issue report, which fundamentally differs from the training paradigm. In other words, even for the same PR, the task prompts are quite different, thus leading to various problem-solving processes and output formats.

Empirical analysis. To address your concerns, we conduct additional experiments to evaluate whether LLMs could reconstruct issue reports. Specifically, we provide the first 30% of each report and ask the evaluated model to complete the remaining text without internet access. Intuitively, LLMs typically read the issue reports during training. As a result, successful continuations that are consistent with original reports are likely to reflect memorization, suggesting that the models may have seen these tasks before. In the implementation, we use GPT-5.5 to assess whether the continuation is consistent with the original report. And the experiment uses a subset of 100 tasks, including 75 simple tasks and 25 hard tasks. For your convenience, we provide the results in Table B.

Table B: Issue-report continuation results on a 100-task subset.

ModelContinuations judged consistent with the original
GLM-5.21%
Kimi-K2.7-Code2%

The results show that only a negligible proportion of continuations match the original reports, suggesting limited detectable memorization of these tasks by the evaluated LLMs.

Q3: The review scope is derived from the files modified by the reference patch, which guarantees that the target bug is contained in the provided scope. However, identifying which files or parts of a repository should be inspected is itself an important aspect of proactive bug discovery. This setting therefore seems easier than fully repository-level proactive bug fixing. The authors should clarify this limitation and discuss its impact on the benchmark's realism.

A3: Thanks for your comments. We have discussed this question in Table 7 (Appendix C). The results show that it is nearly impossible for existing coding agents to discover and resolve a specific recorded bug for a given repository. Motivated by this, we consider that a practical first step toward proactive bug fixing is to review code within a specified scope. Such a setting also has practical applications, such as reviewing recently modified files during software development.

More importantly, we do not consider our setting a limitation for repository-level proactive bug fixing, as the repository-wide task could be decomposed into multiple file-level review subtasks. To support this, we have conducted experiments on two stable and mature repositories, i.e., pflag and Click, each containing ten recorded bugs as verifiable evaluation targets. Specifically, we compared two settings using GLM-5.2: i) global review, which asks the agent to proactively fix bugs across the entire repository in a single run; ii) iterative local review, which asks the agent to review a few source files, e.g., two files, at a time until the entire repository is covered. For your convenience, we provide the corresponding results in Table 7.

Table 7: Comparison of global review and iterative local review on recorded bug fixing.

RepositorySettingLRLPResolved
pflagGlobal review0.00.00.0
pflagIterative local review10.623.820.0
ClickGlobal review0.00.00.0
ClickIterative local review4.733.310.0

From the results, simply performing global review across the entire repository barely identifies recorded bugs, while our design, i.e., iterative local review, substantially improves bug localization and resolved rates. Such results suggest that our setting could be easily extended to the scope of the whole repository and achieve better performance.

Q4: The identification and validation of potential bugs rely substantially on LLM-generated tests and LLM-based judging. Although the authors evaluate multiple judges and include human evaluation, it would be useful to report their agreement more systematically and discuss possible biases introduced by LLM-based evaluation.

A4: Thanks for your comments. In Appendix D.9, we have compared different LLM judges with human experts. To further address your concerns, we conducted an additional experiment to systematically assess their agreement. Specifically, we investigate the instance-level agreement of our default judge, Qwen3.5-397B, with those of GLM-5.2 and human experts. In particular, ''Agreement'' denotes the proportion of tasks on which both judges reach the same decision. ''Rejection Disagreement'' denotes the proportion accepted by Qwen3.5-397B but rejected by the compared judge, while ''Acceptance Disagreement'' denotes the reverse. Cohen’s κ measures agreement beyond chance, with higher values indicating stronger agreement.

For your convenience, we provide the corresponding results in Table C.

Table C: Instance-level agreement with the judge agent powered by Qwen3.5-397B.

JudgeAgreementCohen’s κRejection disagreementAcceptance disagreement
GLM-5.297.5%0.9410.8%1.8%
Human Expert98.5%0.9651.3%0.3%

The results show high agreement between our default judge and both GLM-5.2 and human experts, suggesting that biases introduced by LLM-based evaluation are limited in the evaluation.

[A] Steven Liu et al. TestExplora: Benchmarking LLMs for Proactive Bug Discovery via Repository-Level Test Generation. arXiv:2602.10471, 2026.

Dear Reviewer FE6p,

With one day remaining before the discussion deadline, we would like to know if our response has addressed your concerns. We have provided additional experiments and detailed clarifications to address your questions. If you have any further concerns or suggestions for the paper or our rebuttal, please let us know.

If your concerns have been satisfactorily addressed, we would be grateful if you would consider revising your score accordingly. Thank you again for the time and effort you dedicated to reviewing this work.

Reviewer_k2AVOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper we introduce Active-SWE, a benchmark for evaluating coding agents on proactively discovering and fixing multiple bugs without report guidance, covering 1,663 tasks across six bug categories and eight languages. In addition, the authors propose a novel difficulty-aware task formulation pipeline with a dual-track evaluation framework, facilitating com prehensive evaluation of proactive bug fixing capability.

Critical strengths
  1. The research question is novel.
  2. The design and evaluation of the benchmarking are quite comprehensive.
  3. The paper is well-structured and easy to read.
Critical weaknesses
  1. The research background is insufficiently explained, particularly its connection to existing literature.
  2. The research motivation lacks a solid justification.
  3. The data collection and processing procedures are unclear.
  4. The analysis of the experimental results is difficult to follow.
Questions for authors

Section 1:

  1. The reasoning leading to the conclusion that "issues" should be eliminated contains a logical leap. The introduction characterizes issues as "inherently post-hoc," "prohibitively expensive," and "often ambiguous and incomplete"—points that argue for the improvement of the issue-reporting process, but do not necessarily justify the complete abandonment of issues. Furthermore, the introduction fails to specify the real-world development scenarios—such as code reviews or continuous integration—in which a need for proactive defect discovery might arise despite the absence of formal issue reports.
  2. The introduction also fails to clarify how this work differs from existing research on proactive defect discovery. Although the paper cites TestExplora, a study that also explores proactive defect discovery in the absence of issue reports, the introduction does not explain what distinguishes Active-SWE from that approach. Section 3:
  3. The authors say “consisting of 6 major bug categories and an additional Out-of-Taxonomy category.” However, they do not explain how these 6 categories are defined, what the labeling process for these categories entails, or whether any consistency checks (such as inter-rater reliability assessments) were conducted.
  4. It is said that “we propose a consensus-based curation mechanism for filtering out PRs without discoverable bugs.” the annotators label “a given PR with issue reports” and therefore cannot tell whether a bug remains discoverable once the issue is removed; the section also omits the number of annotators, the voting rule, and the agreement among annotators. Section 4
  5. The origin of the 400 evaluation tasks is not explained. While the authors state that "human experts curate Active-SWE with 400 tasks," they provide no details regarding the selection criteria or the number of experts involved; furthermore, they offer no justification for why these 400 tasks are representative of the full set of 1,663 tasks. Although Figure 3 presents the category distribution for the entire dataset, the breakdown of how many tasks from each category are included in the 400-task subset is not provided. This weakens the representativeness and reproducibility of the benchmark.
  6. Table 3 only specifies "on a subset" without providing the size of that subset.
  7. In Figure 5, the "Resolved" values for the three models are 18.2%, 15.8%, and 10.4%, which are inconsistent with the values of 20.0, 18.5, and 13.0 shown in Table 2.
7 comments

Thanks for your insightful comments and suggestions. We will answer your questions one by one in the following.

Q1.1: The reasoning leading to the conclusion that "issues" should be eliminated contains a logical leap. The introduction characterizes issues as "inherently post-hoc," "prohibitively expensive," and "often ambiguous and incomplete"—points that argue for the improvement of the issue-reporting process, but do not necessarily justify the complete abandonment of issues.

A1.1: Thanks for your comments. We agree that our introduction did not make this step of the argument explicit, and we clarify it below.

We do not claim that issue reports should be abandoned. As you pointed out, improving issue reports helps resolve reported bugs, and we fully agree. However, improving issue reports presupposes that the bug has already been discovered and reported. Our work targets a different stage: bugs that exist in the repository but have not yet been noticed by anyone. For these bugs, there is no issue report to improve. This is what we intended by “post-hoc” in the introduction, i.e., an issue report only becomes available after the bug has been exposed.

Therefore, we regard proactive bug fixing as complementary to, rather than a replacement for, issue-based bug fixing: the former covers the stage before a bug is reported, and the latter covers the stage after. Our core question is “Can coding agents proactively discover and resolve bugs before they are reported?”, which existing issue-based benchmarks are not designed to measure.

We believe this setting is practical for two reasons: i) Unreported bugs can be costly once they escape into production, so discovering them earlier has clear value; ii) It matches existing development workflows in which developers know which code to inspect but not which bugs it contains, e.g., reviewing modified files before opening a pull request (see our response to Q1.2 and Appendix C).

Revision. To avoid this misunderstanding, we have polished the second paragraph in Introduction and the corresponding descriptions in Abstract. It now states two limitations of the issue-based setting: i) coverage, i.e., it only applies to bugs that have already been reported, leaving undiscovered bugs out of scope; ii) quality, i.e., even when reports exist, they may be incomplete or misleading. Correspondingly, we highlight that our work is complementary to existing issue-based benchmarks, rather than a replacement.

Q1.2: Furthermore, the introduction fails to specify the real-world development scenarios—such as code reviews or continuous integration—in which a need for proactive defect discovery might arise despite the absence of formal issue reports.

A1.2: Thanks for your suggestions. In Appendix C, we have discussed two practical scenarios, i.e., code revision before opening a pull request and repository-wide proactive maintenance. We further clarify the practical scenarios below.

Code revision before opening a pull request. Developers inspect recently modified files to identify and fix unnoticed defects before submitting changes. This process does not require an existing bug report, since its purpose is to discover problems that have not yet been identified. For example, GitHub Copilot supports reviewing uncommitted changes and providing suggested fixes [A].

Repository-wide proactive maintenance. Developers periodically inspect existing modules for latent defects before users encounter and report them. Scheduled repository scanning is an established workflow supported by GitHub code scanning [B]. Proactive bug fixing could extend this workflow to repairing discovered defects, such as resource leaks or incorrect retry logic.

Continuous integration. Code inspection can be triggered when developers push changes or open a pull request, as supported by GitHub code scanning [B]. In this scenario, proactive bug fixing could identify and repair defects missed by existing tests before integration, without waiting for a reported failure.

These scenarios share a practical and common need: developers know which code requires inspection but do not necessarily know which bugs it contains. To address your concern, we have clarified these practical scenarios and added relevant references in the revised introduction.

Q2: The introduction also fails to clarify how this work differs from existing research on proactive defect discovery. Although the paper cites TestExplora, a study that also explores proactive defect discovery in the absence of issue reports, the introduction does not explain what distinguishes Active-SWE from that approach.

A2: Thanks for your comment. In Related Work (Section 2), we have categorized TestExplora [C] as a test-generation benchmark, where LLMs are instructed to derive executable tests based on the curated documents. In brief, TestExplora differs from Active-SWE in the following aspects:

i) Different tasks. TestExplora generates executable tests to expose bugs, whereas Active-SWE further requires agents to proactively discover and fix bugs. In other words, beyond exposing bugs, coding agents must further generate correct repair patches, which poses higher demands on code exploration and implementation capabilities.

ii) Different guidance. TestExplora relies on designated test interfaces and detailed documents generated by DocAgent, which specify instance-specific expected behaviors, inputs, and outputs ([C], Section 3.2 and Appendix B). In contrast, Active-SWE requires coding agents to complete tasks without detailed instance-specific guidance.

iii) Different evaluation scopes. TestExplora focuses on whether generated tests could expose recorded bugs. In contrast, Active-SWE enables a more in-depth evaluation from two perspectives: i) discovering and resolving multiple bugs rather than a specific recorded bug; ii) revealing potential bugs beyond those recorded in historical fixes.

Q3: The authors say “consisting of 6 major bug categories and an additional Out-of-Taxonomy category.” However, they do not explain how these 6 categories are defined, what the labeling process for these categories entails, or whether any consistency checks (such as inter-rater reliability assessments) were conducted.

A3: Thanks for your comments. In the following, we will answer your questions one by one.

Taxonomy definitions. The definitions of all six categories and their representative subtypes are provided in Appendix E. To establish this taxonomy, we collect 11,634 PRs and employ three LLMs, namely Gemini-3.1-Pro, GPT-5.4, and Claude Opus 4.8, to characterize and categorize their underlying bugs, using the MITRE CWE Software Development taxonomy [D] as a reference. After that, seven experts refine the categories by reviewing their boundaries, excluding infrequent bug types, and grouping the remaining 18 subtypes into six major categories.

Labeling procedure. Following the Taxonomy Curation Template in Appendix E, we adopt three LLMs, namely Gemini-3.1-Pro, GPT-5.4, and Claude Opus 4.8, to assign each PR to exactly one category. After that, the final labels are determined through consensus-based voting, i.e., majority voting. Specifically, each LLM casts one vote, and the category receiving the most votes is selected as the final label.

Consistency checks. To address your concerns, we additionally report the voting results among the three LLM annotators in Table A. From the results, the three annotators achieve a high level of agreement in labeling consistency.

Table A: Voting agreement among the three LLM annotators.

Voting agreementProportion
All three annotators agree95.9%
Exactly two annotators agree4.1%
All three annotators disagree0.0%

To further address your concern, we additionally asked seven experts to independently classify all 400 tasks in Active-SWE. As shown in Table B, their classifications agree with the LLM-based consensus labels on nearly all the tasks, supporting the reliability of our taxonomy annotation for benchmark construction.

Table B: Agreement between expert annotations and LLM-based consensus labels.

Classification resultNumber of tasksProportion
Agreement39799.2%
Disagreement30.8%

We will clarify these details and include the taxonomy definitions, labeling procedure, consistency checks in the revised manuscript.

Q4: It is said that “we propose a consensus-based curation mechanism for filtering out PRs without discoverable bugs.” the annotators label “a given PR with issue reports” and therefore cannot tell whether a bug remains discoverable once the issue is removed; the section also omits the number of annotators, the voting rule, and the agreement among annotators.

A4: Thanks for your comments. In the following, we will answer your questions one by one.

Discoverability after removing issue reports. During the definition of the bug categories, we have considered discoverability under the taxonomy, i.e., bugs assigned to these categories should be discoverable from the repository context alone. As a result, the issue reports provided in the Taxonomy Curation Template merely help annotators quickly understand the bugs and assign their categories.

To verify this, we additionally repeated taxonomy annotation on 400 tasks after removing the issue reports while keeping the other inputs unchanged. As shown in Table C, only limited tasks received different labels, suggesting taxonomy assignments are largely stable without issue reports.

Table C: Consistency of taxonomy labels with and without issue reports.

Classification resultNumber of tasksProportion
Unchanged labels39899.5%
Changed labels20.5%

Annotators and voting rule. As described in Section 4.1 (Implementation Details), we have specified three closed-source models as LLM-based annotators, namely Gemini-3.1-Pro, GPT-5.4, and Claude Opus 4.8. Besides, we adopt majority voting across the three annotators. Following the notation in Section 3.2, the final label is

yi=V(yi1,…,yiK)=arg⁡max⁡y∈Y∑k=1KI[yik=y],K=3.y_i = \mathcal{V}(y_i^1,\ldots,y_i^K) = \underset{y\in\mathcal{Y}}{\arg\max}\sum_{k=1}^{K}\mathbb{I}[y_i^k=y], \quad K=3.

where Y\mathcal{Y} contains the six bug categories and the Out-of-Taxonomy label. Each model contributes one vote, and the label receiving the most votes is selected.

Agreement among annotators. To address your concerns, we conduct more experiments on the agreement among annotators in Table A. The results show a high level of agreement among the annotators, supporting the consistency of our taxonomy labeling procedure.

Table A: Voting agreement among the three LLM annotators (reproduced from A3 for convenience).

Voting agreementProportion
All three annotators agree95.9%
Exactly two annotators agree4.1%
All three annotators disagree0.0%

We have clarified the voting rule in the revised manuscript.

Q5: The origin of the 400 evaluation tasks is not explained. While the authors state that "human experts curate Active-SWE with 400 tasks," they provide no details regarding the selection criteria or the number of experts involved; furthermore, they offer no justification for why these 400 tasks are representative of the full set of 1,663 tasks. Although Figure 3 presents the category distribution for the entire dataset, the breakdown of how many tasks from each category are included in the 400-task subset is not provided. This weakens the representativeness and reproducibility of the benchmark.

A5: Thanks for your suggestions. In Appendix A, we have provided the language distributions, code patch statistics, and unit test statistics of both 400-task subset and 1663-task fullset. These statistics show that the 400-task subset preserves the multilingual and heterogeneous nature of the full 1,663-task dataset while reducing evaluation cost. Specifically, seven experts selected tasks across the eight programming languages, aiming to maintain similar distributions of languages, patch characteristics, and test characteristics to the full dataset.

To further address your concerns, we additionally compare the bug category distributions of the full dataset and the selected subset. For your convenience, we provide the corresponding results in Table D.

Table D: Bug category distributions of the full dataset and the selected subset.

Bug categoryFull dataset (1,663 tasks)Selected subset (400 tasks)
Logic & Computation27.0%25.0%
Reference & Data Flow18.8%20.2%
Data Processing & Type18.3%20.2%
Domain-specific Workflow15.0%14.7%
Exception Safety10.5%10.0%
State & Lifecycle10.4%9.9%

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.