acceptodds
Under review as a conference paper at ICLR 2027

AgentBug-Smith: Automatically Reproducing Real-World Harness Bugs in Agentic Systems

13 views

Abstract

Agent harness bugs exhibit unique characteristics and remain challenging for state-of-the-art software agents to repair. Progress in this area is further hindered by existing benchmarks, which contain only a small and fixed number of executable harness bugs while requiring hundreds of human hours to construct. This work presents AGENTBUG-SMITH, an automated harness bug reproduction approach that continuously discovers and reproduces real-world harness bugs from open-source agentic systems. Across different backbone LLMs, AGENTBUGSMITH consistently outperforms existing bug reproduction techniques designed for general software, achieving 10.67% - 27.56% higher success rates of reproducing harness bugs. By applying AGENTBUG-SMITH to open-source agentic systems in the wild, we construct LIVE-HARNESS-BENCH, a live and extensible benchmark that currently contains 200 reproducible harness bugs. We further demonstrate the utility of LIVE-HARNESS-BENCH through two downstream applications. First, we use LIVE-HARNESS-BENCH as the evaluation benchmark to systematically evaluate state-of-the-art software agents, revealing their limited capabilities in repairing real-world harness bugs. Second, we use LIVE-HARNESSBENCH as a knowledge base of real-world harness bug fixes, from which reusable repair skills can be distilled to improve existing software agents, increasing their harness-bug repair rates by 6.32%. Together, AGENTBUG-SMITH and LIVEHARNESS-BENCH establish a scalable foundation for continuously evaluating and improving software agents on harness bug repair, turning real-world agent failures into executable evaluation instances and reusable knowledge for harness improvement, thus contributing to the ultimate goal of recursively self-improving agents

Primary area:
datasets and benchmarks
Keywords:
Agent Harness, Harness Bug Reproduction, Automated Benchmark Construction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_K1pFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work presents AgentBug-Smith, a pipeline for synthesizing executable software engineering environments from GitHub pull requests and issues with a focus on harness bugs. The pipeline produces 200 bug environments from 11 repositories. The authors further provide two downstream use cases, LIVE-HARNESS-BENCH that evaluates an agent's capability to resolve harness bugs, and a textual skill to improve such capability.

Critical strengths
  • Identified agent harness bugs are a kind of bug that require special treatment in the standard bug-to-environment pipeline, because of interaction with external resources and dynamically changing environments.
  • The environment synthesis pipeline is generally sound with the use of common techniques from existing SWE pipelines (e.g. SWE-rebench / SWE-gym / R2E-gym / SWE-factory etc).
Critical weaknesses
  • There appears to be little special treatment to harness bugs in the pipeline, aside from providing an external LLM API and key for unit tests. It mentions mocking of services and network isolation, but that is just a prompt telling the test-writing LLM to mock instead of using real online services. And the technique can be applied to general software engineering bugs too.
  • The quality of the resulting benchmark is questionable. The criterion of successful environment production in the paper is merely having a unit test that passes with a golden patch and fails without. There are no solubility, description-test consistency, or other quality checks in the pipeline. The authors' manual check shows that only a small portion of agent solutions are actually correct, out of those passing the unit tests (e.g. 1.27% Correctly resolved solutions out of 35.44% Plausibly resolved solutions). Also, the test split issues in Sec 4.4 appears to be all from the same repository.
  • The authors conclude that state-of-the-art software agents exhibit limited resolution rates in harness bugs but the agents were powered by GPT-4.1-mini instead of a stronger frontier model.
Questions for authors

1-2. Please address the Weakness 1~2. For Weakness 1, what if the llm output differs every time and the unit test results are not stable?

  1. What do you think is the novelty (with an ablated gain) of your environment production pipeline compared to existing works (e.g. SWE-rebench / SWE-gym / R2E-gym / SWE-factory / Scale-SWE etc)? In Table 3 it appears that the higher success rate in environment production compared to SWE-Factory is mainly contributed by pull requests without a unit test, and the gain comes from letting LLMs synthesize unit tests unseen in the the repository code. Yet this capability is not novel (e.g. R2E-Gym has it).

  2. What is the LLM used to iterate the skill prompt in Sec 4.4? Do you prevent the LLM to view the issues in the test set?

6 comments

Q1: There appears to be little special treatment to harness bugs in the pipeline, aside from providing an external LLM API and key for unit tests. And the technique can be applied to general software engineering bugs too. What if the LLM output differs every time and the unit test results are not stable?

Our pipeline includes two key stages, i.e., harness bug identification, and harness bug reproduction. (1) For the harness bug identification component, it is the first approach that can automatically and precisely identify harness bug related Github issues, which is designed just for agentic software and not applicable for general software. (2) For the harness bug reproduction, the joint optimization mechanism is actually tailored for harness bug reproduction, as the generated failure-triggering harness tests often require additional external resources (which is less common in general software compared to agentic software) and thus requires back-forth refinement of the environment and tests. Our later ablation experiments in Q3 also confirm the benefits of such a setting. Though applying our approach to general software is not the focus of this work, it could be an interesting extension in the future work. For LLM outputs, we record the LLM responses for each prompt and mock them in subsequent test executions.

Q2: Quality of the resulting benchmark. The criterion of successful environment production in the paper is merely having a unit test that passes with a golden patch and fails without. A manual check shows that only a small portion of agent solutions are actually correct…

There is actually a misunderstanding in the review: we not just use the test outcome fail-to-pass as the only signal for the successful reproduction, as mentioned in line 275 (i.e., “if it fails on the buggy commit with the same failure reported in the issue description”), the pipeline also leverages LLMs to check if the failed tests thrown out the same failure described in the issue description. Moreover, as mentioned in Line 353, we further manually checked all the reproduced bugs in RQ2, and 98.67% of them are exactly reproducing the same bug described in the issue descriptions as well as with high-quality issue description. The pipeline design and our manual inspection confirm the high quality of our benchmarks.

In addition, the gap between test-passing patches and correct patches is actually a common and long standing challenge due to test insufficiency in test-driven automated program repair[a,b,c, e] and has been observed in many peer-reviewed and widely-used benchmarks, e.g., AgentIssueBench[e], SWE-Bench[f], KGym[g]. We agree this prevalent limitation could be an important future direction to improve and will discuss it in the paper.

Reference

[a] A Large-Scale Empirical Review of Patch Correctness Checking Approaches. FSE’23.

[b] SWE-ABS: Adversarial Benchmark Strengthening Exposes Inflated Success Rates on Test-based Benchmark. ICML’26.

[c] Investigating Test Overfitting on SWE-bench. FSE’26

[d] Are “Solved Issues” in SWE-bench Really Solved Correctly? An Empirical Study. ICSE’26

[e] Can Agent Fix Agent Bugs? NeurIPS’25.

[f] SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR’24

[g] kGym: A Platform and Dataset to Benchmark Large Language Models on Linux Kernel Crash Resolution. NeurIPS’26

Q3: What do you think is the novelty (with an ablated gain) of your environment production pipeline compared to existing works.

Thanks for the suggestion and we believe the joint optimization loop between environment and test generation is the major difference of AgentBug-Smith compared to existing baselines. In particular, existing baselines either do not support test generation components (e.g., SWE-Factory,SWE-rebench) or just include the test generation component sequentially after the environment generation. Our insight is that the generated failure-triggering harness tests often require additional external resources (which is less common in general software compared to agentic software) and thus the executability of generated tests and environment are dependent on each other, requiring back-forth refinement of the environment and tests.

To validate this intuition, we perform additional experiments: (a) removing joint optimization loop: after replacing the join refinement with the sequential refinement of environment and tests, the reproduction success rate of AgentBug-Smith drops from 20% to 9.78%, indicating the critical contribution of the join refinement design; (b) enhancing the best baseline (e.g., SWE-Factory) with the test generation component: we follow the reviewer’s suggestion to adopt the state-of-the-art test generation component (e.g., the Agentless style approach) used in the existing work R2E-Gym. After the enhancement, the reproduction success rate of SWE-Factory increases from 9.33% to 10.67%, which however is still much worse than AgentBug-Smith (i.e., 20%). The results further confirm that (i) the improvement of AgentBug-Smith does not merely come from the additional test generation step; and (ii) the test generation strategy in AgentBug-Smith outperforms existing test generation components in general baselines (e.g., R2E-Gym) for harness bugs.

Q4: What is the LLM used to iterate the skill prompt in Sec 4.4? Do you prevent the LLM from viewing the issues in the test set?

In Sec 4.4, we use the same model for skill iteration (e.g., GPT-4-mini) and to support the mini-swe-agent, that is, no stronger model is involved in the skill iteration process. To prevent any potential data leakage, we definitely prevent LLM from viewing issues in the test set and the skill iteration only happens within the issues in the training set.

Reviewer_K1pFOpenReview ↗

Official Comment by Reviewer K1pF

Thank you for the responses. I have read them together with the paper and the anonymized code release.

Q1/Q3. The response's case is that harness tests "often require additional external resources". That is exactly where harness-specific treatment would show: provisioning those resources, recording provider responses and replaying them so the test is deterministic, stubbing tools and services. The paper and the release do none of this. The generated test gets a prompt instruction to mock the model client, the evaluation container gets live provider keys, and the recording and replay of model responses that the authors claim in the rebuttal does not exist in the release. The pipeline does not handle external resources; it tells an LLM to avoid them. That is why I wrote there is little special treatment, and the new ablation numbers do not bear on this.

Q2. I do not think there is a misunderstanding. My concern is not whether a generated test reproduces the reported failure; it is whether the test can tell a correct patch from a wrong one. Such gaps exist in SWE-bench and similar benchmarks, and I do not dispute that. What the paper reports is a different situation. On the 79-instance test split, an off-the-shelf agent produces 28 plausible patches of which 1 is correct (35.44% vs 1.27%, Table 6). When 27 of 28 passing patches are wrong, the generated tests do not cover the patterns a wrong patch takes; they are not measuring repair, and the resolution rates built on them do not support the paper's conclusions. Related, Sec. 4.4 states that correctness is determined by manual inspection, while Appendix K says "the AI judge marks the patch correct"; please clarify which applies to Tables 5 and 6.

Q4. I asked this question for a reason. The response states that the skill iteration "definitely" never sees test issues. The released skill says otherwise. The test split is the 79 harness-sdk issues. The skill's five capability scanners (C1 to C5) are each tied, in the release's manifest and capability documentation, to one held-out issue from that split (#1705, #2122, #2173, #2205, #362), with the gold PR cited, and the documentation says the scanners were distilled from those held-out issues. Those five issues are exactly the five new successes behind the 6.32% gain, and Appendix K presents #362 as one of them. The scanners also contain the vocabulary of those issues (system_prompt, toolResult, window_size, "trim conversation", ARN), and the skill names the test repository, which Appendix I's own constraint forbids. With the five scanners removed there is no measured improvement. Unless the authors can show the scanners were derived from training issues only, the enhancement result is test-set leakage, and the statement in the response is incorrect.

I am keeping my score.

Reviewer_XwHuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AgentBug-Smith, a system for automatically reproducing real bugs in agent harnesses and building an executable benchmark that can grow over time. It finds open-source agent repositories and resolved bug reports, sets up isolated environments, and generates tests that fail on the buggy version but pass on the fixed one. If a test does not reproduce the bug, the system can revise both the test and the environment. The authors evaluate AgentBug-Smith on 225 candidate bugs using three backbone models. It reproduces 20–28% of the bugs, outperforming SWE-Factory and SWE-bench-Live. Across the three models, it reproduces 75 distinct bugs; manual review confirms that 74 match the reported failures. The authors also build Live-Harness-Bench, a benchmark containing 200 executable bug cases, and use it to evaluate code-fixing agents and a method for turning past fixes into reusable skills.

Critical strengths
  1. The system combines environment setup, isolation of external services, and reproduction-test generation in one pipeline. For bugs whose patches contain no tests, Table 3 reports 21, 26, and 29 successful reproductions with the three models, respectively. Neither baseline reproduces any bugs in this group.
  2. Across the three models, the system reproduces 75 distinct bugs. The authors manually check all 75 and confirm that 74 match the failures described in the reports. This gives more confidence that the tests capture the original problems than a fail-to-pass result alone, and supports the benchmark’s focus on real bugs.
Critical weaknesses
  1. AgentBug-Smith costs more per issue than the baselines. With GPT-4.1-mini, for example, the average cost is 0.56, compared with 0.09 for the baseline. The paper also leaves it unclear whether the methods have comparable limits on attempts, context length, and execution budget. Table 3 mainly compares stages across complete systems, without controlled ablations of key AgentBug-Smith components such as joint test and environment optimization. This makes it hard to tell how much of the improvement comes from AgentBug-Smith’s design and how much comes from additional attempts or resources.
Questions for authors
  1. Could you report the limits on attempts, context length, and token or time budgets for each method, and clarify whether they are comparable? It would also help to see a controlled ablation of AgentBug-Smith’s joint optimization of tests and environments. Do the two baselines support generating reproduction tests when no in-patch test is available?
  2. Beyond targeting a different problem, how does AgentBug-Smith differ in its core approach from SWE-Smith, SWE-Gym, and R2E-Gym?
4 comments

Q1.1: Report limits on attempts, context length, and token or time budgets for each method, and clarify whether they are comparable?

As different methods follow very different pipeline designs, it is hard or can be unfair to use a unified attempt limit across different methods. Therefore, for all the baselines, we directly adopted their released implementation, thus using the original limit settings in their implementation, meanwhile we did not add any additional token or time limits to any method. This allows each method to operate under its intended settings and avoids artificially constraining its effectiveness. In particular, (1) SWE-Factory: as shown in its original implementation, up to 15 iterations for both the enviroment dockerbuild and test execution; (2) for Live-SWE-bench, as shown in its original implementation, up to 2 iteration for both environment dockerbuild and test execution, and up to 60 command steps within each iteration. (3) for our approach AgentBut-Smith, we set up to 5 iteration of environment generation, test generation, and join optimization.

Q1.2. Ablating join optimization

Thanks for the suggestion and we now add the ablation experiments on joint optimization components. After removing the join refinement (only keeping the sequential refinement of environment and tests), the reproduction success rate of AgentBug-Smith significantly drops from 20% to 9.78%, indicating the critical contribution of the join refinement design, which is also the key novelty of AgentBug-Smith from the existing reproduction pipeline.

Q1.3. Do the two baselines support generating reproduction tests when no in-patch test is available?

The two baselines do not support reproduction test generation and how to effectively generate the failure-triggering tests for harness bugs is exactly one of the major contributions of our pipeline. To further justify the benefits of our test generation approach, we further enhanced the best baseline (i.e., SWE-Factory) with a test generation component. In particular, we follow the reviewer K1pF’s suggestion to adopt the state-of-the-art test generation component (e.g., the Agentless style approach) used in the existing work R2E-Gym. After the enhancement, the reproduction success rate of SWE-Factory increases from 9.33% to 10.67%, which however is still much worse than AgentBug-Smith (i.e., 20%). The results further confirm that (i) the improvement of AgentBug-Smith does not merely come from the additional test generation step; and (ii) the test generation strategy in AgentBug-Smith is better than existing test generation components in baselines.

Q2:Beyond targeting a different problem, how does AgentBug-Smith differ in its core approach from SWE-Smith, SWE-Gym, and R2E-Gym?

Beyond targeting different software systems (e.g., traditional software v.s. agentic systems), agentbug-smith differs from these baselines from two perspectives: (i) Harness Bug Identification, AgentBug-Smith presents the first method for automatically identifying harness bug issues, which achieves high accuracy 95% based on our manual evaluation (Section 4.1); (ii) Joint optimization loop between environment and tests, while existing general reproduction baselines either focus on environment generation (SWE-Smith, SWE-Gym) or only following the sequential optimization between environment and test generation (R2E-Gym), AgentBug-Smith proposes a joint optimization loop between environment and test generation. The joint optimization mechanism is actually designed for harness bug reproduction, as the generated failure-triggering harness tests often require additional external resources (which is less common in general software compared to agentic software) and thus requires back-forth refinement of the environment and tests. According to our ablation experiments in Q 1.2 and Q 1.3, such a design substantially contributes to the reproduction success rates.

Dear reviewer XwHu,

The discussion period will be closed soon. We've added new ablation experiments, setting details, and clarification required by your review. We will appreciate it a lot if you can give us feedback and let us know if anything more is needed from our side.

Thanks!

Thank you for the detailed response, which has addressed most of my concerns. However, I’m still not quite sure what sets this work apart from related SWE work like R2E-Gym, SWE-smith, and SWE-Factory. Is the main difference simply that those methods target general software development, while this paper focuses on agent harnesses? It seems like a standard SWE pipeline with a few tweaks could achieve similar results. I’m leaning toward keeping my original score, but I’ll also consider the other reviewers’ and the AC’s views before making my final recommendation.

Dear Reviewer XwHu,

Thank you for your prompt feedback and for acknowledging that our previous responses have addressed most of your concerns. Regarding the novelty, we would like to further clarify two points:

(1) Technical novelty. Our approach introduces two key technical novelty: harness bug identification and the joint optimization loops, we have explained their difference from general techniques with additional ablation results in our previous responses, so we will not repeat those details here. Meanwhile, we would like to emphasize that these contributions are not simply minor tweaks to existing SWE pipelines, instead, they took large engineering effort to implement. Therefore, we believe releasing such an automated pipeline can still benefit the community for further research.

(2) Benchmark novelty. Beyond the technical contributions, a central novelty of our work includes the resulting large-scale benchmark of reproducible, real-world agent harness bugs, which has been largely missing in existing research. We believe this benchmark can serve as a valuable foundation for future research on agent harness reliability.

We fully respect your cautious assessment and appreciate your willingness to consider the broader discussion before making your final recommendation. Thank you again for your thoughtful feedback and engagement throughout the rebuttal process.

Reviewer_y4EpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Recent advances in LLM-based agents have increased the importance of the software harnesses that manage tools, memory, and execution. The paper addresses the need for scalable and continuously extensible benchmarks of real-world harness bugs to support reliability evaluation. It introduces AgentBug-Smith, an automated bug reproduction pipeline, and applies it to construct Live-Harness-Bench. The pipeline identifies agent repositories and harness-related issues, builds isolated execution environments, and generates failure-reproducing tests, with joint refinement of the environment and test when necessary. The experiments examine harness bug identification, environment construction, and test generation, and compare reproduction performance with existing methods. The paper further evaluates software agents on the benchmark and explores skill distillation as a way to improve harness bug repair.

Critical strengths

S1. An important benchmark problem for agent harness reliability

As LLM-based agents become more capable, failures in the harness components that manage tools, memory, and orchestration are an important reliability concern. The paper addresses this problem by providing 200 executable real-world bug instances that support systematic evaluation of harness bug repair. It also proposes an automated construction process that can incorporate newly emerging cases, supporting continued benchmark expansion as agent systems evolve.

S2. Useful empirical analyses and improved reproduction coverage

Comparisons across multiple backbone LLMs and analyses of bug identification, environment construction, and test generation provide useful empirical support for the pipeline’s effectiveness. AgentBug-Smith consistently achieves higher bug reproduction success than the evaluated baselines. Its ability to reproduce bugs without developer-provided tests further broadens benchmark coverage and provides additional executable evaluation cases.

Critical weaknesses

W1. Guidelines for adapting heuristic design choices

The method uses several heuristic choices across its stages, including thresholds of at least 50 GitHub stars and 20 changed patch lines, alongside sampling for environment validation. Such practical choices are understandable given the challenges of reproducing real-world harness bugs, and the paper explains the motivation behind filtering and sampling. Providing clearer guidelines for adapting these settings, or more general procedures for selecting them, would support future extensions of the pipeline and its generalization across repositories with different scales, architectures, and testing practices.

W2. Further ablations of the individual components

Section 4.3 and Table 3 compare complete pipelines, and additional analyses of individual components would provide further empirical validation of the proposed method. Ablating rule-guided initialization and joint refinement would help clarify the contribution of each design choice. A comparison with a baseline augmented with generic reproduction-test generation under comparable budgets would also help assess the benefits of the harness-specific design.

Questions for authors

Q1. Improving generated tests for repair validation

Table 5 shows a gap between plausible and correct resolution rates, such as 19.50% versus 9.00% for mini-SWE-agent after manual assessment. Could you characterize the main failure modes among patches that pass the generated tests but are rejected during manual validation? Analyzing these cases could help identify how additional issue-specific or regression tests might strengthen the reliability of repair evaluation.

Q2. Construction time, scalability, and data distribution

What were the collection window, end-to-end construction time, and computational resources required to assemble the 200 instances, and what update cadence has been used? Reporting candidate and retention counts at each stage, together with the distribution across repositories and programming languages, would help assess practical scalability and potential concentration in particular projects.

4 comments

Q1. Improving generated tests for repair validation.

Thank you for proposing this analysis perspective. We further characterized the overfitting patches based on your suggestion. In particular, we find there are two major categories for the gap. (1) The generated test coverage is insufficient, while passing patch may modify some code locations not covered by generated tests. For example, one overfitting patch additionally modified the memory operation (by deleting Mem0Storage.save() and reset() function calls), which cannot be detected by existing tests. (2) The generated tests do not cover different configuration settings, thus cannot distinguish the patches overfitting to specific configuration settings. For example, one overfitting patch hardcodes the provider settings “azure/o1-preview-” and “openai/o1-preview”, while the tests do not include other potential configurations like “o1-”.

Such a limitation (overfitting patches due to imperfect test sufficiency) is actually a common and long standing challenge in test-driven automated program repair[a,b,c,d] and has been prevalent in many existing or similar benchmarks, e.g., AgentIssueBench[e], SWE-Bench[f], KGym[g]. Based on this analysis, we believe that a promising direction for future work is to enhance the benchmark through automated test augmentation, which could increase both the number of tests and their diversity, particularly in terms of external configuration settings.

Q2. Construction time, scalability, and data distribution.

As shown in Figure-6, the collection window of the current benchmark includes the bugs from 2023/04 to 2026/08. The end-to-end construction of the entire benchmark took approximately 900 hours of wall-clock time on a single machine with modest computational resources (i.e., only 2 vCPUs and 8 GB RAM), incurring a total LLM API cost of $520.24. For the candidate and retention counts at each stage, there are 429 issues at the candidate stage, 303 after environment construction, 200 after final test generation. The benchmark currently spans over 10 widely-used agents, with the detailed distribution in the following table.

Agent RepoRepo LinkStarsLicenseTotal F2P
AntonOsika/gpt-engineerhttps://github.com/AntonOsika/gpt-engineer55kMIT license11
MLSysOps/MLE-agenthttps://github.com/MLSysOps/MLE-agent/1.6kMIT license15
agentscope-ai/agentscopehttps://github.com/agentscope-ai/agentscope/35.5kApache-2.0 license50
dapr/dapr-agentshttps://github.com/dapr/dapr-agents/771Apache-2.0 license3
OpenInterpreter/open-interpreterhttps://github.com/OpenInterpreter/open-interpreter68.8kApache-2.0 license2
Aider-AI/aiderhttps://github.com/Aider-AI/aider50kApache-2.0 license8
strands-agents/harness-sdkhttps://github.com/strands-agents/sdk-python9.1kApache-2.0 license80
crewAIInc/crewAIhttps://github.com/crewAIInc/crewAI60.6kMIT license24
Significant-Gravitas/AutoGPThttps://github.com/Significant-Gravitas/AutoGPT187.9kMIT License & PolyForm Shield License 1.0.02
langchain-ai/langgraphhttps://github.com/langchain-ai/langgraph44.6kMIT license5

Reference

[a] A Large-Scale Empirical Review of Patch Correctness Checking Approaches. FSE’23.

[b] SWE-ABS: Adversarial Benchmark Strengthening Exposes Inflated Success Rates on Test-based Benchmark. ICML’26.

[c] Investigating Test Overfitting on SWE-bench. FSE’26

[d] Are “Solved Issues” in SWE-bench Really Solved Correctly? An Empirical Study. ICSE’26

[e] Can Agent Fix Agent Bugs? NeurIPS’25.

[f] SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR’24

[g] kGym: A Platform and Dataset to Benchmark Large Language Models on Linux Kernel Crash Resolution. NeurIPS’26

W1. Guidelines for adapting heuristic design choices

We agree that providing clearer guidance on adapting the heuristic settings would improve the extensibility of AgentBug-Smith. We will expand the discussion to clarify the purpose and adaptation guidelines for each heuristic. Specifically, (i) the repository popularity threshold serves as a lightweight proxy for community adoption and can be relaxed for emerging or domain-specific agent frameworks, or increased when processing large candidate pools under limited budgets; (ii) the patch-size threshold is intended to filter trivial changes rather than impose an inherent requirement on harness bugs, and can be relaxed or combined with LLM-as-judge component to filter out less meaningful fixes; and (iii) the environment-validation sample size balances basic environment validation against execution overhead and can be adjusted based on test-suite size and heterogeneity. For smaller repositories, executing all available tests may be practical, while larger repositories may benefit from sampling across different test suites or components. More generally, these heuristics primarily control candidate selection and intermediate validation costs; they do not alter the final correctness criteria, which require the generated test to reproduce the reported failure on the buggy version and pass on the patched version.

W2. Further ablations of the individual components.

Thanks for the great suggestion! We now added all the ablation or comparison experiments based on the reviewer’s suggestion.

  • Ablating rule-guided initialization. After removing the rule-guided initialization, the reproduction success rate of AgentBug-Smith drops from 20% to 10.22%, and the cost increases from $0.56 to $0.73.
  • Ablating join refinement. After removing the join refinement (only keeping the sequential refinement of environment and tests), the reproduction success rate of AgentBug-Smith drops from 20% to 9.78%, indicating the critical contribution of the join refinement design, which is also the key difference of AgentBug-Smith from the existing reproduction pipeline.
  • Enhancing baseline with test generation component. We enhanced the best baseline (i.e., SWE-Factory) with a test generation component. In particular, we follow the reviewer K1pF’s suggestion to adopt the state-of-the-art test generation component (e.g., the Agentless style approach) used in the existing work R2E-Gym. After the enhancement, the reproduction success rate of SWE-Factory increases from 9.33% to 10.67%, which however is still much worse than AgentBug-Smith (i.e., 20%). The results further confirm that the improvement of our approach AgentBug-Smith does not merely come from the additional test generation step.

Thank you for the thoughtful rebuttal. The discussion of the heuristic settings, the guidelines for adapting them, and the additional ablations were helpful. I also appreciate the detailed responses to my questions. I will therefore maintain my original rating of 3 (Weak acceptance).

Dear Reviewer y4Ep,

We truly appreciate your acknowledgement to our work and response. We will definitely incorporate all the discussion into paper revision.

Thanks!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.