- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work presents AgentBug-Smith, a pipeline for synthesizing executable software engineering environments from GitHub pull requests and issues with a focus on harness bugs. The pipeline produces 200 bug environments from 11 repositories. The authors further provide two downstream use cases, LIVE-HARNESS-BENCH that evaluates an agent's capability to resolve harness bugs, and a textual skill to improve such capability.
Critical strengths
- Identified agent harness bugs are a kind of bug that require special treatment in the standard bug-to-environment pipeline, because of interaction with external resources and dynamically changing environments.
- The environment synthesis pipeline is generally sound with the use of common techniques from existing SWE pipelines (e.g. SWE-rebench / SWE-gym / R2E-gym / SWE-factory etc).
Critical weaknesses
- There appears to be little special treatment to harness bugs in the pipeline, aside from providing an external LLM API and key for unit tests. It mentions mocking of services and network isolation, but that is just a prompt telling the test-writing LLM to mock instead of using real online services. And the technique can be applied to general software engineering bugs too.
- The quality of the resulting benchmark is questionable. The criterion of successful environment production in the paper is merely having a unit test that passes with a golden patch and fails without. There are no solubility, description-test consistency, or other quality checks in the pipeline. The authors' manual check shows that only a small portion of agent solutions are actually correct, out of those passing the unit tests (e.g. 1.27% Correctly resolved solutions out of 35.44% Plausibly resolved solutions). Also, the test split issues in Sec 4.4 appears to be all from the same repository.
- The authors conclude that state-of-the-art software agents exhibit limited resolution rates in harness bugs but the agents were powered by
GPT-4.1-miniinstead of a stronger frontier model.
Questions for authors
1-2. Please address the Weakness 1~2. For Weakness 1, what if the llm output differs every time and the unit test results are not stable?
-
What do you think is the novelty (with an ablated gain) of your environment production pipeline compared to existing works (e.g. SWE-rebench / SWE-gym / R2E-gym / SWE-factory / Scale-SWE etc)? In Table 3 it appears that the higher success rate in environment production compared to SWE-Factory is mainly contributed by pull requests without a unit test, and the gain comes from letting LLMs synthesize unit tests unseen in the the repository code. Yet this capability is not novel (e.g. R2E-Gym has it).
-
What is the LLM used to iterate the skill prompt in Sec 4.4? Do you prevent the LLM to view the issues in the test set?
6 comments
Q3: What do you think is the novelty (with an ablated gain) of your environment production pipeline compared to existing works.
Thanks for the suggestion and we believe the joint optimization loop between environment and test generation is the major difference of AgentBug-Smith compared to existing baselines. In particular, existing baselines either do not support test generation components (e.g., SWE-Factory,SWE-rebench) or just include the test generation component sequentially after the environment generation. Our insight is that the generated failure-triggering harness tests often require additional external resources (which is less common in general software compared to agentic software) and thus the executability of generated tests and environment are dependent on each other, requiring back-forth refinement of the environment and tests.
To validate this intuition, we perform additional experiments: (a) removing joint optimization loop: after replacing the join refinement with the sequential refinement of environment and tests, the reproduction success rate of AgentBug-Smith drops from 20% to 9.78%, indicating the critical contribution of the join refinement design; (b) enhancing the best baseline (e.g., SWE-Factory) with the test generation component: we follow the reviewer’s suggestion to adopt the state-of-the-art test generation component (e.g., the Agentless style approach) used in the existing work R2E-Gym. After the enhancement, the reproduction success rate of SWE-Factory increases from 9.33% to 10.67%, which however is still much worse than AgentBug-Smith (i.e., 20%). The results further confirm that (i) the improvement of AgentBug-Smith does not merely come from the additional test generation step; and (ii) the test generation strategy in AgentBug-Smith outperforms existing test generation components in general baselines (e.g., R2E-Gym) for harness bugs.
Q4: What is the LLM used to iterate the skill prompt in Sec 4.4? Do you prevent the LLM from viewing the issues in the test set?
In Sec 4.4, we use the same model for skill iteration (e.g., GPT-4-mini) and to support the mini-swe-agent, that is, no stronger model is involved in the skill iteration process. To prevent any potential data leakage, we definitely prevent LLM from viewing issues in the test set and the skill iteration only happens within the issues in the training set.
Official Comment by Reviewer K1pF
Thank you for the responses. I have read them together with the paper and the anonymized code release.
Q1/Q3. The response's case is that harness tests "often require additional external resources". That is exactly where harness-specific treatment would show: provisioning those resources, recording provider responses and replaying them so the test is deterministic, stubbing tools and services. The paper and the release do none of this. The generated test gets a prompt instruction to mock the model client, the evaluation container gets live provider keys, and the recording and replay of model responses that the authors claim in the rebuttal does not exist in the release. The pipeline does not handle external resources; it tells an LLM to avoid them. That is why I wrote there is little special treatment, and the new ablation numbers do not bear on this.
Q2. I do not think there is a misunderstanding. My concern is not whether a generated test reproduces the reported failure; it is whether the test can tell a correct patch from a wrong one. Such gaps exist in SWE-bench and similar benchmarks, and I do not dispute that. What the paper reports is a different situation. On the 79-instance test split, an off-the-shelf agent produces 28 plausible patches of which 1 is correct (35.44% vs 1.27%, Table 6). When 27 of 28 passing patches are wrong, the generated tests do not cover the patterns a wrong patch takes; they are not measuring repair, and the resolution rates built on them do not support the paper's conclusions. Related, Sec. 4.4 states that correctness is determined by manual inspection, while Appendix K says "the AI judge marks the patch correct"; please clarify which applies to Tables 5 and 6.
Q4. I asked this question for a reason. The response states that the skill iteration "definitely" never sees test issues. The released skill says otherwise. The test split is the 79 harness-sdk issues. The skill's five capability scanners (C1 to C5) are each tied, in the release's manifest and capability documentation, to one held-out issue from that split (#1705, #2122, #2173, #2205, #362), with the gold PR cited, and the documentation says the scanners were distilled from those held-out issues. Those five issues are exactly the five new successes behind the 6.32% gain, and Appendix K presents #362 as one of them. The scanners also contain the vocabulary of those issues (system_prompt, toolResult, window_size, "trim conversation", ARN), and the skill names the test repository, which Appendix I's own constraint forbids. With the five scanners removed there is no measured improvement. Unless the authors can show the scanners were derived from training issues only, the enhancement result is test-set leakage, and the statement in the response is incorrect.
I am keeping my score.
Our pipeline includes two key stages, i.e., harness bug identification, and harness bug reproduction. (1) For the harness bug identification component, it is the first approach that can automatically and precisely identify harness bug related Github issues, which is designed just for agentic software and not applicable for general software. (2) For the harness bug reproduction, the joint optimization mechanism is actually tailored for harness bug reproduction, as the generated failure-triggering harness tests often require additional external resources (which is less common in general software compared to agentic software) and thus requires back-forth refinement of the environment and tests. Our later ablation experiments in Q3 also confirm the benefits of such a setting. Though applying our approach to general software is not the focus of this work, it could be an interesting extension in the future work. For LLM outputs, we record the LLM responses for each prompt and mock them in subsequent test executions.
There is actually a misunderstanding in the review: we not just use the test outcome fail-to-pass as the only signal for the successful reproduction, as mentioned in line 275 (i.e., “if it fails on the buggy commit with the same failure reported in the issue description”), the pipeline also leverages LLMs to check if the failed tests thrown out the same failure described in the issue description. Moreover, as mentioned in Line 353, we further manually checked all the reproduced bugs in RQ2, and 98.67% of them are exactly reproducing the same bug described in the issue descriptions as well as with high-quality issue description. The pipeline design and our manual inspection confirm the high quality of our benchmarks.
In addition, the gap between test-passing patches and correct patches is actually a common and long standing challenge due to test insufficiency in test-driven automated program repair[a,b,c, e] and has been observed in many peer-reviewed and widely-used benchmarks, e.g., AgentIssueBench[e], SWE-Bench[f], KGym[g]. We agree this prevalent limitation could be an important future direction to improve and will discuss it in the paper.
[a] A Large-Scale Empirical Review of Patch Correctness Checking Approaches. FSE’23.
[b] SWE-ABS: Adversarial Benchmark Strengthening Exposes Inflated Success Rates on Test-based Benchmark. ICML’26.
[c] Investigating Test Overfitting on SWE-bench. FSE’26
[d] Are “Solved Issues” in SWE-bench Really Solved Correctly? An Empirical Study. ICSE’26
[e] Can Agent Fix Agent Bugs? NeurIPS’25.
[f] SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR’24
[g] kGym: A Platform and Dataset to Benchmark Large Language Models on Linux Kernel Crash Resolution. NeurIPS’26