acceptodds
Withdrawn from ICLR 2027

BugWitGym: Evaluating Coding Agents for Executable Bug Validation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Bug validation asks a coding agent to produce an executable witness for a reported bug. The witness combines a concrete input with a testing harness and exposes faulty behavior during execution. Such evidence makes audit findings actionable, yet benchmark evaluation is difficult when cases reuse public historical bugs and witnesses or require manual construction. We present BugWitGym, an automated framework for constructing bug-validation benchmarks through bug injection. It injects bugs into test-reached paths of real projects, rebuilds each project, and retains cases exposed by a construction-time witness. Bug specifications and execution adapters allow extension to additional bug types and languages. Bug-preserving transformations vary the surrounding structure while preserving the witness behavior. Based on real-world Java projects with test suites, BugWitGym automatically constructs 1,300 benchmark cases. The injected patches resemble historical bug patches and are difficult for the two evaluated models to distinguish in blinded comparisons. We evaluate four coding agent frameworks in six framework/model pairings across bug types, execution contexts, and transformation depths. Witness construction remains difficult even when the bug pattern is known. Our framework, benchmark cases, and evaluation scripts are available.

Primary area:
datasets and benchmarks
Keywords:
Coding Agents, Large Language Models, Bug Validation, Agent Evaluation, Test Generation, Benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KjuZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces BugWitGym, a pipeline for generating a semi-synthetic benchmark, along with a dataset generated by BugWitGym, and experiments with that dataset. The "Wit" in the name stands for "witness": the task of the benchmark is to generate a bug reproduction test. The idea is to take a real repo with a real regression test, which later becomes the hidden held-out witness. The pipeline injects a fault such that the golden witness fails, and use some additional bug-preserving rewrites to make the injected fault less obvious. BugWitGym focuses on Java.

Critical strengths

S1. The benchmark is based on real repos and real tests.

S2. The injected faults are fresh in the sense that they do not correspond to existing issues on code repositories.

S3. The focus on Java is nice to see, given the prevalent overfitting of AI benchmarks to Python. Also, the ability to generate a sizable number of instances is useful.

Critical weaknesses

W1. The golden witness itself is not fresh: the benchmark uses an existing test from the repo as a starting point, and that test has been seen by LLMs if they are before their knowledge cut-off.

W2. The injected faults are synthetic and have limited variety compared to real issues.

W3. The validation in the benchmark appears weak, in that it does not check for the fail-to-pass property, nor does it measure coverage. In other words, as far as I can tell, it only measures whether the witness makes the buggy program fail, not whether the non-buggy program passes the witness test.

W4. The benchmark reveals the buggy location as part of the input. This means it cannot evaluate the difficult sub-task of guessing that location.

W5. The paper misses important closely related work, SWT-Bench and TDD-Bench Verified. Both of those are benchmarks that also measure bug witness creation ability. Furthermore, both of these avoid the above-stated weaknesses W1 thru W4. Furthermore, SWT-Bench has an active leaderboard with several solutions that the paper does not cite. And TDD-Bench Verified recently added a Java variant.

Questions for authors

Q1. What is the creation date distribution of the tests that serve as the seed golden witnesses, and how does that compare to the knowledge cut-off dates of the LLMs used for evaluation?

Q2. Is the evaluation containerized, or are there any other mechanisms to prevent harmful execution of untrusted and possibly malicious generated witnesses during evaluation?

Q3. Do you evaluate the fail-to-pass property?

Q4. Are there any measures in place to prevent prevalent forms of reward hacking, such as exploring Git history, web browsing, or even just generating a trivial "assert false"?

Reviewer_bzoVOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper „BugWitGym: Evaluating Coding Agents for Executable Bug Validation” presents a benchmark that is constructed by first collecting test coverage data (in the paper called “execution context”), and then generating bugs within the covered range by letting an agent modify the code until the test fails. There are also bug-preserving transformations that should modify the code to make the data more diverse. Through this method, 1300 bugs are generated in different subsets are used for exploring different research questions within the paper, i.e., how well agents can find these bugs with tests, how this relates to bug types, how this is related to the code that is covered by the initially constructing tests, and how realistic the bugs are.

Critical strengths
  1. The goal to generate realistic, new bug data at scale to have high-quality benchmarks for test generation is important
  2. The aspects that are considered in the evaluation are relevant, i.e., general performance on the benchmark, as well as factors that may influence this like the bug types and aspects from the construction of the data set.
Critical weaknesses
  1. The paper is completely disconnected from the software testing literature. The most extreme case here is the “execution context”, which is seemingly obtained by directly altering code to insert logging statements. This is an extremely weak method to collect test coverage data, which is typically done through instrumentation of the byte code and that can in this way record for every statement which was covered, i.e., yield a lot more detailed information than just having string-logging. Bytecode instrumentation also yields smaller probe effects in comparison. The paper also ignores all software testing literature on constructing artificial bugs, including all mutation testing literature, learned mutation literature, and LLM-guided bug induction literature. Finally, even the goal of the benchmark is now clearly stated: it seems to be to measure if LLMs can generate effective tests for specific bugs, i.e., tests that can find a pre-defined bug. This is narrow, typical testing benchmarks should also be able to assess whole test-suite generation as well. There is also no comparison how any of the many test generation methods that are cited in the related work would work here, further highlighting the disconnect.
  2. The evaluation for all research questions is not sound. It is unclear why subsamples are collected for all research questions, why the subsample sizes are different for all research questions and what the impact of this choice is. None of the data ever seems to have been manually validated, i.e., there was no manual check if the purely LLM-guided bug generation and transformation works as intended. Especially for the transformations this is a big problem, since they do not rely on sound mechanisms that are ensured to not modify behavior (like automated refactorings), but rather on LLM-based modifications which might alter underlying behavior.
Questions for authors
  1. How does your work relate to existing software testing literature and why were basic test concepts like coverage not used here?
  2. What was the reason for using subsamples for the evaluation and how does this affect the soundness?
  3. Was there any human validation regarding the soundness of the results of the LLM outputs?
Reviewer_SNcMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The manuscript introduces BugWitGym, an automated framework for constructing benchmarks that evaluate whether coding agents can produce executable bug witnesses. Instead of resting historical bugs, BugWitGym instruments real-world Java projects to collect the functions executed by existing tests, and uses LLM agent to inject bugs of ten types adapted from CodeQL Java queries into these paths and keeps test cases only if the test passes on the clean code and fails on the buggy code so that the constructed benchmark is confirmed with test cases to trigger the bug. Then applies transformation around the injected buggy code area and preserves the bug is un-changed, to test the bug validation capabilities of coding agents/models under such sceanario. The manuscript constructed 1,300 test cases fro six even projects and evaluated six agent/model pairing with and without execution context. The result shows the success rate decreases as the execution context extends and the transformation depth grows.

Critical strengths
  1. The manuscript targets a practical problem: evaluating whether coding agents can construct executable witnesses for reported bugs. Existing benchmarks either rely on manually crafted tasks, which are labor-intensive, and usually reuse disclosed historical bugs. By injecting fresh bugs into real-world projects automatically, BugWitGym offers an automated framework to generate new evaluation cases for LLM auditors.

  2. BugWitGym, compared to previous bug-injection benchmark constructing methods, BugWitGym applies bug-preserving transformations that vary the call, data-flow, and control-flow structure around the injected bug, making task difficulty adjustable.

Critical weaknesses
  1. The evaluation oracle is one-sided, and the benchmark does not contain bug-free controls. A generated witness is counted as successful as long as the replay command fails with a semantic test failure on the buggy code. The paper does not describe evaluating the witness on the clean code. However, this process is applied during construction phase. Without checking the test case on the clean code, a test case that fails on the buggy code is not necessarily a valid witness, since its failure may not be caused by the injected bug. A valid witness should pass on the clean code and fail on the buggy code, and only this differential behaviour can confirm the test case is failed to the bug. Thus, the reported success rate may overestimate the agent's actual validation capability.

  2. During evaluation, although the contents of the construction time witness are removed, the agent is still given its path and class name, and instructed to rewrite the test at the same location, and is validated by a replay command that only runs this test class. This witness is an existing test for the original public repository, and an injected bug is retained only if this test exposes it. In addition, since bug-preserving transformations must keep the same witness valid and preserve public API signatures, the original test remains a valid witness at every transformation depth. As a result, a model that has seen the original test during pretraining may reproduce it and obtain a valid witness without reasoning about the injected bug. Which undermines the paper's claim that fresh cases reduce prior exposure to solutions.

  3. The evaluation conclusions are not well supported by the reported data. Some conclusions contradict the paper's own results. In introduction, the manuscript states that Value Flow and Logic bugs are generally harder than API contract bugs, and Sec. 4.3 states open-source agents are generally less effective on these two categories. However, according to the results shown in Table 4 by category under WithEC does not show a consistent pattern, Value Flow is the easiest category for A1, Logic is the easiest for A3 and A4, and A5 and A6 perform better on Value Flow than On API contract. And the evaluation results are easily to be affected by a single test case, in the evaluation, for each cell in Table 4 contains 25 cases, so a single case changes the success rate by 4 pp.

  4. The evaluation design cannot seperate the contributions of agent frameworks and models. Each model is evaluated under only one framework, so differences across pairings, such as the differences between close source and open source agents may come from the framework, the model, or both.

Questions for authors
  1. Is the generated witness also execute on the clean code during evaluation? If not, how many of the reported successes remain valid under this scenario?
  2. How similar are the generated witnesses to the original tests, and how does performance change if the original test path and class name are hidden from the agents? If it is possible, could you run the same test for public repositories that published after the model training time?
  3. How were the category level difficulty conclusions derived, the reported results in Table 4 are inconsistent with the conclusions, and can the authors report confidence intervals or significance tests to show whether the differences between categories are statistically meaningful?
  4. Can the authors evaluate the same model under different frameworks to separate the contributions of frameworks and models, such as in open source agents with industrial models including GPT and claude models?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.