- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces BugWitGym, a pipeline for generating a semi-synthetic benchmark, along with a dataset generated by BugWitGym, and experiments with that dataset. The "Wit" in the name stands for "witness": the task of the benchmark is to generate a bug reproduction test. The idea is to take a real repo with a real regression test, which later becomes the hidden held-out witness. The pipeline injects a fault such that the golden witness fails, and use some additional bug-preserving rewrites to make the injected fault less obvious. BugWitGym focuses on Java.
Critical strengths
S1. The benchmark is based on real repos and real tests.
S2. The injected faults are fresh in the sense that they do not correspond to existing issues on code repositories.
S3. The focus on Java is nice to see, given the prevalent overfitting of AI benchmarks to Python. Also, the ability to generate a sizable number of instances is useful.
Critical weaknesses
W1. The golden witness itself is not fresh: the benchmark uses an existing test from the repo as a starting point, and that test has been seen by LLMs if they are before their knowledge cut-off.
W2. The injected faults are synthetic and have limited variety compared to real issues.
W3. The validation in the benchmark appears weak, in that it does not check for the fail-to-pass property, nor does it measure coverage. In other words, as far as I can tell, it only measures whether the witness makes the buggy program fail, not whether the non-buggy program passes the witness test.
W4. The benchmark reveals the buggy location as part of the input. This means it cannot evaluate the difficult sub-task of guessing that location.
W5. The paper misses important closely related work, SWT-Bench and TDD-Bench Verified. Both of those are benchmarks that also measure bug witness creation ability. Furthermore, both of these avoid the above-stated weaknesses W1 thru W4. Furthermore, SWT-Bench has an active leaderboard with several solutions that the paper does not cite. And TDD-Bench Verified recently added a Java variant.
Questions for authors
Q1. What is the creation date distribution of the tests that serve as the seed golden witnesses, and how does that compare to the knowledge cut-off dates of the LLMs used for evaluation?
Q2. Is the evaluation containerized, or are there any other mechanisms to prevent harmful execution of untrusted and possibly malicious generated witnesses during evaluation?
Q3. Do you evaluate the fail-to-pass property?
Q4. Are there any measures in place to prevent prevalent forms of reward hacking, such as exploring Git history, web browsing, or even just generating a trivial "assert false"?