- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper propose TestJack, which examines individual patches and writes additional tests for suspected bugs. These tests must pass the gold patch, fail the candidate patch, and be accepted by an LLM reviewer as consistent with the task description. To reduce cost, a lightweight version audits a few patches and shares the resulting tests across other patches for the same task. Across five benchmarks, the authors report refuting 34.4% of 4,487 originally passing trials. Their experiments also compare TestJack with SWE-ABS and show that the lightweight version recovers many of the full audit’s findings at substantially lower API cost.
Critical strengths
-
The paper addresses an important problem: passing benchmark tests does not necessarily mean satisfying the task. The concrete examples make a convincing case that existing evaluations can miss substantial defects.
-
Each refutation comes with an executable test, making the findings easier to inspect and reproduce. The broad audit and detailed case studies provide useful material for improving benchmark evaluation.
Critical weaknesses
- The assumption directly affects the reported conclusions. I am concerned that the assumption of consistency between the original tests, task prompt, and gold patch is doing too much work. If the gold patch misses a requirement, a valid new test could be discarded; if the original tests reject a valid solution, that patch never enters the audit. These cases directly affect the reported results, so placing them outside the scope does not fully address the concern. I therefore find the interpretation of 33.2% as a “corrected” resolution rate insufficiently supported.
- The generated tests lack independent validation against gold-patch overfitting. It is unclear how reliably TestJack distinguishes genuine bugs from tests that fit details specific to the gold patch. Passing the gold patch and failing a candidate establishes a behavioral difference, while its relevance to the task still depends on an LLM reviewer. Adding human review could help estimate errors, but would not fully resolve this underlying problem: many original benchmark tests were also written or reviewed by humans. This risks moving the same trust problem from the original tests to the generated tests and then to their reviewers, without establishing an independent basis for correctness. The paper does not explain how this chain of validation is resolved, leaving the interpretation of the 34.4% refutation rate uncertain.
Questions for authors
-
Did you check whether the gold patches satisfy all explicit requirements in their task prompts?
-
Did you examine any patches rejected by the original tests to determine whether they were actually valid solutions?
-
How many of the 1,545 reported refutations were independently checked by human reviewers?
-
Did you test the generated tests on any independently verified correct implementations other than the gold patch?
-
How many refutations remain when you count only tests justified by explicit requirements in the prompt?
-
What model and reasoning configuration did you use for the reviewer agent?
6 comments
Response to Reviewer jcvN (contd.)
Q3: How many of the 1,545 reported refutations were independently checked by human reviewers?
We independently audited 100 randomly sampled refuted trials manually from our per-trial auditing corpus (spanning all five benchmarks and diverse tasks), in addition to 50 randomly sampled refuted trials from our lightweight approach (reported under Q5).
Table R1: Human verification of randomly sampled refutations in per-trial auditing (RQ1).
| Benchmark | #Selected Trials | #Sound | #Incorrect | #Overfit |
|---|---|---|---|---|
| DeepSWE | 70 | 67 | 0 | 3 |
| SkillsBench | 3 | 3 | 0 | 0 |
| SWE-Marathon | 2 | 2 | 0 | 0 |
| SWE-bench Pro | 13 | 12 | 0 | 1 |
| SWE-bench Verified | 12 | 10 | 0 | 2 |
| Total | 100 | 94 | 0 | 6 |
Here, each sampled refutation is categorized under a strict standard:
- #Sound: The witness test is valid, strictly justified by an explicit requirement in the prompt, and exposes a genuine bug in the candidate patch.
- #Incorrect: The witness test itself contains erroneous test logic, broken setup, or false expectations.
- #Overfit: The witness test passes and fails , but enforces an implicit requirement or an unstated/gold-specific detail not explicitly written in the prompt.
As Table R1 shows, 94.0% (94/100) of the per-trial refutations are confirmed sound under strict explicit prompt requirements, 0% (0/100) involve broken or incorrect tests, and only 6.0% (6/100) test implicit or unstated details.
Q4: Did you test the generated tests on any independently verified correct implementations other than the gold patch?
We did not manually author a second set of reference patches from scratch. Across all the repository-level tasks involving multi-file changes in unfamiliar codebases, manually writing alternative implementations is itself error-prone, and certifying their correctness independently of the test suite would reintroduce the same verification circularity we aim to solve.
Instead, we checked the correctness of the gold patches as described in Q1. Moreover, TestJack prevents gold-patch overfitting without requiring a second hand-written reference patch through solution-blind review and cross-implementation validation across frontier models.
Q5: How many refutations remain when you count only tests justified by explicit requirements in the prompt?
Because distinguishing explicit from implicit requirements in natural-language specifications is a semantic judgment that cannot be purely machine-checked without relying on another LLM classifier, we evaluate explicit-only retention directly through human-expert verification on both the per-trial sample (Table R1 above) and a random sample of 50 refuted trials from the lightweight approach across the three RQ2 benchmarks (Table R2 below).
Table R2: Human verification of randomly sampled refutations in the lightweight approach (RQ2).
| Benchmark | #Selected Trials | #Sound | #Incorrect | #Overfit |
|---|---|---|---|---|
| DeepSWE | 45 | 38 | 0 | 7 |
| SkillsBench | 2 | 2 | 0 | 0 |
| SWE-Marathon | 3 | 2 | 0 | 1 |
| Total | 50 | 42 | 0 | 8 |
In both Table R1 and Table R2, #Sound counts only refutations justified by explicit prompt requirements, while any witness test relying on implicit requirements or non-requirements is counted as #Overfit:
- Per-trial auditing (Table R1): 94.0% (94/100) of refutations remain when counting only tests justified by explicit prompt requirements. Scaling this rate across all 1,545 reported refutations yields ~1,452 explicit-requirement refutations, which still refutes ~32.4% of passing trials (vs. 34.4%) and lowers the overall resolution rate from 50.6% to ~34.2% (vs. 33.2%). Thus, our core empirical conclusions hold even under a strict explicit-only criterion.
- Lightweight approach (Table R2): 84.0% (42/50) of refutations remain under explicit-only requirements, with 0% incorrect tests and 16.0% (8/50) reflecting implicit or underspecified behaviors. This directly corroborates our analysis in §4.3 and Appendix B.2 that lower agreement in the lightweight variant stems mainly from prompt underspecification rather than invalid test logic.
Response to Reviewer jcvN (contd.)
Q6: What model and reasoning configuration did you use for the reviewer agent?
We use GPT-5.5 with high reasoning effort for every agent in TestJack, including the Reviewer Agent, the Audit Agent, and the Task Analysis Agent. We will state this explicitly for all three agents in §4.1 in the revised paper.
References
[1] Liu, Jiawei, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. "Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation." Advances in neural information processing systems 36 (2023): 21558-21572.
[2] Aleithan, Reem, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin, and Song Wang. "Swe-bench+: Enhanced coding benchmark for llms." arXiv preprint arXiv:2410.06992 (2024).
[3] Skalse, Joar, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. "Defining and characterizing reward gaming." Advances in neural information processing systems 35 (2022): 9460-9471.
[4] Zhao, Bingchen, Dhruv Srikanth, Yuxiang Wu, and Zhengyao Jiang. "Specbench: Measuring reward hacking in long-horizon coding agents." arXiv preprint arXiv:2605.21384 (2026).
[5] McConnell, Ross M., Kurt Mehlhorn, Stefan Näher, and Pascal Schweitzer. "Certifying algorithms." Computer Science Review 5, no. 2 (2011): 119-161.
[6] Beyer, Dirk, Matthias Dangl, Daniel Dietsch, Matthias Heizmann, and Andreas Stahlbauer. "Witness validation and stepwise testification across software verifiers." In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, pp. 721-733. 2015.
We first share these responses here for your timely review. We are currently revising the paper and will upload it shortly.
Response to Reviewer jcvN
We thank the reviewer for the thoughtful and rigorous feedback, and for recognizing the importance of the evaluation problem we study as well as the value of backing every refutation with a concrete, replayable test.
To directly address your concerns regarding the well-formedness assumption and validation against gold-patch overfitting, we first clarify how TestJack's design resolves these two issues in principle, and then present four new human-expert verification studies covering 300 manually audited tasks and trials. We will incorporate all of these analyses and clarifications into the revised paper.
Clarifying the Two Core Soundness Concerns
Witness verification (completeness vs. soundness):
Static benchmark suites do not fail because humans or LLMs cannot tell whether one concrete test matches the prompt; they fail due to under-coverage (omission)[1,2]. Writing T0 before seeing any submission requires completeness, anticipating every shortcut, ghost branch, or reward hack an agent might take across an open-ended implementation space[3,4]. By contrast, refuting a trial in TestJack requires soundness of just one concrete counterexample (Eq. 2): once a candidate witness test t is found against a specific patch s, verifying that t checks an explicit clause of P is a bounded, local check rather than an open-ended search[5,6].
Prevent overfitting to G:
An LLM opinion alone never refutes a trial in TestJack. Every refutation must clear three independent gates (Eq. 2, §3.2):
Responses to Specific Questions
Yes. Both benchmark provenance and our direct human inspection confirm that the gold patches reliably satisfy explicit prompt requirements:
Yes. Both benchmark construction and a new human-expert study confirm that patches rejected by T0 are genuinely defective: