acceptodds
Under review as a conference paper at ICLR 2027

TestJack: Should you trust the results in coding benchmarks? Agentic Coding Benchmarks Auditing via Evaluator Evolution

Abstract

Large language model (LLM) agents are rapidly reshaping software engineering, accompanied by an explosion of new code benchmarks. Yet nearly all existing benchmarks still rely on the same decades-old criterion: a solution is correct if it passes a fixed set of unit tests. Such tests are often insufficient: they check only part of what the task requires, so agents can reward hack them or silently miss required behavior while still passing every test. As a result, higher benchmark scores may partly reflect better adaptation to the evaluator rather than better problem solving. Existing works focus on static test augmentation: they strengthen each task’s tests once, before any trial is seen, and thus overlook how real trials actually fail. We introduce TestJack, a scalable framework for evaluating patches beyond fixed tests. For each trial, TestJack generates tests targeting prompt requirements the patch may violate, retains only tests passed by the ground-truth patch, and re-examines any trial failures. Each confirmed failure is thus supported by a replayable test. To reduce evaluation cost, we also introduce a lightweight variant which audits a random sample of trials in depth and reuses the resulting tests across all trials for the same task. Across 6 frontier model backends and 5 benchmarks such as DeepSWE and SWE Marathon, we find that about 34.4% of the model trials currently judged correct violate the task requirements, lowering the overall resolution rate from 50.6% to 33.2%. Our results reveal a fundamental limitation of current coding-agent evaluation: as LLMs become better at optimizing against fixed evaluators, those evaluators themselves must become more adaptive.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Test Augmentation, Large Language Model, Coding Agent

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_jcvNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper propose TestJack, which examines individual patches and writes additional tests for suspected bugs. These tests must pass the gold patch, fail the candidate patch, and be accepted by an LLM reviewer as consistent with the task description. To reduce cost, a lightweight version audits a few patches and shares the resulting tests across other patches for the same task. Across five benchmarks, the authors report refuting 34.4% of 4,487 originally passing trials. Their experiments also compare TestJack with SWE-ABS and show that the lightweight version recovers many of the full audit’s findings at substantially lower API cost.

Critical strengths
  1. The paper addresses an important problem: passing benchmark tests does not necessarily mean satisfying the task. The concrete examples make a convincing case that existing evaluations can miss substantial defects.

  2. Each refutation comes with an executable test, making the findings easier to inspect and reproduce. The broad audit and detailed case studies provide useful material for improving benchmark evaluation.

Critical weaknesses
  1. The assumption directly affects the reported conclusions. I am concerned that the assumption of consistency between the original tests, task prompt, and gold patch is doing too much work. If the gold patch misses a requirement, a valid new test could be discarded; if the original tests reject a valid solution, that patch never enters the audit. These cases directly affect the reported results, so placing them outside the scope does not fully address the concern. I therefore find the interpretation of 33.2% as a “corrected” resolution rate insufficiently supported.
  2. The generated tests lack independent validation against gold-patch overfitting. It is unclear how reliably TestJack distinguishes genuine bugs from tests that fit details specific to the gold patch. Passing the gold patch and failing a candidate establishes a behavioral difference, while its relevance to the task still depends on an LLM reviewer. Adding human review could help estimate errors, but would not fully resolve this underlying problem: many original benchmark tests were also written or reviewed by humans. This risks moving the same trust problem from the original tests to the generated tests and then to their reviewers, without establishing an independent basis for correctness. The paper does not explain how this chain of validation is resolved, leaving the interpretation of the 34.4% refutation rate uncertain.
Questions for authors
  1. Did you check whether the gold patches satisfy all explicit requirements in their task prompts?

  2. Did you examine any patches rejected by the original tests to determine whether they were actually valid solutions?

  3. How many of the 1,545 reported refutations were independently checked by human reviewers?

  4. Did you test the generated tests on any independently verified correct implementations other than the gold patch?

  5. How many refutations remain when you count only tests justified by explicit requirements in the prompt?

  6. What model and reasoning configuration did you use for the reviewer agent?

6 comments
AuthorsOpenReview ↗

Response to Reviewer jcvN

We thank the reviewer for the thoughtful and rigorous feedback, and for recognizing the importance of the evaluation problem we study as well as the value of backing every refutation with a concrete, replayable test.

To directly address your concerns regarding the well-formedness assumption and validation against gold-patch overfitting, we first clarify how TestJack's design resolves these two issues in principle, and then present four new human-expert verification studies covering 300 manually audited tasks and trials. We will incorporate all of these analyses and clarifications into the revised paper.


Clarifying the Two Core Soundness Concerns

  1. Witness verification (completeness vs. soundness):

    Static benchmark suites do not fail because humans or LLMs cannot tell whether one concrete test matches the prompt; they fail due to under-coverage (omission)[1,2]. Writing T0T_0 before seeing any submission requires completeness, anticipating every shortcut, ghost branch, or reward hack an agent might take across an open-ended implementation space[3,4]. By contrast, refuting a trial in TestJack requires soundness of just one concrete counterexample (Eq. 2): once a candidate witness test tt is found against a specific patch ss, verifying that tt checks an explicit clause of PP is a bounded, local check rather than an open-ended search[5,6].

  2. Prevent overfitting to GG:

    An LLM opinion alone never refutes a trial in TestJack. Every refutation must clear three independent gates (Eq. 2, §3.2):

    • Executable oracle check (t(R[G])=pass∧t(R[s])=failt(R[G]) = \textsf{pass} \land t(R[s]) = \textsf{fail}): Candidate tests are executed in the official harness against both the reference patch GG and trial ss. Any test with hallucinated expectations or broken setup fails on R[G]R[G] and is automatically discarded.
    • GG-blind prompt grounding (tt checks q∈Q(P)q \in \mathcal{Q}(P)): To prevent overfitting to incidental implementation choices in B(G)∖B(P)\mathcal{B}(G) \setminus \mathcal{B}(P) (e.g., private helpers or exact error strings), a separate reviewer agent re-judges each failing test without access to GG, treating PP as the sole authority and rejecting any test not required by PP.
    • Ambiguity filtering (Δ(P)\Delta(P) withheld from refutation): When a behavior implemented by GG is only implied or left ambiguous by PP, TestJack flags it as an underspecified prompt defect rather than refuting the trial (§3.2, Appendix A.2).

Responses to Specific Questions

Q1: Did you check whether the gold patches satisfy all explicit requirements in their task prompts?

Yes. Both benchmark provenance and our direct human inspection confirm that the gold patches reliably satisfy explicit prompt requirements:

  • Community and curator verification: In SWE-bench Verified and SWE-bench Pro, tasks are curated from merged GitHub pull requests and commits whose patches already underwent upstream maintainer review, followed by dedicated human screening during benchmark construction. In DeepSWE, SkillsBench, and SWE-Marathon, tasks and oracle solutions were co-designed from scratch by domain experts and verified prior to release.
  • Human-expert audit of 50 tasks: We randomly sampled 50 tasks across all five benchmarks (30 from DeepSWE, 3 from SkillsBench, 2 from SWE-Marathon, 7 from SWE-bench Pro, and 8 from SWE-bench Verified) and had human experts manually check each gold patch line-by-line against all explicit requirements in its prompt. We found 0 issues (0/50) where a gold patch missed or violated an explicit requirement.

Q2: Did you examine any patches rejected by the original tests to determine whether they were actually valid solutions?

Yes. Both benchmark construction and a new human-expert study confirm that patches rejected by T0T_0 are genuinely defective:

  • Careful design of original verifiers: Across these five benchmarks, T0T_0 consists either of maintainer-authored PR tests screened by human annotators (SWE-bench Verified and Pro) or hand-written, implementation-agnostic functional verifiers (DeepSWE, SkillsBench, and SWE-Marathon). While these static suites frequently leave required edge cases unchecked (B(T0)⊊B(P)\mathcal{B}(T_0) \subsetneq \mathcal{B}(P)), they rarely assert invalid expectations.
  • Human-expert audit of 100 rejected trials: We randomly sampled 100 trials that failed the original tests T0T_0 across all five benchmarks (60 from DeepSWE, 6 from SkillsBench, 5 from SWE-Marathon, 15 from SWE-bench Verified, and 14 from SWE-bench Pro) for manual expert review. All 100 out of 100 failed trials were confirmed to genuinely violate the prompt requirements; none (0/100) was a valid solution unfairly rejected by T0T_0.
AuthorsOpenReview ↗

Response to Reviewer jcvN (contd.)

Q3: How many of the 1,545 reported refutations were independently checked by human reviewers?

We independently audited 100 randomly sampled refuted trials manually from our per-trial auditing corpus (spanning all five benchmarks and diverse tasks), in addition to 50 randomly sampled refuted trials from our lightweight approach (reported under Q5).

Table R1: Human verification of randomly sampled refutations in per-trial auditing (RQ1).

Benchmark#Selected Trials#Sound#Incorrect#Overfit
DeepSWE706703
SkillsBench3300
SWE-Marathon2200
SWE-bench Pro131201
SWE-bench Verified121002
Total1009406

Here, each sampled refutation is categorized under a strict standard:

  • #Sound: The witness test is valid, strictly justified by an explicit requirement in the prompt, and exposes a genuine bug in the candidate patch.
  • #Incorrect: The witness test itself contains erroneous test logic, broken setup, or false expectations.
  • #Overfit: The witness test passes GG and fails ss, but enforces an implicit requirement or an unstated/gold-specific detail not explicitly written in the prompt.

As Table R1 shows, 94.0% (94/100) of the per-trial refutations are confirmed sound under strict explicit prompt requirements, 0% (0/100) involve broken or incorrect tests, and only 6.0% (6/100) test implicit or unstated details.

Q4: Did you test the generated tests on any independently verified correct implementations other than the gold patch?

We did not manually author a second set of reference patches from scratch. Across all the repository-level tasks involving multi-file changes in unfamiliar codebases, manually writing alternative implementations is itself error-prone, and certifying their correctness independently of the test suite would reintroduce the same verification circularity we aim to solve.

Instead, we checked the correctness of the gold patches as described in Q1. Moreover, TestJack prevents gold-patch overfitting without requiring a second hand-written reference patch through solution-blind review and cross-implementation validation across frontier models.

Q5: How many refutations remain when you count only tests justified by explicit requirements in the prompt?

Because distinguishing explicit from implicit requirements in natural-language specifications is a semantic judgment that cannot be purely machine-checked without relying on another LLM classifier, we evaluate explicit-only retention directly through human-expert verification on both the per-trial sample (Table R1 above) and a random sample of 50 refuted trials from the lightweight approach across the three RQ2 benchmarks (Table R2 below).

Table R2: Human verification of randomly sampled refutations in the lightweight approach (RQ2).

Benchmark#Selected Trials#Sound#Incorrect#Overfit
DeepSWE453807
SkillsBench2200
SWE-Marathon3201
Total504208

In both Table R1 and Table R2, #Sound counts only refutations justified by explicit prompt requirements, while any witness test relying on implicit requirements or non-requirements is counted as #Overfit:

  • Per-trial auditing (Table R1): 94.0% (94/100) of refutations remain when counting only tests justified by explicit prompt requirements. Scaling this rate across all 1,545 reported refutations yields ~1,452 explicit-requirement refutations, which still refutes ~32.4% of passing trials (vs. 34.4%) and lowers the overall resolution rate from 50.6% to ~34.2% (vs. 33.2%). Thus, our core empirical conclusions hold even under a strict explicit-only criterion.
  • Lightweight approach (Table R2): 84.0% (42/50) of refutations remain under explicit-only requirements, with 0% incorrect tests and 16.0% (8/50) reflecting implicit or underspecified behaviors. This directly corroborates our analysis in §4.3 and Appendix B.2 that lower agreement in the lightweight variant stems mainly from prompt underspecification rather than invalid test logic.
AuthorsOpenReview ↗

Response to Reviewer jcvN (contd.)

Q6: What model and reasoning configuration did you use for the reviewer agent?

We use GPT-5.5 with high reasoning effort for every agent in TestJack, including the Reviewer Agent, the Audit Agent, and the Task Analysis Agent. We will state this explicitly for all three agents in §4.1 in the revised paper.


References

[1] Liu, Jiawei, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. "Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation." Advances in neural information processing systems 36 (2023): 21558-21572.

[2] Aleithan, Reem, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin, and Song Wang. "Swe-bench+: Enhanced coding benchmark for llms." arXiv preprint arXiv:2410.06992 (2024).

[3] Skalse, Joar, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. "Defining and characterizing reward gaming." Advances in neural information processing systems 35 (2022): 9460-9471.

[4] Zhao, Bingchen, Dhruv Srikanth, Yuxiang Wu, and Zhengyao Jiang. "Specbench: Measuring reward hacking in long-horizon coding agents." arXiv preprint arXiv:2605.21384 (2026).

[5] McConnell, Ross M., Kurt Mehlhorn, Stefan Näher, and Pascal Schweitzer. "Certifying algorithms." Computer Science Review 5, no. 2 (2011): 119-161.

[6] Beyer, Dirk, Matthias Dangl, Daniel Dietsch, Matthias Heizmann, and Andreas Stahlbauer. "Witness validation and stepwise testification across software verifiers." In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, pp. 721-733. 2015.


We first share these responses here for your timely review. We are currently revising the paper and will upload it shortly.

Reviewer_rGfgOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TestJack uses differential testing to review coding-agent patches that pass a benchmark's original tests. An audit agent generates witness tests that fail on a candidate patch but pass on the reference solution, and a reviewer agent checks whether the exposed differences violate task requirements. Across five benchmarks, the paper reports refuting 1,545 of 4,487 originally passing trials. A lightweight variant combines task-level test augmentation with sampled audits and shares witness tests across trials for the same task. On three datasets, it recovers 72.4% of the per-trial audit's refutations at 15.4% of its estimated API cost.

Critical strengths
  1. Sections 3.1–3.2 clearly motivate TestJack by distinguishing task compliance from test passing and semantic validity from differential execution. These distinctions help explain why neither passing the original tests nor differing from the reference solution is sufficient to establish correctness or incorrectness.
  2. The SWE-ABS comparison and case studies illustrate how inspecting candidate patches can complement static test augmentation.
  3. The lightweight variant offers a useful tradeoff between cost and recovery of per-trial findings. Sharing witness tests across candidates reduces repeated test generation and could make broader auditing more affordable.
Critical weaknesses
  1. The claimed advance over PatchDiff depends on determining whether behavioral differences exposed by generated tests violate task requirements. This is the role of the reviewer agent in Figure 3, but the paper provides neither a reviewer ablation nor independent validation of its judgments. It is therefore difficult to assess this component's contribution and TestJack's advance over PatchDiff. A subset independently labeled by human reviewers would help assess the method's actual defect-detection accuracy.
  2. The SWE-ABS comparison does not establish better detection of genuine defects. Table 4's precision and recall measure agreement with TestJack's per-trial audit instead of independently labeled ground truth. Higher recall therefore indicates better recovery of TestJack's findings, while their correctness remains unverified. Both methods should be evaluated against the same ground-truth labels to compare the performance.
  3. The description of reviewer involvement in the lightweight method is inconsistent. Figure 3 routes the initial augmented tests generated by task analysis through the reviewer agent, but Appendix C.2 states that task-analysis tests are validated against the reference solution without a separate reviewer re-judgment.
  4. The released artifacts raise implementation and reproducibility concerns. The lightweight implementation counts successful discoveries rather than audit attempts toward its budget, but Section 4.3 explicitly states that the budget is finite. This affects the interpretation of the reported recovery of 72.4% of the per-trial audit's refutations at 15.4% of its cost. In addition, the promised experimental data and result-reconstruction materials are absent.
Questions for authors
  1. What fractions of generated differential tests does the reviewer agent label as valid, questionable, and invalid? How does this filtering affect the final number of refuted trials, and what changes when the reviewer is removed?
  2. Can you evaluate the reviewer agent on an independently human-labeled subset and compare TestJack with SWE-ABS or PatchDiff against the same human-labeled ground truth?
  3. Can you clarify the intended budget accounting, check whether the released implementation matches the experiments, and provide the experimental data and reconstruction materials promised in the reproducibility statement?
7 comments
AuthorsOpenReview ↗

Response to Reviewer rGfg

We sincerely thank the reviewer for the constructive feedback and for recognizing our theoretical motivation, the value of dynamic patch inspection over static test augmentation, and the practical cost-recovery tradeoff of cross-trial witness sharing. Below, we clarify the role of prompt-grounded auditing versus the reviewer agent, present new human-labeled validation and ablation experiments, and address the implementation and reproducibility questions.


W1: The claimed advance over PatchDiff depends on determining whether behavioral differences exposed by generated tests violate task requirements. This is the role of the reviewer agent in Figure 3...

We would like to clarify an important architectural distinction: in TestJack, determining whether behavioral differences violate task requirements is not solely the responsibility of the reviewer agent, but is enforced throughout our prompt-grounded generation and validation pipeline from the outset:

  1. Prompt-Grounded Audit Agent (Per-Trial Auditing, §3.2): PatchDiff applies LLM-based differential testing directly between the candidate patch and the ground-truth patch while paying minimal attention to the task prompt, causing many divergences to reflect incidental implementation choices rather than genuine bugs. In contrast, TestJack's audit agent treats the task prompt PP as the primary specification: it decomposes PP into an atomic requirements checklist and writes tests specifically targeting prompt clauses suspected to be violated. The ground-truth code GG is used strictly as an executable validation oracle (t(R[G])=passt(R[G]) = \text{pass}) to check test expectations and provide execution feedback when a candidate test or analysis is flawed.
  2. Task-Analysis Agent (Lightweight Approach, §3.3): Similarly, in Stage 1 of the lightweight approach, the task-analysis agent generates tests from the prompt PP, repository RR, and existing suite T0T_0, without access to the candidate patches, relying on an isolated runner to validate tests against R[G]R[G].
  3. Reviewer Agent as a Secondary Safeguard: Because the per-trial audit agent inspects R[G]R[G] during oracle validation (and LLMs can occasionally over-infer constraints), the reviewer agent acts as an independent final check in a fresh, solution-blind context (without access to GG). It evaluates candidate probes solely against PP and T0T_0 to remove residual errors or ground-truth overfitting introduced upstream.

W1 & Q1: What fractions of generated differential tests does the reviewer agent label as valid, questionable, and invalid? How does this filtering affect the final number of refuted trials, and what changes when the reviewer is removed?

Following our filtering rules (§3.2, Appendix D), TestJack retains only probes labeled VALID and rejects both QUESTIONABLE and INVALID probes (#Objected). Table R1 summarizes the reviewer agent's filtering statistics in the lightweight approach across DeepSWE, SkillsBench, and SWE-Marathon.

Table R1: Reviewer agent review counts and objection rates (QUESTIONABLE + INVALID) in the lightweight approach.

Benchmark#Reviewed#Valid (%)#Objected (%)
DeepSWE481392 (81.5%)89 (18.5%)
SkillsBench6256 (90.3%)6 (9.7%)
SWE-Marathon109 (90.0%)1 (10.0%)
Total553457 (82.6%)96 (17.4%)

Two findings stand out:

  • First, 82.6% (457/553) of generated probes are judged VALID, confirming that upstream prompt-grounding already aligns most generated tests with task requirements before the reviewer step.
  • Second, filtering out the remaining 17.4% (96/553) objected probes is critical, particularly in the lightweight approach where witness tests are shared across all trials of a task, meaning even a single unfiltered overfit test can falsely refute multiple trials in bulk.

To quantify what changes when the reviewer agent is removed, we conducted an ablation study without the reviewer agent and evaluated it on an independently human-checked subset of 50 trials (45 from DeepSWE, 2 from SkillsBench, and 3 from SWE-Marathon as shown in Table R2. #Sound counts trials refuted by valid, prompt-grounded witness tests; #Incorrect counts broken tests encoding wrong expectations (which is 0 because oracle validation on R[G]R[G] eliminates wrong test expectations); and #Overfit counts refutations caused by tests that pass R[G]R[G] but overfit to incidental ground-truth details outside the prompt requirements.

AuthorsOpenReview ↗

Response to Reviewer rGfg (contd.)

Table R2: Human verification of the lightweight approach without the reviewer agent on the 50 human-checked trials.

Benchmark#Selected Trials#Sound#Incorrect#Overfit
DeepSWE4528017
SkillsBench2101
SWE-Marathon3102
Total5030 (60.0%)0 (0.0%)20 (40.0%)

Comparing Table R2 (without reviewer) against TestJack - Lightweight in Table R3 (with reviewer) on the same 50 trials shows the reviewer agent's clear contribution: removing the reviewer causes overfit cases to jump from 8 (16.0%) to 20 (40.0%), while sound refutations drop from 42 (84.0%) to 30 (60.0%).


W1, W2 & Q2: Can you evaluate the reviewer agent on an independently human-labeled subset and compare TestJack with SWE-ABS or PatchDiff against the same human-labeled ground truth?

Yes. To evaluate defect-detection accuracy against independent human ground truth rather than inter-method agreement, we manually inspected the 50 benchmark-passing trials above, where each human-verified as genuinely defective ("incorrect despite passing"), and compared SWE-ABS, TestJack - Lightweight, and TestJack - Per-trial against the same human labels (Table R3). (We compare against SWE-ABS as the automated end-to-end baseline applicable across our multi-language, container-based benchmarks, whereas PatchDiff targets Python SWE-bench patches and relies on manual inspection to adjudicate divergences.)

In Table R3:

  • #Correct and Sound counts defective-despite-passing trials confirmed by the method via a sound, prompt-grounded witness test.
  • #Miss counts defective-despite-passing trials that the method fails to catch (they still pass all augmented tests).
  • #Overfit counts trials refuted only by overfit tests enforcing incidental details of GG not required by PP.

Table R3: Comparison between SWE-ABS and TestJack on the 50 independently human-checked defective trials.

Method#Correct and Sound#Miss#Overfit
SWE-ABS8 (16.0%)35 (70.0%)7 (14.0%)
TestJack - Lightweight42 (84.0%)0 (0.0%)8 (16.0%)
TestJack - Per-trial49 (98.0%)0 (0.0%)1 (2.0%)

These results establish three conclusions on human-labeled ground truth:

  1. High fidelity of Per-trial auditing: TestJack - Per-trial soundly confirms 49/50 (98.0%) defects with 0 misses and only 1 (2.0%) overfit case. This also validates using TestJack - Per-trial as a reliable reference for large-scale evaluation across all 2,931 trials in Table 4, where exhaustive human labeling is infeasible.
  2. Effectiveness of the Lightweight pipeline and Reviewer Agent: TestJack - Lightweight achieves 42/50 (84.0%) sound confirmations and 0 misses, cutting overfitting by 60% relative to the no-reviewer ablation (8 vs. 20 in Table R2).
  3. Substantial advance over SWE-ABS: On the exact same human ground truth, SWE-ABS confirms only 8/50 (16.0%) defects soundly, misses 35/50 (70.0%) defective trials, and overfits on 7/50 (14.0%). TestJack thus yields 5.25×5.25\times (Lightweight) to 6.1×6.1\times (Per-trial) more sound defect detections than SWE-ABS.

W3: The description of reviewer involvement in the lightweight method is inconsistent. Figure 3 routes the initial augmented tests generated by task analysis through the reviewer agent, but Appendix C.2 states that task-analysis tests are validated against the reference solution without a separate reviewer re-judgment.

Thank you for catching this inconsistency. Figure 3 (right) accurately reflects our actual lightweight pipeline and experiments: the initial augmented tests generated by the task-analysis agent are indeed routed through the reviewer agent before deduplication and merging. The sentence in the case study in Appendix C.2 was an outdated description from an earlier draft that was inadvertently left un-updated. We will correct Appendix C.2 and make Stage 1 reviewer filtering explicit in Section 3.3 in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer rGfg (contd.)

W4 & Q3: Can you clarify the intended budget accounting, check whether the released implementation matches the experiments, and provide the experimental data and reconstruction materials promised in the reproducibility statement?

Budget accounting and code-experiment consistency. We confirm that the released implementation matches our reported experiments. Specifically, the budget B=3B = 3 is not "successful test discoveries" broadly, nor total sampling attempts, but the count of confirmed-issue trial audits per task (i.e., sampled trials audited and confirmed as incorrect-despite-passing with valid witness tests). And this budget is more meaningful than just sample attempts because some samples may not inccur useful trial information. We will clarify the exact stopping criterion in §3.3, §4.3, and Algorithm 1 in the revised paper.

Experimental data and reconstruction materials. We have uploaded an anonymized subset of our experimental data at https://figshare.com/s/34bf53abfc3d20c26fae along with the result-reconstruction scripts at https://anonymous.4open.science/r/testjack-C1A7. Because the full raw dataset across all the audited trials (containing execution logs, generated probes, full trajectories, and workspace artifacts) is very large, we provide a representative sampled subset for immediate inspection, and we are actively working on releasing the full dataset.


We first share these responses here for your timely review. We are currently revising the paper and will upload it shortly.

Reviewer_S9X1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work aims to address the reliability of graders/verifiers in existing code benchmarks. The authors find that current UTs may overlook cases or requirements, letting incorrect or hacking solution to pass the tests. To address this, a prompt-grounded differential testing method is proposed to augment the test suite as well as randomized audit with cross-trial witness sharing. The experimental results reveal that ~20% solutions previously passing the tests are actually incorrect or hacking solutions.

Critical strengths
  1. The reliability of verifier/grader/tests is of significant importance. This work is a good reference for subsequent work.
  2. This work reveals there are substantial issues in the tests of existing coding benchmarks.
Critical weaknesses

According to Table 3, the per-trail results and those by the lightweight approach seem to be in low consistency.

How to ensure the TestJack augmented test suits is indeed more reliable and helpful? Is a further evaluation needed to be designed?

What are the main problems in model capabilities that are previously ignored by the unreliable UTs? Detailed and systematic analysis on this could help draw attention to overlooked issues and show differences in behavior patterns of different models.

The analysis of test cases in existing benchmarks could be further enhanced. Currently supp section C provides some case studies, it would be better if some typical categories could be abstracted and statistical analysis could be presented.

Comparison with UT generation methods could be added to show the pros and cons of the proposed TestJack method.

How can the TestJack be extended to more general agentic tasks, like the Claw, work or other scenarios?

Will TestJack be helpful for training scenarios? For example, if it is used to create more robust grader/verifier for RL training tasks, will the trained models be more capable?

Questions for authors

Can the proposed method adapted to multi-modal cases like web development?

7 comments
AuthorsOpenReview ↗

Response to Reviewer S9X1

We sincerely thank the reviewer for the constructive feedback and positive assessment, particularly for recognizing the importance of verifier reliability and our empirical finding that substantial flaws exist in current coding benchmark test suites. Below, we address each comment and present new human-verification results that further validate our framework.


W1: According to Table 3, the per-trail results and those by the lightweight approach seem to be in low consistency.

The apparent discrepancy reflects a trade-off between auditing cost and precision, rather than unreliable test generation. The lightweight approach reduces cost by auditing only a subset of trajectories. For underspecified prompts, the two approaches may therefore consider different evidence, and they also differ in how they carry witnesses from one trial to another. These design differences can lead to inconsistency in Table 3.

  1. Procedural difference: This explains the 72.4% recall at 84.6% lower cost. The per-trial procedure inspects every individual candidate patch ss and subjects its generated witnesses to per-trial scope validation by the reviewer agent. In contrast, the lightweight procedure builds a task-level augmented suite T1T_1 once and audits at most some randomly sampled surviving patches per task, replaying confirmed witnesses across all remaining trials. Recovering 72.4% of per-trial refutations at 15.4% of the API cost shows strong failure-mode overlap across trials. The remaining 27.6% reflects defects unique to unsampled trials when a task spans multiple distinct failure modes or has too few trials to amortize uniform sampling (e.g., SWE-Marathon with only 28 passing trials).
  2. Prompt underspecification + witness diffusion: This explains the 49.2% agreement precision and why #Lightweight > #Per-trial. In the lightweight setting, task-analysis tests or a witness found on one sampled patch diffuses across the entire surviving trial pool via witness sharing, flipping up to dozens of trials in a single task and inflating #Lightweight (1,863 vs. 1,265).

W2: How to ensure the TestJack augmented test suits is indeed more reliable and helpful? Is a further evaluation needed to be designed?

TestJack enforces a three-stage verification gate by design, these three-stage verification design enable TestJack to generate more reliable test suits. We now complement with an independent human audit of 150 refuted trials (Tables R1 and R2).

First, TestJack did not simply equates "the candidate differs from the reference" with "the candidate is wrong." Under Eq. (2) (§3.1–3.2), every per-trial refutation requires: (i) grounding the test in an atomic requirement q∈Q(P)q \in \mathcal{Q}(P) of the prompt, (ii) executable oracle certification (t(R[G])=passt(R[G]) = \text{pass}) inside the benchmark's official container, and (iii) candidate failure (t(R[s])=failt(R[s]) = \text{fail}) followed by solution-blind Reviewer Agent verification, which inspects only the agent-visible prompt PP, base repository, and T0T_0 (never GG) to reject solution-specific overfitting (exact error text, private helpers, internal ordering; Appendix D).

Second, to independently evaluate the reliability of TestJack's verdicts and generated test suites, we conducted a human verification study. We randomly sampled 100 refuted trials across diverse tasks from all 5 benchmarks in the per-trial setting, and 50 refuted trials across the 3 benchmarks in the lightweight setting. Human evaluators inspected each trial, prompt, reference patch, and witness test under a strict explicit-requirement standard, classifying each result into three mutually exclusive categories:

  • #Sound: The witness test and refutation are fully valid and check a behavior explicitly required by the task prompt (or established public contract).
  • #Incorrect: The generated witness test contains a buggy expectation that fails on the ground-truth reference implementation R[G]R[G]. (Because TestJack certifies every test on R[G]R[G], this is 0 across all samples.)
  • #Overfit: The witness test passes on R[G]R[G] and fails on R[s]R[s], but enforces behavior not explicitly mandated by the prompt (conservatively counting all prompt-underspecification cases as #Overfit).

Table R1: Human verification of 100 randomly sampled refutations in the Per-Trial Audit.

Benchmark#Selected Trials#Sound#Incorrect#Overfit
DeepSWE706703
SkillsBench3300
SWE-Marathon2200
SWE-bench Pro131201
SWE-bench Verified121002
Total10094 (94.0%)0 (0.0%)6 (6.0%)
AuthorsOpenReview ↗

Response to Reviewer S9X1 (contd.)

Table R2: Human verification of 50 randomly sampled refutations in the Lightweight Approach.

Benchmark#Selected Trials#Sound#Incorrect#Overfit
DeepSWE453807
SkillsBench2200
SWE-Marathon3201
Total5042 (84.0%)0 (0.0%)8 (16.0%)

Even under this conservative explicit-only criterion, TestJack achieves 94.0% precision in the per-trial setting and 84.0% in the lightweight setting, with 0% broken tests, confirming that TestJack's augmented suites reliably isolate genuine specification violations. We will add Tables R1–R2 and the evaluation protocol to the revised paper.


W3: What are the main problems in model capabilities that are previously ignored by the unreliable UTs? Detailed and systematic analysis on this could help draw attention to overlooked issues and show differences in behavior patterns of different models.

Based on our evaluation results, weak static tests mask six recurring capability gaps in frontier coding agents, spanning incomplete system integration, execution-feedback gaming, and compositional reasoning failures:

  1. Missing cross-component / end-to-end integration (Stub-with-tested-veneer): Agents implement the surface function exercised by unit tests while omitting downstream architectural layers explicitly mandated by the prompt. In navidrome-0130c6dc (gpt-5.6-sol, Appendix C.1, Table 7), the model implements in-memory image discovery (satisfying the sole scanner assertion, with only 6.7% line coverage of the diff) while omitting the explicitly required SQLite migration, repository persistence, and full-rescan trigger.
  2. Fixture-specific shortcuts & partial implementations: When T0T_0 uses minimal fixtures, agents satisfy the observable case via brittle shortcuts. In openlibrary-5069b09e (gpt-5.6-sol, Appendix C.1, Table 8), because the official fixture tests only a single collision with no non-conflicting records, returning a literal constant failed_deletes = 1 and rolling back the entire transaction passes T0T_0 while violating per-record conflict counting and partial updates.
  3. Superficial invariant satisfaction (execution-driven reward hacking): Interactive agents observe test failures and optimize directly against weak assertions. In DeepSWE wazero-multi-module-snapshots (Gemini-3.1-Pro, Figure 1), where T0T_0 checks only compressed stream length (len(compData) < len(baseComp)), the patch slices off the final byte (compData[:len(baseComp)-1]) with the comment // Just force it to be strictly smaller if possible, destroying gzip validity to pass the test.
  4. Aliasing and state-mutation edge cases: Agents handle independent literal inputs correctly but miss aliasing semantics—e.g., GPT-5.5 mutating array elements in place during negative-step slice assignment in abs-stepped-slices (Figure 1), breaking self-referential assignments (a[::-1] = a).
  5. Cross-feature interaction failures: Agents pass isolated feature tests while breaking invariants when features compose. In httpx-deterministic-cookie-store (claude-opus-4.7, Appendix C.2), cookie import and Secure enforcement each pass separately, yet importing cookies from a CookieJar routes through a public setter (self.set) that silently drops the Secure attribute; similarly, 9 trials handle CookieStore on the client constructor but crash on per-request _merge_cookies.
  6. Boundary and protocol-specification drift: Agents substitute loose heuristics for strict protocol rules—e.g., on the same HTTPX task (Appendix C.2), gpt-5.5 strips both leading and trailing dots (.lstrip(".").rstrip(".")), improperly accepting invalid trailing-dot cookie domains, while claude-opus-4.8 widens scheme checks to ("https", "wss") despite an explicit HTTPS-only rule.

Behavioral patterns across models: Across all six evaluated frontier models (Claude Opus 4.6/4.7/4.8, GPT-5.5/5.6, and Gemini 3.1 Pro), every model family exhibits false-positive passes because coding agents iteratively adapt to test execution feedback and often stop once the existing unit tests pass (§1). In our audited trials, these failures fall into two clear behavioral patterns: (i) shortcuts and partial implementations that satisfy weak test checks; and (ii) subtle edge-case and feature-interaction bugs inside otherwise complete solutions.

AuthorsOpenReview ↗

Response to Reviewer S9X1 (contd.)

W4: The analysis of test cases in existing benchmarks could be further enhanced. Currently supp section C provides some case studies, it would be better if some typical categories could be abstracted and statistical analysis could be presented.

We can abstract benchmark evaluator weaknesses into four principled categories grounded in our behavior-space formulation (B(G)∖B(T0)\mathcal{B}(G) \setminus \mathcal{B}(T_0), Eq. 3), supported by corpus-wide statistics from our audit:

  1. (C1) Unexercised Explicit Surface & Missing Layer Selection (Static Task-Visible Gaps): T0T_0 asserts only one entry point or omits existing repository integration tests (e.g., navidrome grading 1 of 6 required behaviors, Table 7; httpx omitting enumerated wrapper/per-request call sites, Appendix C.2). Statistically, our solution-blind TestJack-Task Analysis stage isolates this exact category before seeing any trial, refuting 345 / 2,931 (11.8%) passing trials and accounting for 13.4% of per-trial refutations (Table 4).
  2. (C2) Degenerate Fixtures & Proxy Oracles: T0T_0 executes the target code path but uses single-case fixtures (openlibrary, Table 8) or asserts proxy properties like output length rather than semantic validity (wazero, Figure 1).
  3. (C3) Missing Feature-Composition & Negative-Boundary Tests: T0T_0 checks features on disjoint happy paths without testing feature interactions or protocol rejection rules (httpx, Appendix C.2). Statistically, adding dynamic trial auditing to static task analysis boosts recall 5.4$\times$ (from 13.4% to 72.4%, refuting 63.6% of passing trials; Table 4), demonstrating quantitatively that most benchmark test blind spots (C2–C3) are implementation-dependent rather than statically obvious from PP and T0T_0 alone.
  4. (C4) Prompt Underspecification (Δ(P)\Delta(P)): GG implements unstated assumptions that T0T_0 leaves unchecked. Statistically, this affects 29/113 (25.7%) DeepSWE tasks, 3/13 (23.1%) SkillsBench tasks, and 2/2 SWE-Marathon tasks (34/128 = 26.6% overall; Table 6).

As task horizon and specification breadth grow, the untested behavior gap B(P)∖B(T0)\mathcal{B}(P) \setminus \mathcal{B}(T_0) expands faster than static test authoring can anticipate. We will consolidate this taxonomy and statistical synthesis into the main paper.


W5: Comparison with UT generation methods could be added to show the pros and cons of the proposed TestJack method.

Section 4.4 (Table 4) already provides a controlled head-to-head comparison against SWE-ABS [2], the state-of-the-art static coverage- and mutation-driven UT augmentation method.

  • Why SWE-ABS instead of UTBoost [1]: SWE-ABS (by the same research group [2]) directly extends and substantially outperforms UTBoost [1] by combining coverage-guided test generation with adversarial mutation killing. Moreover, UTBoost only supports Python (SWE-bench), whereas our evaluation spans multi-language repositories.
  • Summary of Pros and Cons (Static UT Generation vs. TestJack):
    • Static UT Generation (UTBoost, SWE-ABS): Pros: Zero trial-time LLM inference cost; produces a fixed suite once before seeing submissions. Cons: Must guess failure modes a priori, missing implementation-dependent bugs and adaptive reward hacking (9.9% vs. 72.4% recall); derives assertions from GG, risking overfitting to incidental gold-patch choices (B(G)∖B(P)\mathcal{B}(G) \setminus \mathcal{B}(P)).
    • TestJack: Pros: Dynamically targets actual agent failure modes and ghost branches; grounds tests in PP via a solution-blind Reviewer Agent while certifying oracle validity on GG (#Incorrect = 0); Lightweight amortizes audit cost by 84.6% via cross-trial witness sharing and yields a reusable augmented suite. Cons: Higher compute cost than purely static augmentation, and requires a reference implementation/oracle GG (Appendix E).

Comments on the paper

3 comments
Area_Chair_rYF9OpenReview ↗

Discussion

Dear Reviewers,

We are reaching the end of the discussion period! If the authors have responded to your review, please read their response, leave a comment, and update your rating if your assessment has changed.

Thank you,

AC.

AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their time, thoughtful critiques, and constructive feedback. Your insightful comments have helped us substantially strengthen both the empirical validation and presentation of our work. We have uploaded the revised manuscript, with all newly added and updated content highlighted in blue. Below, we summarize (1) the core contributions and novelty of our paper as recognized by the reviewers, and (2) the new experiments and revisions we conducted to address the reviewers' concerns.


1. Core Contributions, Novelty, and Reviewer Consensus

Our paper introduces TestJack, the first automated and dynamic benchmark auditing framework that evaluates agentic coding submissions via evaluator evolution rather than trusting static unit test suites fixed prior to benchmark release. Across the reviews, the reviewers consistently recognized the importance, conceptual clarity, and practical value of our contributions:

  • Exposing a fundamental limitation of static test-based evaluation on frontier benchmarks: As coding agents iteratively observe test feedback and adapt to fixed verifiers, passing T0T_0 no longer guarantees satisfying the task. Auditing 4,487 passing trials from 6 frontier model families across 5 recent benchmarks, TestJack reveals that 34.4% (1,545 / 4,487) of reported successes violate task requirements, lowering overall resolution from 50.6% to 33.2%. Reviewers S9X1 and jcvN emphasized that verifier reliability "is of significant importance" (S9X1), that our study "reveals there are substantial issues in the tests of existing coding benchmarks" and serves as "a good reference for subsequent work" (S9X1), and that our concrete examples "make a convincing case that existing evaluations can miss substantial defects" (jcvN).
  • Prompt-grounded differential testing backed by executable witness tests: Unlike static test augmentation (which must guess failure modes before seeing submissions) or plain differential testing (which conflates incidental gold-patch differences with bugs), TestJack separates what to test (prompt requirements Q(P)\mathcal{Q}(P)) from whether expectations are valid (executable oracle certification on R[G]R[G] plus solution-blind Reviewer Agent filtering). Every refutation is thus backed by a concrete, replayable witness test. Reviewer rGfg praised how our formulation clearly distinguishes "task compliance from test passing and semantic validity from differential execution" and demonstrates "how inspecting candidate patches can complement static test augmentation," while Reviewer jcvN highlighted that "each refutation comes with an executable test, making the findings easier to inspect and reproduce."
  • Scalable auditing via Lightweight Task Analysis and Cross-Trial Witness Sharing: By combining solution-blind task analysis with randomized trial auditing and replaying confirmed witnesses across all trials of a task, TestJack-Lightweight recovers 72.4% of per-trial refutations at 15.4% of the API cost (7.3×7.3\times the recall of SWE-ABS). Reviewer rGfg noted that this design "offers a useful tradeoff between cost and recovery of per-trial findings" and "could make broader auditing more affordable."
AuthorsOpenReview ↗

General Response (contd.)

2. Summary of New Experiments and Revisions Addressing Reviewer Concerns

To thoroughly address the reviewers' questions regarding independent ground-truth validation, component ablations, qualitative synthesis, and reproducibility, we added four human-expert verification studies (300 manually audited tasks and trials), Reviewer Agent ablations, corpus-wide taxonomies, and methodological clarifications:

  1. Independent Human-Expert Verification Across 300 Tasks & Trials (§4.6, Tables 5–6; addressing Reviewers S9X1, rGfg, and jcvN):

    • Validating the task well-formedness assumption (B(T0)⊆B(P)⊆B(G)\mathcal{B}(T_0) \subseteq \mathcal{B}(P) \subseteq \mathcal{B}(G), Reviewer jcvN): Human experts inspected 50 randomly sampled gold patches line-by-line against explicit prompt requirements across all 5 benchmarks and found 0/50 violations. Similarly, manual inspection of 100 T0T_0-rejected trials confirmed that 100/100 genuinely violated prompt requirements.
    • Refutation precision under strict explicit requirements (Reviewers S9X1, jcvN): Experts audited 100 per-trial and 50 lightweight refutations under a conservative explicit-only criterion. Executable oracle certification yields 0.0% broken tests across all 150 samples, with 94.0% (94/100) explicit precision in per-trial auditing and 84.0% (42/50) in lightweight auditing. Scaling 94.0% across all 1,545 RQ1 refutations retains $\sim$1,452 explicit violations.
    • Head-to-head comparison against SWE-ABS on human ground truth (Reviewers rGfg, S9X1): Evaluating directly on 50 independently human-verified defective-despite-passing trials (Table 6), TestJack-Per-trial soundly confirms 49/50 (98.0%) defects with 0 misses, and TestJack-Lightweight confirms 42/50 (84.0%) with 0 misses, whereas SWE-ABS catches only 8/50 (16.0%) and misses 70.
  2. Reviewer Agent Filtering Statistics, Ablation Study, and Defense Against Gold-Patch Overfitting (§3.2–§3.3, Appendix B.3, Tables 9–10; addressing Reviewers rGfg, jcvN, and S9X1):

    • Theoretical & architectural clarification (§3.2): We clarified why refuting a concrete patch turns open-ended test-suite completeness into a local counterexample soundness check, and how TestJack prevents PatchDiff-style overfitting to GG via three independent gates: upstream prompt-grounded generation from Q(P)\mathcal{Q}(P), executable oracle validation on R[G]R[G], and GG-blind Reviewer Agent filtering paired with ambiguity withholding.
    • Filtering statistics & w/o Reviewer ablation (Appendix B.3): Across the lightweight experiments, 82.6% (457/553) of generated probes pass the Reviewer Agent as VALID, while 17.4% (96/553) are filtered out (Table 9). Removing the Reviewer Agent on the 50 human-checked trials causes overfit refutations to jump from 16.0% (8/50) to 40.0% (20/50) and sound refutations to drop from 84.0% to 60.0% (Table 10), demonstrating that the solution-blind reviewer eliminates 60% of residual overfitting.
  3. Taxonomies of Benchmark Weaknesses, Agent Capability Gaps, and Broader Extensions (§4.2, §4.4, Appendix A.4, Appendix B.4; addressing Reviewer S9X1):

    • Evaluator & agent failure taxonomies (§4.2, Appendix B.4): We synthesized static benchmark blind spots into four categories (C1–C4) backed by corpus-wide statistics and abstracted six recurring capability gaps masked across all six frontier model families.
    • UT generation trade-offs & broader agentic/RL extensions (§4.4, Appendix A.4): We clarified why SWE-ABS extends UTBoost and summarized static-vs.-dynamic pros and cons (§4.4), and added Appendix A.4 detailing how trial-conditioned evaluator evolution extends to RLVR curriculum verifier co-evolution, general computer-use/workspace agents, and multi-modal web/UI development.
  4. Pipeline Consistency, Budget Accounting, and Reproducibility Artifacts (§3.3, §4.1, §4.3, Appendix A.3, Appendix C.2, Reproducibility Statement; addressing Reviewers rGfg, S9X1, and jcvN):

    • We aligned the description of solution-blind Stage 1 task analysis and reviewer filtering across §3.3, Algorithm 1, and Appendix C.2, and explained how cross-trial witness diffusion on underspecified tasks accounts for #Lightweight > #Per-trial in Table 3.
    • We explicitly defined budget as confirmed-issue trial audits, stated in §4.1 that all three agents use GPT-5.5 with high reasoning effort, and provided an anonymized subset of our experimental dataset alongside our result-reconstruction scripts in the Reproducibility Statement.

We thank the reviewers again for their constructive engagement!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.