- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper benchmarks whether coding agents can generate tests for a given atomic natural-language specification, using repository code as context. Agents generate test inputs and a reusable checker. Evaluation measures whether the resulting harness detects violations of the target specification without rejecting implementations that satisfy it.
Critical strengths
- The paper studies a practical verification problem: whether coding agents can translate a specified requirement into tests that detect violations without imposing additional requirements.
- The paper provides useful ablations and error analysis that help explain the observed failures.
Critical weaknesses
-
The evidence supporting data quality remains insufficient in two respects.
First, the semantic labels rely substantially on GPT-5.6-sol auditing candidates generated by Claude Opus 5. Majority voting over three calls may reduce response instability, but does not address systematic errors in interpreting the specification. The paper does not report voting disagreement statistics or establish whether voting improves agreement with human judgments. Robustness to the choice of judge also remains unclear.
Second, the human audit involves two authors reviewing 50 tasks. The reported agreement provides useful evidence, but the annotators’ relevant expertise, audit blinding, and disagreement cases are not sufficiently documented. More detailed audit statistics are needed to assess the reliability of the labels, particularly the validity of alternative implementations, which directly determines whether a rejection is counted as a false alarm.
-
The baseline coverage is too limited for a thorough evaluation of the benchmark. The main comparison includes only four coding-agent configurations. Broader coverage across model families and capability levels is needed to establish the benchmark’s difficulty and discriminative value, and to assess whether the reported failure patterns extend beyond the selected systems.
Questions for authors
- During dataset construction, GPT-5.6-sol is called three times to audit each candidate mutation. How often do these calls return different verdicts? On the human-audited sample, could you report agreement with the adjudicated human labels for each individual call and for the majority verdict?
- Please clarify the annotators’ relevant expertise and whether their initial judgments were blinded to the pipeline’s verdicts. Could you provide the underlying counts for the reported 94.7% agreement and 98% task-level confirmation, with separate results for target violations and alternative-implementation validity? Please also describe the disagreement cases and how they were adjudicated.
7 comments
W3 (Limited Baseline Coverage): The main comparison includes only four coding-agent configurations. Broader coverage across model families and capability levels is needed to establish the benchmark’s difficulty and discriminative value, and to assess whether the reported failure patterns extend beyond the selected systems.
Response on baseline coverage. We added four agent configurations, so the main comparison now covers eight agents from six model families. Two of these families now contribute both a larger and a smaller model. For a fair comparison, we chose models released in the same period as the four original ones, all before August 14, 2026. This is the release date of GLM-5.3, the most recently released of the four original models. The four new agents are:
- Codex with GPT-5.6-terra;
- Claude Code with Claude Sonnet 5;
- OpenCode with Kimi K3;
- OpenCode with Gemini 3.7 Flash.
Each new agent goes through the main experiment on all 291 tasks, the failure attribution of Section 4.3, and the violating-reference ablation of Section 4.4.1, with the same settings as in the paper.
Table R3. Results on the Full Set (291 tasks), in %. For ease of comparison, we also include the four original agents from Table 1 of the paper as the first four rows.
| Agent | Detection ↑ | False Alarm ↓ | Faithfulness ↑ | Edge Cov. ↑ |
|---|---|---|---|---|
| Codex / GPT-5.6-sol | 81.8 | 23.0 | 61.9 | 82.0 |
| Claude Code / Claude Opus 5 | 65.3 | 20.6 | 50.2 | 72.4 |
| OpenCode / DeepSeek-V4-Pro | 66.7 | 26.8 | 45.0 | 75.1 |
| OpenCode / GLM-5.3 | 74.2 | 15.1 | 62.9 | 84.3 |
| Codex / GPT-5.6-terra | 74.6 | 16.5 | 62.5 | 80.8 |
| Claude Code / Claude Sonnet 5 | 57.0 | 23.7 | 42.3 | 70.6 |
| OpenCode / Kimi K3 | 76.6 | 31.6 | 51.2 | 86.4 |
| OpenCode / Gemini 3.7 Flash | 57.4 | 17.2 | 45.7 | 74.5 |
Difficulty. None of the eight agents produces a faithful harness on more than 63% of the tasks, and faithfulness ranges from 42% to 63%.
Discriminative value. GPT-5.6-sol, GPT-5.6-terra, and GLM-5.3 are significantly more faithful than each of the other five agents, and Claude Opus 5 is significantly more faithful than Claude Sonnet 5. Faithfulness also separates agents that detection does not. Kimi K3, for example, detects about as often as GPT-5.6-sol but is 11 points less faithful. Prior test-generation evaluations, which validate tests only against a single correct implementation, would miss this gap.
Failure patterns. The patterns reported in Sections 4.2 to 4.4 carry over to the new agents, showing that our findings generalize beyond the originally selected systems.
- False alarms make up 34–66% of each agent's failures, and detection exceeds faithfulness by 11–25 points for every agent.
- Faithfulness generally rises with edge coverage, consistent with checks exercised through more thorough scenario exploration. The exception is Kimi K3, which reaches the highest edge coverage but also raises the most false alarms.
- The bottleneck still lies less in exploring test scenarios than in constructing checks. After we replace the generated scenarios with witness scenarios proven to expose the violation, most missed violations persist for every agent except the two GPT-5.6 models.
- As Table R4 shows, every agent has a clear tendency to mirror the reference implementation. When the reference violates the specification, each agent's faithfulness falls by 19–31 points, in the same two ways as for the original agents.
- Kimi K3, Claude Sonnet 5, and Gemini 3.7 Flash, like Claude Opus 5 and GLM-5.3, more often accept the violating reference, and their detection falls by 25–40 points.
- GPT-5.6-terra, like GPT-5.6-sol and DeepSeek-V4-Pro, keeps most of its detection but often rejects the gold implementation along with the violating one on 19 tasks. Its detection is still less robust than GPT-5.6-sol's, falling by 14 points rather than 5.
Table R4. Results on the Lite Set (100 tasks), gold reference → violating reference, in %. The gold-reference values are Table R3 restricted to the Lite Set. For ease of comparison, we also include the four original agents from Table 2 of the paper as the first four rows.
| Agent | Detection ↑ | False Alarm ↓ | Gold Rejection ↓ | Faithfulness ↑ |
|---|---|---|---|---|
| Codex / GPT-5.6-sol | 83 → 78 | 21 → 37 | 4 → 29 | 65 → 46 |
| Claude Code / Claude Opus 5 | 66 → 33 | 19 → 38 | 4 → 31 | 52 → 21 |
| OpenCode / DeepSeek-V4-Pro | 70 → 67 | 23 → 48 | 15 → 41 | 52 → 28 |
| OpenCode / GLM-5.3 | 75 → 44 | 12 → 24 | 0 → 12 | 68 → 39 |
| Codex / GPT-5.6-terra | 75 → 61 | 12 → 32 | 2 → 27 | 67 → 39 |
| Claude Code / Claude Sonnet 5 | 58 → 27 | 21 → 42 | 3 → 33 | 46 → 16 |
| OpenCode / Kimi K3 | 79 → 39 | 28 → 44 | 3 → 31 | 56 → 25 |
| OpenCode / Gemini 3.7 Flash | 54 → 29 | 12 → 27 | 3 → 18 | 47 → 19 |
We thank the reviewer for the valuable suggestions and will add these new results and analyses for the four agents to the revised version.
Thank you for the detailed responses and additional experiments. Your clarifications on voting disagreement and the human audit (Q1–Q2), together with the expanded eight-agent evaluation, address our concerns regarding annotation details and baseline coverage.
However, one concern from our first weakness remains unaddressed: robustness to the choice of judge. Dataset filtering relies solely on GPT-5.6-sol (which also belongs to one of the top-performing evaluated model families), and the response provides no evidence on whether the retained labels are robust to using an alternative judge from another model family (e.g., on the 50-task audited sample).
Finally, since the submission PDF has not yet been updated, please upload the revised manuscript incorporating the eight-agent results (including the Section 4.3 numbers for the four new agents) and the audit details during the discussion period so that we can review the changes before finalizing our assessment.
We sincerely thank Reviewer F1wZ for the valuable review and suggestions. Here are our responses to your questions:
The three calls return different verdicts in 6.9% of audits. On the human-audited sample, the individual calls agree with the adjudicated label on 150, 148, and 151 of the 152 judgments (98.7%, 97.4%, 99.3%), and the majority verdict agrees on 151 (99.3%).
We will add these statistics to the revised version.
Annotators’ expertise and blinded judgments. Both annotators received PhD-level training in computer science and have at least four years of Python development experience. Their initial judgments were independent and blinded to the pipeline's verdicts. Disagreements were resolved by discussion. Annotators with comparable backgrounds (VisCodex [1], NL2Repo-Bench [2], CoreCodeBench [3]), independent judgments by two or more annotators (CyberGym [4], CentaurEval [5], SWE-ABS [6]), and disagreements resolved by adjudication or discussion (VerifyBench [7], CoreCodeBench [3]) are widely adopted in recent benchmarks.
Underlying counts. The underlying counts for the reported 94.7% agreement are 144 of 152 judgments: 48 of 50 on target violations (96.0%) and 96 of 102 on alternative-implementation validity (94.1%). After adjudication, 49 of 50 target violations and all 102 alternative implementations were confirmed, so 49 of 50 tasks (98%) were confirmed at the task level.
Disagreement cases breakdown. The initial disagreement cases between two annotators are 8 judgments on 5 tasks: 2 on target violations, and 6 on alternative-implementation validity. All were adjudicated after the two annotators explained their readings of the cases to each other.
One disagreement case on alternative validity comes from a seaborn task.
Its target specification (S85) describes the
sizeparameter ofPlot.layout: “Size of the resulting figure, in inches. Size is inclusive of legend when using pyplot, but not otherwise.”The alternative violates a sibling specification (S89), which states the method "returns a copy of the Plot instance with modified layout settings".
Table R2. Annotators’ verdicts and reasons.
Adjudication: valid for S85. After discussion, Annotator 1 accepted Annotator 2's reading, and the alternative was adjudicated as valid, which aligns with the pipeline's three-to-none verdict.
We will add these details, including this disagreement case, to a new appendix in the revised version.
References:
[1] Jiang et al. VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Models. ICLR 2026.
[2] Ding et al. NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents. ICML 2026.
[3] Fu et al. CoreCodeBench: Decoupling Code Intelligence via Fine-Grained Repository-Level Tasks. ACL 2026.
[4] Wang et al. CyberGym: Evaluating AI Agents’ Real-World Cybersecurity Capabilities at Scale. ICLR 2026.
[5] Luo et al. CentaurEval: Benchmarking Human-in-the-Loop Value in Agentic Coding. ICML 2026.
[6] Yu et al. SWE-ABS: Adversarial Benchmark Strengthening Exposes Inflated Success Rates on Test-based Benchmark. ICML 2026.
[7] Yan et al. VerifyBench: Benchmarking Reference-based Reward Systems for Large Language Models. ICLR 2026.