acceptodds
Under review as a conference paper at ICLR 2027

SpecGuardBench: Can Coding Agents Construct Reliable Validation Harnesses from Natural Language Specifications?

Abstract

As coding agents become more capable, tests remain the primary evidence for judging whether their code is correct. Yet passing available tests does not establish that an implementation satisfies its natural language specification. Reliable validation requires checks that faithfully capture the specification and scenarios that thoroughly exercise the code, but current benchmarks provide limited insight into whether agents can construct such evidence themselves. We introduce SpecGuardBench, a benchmark of 291 tasks from 13 real-world repositories, covering 8 types of verification logic such as functional postconditions and interaction protocols. Given a natural language specification, repository context, and a reference implementation, an agent constructs a validation harness comprising executable checks and test scenarios. The checks should reject implementations that violate the specification while accepting those that satisfy it, and the scenarios should thoroughly exercise the code. Across four proprietary and open-weight coding agents, none constructs a harness scored as faithful on the benchmark's implementations for more than 63% of the tasks. The bottleneck lies less in exploring test scenarios than in constructing checks that faithfully reflect the specification. Specifically, the agents show a clear tendency to write checks that mirror the reference implementation rather than the specification, and therefore reject implementations that the specification permits.

Primary area:
datasets and benchmarks
Keywords:
Large Language Model, Coding Agent, Benchmark, Code Validation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_F1wZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper benchmarks whether coding agents can generate tests for a given atomic natural-language specification, using repository code as context. Agents generate test inputs and a reusable checker. Evaluation measures whether the resulting harness detects violations of the target specification without rejecting implementations that satisfy it.

Critical strengths
  1. The paper studies a practical verification problem: whether coding agents can translate a specified requirement into tests that detect violations without imposing additional requirements.
  2. The paper provides useful ablations and error analysis that help explain the observed failures.
Critical weaknesses
  1. The evidence supporting data quality remains insufficient in two respects.

    First, the semantic labels rely substantially on GPT-5.6-sol auditing candidates generated by Claude Opus 5. Majority voting over three calls may reduce response instability, but does not address systematic errors in interpreting the specification. The paper does not report voting disagreement statistics or establish whether voting improves agreement with human judgments. Robustness to the choice of judge also remains unclear.

    Second, the human audit involves two authors reviewing 50 tasks. The reported agreement provides useful evidence, but the annotators’ relevant expertise, audit blinding, and disagreement cases are not sufficiently documented. More detailed audit statistics are needed to assess the reliability of the labels, particularly the validity of alternative implementations, which directly determines whether a rejection is counted as a false alarm.

  2. The baseline coverage is too limited for a thorough evaluation of the benchmark. The main comparison includes only four coding-agent configurations. Broader coverage across model families and capability levels is needed to establish the benchmark’s difficulty and discriminative value, and to assess whether the reported failure patterns extend beyond the selected systems.

Questions for authors
  1. During dataset construction, GPT-5.6-sol is called three times to audit each candidate mutation. How often do these calls return different verdicts? On the human-audited sample, could you report agreement with the adjudicated human labels for each individual call and for the majority verdict?
  2. Please clarify the annotators’ relevant expertise and whether their initial judgments were blinded to the pipeline’s verdicts. Could you provide the underlying counts for the reported 94.7% agreement and 98% task-level confirmation, with separate results for target violations and alternative-implementation validity? Please also describe the disagreement cases and how they were adjudicated.
7 comments

We sincerely thank Reviewer F1wZ for the valuable review and suggestions. Here are our responses to your questions:

Q1 (Disagreement Statistics): During dataset construction, GPT-5.6-sol is called three times to audit each candidate mutation. How often do these calls return different verdicts? On the human-audited sample, could you report agreement with the adjudicated human labels for each individual call and for the majority verdict?

The three calls return different verdicts in 6.9% of audits. On the human-audited sample, the individual calls agree with the adjudicated label on 150, 148, and 151 of the 152 judgments (98.7%, 97.4%, 99.3%), and the majority verdict agrees on 151 (99.3%).

We will add these statistics to the revised version.


Q2 (Details in Human Annotation): Please clarify the annotators’ relevant expertise and whether their initial judgments were blinded to the pipeline’s verdicts. Could you provide the underlying counts for the reported 94.7% agreement and 98% task-level confirmation, with separate results for target violations and alternative-implementation validity? Please also describe the disagreement cases and how they were adjudicated.

Annotators’ expertise and blinded judgments. Both annotators received PhD-level training in computer science and have at least four years of Python development experience. Their initial judgments were independent and blinded to the pipeline's verdicts. Disagreements were resolved by discussion. Annotators with comparable backgrounds (VisCodex [1], NL2Repo-Bench [2], CoreCodeBench [3]), independent judgments by two or more annotators (CyberGym [4], CentaurEval [5], SWE-ABS [6]), and disagreements resolved by adjudication or discussion (VerifyBench [7], CoreCodeBench [3]) are widely adopted in recent benchmarks.

Underlying counts. The underlying counts for the reported 94.7% agreement are 144 of 152 judgments: 48 of 50 on target violations (96.0%) and 96 of 102 on alternative-implementation validity (94.1%). After adjudication, 49 of 50 target violations and all 102 alternative implementations were confirmed, so 49 of 50 tasks (98%) were confirmed at the task level.

Disagreement cases breakdown. The initial disagreement cases between two annotators are 8 judgments on 5 tasks: 2 on target violations, and 6 on alternative-implementation validity. All were adjudicated after the two annotators explained their readings of the cases to each other.

One disagreement case on alternative validity comes from a seaborn task.

  • Its target specification (S85) describes the size parameter of Plot.layout: “Size of the resulting figure, in inches. Size is inclusive of legend when using pyplot, but not otherwise.”

  • The alternative violates a sibling specification (S89), which states the method "returns a copy of the Plot instance with modified layout settings".

  • Table R2. Annotators’ verdicts and reasons.

    AnnotatorVerdictReason
    Annotator 1InvalidThe alternative writes the new settings, including the size, into the original plot before copying it, so the original is modified as well.
    Annotator 2ValidThe alternative satisfies S85, which does not require preserving the original plot.
  • Adjudication: valid for S85. After discussion, Annotator 1 accepted Annotator 2's reading, and the alternative was adjudicated as valid, which aligns with the pipeline's three-to-none verdict.

We will add these details, including this disagreement case, to a new appendix in the revised version.

References:

[1] Jiang et al. VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Models. ICLR 2026.

[2] Ding et al. NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents. ICML 2026.

[3] Fu et al. CoreCodeBench: Decoupling Code Intelligence via Fine-Grained Repository-Level Tasks. ACL 2026.

[4] Wang et al. CyberGym: Evaluating AI Agents’ Real-World Cybersecurity Capabilities at Scale. ICLR 2026.

[5] Luo et al. CentaurEval: Benchmarking Human-in-the-Loop Value in Agentic Coding. ICML 2026.

[6] Yu et al. SWE-ABS: Adversarial Benchmark Strengthening Exposes Inflated Success Rates on Test-based Benchmark. ICML 2026.

[7] Yan et al. VerifyBench: Benchmarking Reference-based Reward Systems for Large Language Models. ICLR 2026.

W3 (Limited Baseline Coverage): The main comparison includes only four coding-agent configurations. Broader coverage across model families and capability levels is needed to establish the benchmark’s difficulty and discriminative value, and to assess whether the reported failure patterns extend beyond the selected systems.

Response on baseline coverage. We added four agent configurations, so the main comparison now covers eight agents from six model families. Two of these families now contribute both a larger and a smaller model. For a fair comparison, we chose models released in the same period as the four original ones, all before August 14, 2026. This is the release date of GLM-5.3, the most recently released of the four original models. The four new agents are:

  • Codex with GPT-5.6-terra;
  • Claude Code with Claude Sonnet 5;
  • OpenCode with Kimi K3;
  • OpenCode with Gemini 3.7 Flash.

Each new agent goes through the main experiment on all 291 tasks, the failure attribution of Section 4.3, and the violating-reference ablation of Section 4.4.1, with the same settings as in the paper.

Table R3. Results on the Full Set (291 tasks), in %. For ease of comparison, we also include the four original agents from Table 1 of the paper as the first four rows.

AgentDetection ↑False Alarm ↓Faithfulness ↑Edge Cov. ↑
Codex / GPT-5.6-sol81.823.061.982.0
Claude Code / Claude Opus 565.320.650.272.4
OpenCode / DeepSeek-V4-Pro66.726.845.075.1
OpenCode / GLM-5.374.215.162.984.3
Codex / GPT-5.6-terra74.616.562.580.8
Claude Code / Claude Sonnet 557.023.742.370.6
OpenCode / Kimi K376.631.651.286.4
OpenCode / Gemini 3.7 Flash57.417.245.774.5

Difficulty. None of the eight agents produces a faithful harness on more than 63% of the tasks, and faithfulness ranges from 42% to 63%.

Discriminative value. GPT-5.6-sol, GPT-5.6-terra, and GLM-5.3 are significantly more faithful than each of the other five agents, and Claude Opus 5 is significantly more faithful than Claude Sonnet 5. Faithfulness also separates agents that detection does not. Kimi K3, for example, detects about as often as GPT-5.6-sol but is 11 points less faithful. Prior test-generation evaluations, which validate tests only against a single correct implementation, would miss this gap.

Failure patterns. The patterns reported in Sections 4.2 to 4.4 carry over to the new agents, showing that our findings generalize beyond the originally selected systems.

  • False alarms make up 34–66% of each agent's failures, and detection exceeds faithfulness by 11–25 points for every agent.
  • Faithfulness generally rises with edge coverage, consistent with checks exercised through more thorough scenario exploration. The exception is Kimi K3, which reaches the highest edge coverage but also raises the most false alarms.
  • The bottleneck still lies less in exploring test scenarios than in constructing checks. After we replace the generated scenarios with witness scenarios proven to expose the violation, most missed violations persist for every agent except the two GPT-5.6 models.
  • As Table R4 shows, every agent has a clear tendency to mirror the reference implementation. When the reference violates the specification, each agent's faithfulness falls by 19–31 points, in the same two ways as for the original agents.
    • Kimi K3, Claude Sonnet 5, and Gemini 3.7 Flash, like Claude Opus 5 and GLM-5.3, more often accept the violating reference, and their detection falls by 25–40 points.
    • GPT-5.6-terra, like GPT-5.6-sol and DeepSeek-V4-Pro, keeps most of its detection but often rejects the gold implementation along with the violating one on 19 tasks. Its detection is still less robust than GPT-5.6-sol's, falling by 14 points rather than 5.

Table R4. Results on the Lite Set (100 tasks), gold reference → violating reference, in %. The gold-reference values are Table R3 restricted to the Lite Set. For ease of comparison, we also include the four original agents from Table 2 of the paper as the first four rows.

AgentDetection ↑False Alarm ↓Gold Rejection ↓Faithfulness ↑
Codex / GPT-5.6-sol83 → 7821 → 374 → 2965 → 46
Claude Code / Claude Opus 566 → 3319 → 384 → 3152 → 21
OpenCode / DeepSeek-V4-Pro70 → 6723 → 4815 → 4152 → 28
OpenCode / GLM-5.375 → 4412 → 240 → 1268 → 39
Codex / GPT-5.6-terra75 → 6112 → 322 → 2767 → 39
Claude Code / Claude Sonnet 558 → 2721 → 423 → 3346 → 16
OpenCode / Kimi K379 → 3928 → 443 → 3156 → 25
OpenCode / Gemini 3.7 Flash54 → 2912 → 273 → 1847 → 19

We thank the reviewer for the valuable suggestions and will add these new results and analyses for the four agents to the revised version.

Thank you for the detailed responses and additional experiments. Your clarifications on voting disagreement and the human audit (Q1–Q2), together with the expanded eight-agent evaluation, address our concerns regarding annotation details and baseline coverage.

However, one concern from our first weakness remains unaddressed: robustness to the choice of judge. Dataset filtering relies solely on GPT-5.6-sol (which also belongs to one of the top-performing evaluated model families), and the response provides no evidence on whether the retained labels are robust to using an alternative judge from another model family (e.g., on the 50-task audited sample).

Finally, since the submission PDF has not yet been updated, please upload the revised manuscript incorporating the eight-agent results (including the Section 4.3 numbers for the four new agents) and the audit details during the discussion period so that we can review the changes before finalizing our assessment.

Reviewer_tf6mOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SpecGuardBench, a benchmark evaluating the ability of coding agents to construct tests given a repository context, a reference implementation, and a natural language specification about what a passing solution should (and should not) do. Authors show that both proprietary and open source frontier coding agents struggle to construct valid test cases for all tasks across a curated set of metrics.

Critical strengths

One of the biggest impediments to recursive self improvement (RSI) approaches is the lack of quality evaluations, which impact models' ability to receive verifiable feedback that's also high quality. Good evaluations are also important in the prevention and reduction of reward hacking. Teaching models to be better at coming up with evaluations given natural language specifications is therefore highly impactful to help accelerate the RSI loop, which is a challenge addressed by SpecGuardBench.

Critical weaknesses

All the experiments in the paper are limited to 80 steps (model calls) by coding agents. The scaling experiments in Figure 6 shows how the scores of different coding agents change when the number of steps are increased from 30 to 50 to 80. Although this is informative, it's unclear how the coding agents would perform if they aren't subjected to a hard number of steps and are instead set to as long as possible, within a particular wall clock duration.

Questions for authors

I found this work to be quite timely and insightful. My one concern is with respect to the 80 steps limit imposed on the coding agents. This sounds rather arbitrary and at best too few to test the capabilities of coding agents in this task. How long does 80 steps take in terms of wall clock time? Could it be possible for an agent to achieve much better results when run for several hours?

3 comments

We sincerely thank Reviewer tf6m for the valuable review and suggestions. Here are our responses to your questions:

Q1 (Step Budget Justification): My one concern is with respect to the 80 steps limit imposed on the coding agents. This sounds rather arbitrary and at best too few to test the capabilities of coding agents in this task. How long does 80 steps take in terms of wall clock time?

The reason why we set the budget to 80 steps. We empirically set the budget at 80 steps after preliminary runs. The agent is instructed to cover as many program edges as possible, so an agent that stops with steps left does so by its own choice. During the preliminary runs, we found that most of the runs ended within 50 steps. Therefore, we set the step budget to 80, adding a 60% margin.

The budget rarely binds. Across the 291 tasks, GPT-5.6-sol, Claude Opus 5, and DeepSeek-V4-Pro end their sessions within 60 steps in 95%, 89%, and 83% of runs, respectively, leaving at least a quarter of the budget unused. GLM-5.3 uses more steps and hits the budget in 24% of tasks. These runs take 40-61 minutes of wall clock time and achieve a faithfulness rate of 59.4%, close to GLM-5.3's overall rate of 62.9%. Therefore, the 80-step budget rarely binds.


Q2 (Rerun without Step Limit): it's unclear how the coding agents would perform if they aren't subjected to a hard number of steps and are instead set to as long as possible, within a particular wall clock duration. Could it be possible for an agent to achieve much better results when run for several hours?

Removing step limits yields only modest gains. Most runs for agents other than GLM-5.3 already end with at least a quarter of the budget unused. Nevertheless, to test whether longer runs could yield much better results, we reran the tasks that the budget stopped and that were unfaithful, with no step limit or wall clock limit. As shown in Table R1, GLM-5.3 has the largest gain of 3.8 points, while the other agents improve by at most 0.4 points. The longest rerun used 229 steps and took 65 minutes, and it remained unfaithful.

Table R1: Results of rerunning budget-limited, unfaithful tasks.

AgentStopped by the budget and unfaithfulunfaithful → faithfulFaithfulness: 80-step budget → no limit
Codex / GPT-5.6-sol0--
Claude Code / Claude Opus 51 (0.3%)050.2% → 50.2%
OpenCode / DeepSeek-V4-Pro4 (1.4%)145.0% → 45.4%
OpenCode / GLM-5.328 (9.6%)1162.9% → 66.7%

We thank the reviewer again for the valuable suggestion. We will add Table R1 to the appendix and summarize the results in Section 4.4.2.

Thank you for the clarifications on the step limit and the budget. It's also great to see that you have tested a longer run. One exciting extension of the work could be applying SpecGuardBench on top of an active open source software and prove that this approach can have impact on actively maintained software. I had already rated the paper as an accept, so I'm keeping the judgment as is.

Thank you for recognizing the value of our work and for your thoughtful, encouraging, and forward-looking feedback. We particularly appreciate your suggestion of applying SpecGuardBench to actively maintained open-source software. We are already exploring how to integrate insights from SpecGuardBench into both open-source agent harnesses and products developed by startups, with the goal of better validating AI-driven software development. And we are excited about the potential of this direction to help realize a practical RSI loop and make a broader real-world impact.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Reviewers, AC, and SAC,

Thank you once again for your time and constructive feedback. We are encouraged that the reviewers recognized the timeliness and significance of this research for the RSI loop, the practical importance of specification-faithful code validation, and the value of our benchmark design, ablations, and failure analyses.

As the discussion period draws to a close in 2 hours, we would like to briefly summarize our responses and additional efforts.

1. Step Budget and Longer Runs (tf6m Q1–Q2)

  • Unlimited reruns: We clarified the rationale behind the 80-step budget and conducted additional experiments without step or wall-clock limits. The reviewer appreciated these clarifications and additional experiments, and confirmed that they would maintain their accept rating.

2. Benchmark Label Reliability (F1wZ W1, Q1–Q2)

  • LLM judge reliability and human audit: We added voting disagreement statistics, agreement with human judgments, annotator expertise and blinding details, underlying agreement counts, and representative disagreement cases. The reviewer acknowledged that these additions addressed the concerns regarding annotation details.

  • Robustness to judge choice: Following the reviewer's suggestion, we additionally evaluated Kimi K3 as an alternative judge from another model family, demonstrating the robustness of the retained labels. Unfortunately, as of this summary, we have not yet received the reviewer's feedback on this additional analysis.

3. Expanded Baseline Coverage (F1wZ W2)

  • Four additional agents: We expanded the evaluation from four to eight agents across six model families, adding GPT-5.6-terra, Claude Sonnet 5, Kimi K3, and Gemini 3.7 Flash. We also extended the failure attribution and violating-reference ablation analyses to these agents. The reviewer acknowledged that this addressed the concern regarding baseline coverage.

We have also updated the manuscript to incorporate the additional experiments, analyses, and audit details.

We sincerely appreciate the reviewers' thoughtful, forward-looking, and encouraging feedback and hope these clarifications help with the final assessment.

The Authors of SpecGuardBench

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.