acceptodds
Under review as a conference paper at ICLR 2027

ReqAuditBench: Benchmarking Coding Agents on Proactive Discovery of Requirement–Code Inconsistency

Abstract

Coding agents increasingly write code for software projects, but checking whether the code is consistent with documentation is left to human reviewers. Existing benchmarks do not test whether agents can take over this requirement–code con- sistency audit. Implementation benchmarks do the check for the agent with pre- pared tests, and consistency checkers are given the function to examine. In a real repository, the documentation states many requirements and marks none as broken, so the auditor must first decide which promise to check. We introduce ReqAuditBench to measure this proactive investigation. Each task gives an agent a repository and a region of its documentation that contains at least one violated requirement, without saying which one. We build tasks from pull requests that update both a documented requirement and its implementation. We place the up- dated documentation into the old repository, where nothing marks which passage changed. On 109 tasks with 134 requirements from 9 Python repositories, eight coding agents reach only 35.1–57.5% requirement recall, the share of violated re- quirements they report. Seeing a requirement is often not enough. For 60.3–92.6% of missed requirements, the promise had already appeared in the agent’s tool out- put. In case studies, agents stop before checking the code behind the promise. The results suggest that coding agents are not yet ready to take over this audit.

Primary area:
datasets and benchmarks
Keywords:
requirement code consistency, software engineering

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_JZKnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ReqAuditBench, a benchmark for evaluating whether coding agents can discover inconsistencies between a repository documentation and implementation without being told which requirement is violated or where it is implemented. The benchmark is constructed from merged pull requests by placing updated documentation into the corresponding pre-change repository while preserving the original implementation. It contains 109 tasks with 134 violated requirements across nine Python repositories. The authors evaluate eight coding agents, report requirement recall ranging from 35.1% to 57.5%, and analyze their investigation trajectories and failure modes.

Critical strengths

S1. The paper addresses an important and practical problem.

Checking whether an implementation satisfies its documented requirements is an important software maintenance task. In practice, developers may need to investigate a repository without knowing which requirement is violated or where the relevant implementation resides. Evaluating agents in this setting is practically relevant to automated code auditing.

S2. The paper provides substantial experimental analysis.

Beyond aggregate performance, the paper examines investigation trajectories, exposure to violated requirements, documentation-reading behavior, and selected failure cases. These analyses provide useful observations about how agents select requirements for investigation and where their verification processes stop. The comparison of recall, recorded duration, and estimated token fees also provides a practical perspective on the evaluated agents.

S3. The paper is clearly written.

The paper clearly explains the task definition, benchmark construction pipeline, and scoring procedure. The distinction between the information available to the evaluated agents and the reference evidence available to the evaluator is particularly helpful for understanding the benchmark.

Critical weaknesses

W1. The novelty of both the problem formulation and benchmark construction is limited.

The underlying problem has substantial overlap with prior work. RFCAudit [1] and Spec2Code [2] already investigate inconsistencies between natural-language specifications and implementations, including identifying the relevant implementation from a broader codebase. The paper should explicitly compare its task formulation with these works and explain which additional research challenges ReqAuditBench introduces.

The benchmark construction methodology also shares an important principle with TestExplora [3]. Both exploit historical code changes to pair an earlier implementation with documentation representing intended behavior, while withholding the change information that would reveal the target. ReqAuditBench differs in its documentation source, task definition, and expected output, but the paper does not sufficiently establish why these differences constitute a substantial methodological contribution.

Therefore, the overlap in problem formulation and construction methodology leaves the incremental novelty insufficiently justified.

W2. The contribution’s positioning and longer-term value require stronger justification.

The paper would benefit from a clearer explanation of what the benchmark is intended to enable as agent capabilities improve.

If its primary contribution is to characterize current coding-agent capabilities, broader experiments with different prompts and investigation strategies are needed to establish how stable the reported performance gaps and behavioral observations are. A fixed configuration provides limited evidence about the capability boundaries of the underlying models and harnesses.

If the construction methodology is intended to support continued benchmark expansion or generate data for subsequent training and improvement, its scalability needs empirical support. The current pipeline requires two independent human reviewers to examine every reference card, with a third reviewer resolving disagreements. Although the paper states that the collection procedure can be reused for later changes and additional repositories, it does not report construction token usage, human review time, or cost per accepted task.

The reported token fees concern evaluated agent runs and do not answer these construction-cost questions. Therefore, the paper has not yet established whether the methodology can sustainably produce substantially more tasks or support a scalable data-generation workflow.

W3. Key experimental settings are insufficiently justified, and their effects on the conclusions are not evaluated.

The evaluation uses a single underlying prompt strategy, with separate templates for documentation pages and docstrings. The prompt contains detailed instructions governing evidence collection, reporting, confidence ranking, and stopping behavior. However, the paper does not investigate whether alternative prompts or investigation strategies would substantially change performance, agent rankings, or the observed failure modes.

The five-finding cap is another consequential choice that lacks sufficient justification. This issue is particularly relevant because the reference requirements do not exhaustively cover all inconsistencies in a task region. An agent could report genuine inconsistencies outside the reference set, exhaust its reporting allowance, and stop before investigating a registered target. The paper does not quantify this possibility or evaluate sensitivity to the reporting cap.

Consequently, the results characterize the selected configurations, while their implications for the broader capabilities of the evaluated models and harnesses remain uncertain. The scoring-judge sensitivity analysis does not resolve this concern, since it examines grading robustness rather than the robustness of agent behavior.

Questions for authors

Q1. What is the principal distinction from RFCAudit [1], Spec2Code [2], and TestExplora [3]?

Please distinguish the novelty of the auditing problem from that of the benchmark construction methodology. Which challenges or evaluation capabilities are introduced beyond those already addressed by these works?

Q2. How sensitive are the results to the prompt strategy and five-finding cap?

Would alternative prompts, requirement-selection strategies, or reporting limits materially change recall, agent rankings, and the reported failure patterns? How often do agents reach the cap while reporting valid inconsistencies outside the reference set?

Q3. What resources are required to construct and extend the benchmark?

Please report the LLM token usage, human review effort, candidate acceptance rate, and approximate cost per accepted task. Which stages require human intervention, and what evidence supports extending the pipeline to substantially more repositories and tasks?

References

[1] Mingwei Zheng, Chengpeng Wang, Xuwei Liu, Jinyao Guo, Shiwei Feng, and Xiangyu Zhang. RFCAudit: AI Agent for Auditing Protocol Implementations Against RFC Specifications. ASE, 2025. Paper.

[2] Yuekun Wang, Lili Quan, Xiaofei Xie, Junjie Wang, and Jianjun Chen. Spec2Code: Mapping Protocol Specification to Function-Level Code Implementation. ASE, 2025. Paper.

[3] Steven Liu et al. TestExplora: Benchmarking LLMs for Proactive Bug Discovery via Repository-Level Test Generation. ICML, 2026. Paper.

Reviewer_GLSVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents a new benchmark where models are evaluated on finding documented behavior violations in a repository. The task is practical and important, given that not violating requirements is part of successfully solving/reviewing SWE tasks. The authors evaluate frontier models of various sizes on the benchmark, find capability gaps, present the recall/speed/cost frontier of the benchmarked models and analyze why the models fail when they do.

Critical strengths
  1. The benchmark challenges frontier models.
  2. The benchmark is constructed thoughtfully: the data is natural, many more tasks can be built from the available open-source code, response evaluation was designed to prevent cheating, the tasks were human-audited and may fill an important gap in existing benchmarks.
  3. The models are evaluated well and the paper analyzes the reasons behind the failures it observes: the models may observe the requirement and fail to detect a violation, possibly by failing to observe the code sufficiently.
Critical weaknesses
  1. The paper does not present a precision score and does not discuss the perfect pass rate. Both measures matter for assessing how reliably models can detect behavior violations.
  2. The benchmark is not compared to Frontier Code and DeepSWE, two SWE benchmarks which are meant to reflect real-world concerns and are used to evaluate frontier models.
Questions for authors
  • In the text of Sec 2.3, please describe what does the actual task contain. I had to hunt for this information and eventually find it in Figure 2.
  • L200. Please describe the "reference card" contents immediately after introducing the concept.
  • Table 2. What is a "recovered" requirement?
  • L395. "On the 76 tasks with a recorded run from every agent." Why does the analyzed dataset miss some agent/task pairs?
  • The reproducibility statement is missing. I assume the authors plan to ensure the results are reproducible and release the benchmark and the evaluation scripts.
Reviewer_fECgOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The benchmark asks a coding agent to audit a stretch of a project's documentation against its code and report which documented promises the code breaks, without being told that anything is broken or where. Each of the 109 tasks (134 broken promises across nine Python repositories) comes from a merged pull request that changed both documentation and code; the authors write the new documentation back into the old code so the mismatch is real rather than planted, and the agent never sees the patch or the history. Reference answers are drafted by a model and checked by two human reviewers and an adjudicator. Agents may submit up to five findings, each quoting the documentation passage and describing the contradicting behaviour, and a finding is credited if the quote matches verbatim and two of three LLM judges agree it matches the reference. Eight agents recover between 35% and 58% of the targets. The analysis shows that in most missed cases the relevant documentation text was already displayed in the agent's own tool output, and that agents who open more files do worse.

Critical strengths

The overall task construction is well done. Violations occur naturally, the pre-change code is byte-identical, modified files must parse identically once docstrings are stripped, and every reference quotation must appear verbatim in the snapshot.

Critical weaknesses

The score counts only the one violation the maintainers happened to fix, while the paper's own audit shows agents find several others in the same region. Across eight agents there are 3,217 submitted findings, about 3.7 per task, and the LLM auditors judge 73 to 88% of them real. For the top model that is roughly 350 real findings against 77 credited. So the headline "agents recover only 35 to 58%" is largely a measure of whether the agent ranked the maintainers' pick into its top five, not of whether it can detect inconsistencies. The case studies in Section 4.1, where agents "turn to other promises", fit this reading but are framed as failure, and the abstract's conclusion that agents are not ready for this audit is not fully supported.

Questions for authors
  • For the 39 pairs without a completed run, how many were provider or runner failures versus the agent exhausting its two-hour budget, and what is each agent's recall on completed tasks? (Clarification)
  • How many confirmed real inconsistencies does a typical region contain, and does the ranking in Table 2 survive lifting the five-finding cap? (Clarification)
  • What fraction of the 243 candidate cards went to adjudication, and what was reviewer agreement before it? (Clarification)
  • Were PR merge dates compared with each model's training cutoff, and does recall differ for post-cutoff PRs? (Clarification)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.