- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents ReqAuditBench, a benchmark for evaluating whether coding agents can discover inconsistencies between a repository documentation and implementation without being told which requirement is violated or where it is implemented. The benchmark is constructed from merged pull requests by placing updated documentation into the corresponding pre-change repository while preserving the original implementation. It contains 109 tasks with 134 violated requirements across nine Python repositories. The authors evaluate eight coding agents, report requirement recall ranging from 35.1% to 57.5%, and analyze their investigation trajectories and failure modes.
Critical strengths
S1. The paper addresses an important and practical problem.
Checking whether an implementation satisfies its documented requirements is an important software maintenance task. In practice, developers may need to investigate a repository without knowing which requirement is violated or where the relevant implementation resides. Evaluating agents in this setting is practically relevant to automated code auditing.
S2. The paper provides substantial experimental analysis.
Beyond aggregate performance, the paper examines investigation trajectories, exposure to violated requirements, documentation-reading behavior, and selected failure cases. These analyses provide useful observations about how agents select requirements for investigation and where their verification processes stop. The comparison of recall, recorded duration, and estimated token fees also provides a practical perspective on the evaluated agents.
S3. The paper is clearly written.
The paper clearly explains the task definition, benchmark construction pipeline, and scoring procedure. The distinction between the information available to the evaluated agents and the reference evidence available to the evaluator is particularly helpful for understanding the benchmark.
Critical weaknesses
W1. The novelty of both the problem formulation and benchmark construction is limited.
The underlying problem has substantial overlap with prior work. RFCAudit [1] and Spec2Code [2] already investigate inconsistencies between natural-language specifications and implementations, including identifying the relevant implementation from a broader codebase. The paper should explicitly compare its task formulation with these works and explain which additional research challenges ReqAuditBench introduces.
The benchmark construction methodology also shares an important principle with TestExplora [3]. Both exploit historical code changes to pair an earlier implementation with documentation representing intended behavior, while withholding the change information that would reveal the target. ReqAuditBench differs in its documentation source, task definition, and expected output, but the paper does not sufficiently establish why these differences constitute a substantial methodological contribution.
Therefore, the overlap in problem formulation and construction methodology leaves the incremental novelty insufficiently justified.
W2. The contribution’s positioning and longer-term value require stronger justification.
The paper would benefit from a clearer explanation of what the benchmark is intended to enable as agent capabilities improve.
If its primary contribution is to characterize current coding-agent capabilities, broader experiments with different prompts and investigation strategies are needed to establish how stable the reported performance gaps and behavioral observations are. A fixed configuration provides limited evidence about the capability boundaries of the underlying models and harnesses.
If the construction methodology is intended to support continued benchmark expansion or generate data for subsequent training and improvement, its scalability needs empirical support. The current pipeline requires two independent human reviewers to examine every reference card, with a third reviewer resolving disagreements. Although the paper states that the collection procedure can be reused for later changes and additional repositories, it does not report construction token usage, human review time, or cost per accepted task.
The reported token fees concern evaluated agent runs and do not answer these construction-cost questions. Therefore, the paper has not yet established whether the methodology can sustainably produce substantially more tasks or support a scalable data-generation workflow.
W3. Key experimental settings are insufficiently justified, and their effects on the conclusions are not evaluated.
The evaluation uses a single underlying prompt strategy, with separate templates for documentation pages and docstrings. The prompt contains detailed instructions governing evidence collection, reporting, confidence ranking, and stopping behavior. However, the paper does not investigate whether alternative prompts or investigation strategies would substantially change performance, agent rankings, or the observed failure modes.
The five-finding cap is another consequential choice that lacks sufficient justification. This issue is particularly relevant because the reference requirements do not exhaustively cover all inconsistencies in a task region. An agent could report genuine inconsistencies outside the reference set, exhaust its reporting allowance, and stop before investigating a registered target. The paper does not quantify this possibility or evaluate sensitivity to the reporting cap.
Consequently, the results characterize the selected configurations, while their implications for the broader capabilities of the evaluated models and harnesses remain uncertain. The scoring-judge sensitivity analysis does not resolve this concern, since it examines grading robustness rather than the robustness of agent behavior.
Questions for authors
Q1. What is the principal distinction from RFCAudit [1], Spec2Code [2], and TestExplora [3]?
Please distinguish the novelty of the auditing problem from that of the benchmark construction methodology. Which challenges or evaluation capabilities are introduced beyond those already addressed by these works?
Q2. How sensitive are the results to the prompt strategy and five-finding cap?
Would alternative prompts, requirement-selection strategies, or reporting limits materially change recall, agent rankings, and the reported failure patterns? How often do agents reach the cap while reporting valid inconsistencies outside the reference set?
Q3. What resources are required to construct and extend the benchmark?
Please report the LLM token usage, human review effort, candidate acceptance rate, and approximate cost per accepted task. Which stages require human intervention, and what evidence supports extending the pipeline to substantially more repositories and tasks?
References
[1] Mingwei Zheng, Chengpeng Wang, Xuwei Liu, Jinyao Guo, Shiwei Feng, and Xiangyu Zhang. RFCAudit: AI Agent for Auditing Protocol Implementations Against RFC Specifications. ASE, 2025. Paper.
[2] Yuekun Wang, Lili Quan, Xiaofei Xie, Junjie Wang, and Jianjun Chen. Spec2Code: Mapping Protocol Specification to Function-Level Code Implementation. ASE, 2025. Paper.
[3] Steven Liu et al. TestExplora: Benchmarking LLMs for Proactive Bug Discovery via Repository-Level Test Generation. ICML, 2026. Paper.