acceptodds
Under review as a conference paper at ICLR 2027

PM Bench: Verifiable Clarification Benchmark for Coding Agents

Abstract

As coding agents take on increasingly autonomous software engineering, a critical capability emerges: reliably and consistently resolving incomplete and contradictory specifications. We argue that this is a fundamental capability that requires objective ground truth that is not a matter of judgment. We introduce PM Bench, a benchmark that measures two complementary capabilities: detection, a metric for asking the correct clarification that resolves the issues injected into a complete specification, and restraint, a metric for not manufacturing clarification questions about a complete specification. Our central contribution is a fully automated, formalism-first construction in which a computer science formalism, rather than natural language, is the source of truth across both task generation and verification. We begin from a certified, well-specified statechart, plant controlled defects (e.g., an unreachable state, conflicting transitions, a counter overflow), and render the result into a Product Requirements Document that reads as complete prose. Because the answer key is the planted defects, scoring is verifiable: a hidden judge checks each clarifying question against the formal oracle facts. Evaluating eleven models, we find that detection is near the ceiling for frontier models while restraint is not: models are separated almost entirely by their false-alarm rate on complete specifications. We further find that the choice of scoring metric reorders the middle of the roster. A paired pilot shows that re-rendering the same certified chart as human-style prose leaves detection unchanged but erodes restraint in frontier models. The method is demonstrated on controller statecharts but generalizes to any domain expressible as a formal specification.

Primary area:
datasets and benchmarks
Keywords:
benchmark, data, reinforcementlearning, rl, statecharts, verifiability, synthetic

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_uD6QOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Coding agents can often get incomplete or contradictory specifications. Instead of quickly making incorrect assumptions without surfacing them to the user, they should ask clarification questions. This paper introduces PM Bench, a benchmark for evaluating whether these problems can be identified in software specifications and how to avoid asking questions about problems that do not exist. The authors start with a formally verified state chart, introduce defects such as unreachable states, and then turn the state chart into a natural language specification.

These agents are supposed to look at this specification and clarify any problems before implementing it. Since the authors know exactly where these errors were introduced, they can use the original formal specification to check whether the agent identified the correct problems. They can also create clean versions of the same specification to check whether these agents are raising false alarms. The authors evaluate 11 models and find that stronger models are generally pretty good at detecting these defects, but differ widely in how often they raise false alarms on clean specifications (from about ~3% to ~43%).

Critical strengths

I think the problem is pretty important. Coding agents should ideally be able to recognize when something does not make sense and try to clarify it before blindly proceeding with an implementation.

The idea of starting with a formally verified specification and introducing defects is pretty good. There is also a concrete source of ground truth, rather than simply asking for a human evaluation or asking some LLM to check whether a given specification is ambiguous in nature.

The authors have clearly put in effort to create different categories of defects and evaluate them on a set of models. The idea of looking at the differences between detecting problems and false alarms is also interesting.

Critical weaknesses

My main concern is whether the benchmark really measures the ability of these coding agents to clarify incomplete requirements. The experiments are using these automatically generated state charts, which are quite structured and templated in terms of specifications. These specifications are quite different from what LLMs usually get in terms of product requirements, documents that developers actually receive, etc.

The difficult part for an engineer, or now for LLM coding agents, is often understanding what these vague goals are and translating them into fairly well-specified engineering requirements. There is often missing context or unstated assumptions, and a deeper requirement of understanding what the user really wants is often the gap.

If you look at the agent instructions in Appendix A, the authors are instructing the agent to look for the kinds of problems they have introduced, including unreachable states, conflicting transitions, non-terminating loops, and counter overflows. This is a self-fulfilling prophecy, then: I am not entirely convinced that we are measuring the general ability of agents to recognize when a clarification is needed. I think we are just looking at the ability to follow a checklist and audit a state machine for a known set of formal properties.

The benchmark’s defect distribution is also quite narrow compared to the broader problems of writing good specifications in software. Writing good formal specifications has been one of the hardest problems in formal methods for many decades now. Most of the benchmark is concerned with minor structural problems in these state machines rather than the general problem of resolving incomplete or ambiguous user requirements. It is quite difficult to write even a good loop invariant for a nontrivial task, and these specifications are hard to come up with even for human experts.

The authors also conduct a smaller experiment with more human-style prose, which is pretty good. These documents are still generated from the formal state charts and then written by an LLM in a way that mimics how a human would write. However, this pilot only evaluates smaller, simpler charts (without the conflict families or entangled defects) and four models, so I do not think this sufficiently establishes that the benchmark actually does capture the difficulties of clarification in realistic software engineering workflows.

Questions for authors

I would have loved to see how these results change when models are not told to look for these categories of defects that were introduced. Do you have results with a clarification protocol that does not list the defect types?

I have questions about the choice of primary evaluation metric. Balanced accuracy is the headline metric, but Figure 9 shows the ranking changes substantially across metrics. Why is balanced accuracy the right primary metric?

5 comments
AuthorsOpenReview ↗

Reply to Reviewer uD6Q, Part 1/3: Weaknesses (W1, W2)

Thank you for the detailed review.

W1: Does the benchmark measure clarification of real requirements? W2: Vague goals and unstated assumptions.

Only partially, and by design; we narrowed the claim (General response, Item 8). PM Bench measures one verifiable part of clarification: noticing an incomplete or inconsistent specification, and not raising problems that are not there. We restrict ourselves to domains that can be modeled as statecharts, and expect the approach to extend to others, such as dataflow models.

Out of scope. We agree that two things are the harder part of clarification, and PM Bench does not measure them. First, an agent may fill gaps with assumptions based on how systems in the domain usually work. Second, a specification can be coherent yet incomplete, with no observable cue: the gap surfaces only when someone mentions a scenario nobody had considered. A requirement the specification never mentions has no formal answer key, and no method that inspects the specification alone can detect it.

In scope. Incomplete and hidden requirements are the most frequently reported requirements-engineering problem in industry [1, 2], and they often leave traces in the specification. These correspond to the completeness and consistency characteristics of ISO/IEC/IEEE 29148 [3]. Typical traces include (non-exhaustive):

  1. a use case, component or requirement is mentioned but never worked out;
  2. stakeholders hold contradicting assumptions, often because each believes a shared term or function serves their own goals;
  3. a requirement is too vague to determine a single behavior;
  4. scenarios or details nobody has considered.

In the statechart domain, these map to (1) states, events or transitions that are mentioned but not connected (unreachable states, which the paper calls dead states); (2) contradicting behavior (conflicting transitions: eventless and event conflicts; guards that can never be true: vacuous edges); (3) vague specifications that admit multiple implementations, which we cover only where the vagueness is missing information; and (4) missing scenarios (initialization: init omission; termination: livelock; overflow handling: overflow). We left out pure well-formedness defects (dangling edges, undefined variables) because agents caught them reliably in our manual pilot runs and lightweight automated checks already find them [4]. This part matters downstream: at a planted defect, code matches the intended behavior 87% of the time when the agent asked and 9% when it did not (W4).

Vagueness is a scale. At one end the goal does not say what the system must do (e.g., “build a controller”); we do not address it. At the other end, vagueness is missing information about a specific behavior, which PM Bench measures, without claiming full coverage. Vaguer specifications could be generated from the chart, but the LLM that answers the agent’s questions would then need further validation.

Realistic documents. The full-size prose documents have a median of 65 states and 2,012 words, including sections and tables that state no rule (W5). Nine expert annotators rated the compact pilot’s prose 3.26 for realism (1 to 5), against 2.13 for the template of the same tasks and 1.89 for the main-campaign template. Our prose is generated from the chart, which keeps every answer verifiable; we do not claim it matches documents written by engineers. On full-size prose, BA falls by about 0.09, so template scores likely overstate performance on prose documents.

[1] Méndez Fernández et al., Naming the pain in requirements engineering, Empirical Software Engineering 22(5), 2017.

[2] Kalinowski et al., Preventing incomplete/hidden requirements, SWQD 2016.

[3] ISO/IEC/IEEE 29148:2018, Requirements engineering.

[4] H. Femmer, D. Méndez Fernández, S. Wagner, S. Eder. Rapid quality assurance with requirements smells. Journal of Systems and Software, 123:190–213, 2017.

AuthorsOpenReview ↗

Reply to Reviewer uD6Q, Part 2/3: Weaknesses (W3 to W5)

Q1, W3: Does the checklist make detection self-fulfilling? You are right that listing the defect kinds turns part of the task into checklist-following, so the protocol without the list is now primary, running on all 252 tasks for all eleven models (General response, Item 2). Without the list, the top six models still catch a planted defect on 0.79 to 0.99 of defective specifications, and on all 252 tasks run both ways (nine models) Fable 5 does not change, while Opus 4.8 loses 0.147 balanced accuracy (BA) and GPT-5.6 Sol 0.081 without the list. The agent is still asked to review the specification: on 84 tasks, detection is 0.806 with the list, 0.677 with the review request alone and 0.200 with neither, which is why we narrowed the claim.

W4: Narrow, minor structural defects. We agree the seven families do not cover all requirement problems. Every defect planted in a chart changes the observable behavior of the machine, and at a planted defect the code matches the intended behavior 87.1% of the time when the agent asked about it and 8.8% when it did not (General response, Item 5; an association). Two families are minor: an overflow changes only a counter’s value, not the control flow, and a vacuous edge removes one transition; together they are 137 of the 864 planted defects. Dead states are 460 of the 864, and nearly all hide behind a counter invariant (a conserved sum, a fixed difference, a residue lattice) that the agent has to work out. Without the list, Fable 5 catches 91% of residue-lattice dead states and the next model, Sonnet 5, 52%. Also, see the answers to W1.

W5: The prose pilot was small. We extended it to nine models and to 41 full-size charts drawn at random from the main benchmark, with every defect family (17 of the 41 have a conflict, and the entangled rungs are included). On full-size charts detection falls by 0.098 and BA by 0.091 over six models with the checklist; without the list, all eleven models ran on these documents and BA falls by 0.082 (General response, Item 1).

AuthorsOpenReview ↗

Reply to Reviewer uD6Q, Part 3/3: Questions

Q1: Results without the list. See W3 above and General response, Item 2.

Q2: Why balanced accuracy? You are right that the ranking changes across metrics, and the submitted paper understated it by saying only the middle of the roster moves. With the checklist, Opus 4.8 ranks first on balanced accuracy (0.956, not separated from Fable 5 at 0.954) and sixth on all-defects recall. We added this to the paper.

We keep balanced accuracy (BA) as the primary metric for three reasons.

  • It matches the task. Each task yields one defective and one complete specification, and an agent can fail in two ways: miss a planted defect, or claim a defect that is not there. BA weighs the two failures equally.
  • It cannot be gamed by asking more or less. An agent that always claims a defect, or never claims one, scores at most 0.5. Detection alone rewards always asking, and restraint alone rewards never asking.
  • It does not depend on how many specifications are defective. In deployment that share is unknown, and accuracy or F1 would change with it.

Under the primary protocol (without the list), the leader does not depend on the metric: Fable 5 ranks first on BA, detection, all-defects recall and exact success, and MCC gives the same order as BA. Fable 5 catches every planted defect on 62% of defective specifications; the next model, Gemini 3.1 Pro, does so on 29%.

The re-ranking in Figure 9 arises with the checklist. There, Opus 4.8 catches at least one planted defect on 93% of defective specifications but all of them on only 24%, which is why it drops from first on BA to sixth on all-defects recall. Metrics that require every planted defect penalize stopping after the first one. We think this is a real difference between models and not noise, so we report all-defects recall next to BA and do not fold it into one number.

Changes to the paper from your review: the protocol without the defect list becomes primary, the claim is narrowed, the prose study covers full-size charts, and all-defects recall is reported next to BA. We add the references to the paper and replaced Figure 1 with a human-readable benchmark sample to improve clarity and intuition.

Reviewer_rNZ8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In this paper, the authors proposed a provable approach that measures 2 complementary capabilities: detection and restraint. The authors demonstrated the effectiveness of the proposed framework using a set of eleven different models. The authors demonstrated through the experiments that the framework was effective on controller state charts but generalizes on any other domain that can be captured by a temporal logic.

Critical strengths

The authors were clear in their discussion of the novelty of the paper. Clearly the ability of the framework to measure both defect detection and false alarms against verifiable ground truth is well discussed. Through the elaborate experiments, the authors to a large extent demonstrated the practicality of the proposed architecture.

Critical weaknesses
  1. The authors demonstrated the effectives of the framework using only fixed-template generated state charts, rather than less structured real-world requirements. The authors should attempt to expand the evaluation of the proposed work beyond the charts used.

  2. The authors employed formal facts in their grading analysis. In an attempt to establish grading reliability, the authors should attempt using another judge

  3. The framework does not critical requirement issues in common software use cases, such as conflicting business rules or missing functional requirements. The lack of analysis on these requirements significantly limits the discussions on the scope of the proposed framework.

Questions for authors
  1. Can the grading proposed be considered fully objective when an LLM judge still interprets the agent's questions?
  2. Are all formally detected defects genuine requirements problems?
  3. How representative are the generated specifications?
2 comments
AuthorsOpenReview ↗

Reply to Reviewer rNZ8

hank you for the feedback.

W1: Template-written specifications. We extended the human-readable experiment as requested. See Item 1 in the general section. On 41 full-size documents with every defect family, prose lowers detection by 0.098 and balanced accuracy (BA) by 0.091 over six models with the checklist, and BA by 0.082 over all eleven models without the list; on compact charts neither detection nor restraint changes detectably (General response, Item 1, table).

W2: Grading reliability. We agree, and we did both. We validated the judge in two independent ways (see items 3 and 4 of the general comment), a second judge from a different model family, and expert annotators. A second judge agrees with ours on detection for 99.8% of defective specifications and on defect claims for 87.0% of clean ones (rollouts where the agent asked something), and annotators agree with the judge on 98.3% of planted defects (κ\kappa = 0.96), as often as with each other; on false claims they agree with the judge about as often as with each other (79.1% against 80.1%) (General response, Items 3 and 4, table).

W3: Coverage of requirement problems. We agree the seven families do not cover all requirement problems, and we narrowed the claim accordingly. Two of the issues you name are partly covered: the two conflict families are conflicting rules (classed as ambiguity in Table 1 of the submission), and init omission is missing information, though only on four tasks. Missing functional requirements that a specification never mentions are not covered, because they have no formal answer key. We add a table (Table 2) that maps each family to the requirement characteristics of ISO/IEC/IEEE 29148 and the defect classes of IEEE 1028-1997.

Q1: Is grading fully objective? We agree. We made the claim more precise in the paper. Grading divides the work between an LLM and code: the judge marks which planted defects the agent’s questions address and lists the claimed defects the formal facts do not support; code drops any listed claim that the formal facts support and decides the reward. We validated the judge in two independent ways (see items 3 and 4 of the general comment), a second judge from a different model family, and expert annotators.

Q2. Are the planted defects genuine problems? Every defect planted in a chart changes the observable behavior of the machine. It also matters for the code: when the agent asked about a defect, its code matched the intended behavior at that defect in 84% to 97% of cases depending on the family; when it did not ask, in 1% to 25% (and at chance for event conflicts). Two families are minor: an overflow changes only a counter’s value, not the control flow, and a vacuous edge removes one transition; together they are 137 of the 864 planted defects. Dead states also occur in machines people ship: 8 of 321 fully parsed XState machines from public repositories contain one. Expert annotators called a planted defect a real requirements problem in 385 of 429 ratings.

Q3. How representative are the specifications? The main benchmark uses a fixed template, which annotators rated 1.89 for realism on a scale of 1 to 5. The compact pilot’s prose documents rated 3.26, against 2.13 for its template. On full-size prose documents with every defect family, scores fall by 0.09 balanced accuracy (Item 1), so template scores likely overstate what most models do on prose documents. Our prose is generated from the chart, which keeps every answer verifiable; we do not claim it matches documents written by engineers.

Changes to the paper from your review: the full-size prose study, the judge validation, a narrower claim, and a table mapping the families to requirement-defect classes.

Reviewer_rNZ8OpenReview ↗

Maintaining the score

I thank the authors for their response.

Given the authors acknowledge the limited coverage of the requirements problems, I maintain score.

Reviewer_fNfJOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces PM Bench, a benchmark for evaluating coding agents' ability to identify incomplete or inconsistent software specifications while avoiding false alarms. The paper proposes a formalism-first benchmark construction pipeline that generates certified statecharts, injects controlled defects, and converts them into natural-language specifications. By maintaining formally verified ground truth, the benchmark supports automated grading through LLM-based claim extraction and deterministic verification against formal oracle facts. PM Bench consists of 252 base tasks, 504 specifications, and 864 injected defects across seven defect families and seven difficulty levels. The paper evaluate 11 LLMs under the same coding-agent scaffold, measuring their defect detection and restraint capabilities. The results show that while frontier models generally achieve high detection rates, they differ substantially in their false-alarm rates and their ability to identify all defects.

Critical strengths

1. Formalism-first benchmark construction with verifiable ground truth. Unlike existing benchmarks that rely heavily on human annotations or LLM-generated ambiguities, this paper proposes a formalism-first approach that constructs clarification tasks by injecting controlled defects into certified statecharts, which provides formally grounded defect labels and enables automated verification of agents' clarification behavior.

2. Evaluation of both defect detection and false alarms. The paper evaluates not only whether models can identify specification defects but also whether they incorrectly report problems in well-specified tasks. By constructing paired defective and clean specifications, the benchmark captures both detection and restraint capabilities. The evaluation of 11 LLMs are comprehensive and provides useful insights that would be overlooked by detection-only evaluations.

Critical weaknesses

W1: Limited real-world representativeness. The benchmark mainly uses synthetic statechart specifications generated from fixed templates, which differ from real-world software requirements. Moreover, the task instructions explicitly enumerate the types of defects that agents should identify, potentially reducing the task to a predefined specification-checking problem rather than evaluating general clarification capabilities. Although the authors include a human-style prose experiment, it is limited to four models and simpler statecharts. Therefore, it is unclear whether the benchmark can effectively evaluate clarification capabilities in real-world software engineering tasks.

W2: Insufficient details and validation of the evaluation setup. The authors state that all 11 models are evaluated using the same coding-agent scaffold, but they do not specify which scaffold is used, making the evaluation results unreliable. Furthermore, the grading relies on an LLM judge without independent validation, and the primary detection metric can count incorrect defect claims as successful detections. These issues raise concerns about the reliability of the reported results and model comparisons.

Questions for authors

Q1: Real-world representativeness. The benchmark mainly uses synthetic statechart specifications, and the task instructions explicitly list the defect types to look for. How do the authors justify that PM Bench can effectively measure clarification capabilities in real-world software engineering tasks?

Q2: Agent scaffold and evaluation reliability. The authors mention that all 11 models are evaluated under the same coding-agent scaffold, but do not specify which scaffold is used. Different agent scaffold could greatly affect the final evaluation results. Could the authors clarify the agent framework and its configuration? If it is a custom-built agent, how do the authors ensure that the results generalize to widely used coding agents such as Claude Code and Codex, which are widely used and may have different tools, prompts, and workflows?

Q3: Grading reliability. The evaluation relies on an LLM judge to extract defect claims, but its reliability is not independently validated. How do the authors ensure that grading errors do not affect the reported results? Additionally, why does the primary detection metric count incorrect defect claims as detections, rather than requiring them to match the formal oracle?

I would adjust my score if the authors adequately address these questions.

3 comments
AuthorsOpenReview ↗

Reply to Reviewer fNfJ

We appreciate the feedback.

W1 and Q1: Real-world representativeness. We agree, thank you for pointing this out. We re-ran the evaluations without the defect kinds in the prompt. On all 252 tasks and all eleven models, the top six models still catch a planted defect on 0.79 to 0.99 of defective specifications; Fable 5 leads. On the same tasks run both with and without the list (nine models; the with-list runs of Kimi K2.6 and Qwen3.6-27B mostly reached the time limit), detection falls from 0.823 to 0.699 and restraint rises from 0.845 to 0.898, so balanced accuracy (BA) falls by 0.035. Fable 5 does not change; Opus 4.8 loses 0.147 BA and GPT-5.6 Sol 0.081. See General response, Item 2. On representativeness: we narrowed the claim. PM Bench measures one verifiable part of clarification, noticing an incomplete or inconsistent specification and not raising problems that are not there. This part matters for the code an agent writes: each planted defect forces the agent either to ask or to make an assumption, and at a planted defect the code matches the intended behavior 87% of the time when the agent asked about it and 9% when it did not; away from defects there is no detectable difference (General response, Item 5). This is an association, not proof that asking causes better code. We also extended the prose experiment to full-size charts (General response, Item 1), where BA falls by 0.091 over six models with the checklist and by 0.082 over all eleven models without the list.

W2 and Q2: The scaffold and other harnesses. The scaffold is mini-swe-agent 2.4.5 under Harbor 0.3.0, in its stock configuration: the agent’s only tool is a bash shell, each command times out after 30s, and an episode ends when the agent submits or after 1,800s; each model runs at its provider’s default temperature and reasoning setting, one attempt per task. We also ran Opus 4.8 in Claude Code 2.1.325 and GPT-5.6 Sol in Codex 0.169.0, and found no detectable difference (General response, Item 6). On 84 tasks, Opus 4.8 caught a planted defect on 84 of 84 defective specifications under Claude Code against 78 of 83 under mini-swe-agent, and no registered contrast is significant after correction. For Opus 4.8 and GPT-5.6 Sol, with the checklist the intervals rule out a shift in BA larger than about 0.06; for Haiku 4.5, whose detection is lower, they do not rule out a large gain, so mid-table ranks may depend on the harness.

W2 and Q3: Grading reliability. We validated the judge in two independent ways (see items 3 and 4 of the general comment), a second judge from a different model family, and expert annotators. The second judge agrees on detection for 99.8% of defective specifications and on defect claims for 87.0% of clean ones (rollouts where the agent asked something), and ranks the same model first; annotators agree with the judge on 98.3% of planted defects (κ\kappa = 0.96). The submitted judge prompt was stricter than annotators on false claims, so we revised it: a question or a hedge is no longer a defect claim, and the revised prompt agrees with annotators on 84.6% of task rewards, against 85.0% between annotators (General response, Item 4). Under the submitted grading, re-scoring at annotator rates changes the ranking only by two adjacent swaps (τ\tau = 0.93).

The detection metric. You are right, and we changed it. Detection now counts a defective specification only if the agent caught at least one planted defect, by the judge’s per-defect verdict. A defect claim that matches no planted defect no longer counts. We keep the old quantity as a diagnostic.

With the checklist, the two definitions differ on 69 of 2,772 defective specifications (2.5%), most of them from Haiku 4.5, Qwen3-30B and GPT-5.6 Luna. The ranking changes in two adjacent pairs (Sonnet 5 and GPT-5.5, ranks 5 and 6; Kimi K2.6 and Haiku 4.5, ranks 9 and 10). All main-comparison numbers in the revision use the new definition, on all 252 tasks. On comparisons between models: with the checklist the top two are not separated on BA (Opus 4.8 at 0.956, Fable 5 at 0.954); without the list Fable 5 leads; the revision reports confidence intervals on scores.

Changes to the paper from your review: the scaffold and its settings, the harness test, the judge validation, and the stricter detection metric.

I appreciate the rebuttal. I will adjust my score.

Thank you! We really appreciate your time and feedback!

Reviewer_7vR8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents PM Bench, a benchmark for evaluating whether coding agents can identify problems in software requirements and avoid unnecessary clarification questions. The authors generate tasks from verified statecharts and introduce known defects to provide clear ground truth. Experiments on 11 models show that strong models can detect most defects, but they differ significantly in how often they report problems in correct specifications.

Critical strengths

The main strength is the use of formal models to generate tasks with known defects. This makes the evaluation more reliable than relying entirely on human or LLM judgments. The inclusion of correct specifications is also useful, as it measures whether agents can avoid reporting problems that do not exist.

Critical weaknesses
  1. Most experiments use generated statecharts with fixed templates, which are quite different from real software requirements. The experiment using more natural language only tests four models on smaller tasks. I'm not sure whether the results would be similar in real software projects.
  2. I'm also concerned about the grading process. Although the final scores are checked using formal facts, an LLM is still used to identify what problems the agents report. It would be helpful to see how reliable this step is and whether using a different LLM judge would affect the results.
Questions for authors
  1. The main experiments use fixed templates to describe the requirements. Would the results be similar if the authors used more realistic requirement documents? The experiment with natural language descriptions only includes four models and smaller statecharts. Could the authors extend this experiment to larger and more complex tasks?
  2. Although the final scores are checked by code, an LLM is still used to identify the agents' claims. Have the authors tested whether using a different judge would change the results?
  3. The benchmark contains 252 tasks, but the main comparison uses only 207. Could the authors explain whether excluding these tasks affects the model rankings?
  4. The instructions already tell agents what kinds of defects to look for. Would the agents still perform well if they were given less guidance?
  5. The paper focuses on whether agents ask the right questions. Have the authors also checked whether these questions actually help agents produce better code?
1 comment
AuthorsOpenReview ↗

Reply to Reviewer 7vR8

Thank you for the feedback. We agree with the points raised and ran experiments to address them.

W1 and Q1: Realistic requirement documents. We extended the human-readable experiment as requested, to nine models and to 41 full-size charts with every defect family (General response, Item 1). On compact charts, neither detection nor restraint changes detectably. On full-size charts detection falls by 0.098 and balanced accuracy (BA) by 0.091 over six models with the checklist; without the list, all eleven models ran on these documents and BA falls by 0.082. The full-size documents have a median of 65 states and 2,012 words, including sections and tables that state no rule. Expert annotators rated the compact pilot’s prose documents 3.26 for realism on a scale of 1 to 5, against 2.13 for the template of the same tasks (Item 4). Our prose is generated from the chart, which keeps every answer verifiable; we do not claim it matches documents written by engineers.

W2 and Q2: A different judge. We graded the results without the list with a second judge: the two judges agreed on detection for 99.8% of defective specifications and on defect claims for 87.0% of clean ones (rollouts where the agent asked something); both rank Fable 5 first, and the rankings have Kendall’s τb\tau_b = 0.917 (Item 3 in the general comments). Expert annotators agree with the judge on 98.3% of planted defects. The submitted judge prompt was stricter than they are on false claims, so we revised it; the revised prompt agrees with annotators on task rewards about as often as they agree with each other (84.6% against 85.0%; Item 4).

Q3. 207 of 252 tasks. The 45 tasks were left out because the judge returned no valid verdict on one of their answers, not because of any score; the main comparison now covers all 252 tasks (Item 7 in the general comments). Under the submitted metric and grading, adding the 45 tasks does not change the ranking: the one tie on 207 tasks (Sonnet 5 and GPT-5.5, both 0.855) resolves to GPT-5.5 (0.859 against 0.845). The order below also reflects the revised judge prompt and the stricter detection metric (Items 4 and 7). With the checklist, the order on all 252 tasks is Opus 4.8 and Fable 5 (not separated: the 95% interval of their BA difference includes 0), then GPT-5.6 Sol, Gemini 3.1 Pro, Sonnet 5 and GPT-5.5; without the list, Fable 5 leads alone (Item 2).

Q4: Less guidance. Thank you for pointing this out. We re-ran the evaluations without the defect kinds in the prompt. On all 252 tasks and all eleven models, the top six models still catch a planted defect on 0.79 to 0.99 of defective specifications; Fable 5 leads. On the same tasks run both with and without the list (nine models; the with-list runs of Kimi K2.6 and Qwen3.6-27B mostly reached the time limit), detection falls from 0.823 to 0.699 and restraint rises from 0.845 to 0.898, so balanced accuracy falls by 0.035. Fable 5 does not change; Opus 4.8 loses 0.147 BA and GPT-5.6 Sol 0.081. On 84 tasks, detection is 0.806 with the list, 0.677 with the review request alone and 0.200 with neither (eleven models). See General response, Item 2.

Q5: Does asking lead to better code? We ran an experiment as requested (General response, Item 5). We found that the code was right 87% of the time when the agent asked about the planted defect, and 9% when it did not. Within the same program, caught defects are coded correctly 82.6% of the time against 12.8% for missed ones. Away from defects there is no detectable difference (+0.8 points, 95% CI −1.6 to +3.0), so the gain is specific to the defect. This is an association, not proof that asking causes better code.

Changes to the paper from your review: the full-size prose study, the judge validation, all 252 tasks in the main comparison, the protocol without the defect list, and the code study.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response, Part 1/3

We’d like to thank all four reviewers. We will upload a paper reflecting the changes tomorrow morning. The most important change: we removed the list of defect kinds from the agent’s prompt (Item 2). Based on the observations we ran the following experiments to address the concerns. Unless stated otherwise, all numbers use a revised judge prompt, under which a question or a hedge is not a defect claim (Item 4); the code study (Item 5) and the harness test (Item 6) keep the judge prompt of the submitted paper.

  1. Realistic documents. We extended the human-readable pilot to 9 models (on 47 of the pilot’s compact charts), to 41 full-size charts randomly pulled from the main benchmark (6 models with the checklist, all eleven without the list), and to every defect family (17 of the 41 tasks have a conflict, and the entangled rungs are included).

Results:

  • On compact charts, re-rendering the same certified chart as human-style prose changes neither pooled detection nor pooled restraint detectably over the nine models; pooled balanced accuracy (BA) changes by −0.004.
  • On full-size charts, prose lowers both detection (by 0.098) and restraint (by 0.085), and balanced accuracy falls by 0.091. Opus 4.8’s change is not detectable, including on documents written by a second model (GPT-5.5). Without the list, all eleven models ran on the full-size documents, and prose lowers BA by 0.082.

Prose minus structured text on the same tasks and arms, pooled [95% CI]; the first two rows ran with the checklist:

ChartsDetectionRestraintBA
Compact (9 models, 47 tasks)+0.015 [−0.020, +0.048]−0.022 [−0.070, +0.022]−0.004 [−0.033, +0.024]
Full-size (6 models, 41 tasks)−0.098 [−0.146, −0.049]−0.085 [−0.159, −0.012]−0.091 [−0.134, −0.047]
Full-size, without the list (11 models, 41 tasks)−0.058 [−0.087, −0.027]−0.107 [−0.138, −0.075]−0.082 [−0.104, −0.060]

Appendix F of the revised paper gives the details.

  1. Without the list of defect kinds. We removed the defect list from the agent’s prompt (the no-list protocol) and now report the benchmark without it as the primary protocol; we thank the reviewers for pointing this out. All eleven models ran without the list on all 252 tasks (252 defective and 252 clean specifications per model):
ModelDetectionRestraintBA [95% CI]
Fable 50.9920.9290.960 [0.942, 0.976]
Sonnet 50.9520.8330.893 [0.865, 0.919]
Gemini 3.1 Pro0.9010.8850.893 [0.863, 0.921]
GPT-5.50.8130.9330.873 [0.843, 0.901]
Kimi K2.60.7940.9480.871 [0.841, 0.901]
GPT-5.6 Sol0.8250.8770.851 [0.817, 0.883]
Opus 4.80.6670.9520.810 [0.778, 0.841]
Qwen3.6-27B0.6670.9370.802 [0.770, 0.833]
GPT-5.6 Luna0.7780.6980.738 [0.700, 0.776]
Haiku 4.50.3330.9880.661 [0.631, 0.692]
Qwen3-30B0.0320.9840.508 [0.494, 0.522]
  • This changes the main result of the submission. With the list, detection was near the ceiling and the leaders were separated almost entirely by restraint (false alarms from about 3% to 43%). Without the list, restraint varies less and detection more, so models differ in both, and we compare them on BA, which weighs the two equally.
  • Fable 5 leads and is separated from the next two models, Sonnet 5 and Gemini 3.1 Pro (tied second): the 95% interval of the BA difference excludes 0.
  • False alarms range from about 1% to 30% of clean specifications. The top six models differ in both detection (0.79 to 0.99) and restraint (0.83 to 0.95).

What the list adds. All eleven models ran with and without the list on all 252 tasks. Detection with and without the list, and the change in BA (504 paired specifications per model; the pooled row leaves out Kimi K2.6 and Qwen3.6-27B, whose with-list runs mostly reached the time limit):

ModelDetection, with listDetection, withoutBA change [95% CI]
Fable 51.0000.992+0.006 [−0.008, +0.020]
Opus 4.80.9290.667−0.147 [−0.179, −0.117]
GPT-5.6 Sol0.9960.825−0.081 [−0.117, −0.046]
Nine models, pooled0.8230.699−0.035 [−0.047, −0.024]

On these tasks, Fable 5 does not change; Opus 4.8 loses 0.147 BA and GPT-5.6 Sol 0.081, mainly through detection. Pooled, restraint rises from 0.845 to 0.898. With Kimi K2.6 added from a same-week rerun on 84 tasks, the pooled change is the same (−0.035, 95% CI −0.048 to −0.022). The instructions still ask the agent to review the specification; on 84 tasks, detection is 0.806 with the list, 0.677 with the review request alone and 0.200 with neither (eleven models).

AuthorsOpenReview ↗

General response, Part 2/3

3. A second judge. We re-evaluated the no-list results with two LLM judges. Results: a second judge from another model family (GPT-6 Astra), ran through the same grader on all 252 tasks (every rollout in which the agent asked something), agrees with the Opus 4.8 judge on detection for 99.8% of 2,056 defective specifications (Cohen’s κ\kappa = 0.979) and on whether a defect was claimed for 87.0% of 622 clean ones (κ\kappa = 0.740). The two rankings have Kendall’s τb\tau_b = 0.917, and Fable 5 is first under both.

Neither judge detectably favors its own model family: Opus 4.8’s BA minus GPT-6 Astra’s is no higher on the four models of the judge’s own family than on the others (difference −0.006, 95% CI −0.013 to +0.001).

4. Human evaluation

  • Expert (PhDs and PhD students in STEM fields who worked with automata or statecharts in their studies) annotators regraded conversations the Opus 4.8 judge had already graded, answering the judge’s two questions: did the agent catch each planted defect, and did it state a defect that is not there.
  • Size: 53 tasks, each graded by two annotators: 386 conversations and 772 labels from nine annotators. 38 tasks are from the main campaign and 15 from the compact prose pilot.
  • Blinding: annotators saw the specification, the planted defects and the agent’s messages but not the judge’s verdict or the model’s name.
  • Detection is graded reliably: on the 38 main-campaign tasks, the judge agrees with expert annotators on 98.3% of planted defects (κ\kappa = 0.96), as often as annotators agree with each other (97.4%).
  • The submitted judge prompt was stricter than annotators on false claims (the stricter party in 117 of 127 disagreements), often because it counted a question or a hedge as a claim, so we revised it: a defect must be asserted, not asked about, and part of a credited catch is not counted again. The revised prompt agrees with annotators on 84.6% of task rewards on the main campaign (submitted prompt 82.7%; annotators with each other 85.0%). Under the submitted grading, with the checklist, re-scoring at annotator rates moves the ranking by two adjacent swaps (τ\tau = 0.93) after which Fable 5 is first. On compact prose the submitted prompt over-called false claims (reward agreement 64.2%, against 85.0% between annotators); the revised prompt reaches 80.0%.
  • Realism (1 to 5): the main-campaign template scores 1.89; on the compact pilot, the template scores 2.13 and humanized prose 3.26; paired on the same task and annotator, prose scores 1.13 higher.

Agreement per label under the revised judge prompt, Cohen’s κ\kappa in parentheses; the annotator pair is the reference:

ComparisonPlanted defect caughtFalse claimTasks
Second judge vs Opus 4.8 judge99.8% (0.98), per specification87.0% (0.74), clean arm252
Annotators vs judge, main campaign98.3% (0.96)79.1% (0.54)38
Annotator vs annotator, main campaign97.4%80.1%38
Annotators vs judge, compact prose96.3% (0.86)79.2% (0.53)15

We added the results to the paper as a new section in the Appendix, and a summary to Section 4.2.

AuthorsOpenReview ↗

General response, Part 3/3

5. Code generation for the benchmark.

  • We executed the code the agents wrote in the main campaign with the checklist (all 252 tasks, 11 models; 4,855 of 5,544 runs saved code) beside an executable model of the correct chart, on inputs that reach each planted defect.
  • At a planted defect, the code matches the intended behavior 87% of the time when the agent asked about that defect and 9% when it did not.
  • On inputs that never reach a defect, there is no detectable difference (+0.8 points, 95% CI −1.6 to +3.0). This is an association; it does not prove that asking causes the correct code.

Share of code that matches the intended behavior, by whether the agent caught the defect; differences in points within model and family or rung [95% CI]:

ComparisonCaughtMissedDifference
Per defect (6,907 defects)87.1%8.8%+61.8 [+58.1, +65.3]
Same program, caught vs missed defects (1,032 rollouts)82.6%12.8%+69.8 [+66.9, +72.5]
Control: walks that reach no defect (2,423 rollouts)95.1%49.0%+0.8 [−1.6, +3.0]

In the control row, the raw shares differ because weaker models miss more defects; within model and rung there is no detectable difference.

6. Other harnesses. The original paper uses a mini-swe-agent harness; we randomly selected 84 tasks from the 252 (12 per difficulty level) and executed them with Opus 4.8 in Claude Code, and GPT-5.6 Sol in Codex, each against mini-swe-agent runs of the same tasks. Haiku 4.5, a model below the detection ceiling, ran in Claude Code on 42 of them. On 42 of the tasks, each native harness was also run at two reasoning settings, and both models were rerun in mini-swe-agent to measure how much results change from one run to the next.

  • No detectable difference in detection: GPT-5.6 Sol caught a planted defect on 84 of 84 defective specifications in mini-swe-agent and 83 of 84 in Codex, and Opus 4.8 caught 84 of 84 in Claude Code against 78 of 83 in mini-swe-agent. No comparison of reasoning settings or reruns is significant after correction (smallest adjusted p = 0.75).

  • No detectable difference in restraint: GPT-5.6 Sol's restraint in Codex differs from mini-swe-agent by +0.04 on 84 tasks (95% CI -0.05 to +0.12).

  • Without the list (42 tasks; Opus 4.8 and GPT-5.6 Sol), no contrast shows an effect after correction either.

A registered extension added 42 new tasks and Haiku 4.5, a model below the detection ceiling. Native harness minus mini-swe-agent [95% CI]; none of the six contrasts is significant after correction:

Model, harnessTasksDetectionRestraint
GPT-5.6 Sol, Codex84−0.012 [−0.036, +0.000]+0.036 [−0.048, +0.119]
Opus 4.8, Claude Code84+0.060 [+0.012, +0.108]−0.036 [−0.095, +0.024]
Haiku 4.5, Claude Code42+0.098 [−0.098, +0.293]+0.119 [−0.095, +0.333]

Opus 4.8’s detection interval excludes 0, but its exact test gives p = 0.063 before correction.

7. All 252 tasks and a stricter detection metric.

  • The main comparison now covers all 252 tasks instead of 207. The 45 missing tasks came from 52 answers (0.9%) where the judge returned no valid verdict; every answer in which the agent asked something is now regraded with the revised judge prompt, and a reply that contains two verdicts is scored on the last one. With the checklist, Opus 4.8 (BA 0.956) and Fable 5 (0.954) are not separated; without it, Fable 5 leads (Item 2).
  • Detection now counts a defective specification only if the agent caught a planted defect, by the judge’s per-defect verdict; a defect claim that matches no planted defect no longer counts.
  • With the checklist, the two definitions differ on 69 of 2,772 defective specifications (2.5%), and the ranking changes in two adjacent pairs (Sonnet 5 and GPT-5.5, ranks 5 and 6; Kimi K2.6 and Haiku 4.5, ranks 9 and 10).

8. Narrowed claim. PM Bench measures one verifiable part of clarification, noticing an incomplete or inconsistent specification and not raising problems that are not there. The abstract and introduction now say so.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.