- Rating
- 4(+2)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Coding agents can often get incomplete or contradictory specifications. Instead of quickly making incorrect assumptions without surfacing them to the user, they should ask clarification questions. This paper introduces PM Bench, a benchmark for evaluating whether these problems can be identified in software specifications and how to avoid asking questions about problems that do not exist. The authors start with a formally verified state chart, introduce defects such as unreachable states, and then turn the state chart into a natural language specification.
These agents are supposed to look at this specification and clarify any problems before implementing it. Since the authors know exactly where these errors were introduced, they can use the original formal specification to check whether the agent identified the correct problems. They can also create clean versions of the same specification to check whether these agents are raising false alarms. The authors evaluate 11 models and find that stronger models are generally pretty good at detecting these defects, but differ widely in how often they raise false alarms on clean specifications (from about ~3% to ~43%).
Critical strengths
I think the problem is pretty important. Coding agents should ideally be able to recognize when something does not make sense and try to clarify it before blindly proceeding with an implementation.
The idea of starting with a formally verified specification and introducing defects is pretty good. There is also a concrete source of ground truth, rather than simply asking for a human evaluation or asking some LLM to check whether a given specification is ambiguous in nature.
The authors have clearly put in effort to create different categories of defects and evaluate them on a set of models. The idea of looking at the differences between detecting problems and false alarms is also interesting.
Critical weaknesses
My main concern is whether the benchmark really measures the ability of these coding agents to clarify incomplete requirements. The experiments are using these automatically generated state charts, which are quite structured and templated in terms of specifications. These specifications are quite different from what LLMs usually get in terms of product requirements, documents that developers actually receive, etc.
The difficult part for an engineer, or now for LLM coding agents, is often understanding what these vague goals are and translating them into fairly well-specified engineering requirements. There is often missing context or unstated assumptions, and a deeper requirement of understanding what the user really wants is often the gap.
If you look at the agent instructions in Appendix A, the authors are instructing the agent to look for the kinds of problems they have introduced, including unreachable states, conflicting transitions, non-terminating loops, and counter overflows. This is a self-fulfilling prophecy, then: I am not entirely convinced that we are measuring the general ability of agents to recognize when a clarification is needed. I think we are just looking at the ability to follow a checklist and audit a state machine for a known set of formal properties.
The benchmark’s defect distribution is also quite narrow compared to the broader problems of writing good specifications in software. Writing good formal specifications has been one of the hardest problems in formal methods for many decades now. Most of the benchmark is concerned with minor structural problems in these state machines rather than the general problem of resolving incomplete or ambiguous user requirements. It is quite difficult to write even a good loop invariant for a nontrivial task, and these specifications are hard to come up with even for human experts.
The authors also conduct a smaller experiment with more human-style prose, which is pretty good. These documents are still generated from the formal state charts and then written by an LLM in a way that mimics how a human would write. However, this pilot only evaluates smaller, simpler charts (without the conflict families or entangled defects) and four models, so I do not think this sufficiently establishes that the benchmark actually does capture the difficulties of clarification in realistic software engineering workflows.
Questions for authors
I would have loved to see how these results change when models are not told to look for these categories of defects that were introduced. Do you have results with a clarification protocol that does not list the defect types?
I have questions about the choice of primary evaluation metric. Balanced accuracy is the headline metric, but Figure 9 shows the ranking changes substantially across metrics. Why is balanced accuracy the right primary metric?
5 comments
Reply to Reviewer uD6Q, Part 2/3: Weaknesses (W3 to W5)
Q1, W3: Does the checklist make detection self-fulfilling? You are right that listing the defect kinds turns part of the task into checklist-following, so the protocol without the list is now primary, running on all 252 tasks for all eleven models (General response, Item 2). Without the list, the top six models still catch a planted defect on 0.79 to 0.99 of defective specifications, and on all 252 tasks run both ways (nine models) Fable 5 does not change, while Opus 4.8 loses 0.147 balanced accuracy (BA) and GPT-5.6 Sol 0.081 without the list. The agent is still asked to review the specification: on 84 tasks, detection is 0.806 with the list, 0.677 with the review request alone and 0.200 with neither, which is why we narrowed the claim.
W4: Narrow, minor structural defects. We agree the seven families do not cover all requirement problems. Every defect planted in a chart changes the observable behavior of the machine, and at a planted defect the code matches the intended behavior 87.1% of the time when the agent asked about it and 8.8% when it did not (General response, Item 5; an association). Two families are minor: an overflow changes only a counter’s value, not the control flow, and a vacuous edge removes one transition; together they are 137 of the 864 planted defects. Dead states are 460 of the 864, and nearly all hide behind a counter invariant (a conserved sum, a fixed difference, a residue lattice) that the agent has to work out. Without the list, Fable 5 catches 91% of residue-lattice dead states and the next model, Sonnet 5, 52%. Also, see the answers to W1.
W5: The prose pilot was small. We extended it to nine models and to 41 full-size charts drawn at random from the main benchmark, with every defect family (17 of the 41 have a conflict, and the entangled rungs are included). On full-size charts detection falls by 0.098 and BA by 0.091 over six models with the checklist; without the list, all eleven models ran on these documents and BA falls by 0.082 (General response, Item 1).
Reply to Reviewer uD6Q, Part 3/3: Questions
Q1: Results without the list. See W3 above and General response, Item 2.
Q2: Why balanced accuracy? You are right that the ranking changes across metrics, and the submitted paper understated it by saying only the middle of the roster moves. With the checklist, Opus 4.8 ranks first on balanced accuracy (0.956, not separated from Fable 5 at 0.954) and sixth on all-defects recall. We added this to the paper.
We keep balanced accuracy (BA) as the primary metric for three reasons.
- It matches the task. Each task yields one defective and one complete specification, and an agent can fail in two ways: miss a planted defect, or claim a defect that is not there. BA weighs the two failures equally.
- It cannot be gamed by asking more or less. An agent that always claims a defect, or never claims one, scores at most 0.5. Detection alone rewards always asking, and restraint alone rewards never asking.
- It does not depend on how many specifications are defective. In deployment that share is unknown, and accuracy or F1 would change with it.
Under the primary protocol (without the list), the leader does not depend on the metric: Fable 5 ranks first on BA, detection, all-defects recall and exact success, and MCC gives the same order as BA. Fable 5 catches every planted defect on 62% of defective specifications; the next model, Gemini 3.1 Pro, does so on 29%.
The re-ranking in Figure 9 arises with the checklist. There, Opus 4.8 catches at least one planted defect on 93% of defective specifications but all of them on only 24%, which is why it drops from first on BA to sixth on all-defects recall. Metrics that require every planted defect penalize stopping after the first one. We think this is a real difference between models and not noise, so we report all-defects recall next to BA and do not fold it into one number.
Changes to the paper from your review: the protocol without the defect list becomes primary, the claim is narrowed, the prose study covers full-size charts, and all-defects recall is reported next to BA. We add the references to the paper and replaced Figure 1 with a human-readable benchmark sample to improve clarity and intuition.
Reply to Reviewer uD6Q, Part 1/3: Weaknesses (W1, W2)
Thank you for the detailed review.
W1: Does the benchmark measure clarification of real requirements? W2: Vague goals and unstated assumptions.
Only partially, and by design; we narrowed the claim (General response, Item 8). PM Bench measures one verifiable part of clarification: noticing an incomplete or inconsistent specification, and not raising problems that are not there. We restrict ourselves to domains that can be modeled as statecharts, and expect the approach to extend to others, such as dataflow models.
Out of scope. We agree that two things are the harder part of clarification, and PM Bench does not measure them. First, an agent may fill gaps with assumptions based on how systems in the domain usually work. Second, a specification can be coherent yet incomplete, with no observable cue: the gap surfaces only when someone mentions a scenario nobody had considered. A requirement the specification never mentions has no formal answer key, and no method that inspects the specification alone can detect it.
In scope. Incomplete and hidden requirements are the most frequently reported requirements-engineering problem in industry [1, 2], and they often leave traces in the specification. These correspond to the completeness and consistency characteristics of ISO/IEC/IEEE 29148 [3]. Typical traces include (non-exhaustive):
In the statechart domain, these map to (1) states, events or transitions that are mentioned but not connected (unreachable states, which the paper calls dead states); (2) contradicting behavior (conflicting transitions: eventless and event conflicts; guards that can never be true: vacuous edges); (3) vague specifications that admit multiple implementations, which we cover only where the vagueness is missing information; and (4) missing scenarios (initialization: init omission; termination: livelock; overflow handling: overflow). We left out pure well-formedness defects (dangling edges, undefined variables) because agents caught them reliably in our manual pilot runs and lightweight automated checks already find them [4]. This part matters downstream: at a planted defect, code matches the intended behavior 87% of the time when the agent asked and 9% when it did not (W4).
Vagueness is a scale. At one end the goal does not say what the system must do (e.g., “build a controller”); we do not address it. At the other end, vagueness is missing information about a specific behavior, which PM Bench measures, without claiming full coverage. Vaguer specifications could be generated from the chart, but the LLM that answers the agent’s questions would then need further validation.
Realistic documents. The full-size prose documents have a median of 65 states and 2,012 words, including sections and tables that state no rule (W5). Nine expert annotators rated the compact pilot’s prose 3.26 for realism (1 to 5), against 2.13 for the template of the same tasks and 1.89 for the main-campaign template. Our prose is generated from the chart, which keeps every answer verifiable; we do not claim it matches documents written by engineers. On full-size prose, BA falls by about 0.09, so template scores likely overstate performance on prose documents.
[1] Méndez Fernández et al., Naming the pain in requirements engineering, Empirical Software Engineering 22(5), 2017.
[2] Kalinowski et al., Preventing incomplete/hidden requirements, SWQD 2016.
[3] ISO/IEC/IEEE 29148:2018, Requirements engineering.
[4] H. Femmer, D. Méndez Fernández, S. Wagner, S. Eder. Rapid quality assurance with requirements smells. Journal of Systems and Software, 123:190–213, 2017.