- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes VeriSpec, a multi-stage framework for detecting inconsistencies in model specifications through LLM-based rule extraction, graph-based clustering, and LLM-based verification.
Critical strengths
The framework considers contextual information, rule authority, and exceptions when identifying conflicts.
Critical weaknesses
Despite its formalization, inconsistency detection ultimately relies on LLM judgments rather than rigorous verification.
Errors introduced during rule extraction and graph partitioning may propagate through the pipeline, potentially causing missed inconsistencies.
The additional complexity of the pipeline needs stronger justification through comparisons with direct long-context LLM baselines and detailed ablation studies.
Questions for authors
N/A
3 comments
Response to Reviewer DWW2 (2/2)
DWW2-Weakness3: long-context baselines and ablations.
We thank the reviewer for asking for these comparisons. We compare VeriSpec with two long-context baselines, DirectPrompt and DirectAgent, and with four ablations, one of which is new for this rebuttal. All methods use Claude Opus 5 at the same reasoning effort with five attempts.
| Method | Setting | Validated / reported | Precision | Cost | Cost per validated |
|---|---|---|---|---|---|
| DirectPrompt | Whole spec in one long-context call | 0/16 | 0% | $4.92 | — |
| DirectAgent | Whole spec, read by a Claude Code agent | 0/9 | 0% | $27.66 | — |
| Ablations | |||||
| SectionText | Removes rule extraction and the rule graph | 1/10 | 10.0% | $58.64 | $58.64 |
| SectionGroup | Removes the rule graph (one unit per section) | 1/8 | 12.5% | $63.94 | $63.94 |
| EmbedCluster (new) | Replaces the rule graph with embedding clusters | 2/6 | 33.3% | $40.55 | $20.28 |
| PlainDetect | Removes the witness and context-resolution arguments | 3/221 | 1.4% | $47.46 | $15.82 |
| VeriSpec | Full pipeline | 5/13 | 38.5% | $55.62 | $11.12 |
Long-context baselines. DirectPrompt is the direct long-context baseline. It gives the whole spec (about 38,000 words) to the model in a single call and asks for inconsistencies, skipping every stage of VeriSpec. DirectAgent instead lets a Claude Code agent read the spec freely. Neither finds a validated inconsistency. Both often report a conflict while missing a nearby exception that resolves it. For example, they flag a transformation permission as conflicting with content prohibitions, although the transformation passage says that the permission still applies when the output would otherwise be restricted.
Ablations. Each ablation removes or replaces one stage of VeriSpec, and each one loses validated inconsistencies. Every validated inconsistency found by an ablation is among VeriSpec's five. The new ablation, EmbedCluster, replaces the rule graph with embedding clusters. It finds only 2 validated inconsistencies. It misses the output-only, fictional-portrayal, and polite-reassurance cases, because embedding similarity places the two rules of each case in different analysis units. In the revised paper, Table 1 now separates the long-context baselines from the ablations and includes EmbedCluster, Section 5.1 states that all methods use the same reasoning effort, Section 5.3 discusses EmbedCluster, and Appendix C describes its implementation.
Ethics flag. We appreciate the reviewer's attention to ethics. The work analyzes a public document and involves no human subjects or private data. Anonymization only replaces provider and model names, and a programmatic check confirms that the rest of the text is unchanged. Its purpose is to keep models from refusing to analyze the document, not to obtain harmful content. We have reworded "suppresses guardrails" to "avoids refusals to analyze the document" in the Anonymization paragraph of Section 4, and Appendix D (Table 6) now reports refusal rates with and without anonymization. We reported all five inconsistencies to the OpenAI Model Spec developers. They acknowledged all five, and their latest release (October 29, 2026) addresses two of them. The ethics statement of the revised paper now states both the purpose of anonymization and the developers' response.
[1] Mu et al. Rule Based Rewards for Language Model Safety. NeurIPS 2024.
[2] Rahman et al. CellularLint: A Systematic Approach to Identify Inconsistent Behavior in Cellular Network Specifications. USENIX Security 2024.
[3] Sun et al. Clover: Closed-Loop Verifiable Code Generation. SAIV 2024.
Thank you again for your time and your detailed review. As the discussion period draws to a close, we would like to kindly follow up on our response. We have answered each of your points and uploaded a revised paper with all changes highlighted in blue. In particular, we clarify that we ask the LLM to reason like a verifier without claiming formal guarantees, explain why formal verification does not fit natural-language specifications, describe the safeguards against error propagation, compare VeriSpec with two long-context baselines and four ablations, and address the ethics flag.
We would be grateful to know whether our response addresses your concerns. If any question remains, we are happy to clarify it before the discussion period ends.
Response to Reviewer DWW2 (1/2)
We have uploaded a revised paper with all changes highlighted in blue, and we note below where each change appears.
DWW2-Weakness1: LLM judgment rather than rigorous verification.
We thank the reviewer for raising the distinction between LLM judgment and rigorous verification. We do not claim that the LLM is a verifier. We use the term because we ask the LLM to reason like one. Instead of reporting any apparent conflict, it must check each candidate against our definition of inconsistency and provide a report with evidence that a human can check. We agree that this gives no soundness or completeness guarantee, and the revised paper now states this explicitly in Section 1, Section 4.3, and the new Limitations paragraph of Section 6.
Why not formal verification. The inconsistencies in model specs depend on natural-language judgments, such as whether a request is a "transformation", whether code is "for functional use", or whether output is "consumed programmatically". A formal encoding would have to fix every such judgment in advance. It would either lose distinctions that the spec relies on, such as permissions, defaults, exceptions, and soft obligations, or require substantial manual effort.
For this reason, work on behavior policies, natural-language specifications, and verifiable code generation commonly uses language models to judge natural-language predicates, even when the rest of the check is formal. Rule Based Rewards [1] breaks a safety behavior policy into binary propositions about a model response, such as whether it contains an apology. A grader LLM estimates the probability that each proposition holds, and a linear model fit to the policy's rules combines these probabilities into a safety reward. CellularLint [2] fine-tunes language models to judge whether pairs of passages in the 4G and 5G NAS and security specifications are inconsistent. Clover [3] checks generated code against its formal annotations with the Dafny verifier, but uses an LLM to judge whether a regenerated docstring describes the same behavior as the original one. VeriSpec uses LLM judgment for the same reason. Predicates such as whether a witness situation triggers a rule, or whether an exception applies to it, are natural-language judgments of this kind. We added these works to a new paragraph in Section 3 of the revised paper.
What the verifier must provide. Each report must contain the two rule ids, a concrete witness situation, an unsatisfiability argument, and a context-resolution argument. The unsatisfiability argument explains why no behavior satisfies both rules. The context-resolution argument explains why authority, modality, exceptions, and worked examples do not resolve the conflict. For example, the report for our code-transformation case reads as follows.
These requirements matter. On the same analysis units with the same model, PlainDetect drops them and produces 221 reports with 3 valid, whereas VeriSpec produces 13 reports with 5 valid. Precision rises from 1.4% to 38.5%.
DWW2-Weakness2: error propagation.
We thank the reviewer for pointing out the risk of error propagation. Each stage has safeguards against missed inconsistencies.
Empirically, three baselines skip rule extraction (DirectPrompt, DirectAgent, and SectionText), and two also skip clustering. None of them found a validated inconsistency that VeriSpec missed. The revised paper adds this analysis as Appendix E, and Section 4.1.2 now describes what the context of each rule contains and how the topic tests were validated.