acceptodds
Under review as a conference paper at ICLR 2027

Detecting Inconsistencies in Model Specifications with LLM-as-Verifier Reasoning

Abstract

Model specifications define how large language models (LLMs) should behave, guiding alignment training, inference-time behavior, and evaluation. Yet these specifications may themselves contain defects: two individually reasonable principles may prescribe incompatible behavior when applied to the same situation, leaving no response that satisfies both. Detecting such inconsistencies is challenging. Formalizing natural-language specifications risks losing subtle distinctions, while behavior-based testing cannot reliably distinguish specification defects from differences in model behavior. We introduce VeriSpec, the first approach to directly detect inconsistencies in model specifications by auditing the specification text itself. Our key insight is to preserve the specification in natural language while using an LLM as a verifier. VeriSpec extracts structured, context-aware rules, constructs a topic-guided graph to cluster behaviorally related rules at the same authority level, and applies LLM-as-verifier reasoning to detect inconsistencies. Applying VeriSpec to the OpenAI Model Spec, we extract 405 rules and manually validate five inconsistencies, all reported to its developers, who responded positively and have initiated internal discussions. Compared with five baselines, VeriSpec identifies the most validated inconsistencies, achieves the highest precision (38.5%), and incurs the lowest cost per validated inconsistency ($11.12). These results establish direct specification auditing as a practical complement to behavioral alignment evaluation, catching defects at the source before they shape any model.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Model Specification, Model Spec, Inconsistency Detection, Specification Testing, LLM-as-Verifier

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DWW2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VeriSpec, a multi-stage framework for detecting inconsistencies in model specifications through LLM-based rule extraction, graph-based clustering, and LLM-based verification.

Critical strengths

The framework considers contextual information, rule authority, and exceptions when identifying conflicts.

Critical weaknesses

Despite its formalization, inconsistency detection ultimately relies on LLM judgments rather than rigorous verification.

Errors introduced during rule extraction and graph partitioning may propagate through the pipeline, potentially causing missed inconsistencies.

The additional complexity of the pipeline needs stronger justification through comparisons with direct long-context LLM baselines and detailed ablation studies.

Questions for authors

N/A

3 comments
AuthorsOpenReview ↗

Response to Reviewer DWW2 (1/2)

We have uploaded a revised paper with all changes highlighted in blue, and we note below where each change appears.

DWW2-Weakness1: LLM judgment rather than rigorous verification.

We thank the reviewer for raising the distinction between LLM judgment and rigorous verification. We do not claim that the LLM is a verifier. We use the term because we ask the LLM to reason like one. Instead of reporting any apparent conflict, it must check each candidate against our definition of inconsistency and provide a report with evidence that a human can check. We agree that this gives no soundness or completeness guarantee, and the revised paper now states this explicitly in Section 1, Section 4.3, and the new Limitations paragraph of Section 6.

Why not formal verification. The inconsistencies in model specs depend on natural-language judgments, such as whether a request is a "transformation", whether code is "for functional use", or whether output is "consumed programmatically". A formal encoding would have to fix every such judgment in advance. It would either lose distinctions that the spec relies on, such as permissions, defaults, exceptions, and soft obligations, or require substantial manual effort.

For this reason, work on behavior policies, natural-language specifications, and verifiable code generation commonly uses language models to judge natural-language predicates, even when the rest of the check is formal. Rule Based Rewards [1] breaks a safety behavior policy into binary propositions about a model response, such as whether it contains an apology. A grader LLM estimates the probability that each proposition holds, and a linear model fit to the policy's rules combines these probabilities into a safety reward. CellularLint [2] fine-tunes language models to judge whether pairs of passages in the 4G and 5G NAS and security specifications are inconsistent. Clover [3] checks generated code against its formal annotations with the Dafny verifier, but uses an LLM to judge whether a regenerated docstring describes the same behavior as the original one. VeriSpec uses LLM judgment for the same reason. Predicates such as whether a witness situation triggers a rule, or whether an exception applies to it, are natural-language judgments of this kind. We added these works to a new paragraph in Section 3 of the revised paper.

What the verifier must provide. Each report must contain the two rule ids, a concrete witness situation, an unsatisfiability argument, and a context-resolution argument. The unsatisfiability argument explains why no behavior satisfies both rules. The context-resolution argument explains why authority, modality, exceptions, and worked examples do not resolve the conflict. For example, the report for our code-transformation case reads as follows.

  • Witness situation. A developer asks for comments to be added to Python code, returned as code only, and the function is missing a colon.
  • Unsatisfiability. Preserving the code returns invalid code, while adding the colon changes content the user did not ask to change.
  • Context resolution. Nothing in the spec sets a precedence, and the code-only request rules out explaining the defect.

These requirements matter. On the same analysis units with the same model, PlainDetect drops them and produces 221 reports with 3 valid, whereas VeriSpec produces 13 reports with 5 valid. Precision rises from 1.4% to 38.5%.

DWW2-Weakness2: error propagation.

We thank the reviewer for pointing out the risk of error propagation. Each stage has safeguards against missed inconsistencies.

  • Rule extraction, step 1 (annotating principles). A programmatic check confirms that annotation only inserts tags. The pipeline added 126 principles that the original annotations missed, and two authors reviewed every section for missed or over-fragmented principles.
  • Rule extraction, step 2 (transforming principles to rules). A programmatic check confirms that every annotation maps to exactly one rule. The verifier also does not rely on the extracted fields alone. For each rule, it receives the full text of the rule's section, its ancestor sections, and any referenced sections, together with every relevant worked example.
  • Clustering. A rule appears in every topic it fits, and 162 rules have several topics. Two related rules are therefore separated only if they share no topic. Two independent LLM labelers validated the topic tests (Cohen's κ ≥ 0.6, and κ ≥ 0.87 for sub-topics). In a held-out check, rule pairs linked by a cross-reference share a topic 7.16× more often than random pairs (p ≤ 0.001).

Empirically, three baselines skip rule extraction (DirectPrompt, DirectAgent, and SectionText), and two also skip clustering. None of them found a validated inconsistency that VeriSpec missed. The revised paper adds this analysis as Appendix E, and Section 4.1.2 now describes what the context of each rule contains and how the topic tests were validated.

AuthorsOpenReview ↗

Response to Reviewer DWW2 (2/2)

DWW2-Weakness3: long-context baselines and ablations.

We thank the reviewer for asking for these comparisons. We compare VeriSpec with two long-context baselines, DirectPrompt and DirectAgent, and with four ablations, one of which is new for this rebuttal. All methods use Claude Opus 5 at the same reasoning effort with five attempts.

MethodSettingValidated / reportedPrecisionCostCost per validated
DirectPromptWhole spec in one long-context call0/160%$4.92—
DirectAgentWhole spec, read by a Claude Code agent0/90%$27.66—
Ablations
SectionTextRemoves rule extraction and the rule graph1/1010.0%$58.64$58.64
SectionGroupRemoves the rule graph (one unit per section)1/812.5%$63.94$63.94
EmbedCluster (new)Replaces the rule graph with embedding clusters2/633.3%$40.55$20.28
PlainDetectRemoves the witness and context-resolution arguments3/2211.4%$47.46$15.82
VeriSpecFull pipeline5/1338.5%$55.62$11.12

Long-context baselines. DirectPrompt is the direct long-context baseline. It gives the whole spec (about 38,000 words) to the model in a single call and asks for inconsistencies, skipping every stage of VeriSpec. DirectAgent instead lets a Claude Code agent read the spec freely. Neither finds a validated inconsistency. Both often report a conflict while missing a nearby exception that resolves it. For example, they flag a transformation permission as conflicting with content prohibitions, although the transformation passage says that the permission still applies when the output would otherwise be restricted.

Ablations. Each ablation removes or replaces one stage of VeriSpec, and each one loses validated inconsistencies. Every validated inconsistency found by an ablation is among VeriSpec's five. The new ablation, EmbedCluster, replaces the rule graph with embedding clusters. It finds only 2 validated inconsistencies. It misses the output-only, fictional-portrayal, and polite-reassurance cases, because embedding similarity places the two rules of each case in different analysis units. In the revised paper, Table 1 now separates the long-context baselines from the ablations and includes EmbedCluster, Section 5.1 states that all methods use the same reasoning effort, Section 5.3 discusses EmbedCluster, and Appendix C describes its implementation.

Ethics flag. We appreciate the reviewer's attention to ethics. The work analyzes a public document and involves no human subjects or private data. Anonymization only replaces provider and model names, and a programmatic check confirms that the rest of the text is unchanged. Its purpose is to keep models from refusing to analyze the document, not to obtain harmful content. We have reworded "suppresses guardrails" to "avoids refusals to analyze the document" in the Anonymization paragraph of Section 4, and Appendix D (Table 6) now reports refusal rates with and without anonymization. We reported all five inconsistencies to the OpenAI Model Spec developers. They acknowledged all five, and their latest release (October 29, 2026) addresses two of them. The ethics statement of the revised paper now states both the purpose of anonymization and the developers' response.

[1] Mu et al. Rule Based Rewards for Language Model Safety. NeurIPS 2024.

[2] Rahman et al. CellularLint: A Systematic Approach to Identify Inconsistent Behavior in Cellular Network Specifications. USENIX Security 2024.

[3] Sun et al. Clover: Closed-Loop Verifiable Code Generation. SAIV 2024.

Thank you again for your time and your detailed review. As the discussion period draws to a close, we would like to kindly follow up on our response. We have answered each of your points and uploaded a revised paper with all changes highlighted in blue. In particular, we clarify that we ask the LLM to reason like a verifier without claiming formal guarantees, explain why formal verification does not fit natural-language specifications, describe the safeguards against error propagation, compare VeriSpec with two long-context baselines and four ablations, and address the ethics flag.

We would be grateful to know whether our response addresses your concerns. If any question remains, we are happy to clarify it before the discussion period ends.

Reviewer_cHwVOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces an approach to detect inconsistencies in model specification. The approach VERISPEC extracts rules then construct topic-guided graphs to cluster on the authority level, then apply LLM-as-verfier to detect inconsistencies.

Critical strengths

S1: The study is relevant to model alignment, an important topic in the era of LLMs.

S2: The study points out the existing inconsistencies in the existing model spec, informing the design of the specs to be aware of these inconsistencies.

Critical weaknesses

W1: This work proposes an engineering pipeline that is rather standard without introducing anything novel that is inspired by the current task. The conclusions are primarily derived from the case study of model specification, which can lack generalizability to other tasks.

W2: The pipeline involves a graph component for clustering, however, there are no comparisons with a semantic filter baseline to show if the graph component is really necessary.

W3: The results table compares methods on the metric cost per validation, however, the small denominator ranging from 1-5 introduces volatility to the results. Moreover, the results are derived from 1 run, lacking statistical significance. And, since the ground truth is unknown, this work is not able to calculate the recall, masking the true capability of the proposed approach.

Questions for authors

W1-W3

3 comments
AuthorsOpenReview ↗

Response to Reviewer cHwV (1/2)

We have uploaded a revised paper with all changes highlighted in blue, and we note below where each change appears.

cHwV-Weakness1: novelty and generality.

We thank the reviewer for raising this. We agree that the individual techniques in VeriSpec, such as LLM prompting and clustering, are not new. What is new is how the pipeline is designed around the way model specs are written. Related principles often appear in different sections, and an apparent conflict between two principles is often resolved elsewhere in the spec by an exception, a worked example, or the chain of command. Each stage of VeriSpec addresses a failure that we observed when a method ignores these properties. All methods below use Claude Opus 5 with five attempts.

Observed failureMethod without the stageValidated / reported
Reports conflicts that nearby context already resolvesDirectPrompt, DirectAgent (whole spec)0/16, 0/9
Misses conflicts between rules in different sectionsSectionText, SectionGroup (one unit per section)1/10, 1/8
Buries valid inconsistencies among resolved conflictsPlainDetect (no witness or context-resolution argument)3/221
—VeriSpec5/13

Two examples illustrate these failures. Whole-spec prompting repeatedly reports a transformation permission as conflicting with content prohibitions, although the transformation passage says that the permission still applies when the output would otherwise be restricted. Section-based units cannot find the code-transformation inconsistency at all, because its two principles ("do not change unrequested parts of the text" and "code for functional use should be executable") appear in different sections.

cHwV-Weakness2: semantic-filter baseline.

We thank the reviewer for suggesting this baseline, and we show the results below.

New baseline. To test a semantic filter directly, we ran EmbedCluster, which replaces the rule graph with embedding clusters. It embeds each rule's statement, precondition, and postcondition with BGE-large (bge-large-en-v1.5) [1] and clusters the rules within each authority level by size-constrained spherical k-means. For a fair comparison, EmbedCluster matches VeriSpec in the number of analysis units (32), the number of rule–unit memberships (rules also join their second-nearest cluster until the totals match), and the unit sizes (5–26 rules, with a median of 16.5 against VeriSpec's 16). Rule extraction, context, verifier prompt, model, effort, and the five attempts are unchanged, so only the grouping differs.

MethodReportedValidatedPrecisionCostCost per validated
EmbedCluster6233.3%$40.55$20.28
VeriSpec13538.5%$55.62$11.12

EmbedCluster finds the legal-omission and code-transformation cases but misses the other three. The cause is the grouping. In the output-only case, embedding similarity places the two rules ("follow transformation instructions without comment" and "highlight any potential misalignment" in a false premise) in different units, so the LLM verifier never sees them together. The same happens in the fictional-portrayal and polite-reassurance cases. The rule graph places each pair in a shared unit, because both rules govern the same situation even though they address different concerns. Both inconsistencies that EmbedCluster validates are among VeriSpec's five. In the revised paper, EmbedCluster appears in Section 5.1, Table 1, and Section 5.3, with implementation details in Appendix C.

AuthorsOpenReview ↗

Response to Reviewer cHwV (2/2)

cHwV-Weakness3: cost ratio, single run, and recall.

We thank the reviewer for these points on statistical robustness and recall.

Cost per validated inconsistency. We agree that this ratio is sensitive to small counts. With only 1 to 5 validated inconsistencies per method, one more or one fewer changes it a lot. However, our conclusion does not depend on this ratio. VeriSpec validates the most inconsistencies at a total cost similar to the baselines that validate any ($47.46 to $63.94). The revised paper now compares methods mainly by total cost and the number of validated inconsistencies, and reports the ratio only as secondary information in Section 5.1 and Table 1. The table below also gives 95% confidence intervals for precision, which Table 1 of the revised paper now includes.

MethodValidated / reportedPrecision (95% CI)Total cost
PlainDetect3/2211.4% [0.5, 3.9]$47.46
SectionText1/1010.0% [1.8, 40.4]$58.64
SectionGroup1/812.5% [2.2, 47.1]$63.94
DirectPrompt0/160% [0, 19.4]$4.92
DirectAgent0/90% [0, 29.9]$27.66
VeriSpec5/1338.5% [17.7, 64.5]$55.62

VeriSpec's precision is significantly higher than that of PlainDetect (two-sided Fisher exact test, p < 10⁻⁴) and DirectPrompt (p = 0.011). The section-based baselines report too few candidates (10 and 8) for a precision test to be informative, but a clearer difference is in the number of validated inconsistencies, where VeriSpec validates 5 and each of these two baselines validates only 1, at a similar total cost. Section 5.3 of the revised paper now reports these tests.

Runs. The results do not come from a single run. Every method, including each baseline, ran five independent attempts over all analysis units, and the reported counts are the union of the five attempts. For VeriSpec, the number of unique candidate inconsistencies grew as 1, 4, 10, 10, and 13 over the five attempts. Two extra attempts, run only to check saturation and excluded from the cost, found no new candidates. The revised paper now states this in Section 5.2, the caption of Figure 4, and Appendix C.

Recall. We agree that recall cannot be measured without ground truth, and the revised paper now states this in the new Limitations paragraph of Section 6. As an estimate, we report pooled recall, a standard approach in information retrieval when complete labels are unavailable. The pool contains every validated inconsistency found by any of compared methods. All of them are among VeriSpec's five. Pooled recall is therefore 5/5 for VeriSpec, 3/5 for PlainDetect, 1/5 for each SectionText and SectionGroup, and 0/5 for DirectPrompt and DirectAgent. Section 5.1 of the revised paper defines pooled recall, and Table 1 reports it for every method.

All other baseline methods found no inconsistency outside VeriSpec's five, and two extra VeriSpec attempts found no new candidates. This suggests that VeriSpec does not miss inconsistencies that other designs find easily. This evidence is indirect but still convincing since it holds across very different detection designs.

[1] Xiao et al. C-Pack: Packed Resources For General Chinese Embeddings. SIGIR 2024.

Thank you again for your time and your detailed review. As the discussion period draws to a close, we would like to kindly follow up on our response. We have answered each of your points and uploaded a revised paper with all changes highlighted in blue. In particular, we explain how each stage of VeriSpec addresses a failure we observed in simpler designs, add the semantic-filter baseline you suggested (EmbedCluster), and report 95% confidence intervals, significance tests, results across attempts, and pooled recall.

We would be grateful to know whether our response addresses your concerns. If any question remains, we are happy to clarify it before the discussion period ends.

Reviewer_ndBzOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes VeriSpec, which detects inconsistencies in natural-language model specifications by using an LLM (Claude Opus 5) as the verifier. The specification is split into structured natural-language rules, each rule gets its interpretive context attached, and rules are clustered by topic within the same authority level. The authors motivate context attachment by noting that "a verifier shown only the statements reports tensions the document already resolves".

Critical strengths
  1. The idea is useful, and the five reported inconsistencies are of practical value. However, I do not think it fits as an ICLR research paper.
Critical weaknesses
  1. Heavy reliance on AI at every stage, with limited human verification. The AI use statement says generative AI implemented the pipeline, ran the experiments and polished the text. Claude Code also completes the annotations (126 of 405), extracts rules, derives topics and anonymizes the spec. Apart from the final two-person check of 13 candidates, no human evaluation of these intermediate artifacts is reported.

  2. Calling the LLM a verifier misleads the reader. A verifier's output is by definition trusted because it carries a mathematical proof (or a counterexample) that a statement is true (see SMT solver or others). This is not what VeriSpec does. There is no proof or soundness guarantee, and no measure of recall. The LLM may flag false conflicts (62% of its reports were rejected) and may also miss real ones, and the paper has no way to estimate how many it misses.

  3. The method is, in essence, a structured prompt to an LLM. It is more like a tool to flag potential inconsistencies that a human then checks.

  4. All results come from the OpenAI Model Spec, which has explicit authority labels and focus-area annotations that the pipeline depends on.

Questions for authors
  1. How many detection attempts were run: five (as in Section 5.2 and Table 1) or seven (as in Figure 4)? Is the cost of attempts 6–7 included?

  2. Who were the two validators, what rubric did they use, and what was their agreement before discussion?

  3. What refusal rates did you observe with and without anonymization, and for which models?

  4. Are PlainDetect's 3 validated findings, and the legal-disclosure case found by the section baselines, all among VeriSpec's 5? Did any baseline find a valid inconsistency VeriSpec missed?

4 comments
AuthorsOpenReview ↗

Response to Reviewer ndBz (1/3)

We have uploaded a revised paper with all changes highlighted in blue, and we note below where each change appears.

ndBz-Weakness1: reliance on AI and human verification. We thank the reviewer for raising this. AI was heavily used in our pipeline, but all intermediate results and artifacts are verified and validated by the authors. Every pipeline stage has a check, and two authors validate every reported inconsistency.

  1. Anonymization: a programmatic check confirms that only provider and model names are replaced and the rest of the text is unchanged.
  2. Rule extraction, step 1 (annotating principles): a programmatic check confirms that annotation only inserts tags. Two authors then reviewed the completed annotation section by section for missed principles and over-fragmented sentences.
  3. Rule extraction, step 2 (transforming principles to rules): a programmatic check confirms that every annotation maps to exactly one rule.
  4. Graph-based clustering (topic tests): each topic comes with a yes/no test that decides whether a rule belongs to it. Two independent LLM labelers applied each test to a 40-rule sample, and tests were revised until Cohen's κ ≥ 0.6 (≥ 0.87 for sub-topics; the one failing test rose from 0.48 to 0.925). A held-out check, based on evidence that the labelers never saw, also confirmed the tests. Rule pairs linked by a cross-reference share a topic 7.16× more often than random pairs (p ≤ 0.001).
  5. Detection (reported candidates): two authors validated every candidate.

In the revised paper, the new Appendix D lists these checks by stage (Table 5), and Section 4.1.2 now describes the topic tests and the held-out check.

ndBz-Weakness2: "verifier" is misleading. We thank the reviewer for this terminology point. We agree that the LLM provides neither soundness nor completeness, and the revised paper now states this explicitly in Section 1, Section 4.3, and the new Limitations paragraph of Section 6. We use "verifier" in the sense that is common in machine learning, where a verifier is a model that checks whether a candidate is correct, without a formal guarantee. For example, verifiers check candidate solutions to math problems [1], and LLM verifiers check candidate code patches and agent trajectories [2]. VeriSpec follows this pattern. The LLM checks each candidate rule pair against our definition of inconsistency, instead of flagging any apparent tension, and supports each report with evidence that a human can check. Section 4.3 of the revised paper now gives this definition of a verifier with both citations.

Concretely, each report is a certificate: the two rule IDs, a witness situation, why no behavior satisfies both rules, and why the context does not resolve the conflict. A human can check this certificate much as one checks a counterexample. The revised paper now describes each report as such a certificate in the abstract, Section 1, and Section 4.3.

We acknowledge that the LLM can still report false conflicts (8 of its 13 reports were rejected). But the certificate keeps this manageable. On the same units without it, a human would have to check 221 reports to find 3 valid inconsistencies. With it, they check 13 and find 5. The Limitations paragraph of Section 6 in the revised paper now states this trade-off.

AuthorsOpenReview ↗

Response to Reviewer ndBz (2/3)

ndBz-Weakness3: purely a structured prompt.

We thank the reviewer for this perspective. We agree that VeriSpec trains no new model and that each of its stages prompts an LLM. Our contribution is how the pipeline structures the problem for the LLM, namely which rules it sees together, what context it receives, and what evidence it must return. Such pipelines are established research contributions when they find real defects. For example, IRIS [3] prompts an LLM to infer taint specifications for a static analyzer and to filter its false alarms, and finds 55 of 120 known vulnerabilities, compared with 27 for the analyzer alone. Fuzz4All [4] prompts an LLM to generate and mutate test inputs, and found 98 bugs in systems such as GCC, Clang, Z3, and OpenJDK. Agentless [5] replaces a free-acting agent with a fixed sequence of prompts for localization, repair, and patch validation, and outperforms open-source agents at a lower cost. VeriSpec applies the same idea to model specifications. We added this discussion, with these three works, as a new paragraph in Section 3 of the revised paper.

We also agree that VeriSpec flags potential inconsistencies for a human to check, so its value depends on which candidates it flags. The version we analyzed has about 38,000 words, and its 405 rules form 27,589 same-authority pairs. This is far too much for a person to inspect by hand, so the tool is only useful if it flags a short list that contains the real inconsistencies. A structured prompt without our pipeline does not do this, as the baselines below show.

  • Giving the model the whole specification, in one long-context call or as a Claude Code agent, produced 25 reports and no valid inconsistency.
  • Grouping rules by section found only 1 valid inconsistency, because most cases involve rules from different sections.
  • Our groups without the certificate requirement produced 221 reports with only 3 valid, compared with 13 reports and 5 valid with it.
  • Grouping rules by embedding similarity instead of our topic graph found 2 of the 5. Similarity put the two rules of the output-only case, and those of the fictional-portrayal case, in different groups.
  • Removing the context attached to each rule raised the reports from 13 to 22, lowered precision from 38.5% to 9.1%, and missed three cases.

The results show that VeriSpec can save human effort and budget while achieving higher precision. A person checks only 13 reports to find 5 inconsistencies and each validated inconsistency costs $11.12, the lowest among the original baselines, and its precision of 38.5% is the highest of all methods. Table 1 and Section 5.3 of the revised paper now report all of these comparisons, including the new embedding-clustering baseline.

On fit to ICLR: Model specifications are used to generate alignment training data (as in constitutional AI and deliberative alignment), to guide reasoning at inference time, and to define evaluation criteria. An inconsistency in the specification therefore becomes an inconsistency in what models are trained and evaluated on. Prior work tests whether models follow a specification; we test whether the specification itself is consistent. Our findings are also useful in practice: the OpenAI Model Spec developers acknowledged all five inconsistencies, and their latest release (October 29, 2026) addresses two of them. The revised paper reports this in the abstract, Section 1, and the ethics statement.

AuthorsOpenReview ↗

Response to Reviewer ndBz (3/3)

ndBz-Weakness4: dependence on OpenAI's labels and annotations.

We thank the reviewer for raising this point. We first clarify a misunderstanding caused by our paper. The paper lists Claude's Constitution as a model spec, but it is not one, because it does not define concrete model behavior as the OpenAI Model Spec does. We have corrected this in Section 1 and at the start of Section 5 of the revised paper. To our knowledge, the OpenAI Model Spec is the only public document that specifies model behavior for a publicly deployed model as detailed rules at this scale, so it is currently the only available evaluation target.

VeriSpec does not depend on any property specific to the OpenAI Model Spec. The focus-area annotations are not required. The pipeline identifies principles with a general criterion, namely any statement that prescribes, permits, or prohibits a model behavior in a situation concrete enough to judge compliance. For the OpenAI Model Spec, we kept its 279 existing annotations, and the pipeline added the other 126 with this criterion. On a specification without annotations, the same step annotates every principle from scratch, and the 279 existing annotations can serve as few-shot examples, as they did for the OpenAI Model Spec. The authority labels are not required either. They only restrict which rule pairs are compared, because the chain of command already resolves conflicts between levels. We expect any model spec to define authority in some form, even if less explicitly than the OpenAI Model Spec, because the instruction hierarchy is central to model alignment. For example, Claude's Constitution gives different levels of trust to its three types of principals, namely Anthropic, operators, and users. If a specification defines no authority, VeriSpec simply drops this restriction. This only adds rule pairs to compare, so it cannot cause VeriSpec to miss an inconsistency. We added this discussion to the revised paper as Appendix E.

ndBz-Question1: five or seven attempts?

We thank the reviewer for catching this ambiguity. All methods ran five independent attempts, and all reported counts and costs come from those five. We ran two extra attempts only to check saturation. They found no new candidates, and their cost is not included in the $55.62. The revised paper now clarifies this in Section 5.2, the caption of Figure 4, and Appendix C.

ndBz-Question2: validators, rubric, agreement.

We thank the reviewer for asking about the validation protocol. The two validators are authors. Each independently reviewed every candidate of every method, including the rule pair, the witness, and the surrounding specification text. A candidate is valid only if (i) both rules share an authority level, (ii) the witness, or a stronger feasible situation, triggers both rules, and (iii) no behavior satisfies both after exceptions, worked examples, modality, and the chain of command are considered. Disagreements were resolved by discussion.

Before discussion, the validators agreed on 9 of the 13 VeriSpec candidates, and on 252 of the 264 candidates reported in total by the five baselines (DirectPrompt, DirectAgent, SectionText, SectionGroup, and PlainDetect). The revised paper adds these criteria to Section 5.1 and the agreement figures to Appendix D.

ndBz-Question3: refusal rates.

We thank the reviewer for asking about refusals. We ran the detection stage of VeriSpec, namely 160 requests per model (32 analysis units × 5 attempts), once on the original specification and once on the anonymized one. We counted a request as refused when the model declined to analyze the units. Without anonymization, 30 of 160 requests were refused for GPT-6 Astra (18.8%), 48 for Claude Opus 5 (30.0%), and 93 for Claude Fable 5.1 (58.1%). With anonymization, the rates were 0%, 0%, and 36.3%. The revised paper reports these rates in Appendix D (Table 6) and summarizes them in the Anonymization paragraph of Section 4 and in Section 4.3.

ndBz-Question4: baseline overlap.

We thank the reviewer for this precise question. Yes. Every validated inconsistency found by any baseline is among VeriSpec's five, and no baseline found one that VeriSpec missed. SectionText and SectionGroup each found only the legal-disclosure case, and PlainDetect found three of the five. The same holds for the embedding-clustering baseline we added for this rebuttal (DWW2-Weakness3), which found two of the five. The revised paper adds this result to Section 5.3 and a pooled-recall column to Table 1.

[1] Cobbe et al. Training Verifiers to Solve Math Word Problems. arXiv 2021.

[2] Kwok et al. LLM-as-a-Verifier: A General-Purpose Verification Framework. arXiv 2026.

[3] Li et al. IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilities. ICLR 2025.

[4] Xia et al. Fuzz4All: Universal Fuzzing with Large Language Models. ICSE 2024.

[5] Xia et al. Agentless: Demystifying LLM-based Software Engineering Agents. FSE 2025.

Thank you again for your time and your detailed review. As the discussion period draws to a close, we would like to kindly follow up on our response. We have answered each of your points and uploaded a revised paper with all changes highlighted in blue. In particular, we describe the checks at every stage of the pipeline, clarify the sense in which we use "verifier", explain why VeriSpec does not depend on OpenAI's labels and annotations, and report the validation protocol, the refusal rates with and without anonymization, and the overlap between the baselines and VeriSpec.

We would be grateful to know whether our response addresses your concerns. If any question remains, we are happy to clarify it before the discussion period ends.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.