acceptodds
Withdrawn from ICLR 2027

claimcheck: Measuring and Improving Specification Faithfulness

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Formal verification proves that a program satisfies its specification, but says nothing about whether the specification captures what the author meant. As LLMs increasingly write both code and specifications, the risk of a gap between the author's natural-language requirement and the specification emerges. A machine-checked proof of a property narrower than the one requested is worthless but passes every check. Closing this gap is solving the spec formalization problem: given a natural-language requirement, produce a formal artifact whose meaning matches it. To detect this problem, we curate an end-to-end benchmark for spec formalization spanning across domains, Dafny contracts, Lean and SQL queries. Each task pairs a natural-language requirement with a formal artifact and a human-checked label. We measure success with a standardized grading script and establish baselines for frontier models. We introduce claimcheck, a tool that detects incorrect formalizations: the formal artifact is translated back into plain English by a model that never sees the requirement, and a separate model, the judge, decides whether the informalization and the requirement express the same guarantee. Because the informalization is written without sight of the requirement, it reports what the formal artifact says rather than what the requirement leads a reader to expect, and judging against it catches subtle drift such as weakened postconditions and narrowed scope.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
formalization, benchmark, claims, specification mining

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 4 reviews, above 0% of the others.
Reviewer_a8RaOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper tackles the problem of spec formalization faithfulness for formal verification. The authors address the gap between author's natural language requirement (NLR) and the specs expressed in a formal language, which need to be verified for consistency. They build a benchmark of 3234 rows with NLRs, their corresponding formal artifacts and a label stating whether the match is faithful or unfaithful. The benchmark spans 4 formal languages, such as Dafny contracts and Lean 4, Isabelle and Rocq theorem statement. They additionally introduce ClaimCheck - a tool for detecting incorrect formalizations -- the formal artifact is translated back to English by a model that never sees the original requirement, and an LLM-judge decides whether the informalization and the requirement express the same guarantee. ClaimCheck performs well compared with simple baselines like zero-shot judge (direct judge) and direct judge + guidance baseline. However the performance gap shrinks as the judge becoming stronger.

Critical strengths
  • Tackling an important problem that has solid motivation. The paper addresses an important problem for scaling AI-generated code, proofs and formal specs -- checking whether a formal specification actually aligns directly with the original natural language requirement (NLR). That's very important, as there are multiple cases of specifications being formally verified, however they might be expressing the wrong guarantee.

  • Creation of a benchmark and empirical evaluations. The paper constructs a benchmark of 3234 examples across 4 formal languages, with 1029 meaning-preserving transformations that help prevent models from simply detecting edited specifications -- that's very useful, as other people can build upon it. Additionally, the authors provide claimcheck, a 2-stage checking pipeline and demostrate improvements of 4.2 to 6.0 percentage points (pp) in macro recall vs the strongest direct-judging baseline competitor. Ablation experiments additionally demonstrate that removing back-translation stage reduces the macro recall significantly (2.8 to 6.0 pp), and allowing the informalizer to see the original requirement slightly reduces (0.8 to 3.0 pp), which basically suggests that separating the available information helps to improve the scores, which should be used by different protocols.

Critical weaknesses
  • Limited novelty in terms of methodology and insufficient comparison with prior approaches. The proposed ClaimCheck method appears to be very similar to existing semantic checking methods using the back and forth technique. Specifically, Clover (Sun et al., 2024: https://arxiv.org/pdf/2310.17807) already uses the translation of formal annotations back into natural language without seeing the original requirement, and the independent LLM-based equivalence check follows it. Monty (Liu et al., 2026: https://arxiv.org/pdf/2607.13303) also uses a similar approach. Even though the paper acknowledges these methods, it does not directly compare against their checking protocols using the same experimental settings. Additionally, it is difficult to understand what specific methodological improvement ClaimCheck adds beyond evaluating an existing approach on the new benchmark

  • Limited evaluation of practical value and whether the benchmark generalizes well. The experiments mainly compare ClaimCheck agains simple direct-judging baselines and does not compare it to other methods (mentioned above) and other stronger approaches, e.g. based on reasoning. Additionally, the performance gain becomes much smaller with stronger judges in the model sweep (though these experiments use a variant of ClaimCheck and not its main configuration). The benchmark in essence consists of synthetically generated errors, including a big chunk of simple negation versions in Isabelle and Rocq -- it is not clear whether the improvements observed actually generalize to real-work mistakes that are made by LLM-based formalizatoin systems. Lastly, most of the perfomance improvement comes from correctly accepting faithful specifications, when detection of unfaithful ones becomes worse (e.g. on Lean 4 unfaithful recall drops from 95.9% to 88.4%), which is quite concernign for formal verification, as more incorrect specs can be undetected.

Questions for authors
  • Q1\textbf{Q1}. What is fundamentally new comparing (at least) to Clover and Monty (described above)? Could the authors clarify the specific methodological contribution and provide comparisons with these methods?
  • Q2\textbf{Q2}. Could the authors evaluate ClaimCheck on the actual incorrect specs generated by LLMs and not only rule-based variations -- it will help in understanding whether it really helps in practice.
  • Q3\textbf{Q3}. In the model sweep why is the only option being evaluated is the option where the judge sees the artifact -- is it possible to report the main ClaimCheck config using the strongest judge and compare it with the baselines?
  • Q4\textbf{Q4}. It is not clear that the choice fo evaluation metric is the best given the last sentence from weaknesses field -- with the decrease in unfaithful recall on Lean 4, how should the tradeoff between false acceptance and false rejection be interpreted for real verification applications?
2 comments
AuthorsOpenReview ↗

Response to Reviewer a8Ra

Q1 (what is new compared with Clover and Monty). Please see point 1 of our general comment. For Clover and Monty specifically: in both, the back-translation is one component inside a larger pipeline that also involves sampling and, in Monty's case, running a test generator. Clover does score its check alone, but never against a judge shown both the requirement and the specification. claimcheck is only one informalization with haiku-4.5 plus one judge call.

Q2 (real LLM-generated specifications). You are right, and we will address this in the final version. Please see point 2 of our general comment.

Q3 (main configuration with the strongest judge). We thank the reviewer for their suggestion. We have now run this, and the results are in point 3 of our general comment. With gemini-3.7-flash, the main configuration is level with baseline + list on Isabelle and Dafny, ahead on Rocq and behind on Lean.

Q4 (macro recall and the drop on Lean). This is a fair criticism of macro recall. For verification a missed unfaithful specification is the more expensive one. Point 4 of our general comment splits macro recall into its two parts and shows the trade-off. For a strong judge, or wherever a missed unfaithful specification is the more expensive error, showing the specification as well is the safer setting. We will report both parts in the revised version.

Thank you for the detailed response, however I will maintain my assessment.

Reviewer_uD6TOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper construct an autoformalization benchmark in formal verification and formal math, across four formal languages. It also introduces claimcheck, a metric for autoformalization by translating the candidate specification back to natural language and judge its alignment with the ground truth (i.e. auto-informalization).

Critical strengths

None

Critical weaknesses

1/ The benchmark construction is simply a collection of existing dataset and mutation. There's no discussion or guarantee of data contamination, and the existing datasets (Clover and PutnumBench) are already saturated by current LLMs. The scope choice is also not considerate. While the paper introduces this work to be an autoformalization benchmark on software formal verification, there's only 362 instances about coding (from Clover), and other ~3k are all math (from PutnumBench).

2/ Using auto-informalization to check formal statement alignment has an established line-of-work already. Clover [1], Lean Workbook [2], and Herald [3] already use back-translation followed by semantic comparison. FormalAlign [4] and Mathesis [5] also compare this approach with direct judging. The paper failed to clearly explain the difference between claimcheck and the existing work, nor compare with these work with experiments.

[1] Clover: Closed-Loop Verifiable Code Generation. Sun et al.

[2] Lean Workbook: A Large-Scale Lean Problem Set Formalized from Natural Language Math Problems. Ying et al.

[3] Herald: A Natural Language Annotated Lean4 Dataset. Gao et al.

[4] FormalAlign: Automated Alignment Evaluation for Autoformalization. Lu et al.

[5] Mathesis: Towards Formal Theorem Proving from Natural Languages. Yu et al.

Questions for authors

1/ Can you explain the difference between claimcheck and established work? I couldn't understand the related work section. For example, when comparing to Clover, the paper wrote "It scores that check alone, accepting all 60 originals and rejecting 40 to 70% of each incorrect class, but never against a model shown both sides; and on its rows, where every faithful row is untouched, detecting edits would score the whole faithful half.". Could you please explain what does "never against a model shown both sides" mean and connects to claimcheck? And what does "detecting edits would score the whole faithful half"? Also, I believe experimental comparison is also very important, given the similarity of this work and prior ones.

2/ Can you provide evidence or citation for the following claim: "The trouble is that the requirement gives the answer away. A model that has just read sort a list expects the specification to describe sorting, and is biased toward agreeing that it does"?

3/ Can you elaborate the need of meaning-preserving controls? During evaluation, isn't the judge only shown one row at a time, so it cannot know if the current row is a mutation or origin statement?

2 comments
AuthorsOpenReview ↗

Response to Reviewer uD6T

Weakness 1 (contamination and scope). On contamination, this is a fair concern and we should have discussed it. Only 825 of the 3,234 rows come from the public datasets. The other 2,409 are edits we made, so they cannot be in any training data, and knowing the original does not say whether an edit changed the meaning. We cannot rule contamination out, and we will add this discussion to the revised version. On scope, you are right: 362 rows are Dafny contracts and 2,872 are theorem statements. We will state this plainly.

Weakness 2 and Q1 (difference from prior work). We agree the related work section was hard to follow, and we will rewrite it. Our answer on prior work is in point 1 of our general comment. The two sentences you ask about mean the following. "Never against a model shown both sides": Clover scores its check alone, but does not compare it with simply asking a model that sees both the requirement and the specification. "Detecting edits would score the whole faithful half": in Clover's test rows every correct row is the untouched original. A checker that only notices edits gets every correct row right without understanding the meaning. Our benchmark adds rows that are edited but still correct.

Q2 (evidence for "the requirement gives the answer away"). You are right that the sentence needs support where it appears. Our evidence is the ablation in Section 5.2: when the back-translation model also sees the requirement, it catches 6.0 to 8.8 points fewer wrong specifications. Its description follows the requirement and not the specification. We will point to this result and cite the related work from Section 6 where the sentence appears.

Q3 (why the controls are needed). You are right that the judge sees one row at a time. But an edited specification often looks unusual on its own, so without the controls a judge can score well by rejecting anything that looks unusual. The controls are correct specifications written differently. The zero-shot Sonnet judge accepts about three quarters of the originals but only a little over half of the controls, so it does react to how a specification is written.

Thank you for the response and the additional results. My main concerns remain, and I will maintain my score.

Reviewer_L1dxOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper evaluates whether formal specifications match natural-language requirements. It introduces a benchmark across four formal languages and tests a pipeline that back-translates specifications into English before comparing them with the requirements.

Critical strengths
  • Meaning-preserving controls address a useful evaluation concern: distinguishing semantic errors from harmless changes in wording.

  • The paper is clearly written and includes ablations of its main design choices.

Critical weaknesses

The contribution is limited. The back-translation approach has already been used in prior work, and the paper mainly extends its evaluation to more languages and models. Most tests involve synthetic edits, and the results offer limited new insight into detecting errors in actual model-generated specifications.

Claimcheck achieves higher macro recall mainly by accepting more faithful specifications. On Lean, its reported recall on unfaithful specifications drops from 95.9% to 88.4% compared with the informed baseline. For a tool intended to detect specification errors, it is unclear whether fewer false alarms justify missing more incorrect specifications.

Questions for authors
  • The labels need another check. For example, in putnam_1964_a4, a positive integer period can always be doubled, so requiring a period greater than 1 does not change the statement.
  • It would be useful to test the main version with the strongest judge as well. The model sweep currently uses the version that also sees the formal specification.
  • Are variants of the same problem treated as independent samples when computing the confidence intervals?
1 comment
AuthorsOpenReview ↗

Response to Reviewer L1dx

Weaknesses (contribution, synthetic edits, and the drop on Lean). Please see points 1, 2 and 4 of our general comment. On Lean, the wrong specifications the main configuration accepts are mostly one-character edits (58 of 77). With the specification also in view, the judge catches 47 of the 77.

Q1 (label check, putnam_1964_a4). You are right about putnam_1964_a4, and thank you for checking. We have re-checked the labels and found 27 rows with a wrong label. We will correct them, re-run the results and release the corrected benchmark with the revised version.

Q2 (main configuration with the strongest judge). We thank the reviewer for their suggestion. We have now run this, and the results are in point 3 of our general comment.

Q3 (confidence intervals). Yes, in the submission they were, and you are right that they should not be. We recomputed the intervals by resampling whole problems, so all variants of a problem stay together. The gain of claimcheck over baseline + list, with 95% intervals:

ProblemsGain95% interval
Lean293+4.8+2.4 to +7.2
Isabelle285+5.9+3.2 to +8.7
Rocq185+5.0+1.8 to +8.1
Dafny62+4.2−0.1 to +8.8

The gain holds on Lean, Isabelle and Rocq. On Dafny, with 62 problems, the interval now reaches zero. We will report these intervals in the revised version.

Reviewer_3tNnOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper aims to address the problem of how to ensure the alignment of a formal specification and an informal natural-language requirement. The paper proposes a method named claimcheck to address this problem. claimcheck first uses an LLM to transform the formal specification into natural language and then asks an LLM to judge whether the obtained requirement is the same as the original one. The paper also assembles 3,234 requirement–specification pairs based on existing formal program benchmarks.

Critical strengths

The paper clearly states the specification–requirement alignment problem, proposes a pipeline that evaluates the alignment, and constructs a benchmark containing requirement–specification pairs. The writing is generally easy to follow.

Critical weaknesses
  • The main concern is novelty. The claimcheck framework is too direct to constitute a sufficient contribution; perhaps it could serve as a direct baseline.
  • Missing comparisons. The specification–requirement alignment problem has already been investigated in many studies, e.g., the paper “Intent Formalization: A Grand Challenge for Reliable Coding in the Age of AI Agents” and the references therein.
Questions for authors

Does claimcheck work on real LLM-generated specifications, rather than on deliberately mutated examples?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 3tNn

Weakness 1 (novelty). This is a fair concern, and other reviewers raised it too, so we answer it in point 1 of our general comment. Our main goal was a tool that is cheap enough to plug into any pipeline.

Weakness 2 (missing comparisons). Thank you for this reference. We were not aware of it and we will cite it.

Q1 (real LLM-generated specifications). You are right that our incorrect rows are rule-based edits, and we answer this in point 2 of our general comment. claimcheck also surfaced real mismatches in the benchmarks themselves. In CloverBench's online_max, the docstring asks for the index of the first element that exceeds the maximum of the first x elements, but the contract accepts any later element that exceeds everything before it. For a = [1, 5, 9, 0] and x = 1, the docstring requires p = 1 and the contract also admits p = 2. CloverBench labels this pair as correct, and claimcheck disputes it. We will include more examples of this kind in the revised paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comment - To all reviewers

We thank the reviewers for their careful reading. Four points came up in more than one review, so we answer them here and refer to them in the individual replies. We will update the PDF soon.

1. Novelty. The back-translation mechanism is shared with Clover, Monty and others, and we discuss them in Section 6. In those works it is one component inside a larger pipeline. With the exception of Mathesis, which does so with one model on one language, it is not compared against a judge shown both the requirement and the specification, and the test rows contain no correct specifications that have been rewritten. Our contribution is to evaluate this check on its own, against that judge, on a benchmark where a correct specification can also be an edited one.

2. Real LLM-generated specifications. You are right that our incorrect rows are rule-based edits. We built the benchmark this way because we needed rows we know are incorrect. Dafny is the exception, where the verifier gives a partial oracle, and that is where an LLM-generated slice is feasible. We will address this in the final version.

3. The main configuration with the strongest judge (new result). We ran the main configuration from Table 2 with gemini-3.7-flash. Macro recall:

gemini-3.7-flashLeanIsabelleRocqDafny
zero-shot baseline94.694.891.491.4
baseline + list95.695.292.295.6
claimcheck, judge sees artifact95.296.192.895.3
claimcheck (main)89.195.493.695.6

Compared with baseline + list, claimcheck is level on Isabelle and Dafny, ahead on Rocq (p = 0.007) and behind on Lean. The gain is large for weaker judges and disappears for the strongest.

4. Fewer wrong specifications caught on Lean (new analysis). The table splits macro recall into its two parts: correct specifications accepted / wrong specifications caught.

LeanIsabelleRocqDafny
Sonnet, sees artifact75.4 / 94.176.1 / 98.467.9 / 99.188.9 / 94.5
Sonnet, main87.5 / 88.486.4 / 96.177.8 / 98.291.7 / 94.5
Gemini, sees artifact98.3 / 92.293.1 / 99.087.3 / 98.297.2 / 93.4
Gemini, main96.2 / 82.095.4 / 95.592.6 / 94.698.9 / 92.3

Hiding the specification from the judge makes it accept more correct specifications and catch fewer wrong ones. The wrong Lean specifications it accepts are mostly one-character edits, a ≤/< swap or a changed constant (58 of 77 for Sonnet, 98 of 120 for Gemini). For a strong judge, or wherever a missed unfaithful specification is the more expensive error, showing the specification as well is the safer setting. We will report both parts and say this in the revised version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.