- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper tackles the problem of spec formalization faithfulness for formal verification. The authors address the gap between author's natural language requirement (NLR) and the specs expressed in a formal language, which need to be verified for consistency. They build a benchmark of 3234 rows with NLRs, their corresponding formal artifacts and a label stating whether the match is faithful or unfaithful. The benchmark spans 4 formal languages, such as Dafny contracts and Lean 4, Isabelle and Rocq theorem statement. They additionally introduce ClaimCheck - a tool for detecting incorrect formalizations -- the formal artifact is translated back to English by a model that never sees the original requirement, and an LLM-judge decides whether the informalization and the requirement express the same guarantee. ClaimCheck performs well compared with simple baselines like zero-shot judge (direct judge) and direct judge + guidance baseline. However the performance gap shrinks as the judge becoming stronger.
Critical strengths
-
Tackling an important problem that has solid motivation. The paper addresses an important problem for scaling AI-generated code, proofs and formal specs -- checking whether a formal specification actually aligns directly with the original natural language requirement (NLR). That's very important, as there are multiple cases of specifications being formally verified, however they might be expressing the wrong guarantee.
-
Creation of a benchmark and empirical evaluations. The paper constructs a benchmark of 3234 examples across 4 formal languages, with 1029 meaning-preserving transformations that help prevent models from simply detecting edited specifications -- that's very useful, as other people can build upon it. Additionally, the authors provide claimcheck, a 2-stage checking pipeline and demostrate improvements of 4.2 to 6.0 percentage points (pp) in macro recall vs the strongest direct-judging baseline competitor. Ablation experiments additionally demonstrate that removing back-translation stage reduces the macro recall significantly (2.8 to 6.0 pp), and allowing the informalizer to see the original requirement slightly reduces (0.8 to 3.0 pp), which basically suggests that separating the available information helps to improve the scores, which should be used by different protocols.
Critical weaknesses
-
Limited novelty in terms of methodology and insufficient comparison with prior approaches. The proposed ClaimCheck method appears to be very similar to existing semantic checking methods using the back and forth technique. Specifically, Clover (Sun et al., 2024: https://arxiv.org/pdf/2310.17807) already uses the translation of formal annotations back into natural language without seeing the original requirement, and the independent LLM-based equivalence check follows it. Monty (Liu et al., 2026: https://arxiv.org/pdf/2607.13303) also uses a similar approach. Even though the paper acknowledges these methods, it does not directly compare against their checking protocols using the same experimental settings. Additionally, it is difficult to understand what specific methodological improvement ClaimCheck adds beyond evaluating an existing approach on the new benchmark
-
Limited evaluation of practical value and whether the benchmark generalizes well. The experiments mainly compare ClaimCheck agains simple direct-judging baselines and does not compare it to other methods (mentioned above) and other stronger approaches, e.g. based on reasoning. Additionally, the performance gain becomes much smaller with stronger judges in the model sweep (though these experiments use a variant of ClaimCheck and not its main configuration). The benchmark in essence consists of synthetically generated errors, including a big chunk of simple negation versions in Isabelle and Rocq -- it is not clear whether the improvements observed actually generalize to real-work mistakes that are made by LLM-based formalizatoin systems. Lastly, most of the perfomance improvement comes from correctly accepting faithful specifications, when detection of unfaithful ones becomes worse (e.g. on Lean 4 unfaithful recall drops from 95.9% to 88.4%), which is quite concernign for formal verification, as more incorrect specs can be undetected.
Questions for authors
- . What is fundamentally new comparing (at least) to Clover and Monty (described above)? Could the authors clarify the specific methodological contribution and provide comparisons with these methods?
- . Could the authors evaluate ClaimCheck on the actual incorrect specs generated by LLMs and not only rule-based variations -- it will help in understanding whether it really helps in practice.
- . In the model sweep why is the only option being evaluated is the option where the judge sees the artifact -- is it possible to report the main ClaimCheck config using the strongest judge and compare it with the baselines?
- . It is not clear that the choice fo evaluation metric is the best given the last sentence from weaknesses field -- with the decrease in unfaithful recall on Lean 4, how should the tradeoff between false acceptance and false rejection be interpreted for real verification applications?
2 comments
Thank you for the detailed response, however I will maintain my assessment.
Response to Reviewer a8Ra
Q1 (what is new compared with Clover and Monty). Please see point 1 of our general comment. For Clover and Monty specifically: in both, the back-translation is one component inside a larger pipeline that also involves sampling and, in Monty's case, running a test generator. Clover does score its check alone, but never against a judge shown both the requirement and the specification. claimcheck is only one informalization with haiku-4.5 plus one judge call.
Q2 (real LLM-generated specifications). You are right, and we will address this in the final version. Please see point 2 of our general comment.
Q3 (main configuration with the strongest judge). We thank the reviewer for their suggestion. We have now run this, and the results are in point 3 of our general comment. With gemini-3.7-flash, the main configuration is level with baseline + list on Isabelle and Dafny, ahead on Rocq and behind on Lean.
Q4 (macro recall and the drop on Lean). This is a fair criticism of macro recall. For verification a missed unfaithful specification is the more expensive one. Point 4 of our general comment splits macro recall into its two parts and shows the trade-off. For a strong judge, or wherever a missed unfaithful specification is the more expensive error, showing the specification as well is the safer setting. We will report both parts in the revised version.