acceptodds
Under review as a conference paper at ICLR 2027

Isomorphism Is Not Enough: An Agentic Audit of Verified Code Generation in Lean

Abstract

Agentic systems have recently emerged as state-of-the-art approaches for automated theorem proving in formal mathematics. To assess how far these capabilities extend to program verification and how reliably existing benchmarks can measure them, we audit CLEVER, a Lean 4 benchmark for verifiable code generation, using three independent agentic stacks based on Claude Opus 4.6, GPT-5.4, and Kimi K2.5. Our audit finds arguably erroneous reference specifications in 80 of 161 entries, each accepted by the benchmark's authors, distills a taxonomy of benchmark errors, and exposes structural limitations of isomorphism-based scoring, whose signal breaks down when generated and reference specifications formalize different yet defensible readings. In the process, the stacks generate arguably valid specifications for up to 98.8% of entries, certify implementations against sound reference specifications for up to 92.3%, and reach up to 96.2% end-to-end success over entries with sound premises, suggesting that CLEVER no longer stresses modern agentic provers as much as its design intended. Looking ahead, we translate our findings into recommendations for future work, with our error analysis doubling as a repair list for CLEVER and a heuristic checklist for the next generation of benchmarks.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
vericoding, benchmark audit, agentic AI, program verification, Lean 4

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_yPMTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper highlights a critical issue in code verification: reference specifications may be incorrect or capture only one of several reasonable interpretations of the requirements, making equivalence to a single reference a potentially misleading criterion for evaluating generated specifications. The authors audit CLEVER, a benchmark for verified code generation in Lean, using three agentic systems to generate specifications, check equivalence, generate implementations, and prove correctness. They report errors in 80 of 161 reference specifications, 13 entries where reasonable interpretations differ from the reference, and four entries with issues in fixed benchmark components, with some overlap between categories. The main contributions are a detailed audit, an error taxonomy, and an analysis of the limitations of equivalence-based scoring. The findings are supported by Lean checks, comparisons across systems and prompt settings, manual inspection, and reported confirmation by the benchmark authors. The results suggest that much of CLEVER’s remaining difficulty stems from problems in its specifications and evaluation criteria.

Critical strengths
  1. The paper highlights an important problem in evaluating verified code generation. Reference specifications may be incorrect or capture only one reasonable interpretation of the requirements, making equivalence to a single reference a potentially misleading measure of correctness. The audit shows that this concern has substantial consequences in practice. This is a meaningful contribution because it calls into question whether benchmark scores accurately reflect models’ verification capabilities.
  2. The paper provides actionable findings for improving verification benchmarks. Beyond reporting aggregate error counts, the authors provide per-entry diagnoses and a taxonomy of recurring specification errors, with the findings reportedly reviewed and accepted by the benchmark authors. These results offer a concrete basis for repairing CLEVER and a useful checklist for constructing future benchmarks, giving the work practical value beyond its critique of the evaluation method.
Critical weaknesses
  1. The methodological contribution is limited. The paper identifies an important problem, but its main contributions are an audit and an error taxonomy. It recommends improvements to evaluation without implementing and validating an alternative. While the findings are useful, the paper leaves open how to reliably evaluate generated specifications when the reference is flawed or the requirements are ambiguous, which limits its contribution toward resolving the problem it highlights.
  2. The evidence for benchmark saturation is less convincing than the evidence for benchmark defects. The headline 96.2% end-to-end success rate combines two prompt settings and evaluates implementations against the systems’ own specifications, whose fidelity to the requirements relies on agent judgments and author inspection. These results do not directly establish comparable performance on a repaired benchmark with independently validated specifications. Such an evaluation would more convincingly support the claim that CLEVER no longer meaningfully challenges modern agentic systems.
Questions for authors
  1. Proof validation: Did you use Lean Comparator or an equivalent mechanism to verify that accepted proofs establish the intended theorems under the original definitions and permitted axioms? Were the same validation checks applied to all three systems?
  2. Specification validity: For the reported 96.2% end-to-end success rate, how did you establish that the generated specifications faithfully capture the requirements? What independent checks ruled out missing constraints or overly weak specifications?
  3. Benchmark saturation: Do you have results on corrected reference specifications with clarified requirements? If not, what evidence supports the saturation claim beyond success against the systems’ own generated specifications?
3 comments
AuthorsOpenReview ↗

Response to Reviewer yPMT (Part 1/2)

Part 1/2

We thank the reviewer for the careful review, for recognizing that equivalence to a single reference can be a misleading measure of correctness, and for finding our per-entry diagnoses and taxonomy a practical basis for repairing CLEVER. We answer Q1 and Q2 below, and Q3 and W1 in part 2 of our response. The revised PDF marks new and revised text in blue. We hope our responses resolve your concerns and, if so, would be grateful if you would reconsider your soundness and overall scores.

Q1: Proof validation

CLEVER's own scoring. Every accepted result is scored by CLEVER's evaluation interface: each solution is rebuilt from the benchmark source, keeping only its editable sections (generated specification, implementation, proofs, and helper lemmas), so the theorem statements and reference definitions are the original ones. The rebuilt file is rejected if it does not compile or any sorry remains. On top of this, we checked that no accepted solution declares an axiom. For all three stacks, agent-reported successes and harness acceptance coincide on every run, up to two benign exceptions for Claude explained in Sec. 2.2 (lines 235–239).

Comparator. We thank the reviewer for this suggestion. We have now checked all 1,415 accepted proofs of the three stacks with Comparator, against the original statements and permitted axioms, and report this check in Sec. 2.2 (lines 242–245) and Appendix C.4. 55 proofs (32 by Claude, 17 by GPT-5.4, 6 by Kimi) use native_decide, a standard Lean tactic accepted by CLEVER but rejected by Comparator by default, since it trusts compiled code through two extra axioms. With Comparator adapted to permit these two axioms besides the standard ones (propext, Quot.sound, Classical.choice), all accepted proofs pass except four of Claude’s runs (entries 95, 50, 118, and 163), already reported in Sec. 2.2. In each, a benchmark test contradicts the specification the implementation is proved against, so the file does not compile; all four pass once that test is removed. Counting the native_decide proofs as failures would not change our conclusions. Certification against the 78 sound references (CgtC_\mathrm{gt}) would drop only slightly (Claude 70 →\rightarrow 66, GPT-5.4 72 →\rightarrow 69, Kimi 53 →\rightarrow 52), as would end-to-end successes on the 157 sound premises (CgenC_\mathrm{gen}; 151 →\rightarrow 144, 142 →\rightarrow 138, 118 →\rightarrow 117).

Q2 and W2: Specification validity

Prompt settings. With the guided prompt alone, the end-to-end rate is 92.4% (145 of 157 entries); combining both prompts adds six entries, which only the simple chain solves.

Mechanical evidence. Of Claude's 151 end-to-end successes, 73 are mechanically shown (Appendix D.1) to be at least as strong as the reference: 60 are proved isomorphic to it, and for 13 more Claude proved that the generated specification implies it. For 63 of these 73, neither the CLEVER authors nor we found a flaw in the reference. These isomorphisms are not vacuous, since the certified implementation satisfies both specifications. For 18 others, formal counterexamples show that the generated specification constrains behavior the reference leaves open, which excludes degenerate specifications such as True.

Assessment. For the remaining entries, faithfulness is assessed as described in Sec. 2.2. We inspected Claude's solutions together with the agent's reasoning, including the arguments, often counterexamples, with which it defended its own specification and criticized the reference. The CLEVER authors then independently reviewed and accepted the resulting findings on the references.

Cross-task consistency. Results across tasks also support the specifications. In our alternative pipeline, each end-to-end success also requires an implementation, certified against the generated specification, that passes the benchmark's test cases (save for one of the test-case discrepancies above, entry 163). This rules out unsatisfiable specifications and those that reject the tested behavior, though not overly weak ones, hence the evidence above.

Inherent limit. Ultimately, this difficulty is inherent to evaluating specification autoformalization. Whether a weaker specification misses a constraint or legitimately leaves it open (for instance, behavior outside the valid input domain) often depends on how the NL description is read, and no mechanical check against a single reference can settle it. This ambiguity is why we introduced MISMATCH flags. We therefore agree with the reviewer that our end-to-end rates on the alternative pipeline are conditional on the trustworthiness of these assessments, and the paper scopes them accordingly. Every specification and end-to-end success is scored twice, as valid (its reading assessed as admissible) and as accepted (proved isomorphic to the reference; CLEVER's own criterion), and both counts are reported throughout Sec. 3.

AuthorsOpenReview ↗

Response to Reviewer yPMT (Part 2/2)

Part 2/2

Q3 and W2: Residual difficulty

Framing in the paper. Our main contributions are those noted in the reviewer's summary: the audit, the error taxonomy, and the analysis of equivalence-based scoring. The performance results come as a by-product, and the analysis of residual difficulty stands as a secondary contribution. Rather than saturation of CLEVER's score, our experiments were meant to capture the residual difficulty of the underlying problems, independently of the benchmark's infrastructure and references. This is why the paper states the claim in hedged form. The Abstract reads the results as “suggesting that CLEVER no longer stresses modern agentic provers as much as its design intended”, and the Conclusions find that “much of its remaining difficulty is an artifact of its evaluation”. We understand, however, how the imprecise label “Saturation” of our third contribution (Section 1) may have suggested a stronger claim: we have replaced it with “Residual difficulty”, made the distinction explicit in Section 1, and will hedge the TLDR accordingly in the camera-ready.

Measuring saturation. The very issues we identify make saturation hard to measure within CLEVER's own protocol. Flawed references and mismatches hide performance, so CLEVER's score systematically under-reports it. This motivated our distinction between valid and accepted specifications, and the alternative pipeline that yields the 96.2% end-to-end rate. As the reviewer notes, this rate is not directly comparable with CLEVER's original pipeline by construction.

Sound references. The CLEVER authors are preparing a repaired release based on our findings; we will link the upstream issue in the camera-ready and re-run our experiments once the release is available. Meanwhile, CLEVER's original pipeline on its sound portion offers the closest evidence. On CgtC_\mathrm{gt}, the 78 entries in which neither the CLEVER authors nor we found a flaw, Claude and GPT-5.4 certify implementations for 89.7% and 92.3% of entries. Both also meet CLEVER's own end-to-end criterion on 70.5% of them. Taking the best of the three stacks per entry, these rates rise to 94.9% and 79.5%, respectively. Unlike the 96.2% end-to-end rate, these figures use CLEVER's own scoring and do not depend on our assessment of generated specifications. Our only input is the selection of the 78 entries, which rests on flaw verdicts confirmed by the CLEVER authors. Since sound entries may also be easier than flawed ones, these results hint at saturation rather than establish it. Together, they point to HumanEval-level problems no longer stressing modern agentic provers, and suggest that even once repaired, CLEVER may no longer be informative for these systems, which is why our recommendations end with more challenging tasks (Section 4.4).

W1: Methodological contribution

We agree that the paper diagnoses isomorphism-based scoring rather than replacing it, and this is the scope it claims: our findings "expose structural limitations" of the method and "motivate recommendations" for future benchmarks (Section 1). We see the audit as a necessary first step towards a replacement. Its per-entry findings and taxonomies map the failure modes that a new evaluator of generated specifications must handle, and the confirmed reference flaws provide concrete cases on which to test it.

The audit, however, contributes more than its experimental results:

  • Protocol: separating proof failures, benchmark defects, and defensible divergences required status flags that attribute each failure to its source, and task variants run as diagnostic probes;
  • Diagnosis of the method itself: isomorphism against a single reference presumes an unambiguous NL description, and CLEVER's theorem form accepts vacuous equivalences between unsatisfiable specifications (Section 5);
  • Recommendations that outline how to design new evaluators (Section 4.4): restoring a shared intent for specifications, ruling out vacuous successes, and combining LLM-based and deterministic evaluators.

Moreover, our implementation runs already instantiate one of these recommendations. The witness check of Section 4.4 rules out vacuous isomorphisms. If an implementation can be certified against a specification, that specification is satisfiable, so an isomorphism with it cannot be vacuous. Seven isomorphisms were accepted against flawed references. Four of them, on two entries (67 and 160), are vacuous; since no stack certifies an implementation against either reference, the check flags them. The other three (entries 20, 95, and 97) reproduce a flawed reference that is still satisfiable, so the check rightly passes them. Catching these flaws needs another evaluator, hence our recommendation to combine evaluators.

Implementing and validating a full replacement is a contribution of its own, which we plan to pursue in future work.

AuthorsOpenReview ↗

Kind reminder to reviewer yPMT

Dear Reviewer yPMT,

We once again thank you for your thoughtful review and feedback. As the discussion period ends in about 24 hours, we hope our responses and the revised PDF have addressed your concerns, and we remain available in case you have any follow-up questions.

Best,
Paper Authors

Reviewer_or49OpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper is an audit of CLEVER, a Lean 4 benchmark for verifiable code generation. They find a number of dataset bugs/mistakes and issues with measurement methodology.

Critical strengths

The authors make a very compelling case:

  • CLEVER is indeed a single point of failure for the community, being one of very few end-to-end Lean-for-coding benchmarks.
  • In CLEVER, The authors find a number of mistakes/dataset bugs and find that much of the variance between models is due to these edge cases.
  • The authors further find that the isomorphism-based scoring methodology is flawed as such.
  • Finally, they argue that future benchmarks should be more challenging.
  • In the appendix, they very thoroughly discuss a number of further failure modes, e.g. contamination.

All this taken together, I find myself agreeing with the authors on their recommendations. Despite of its focus on CLEVER, this paper is useful and important to the broader community because of those recommendations. Some of its methodology will also be directly applicable to other, related fields such as the broader math AI community.

Critical weaknesses

No material weakness.

Questions for authors

No questions.

3 comments
AuthorsOpenReview ↗

Response to Reviewer or49

We thank the reviewer for the careful reading and the positive assessment. We share the view that CLEVER, as one of the very few end-to-end benchmarks for verified code generation in Lean, is a single point of failure for the community; this is what motivated the audit. We also agree with the reviewer that parts of our methodology will carry over to related settings such as math AI.

Following the other reviews, we have revised the paper to improve the exposition and further clarify the scope of our performance claims, and have run an additional check of all accepted proofs with Lean Comparator. Note that the revised PDF marks new and revised text in blue.

Thanks for your comments and your revisions which undoubtedly strengthened the paper. My score is already a 4; so no change.

We thank the reviewer once again for their positive assessment and for engaging with us promptly.

Reviewer_DgKVOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper is an audit of CLEVER, a benchmark of problems derived from HumanEval where the goal is end to end code verification in Lean 4. The CLEVER paper shows that frontier models at the time struggle with the benchmark, failing many tasks. This paper argues that the CLEVER benchmark has several issues (e.g. erroneous reference specs) of its own that might be influencing the model's performance on the task. When these shortcomings are addressed, the paper shows that the frontier coding agents solve the majority of the problems in the benchmark so CLEVER is no longer an informative assesment of frontier coding agents capabilites.

Critical strengths

This paper finds concrete issues in CLEVER and shows that fixes targeting them substantially raises frontier coding agents' success on the benchmark. This is a consequential result because CLEVER is used as a measure of progress for verified code generation, but the results of this paper suggests that its measuring ability is impacted heavily by its shortcomings. The paper reports that the authors of the CLEVER library have reviewed and accepted the proposed changes.

Critical weaknesses

Main weakness of the paper is that the main text is hard to follow especially for an audience who don't have direct working experience with CLEVER. For example, a concrete benchmark example in the main text along with a flawed reference could be quite helpful in communicating the findings of the paper to an audience broader than just the CLEVER benchmark experts. As it stands, the paper seems better suited for a direct communication between the CLEVER benchmark authors and the paper authors, as opposed to a machine learning paper with a general audience.

Questions for authors

n/a

5 comments
AuthorsOpenReview ↗

Response to Reviewer DgKV

We thank the reviewer for the review and for recognizing what our audit means for measuring progress in verified code generation. We also sincerely appreciate the reviewer calling our results "consequential".

Presentation. We agree that the main text assumes too much familiarity with CLEVER, and that the findings should reach a general ML audience. With this goal in mind, we have revised the paper using the additional page. Note that the revised PDF marks new and revised text in blue.

  1. Plain-language presentation of CLEVER. The introduction now explains what verified code generation is and why it matters (lines 42–47), and states the core problem in plain terms before our contributions (lines 71–75). Section 2.1 now explains what Lean is, and plainly describes the structure of a CLEVER entry and what the isomorphism theorem asserts (lines 103–107, 135–138, 143–149).
  2. A worked example in the main text. Figure 2 walks through one CLEVER entry: its NL description, the flawed reference specification, a correct generated specification, and why the two cannot be proved isomorphic. A second entry illustrates a description that admits two defensible readings. The text returns to these examples throughout the paper (Sections 1, 2, 3.1, and 4.2).
  3. A clearer overview figure. Figure 1 now includes a description of CLEVER’s protocol and of its four tasks in plain words (e.g., “prove spec ≡ reference”), and highlights the human-written reference specification, against which CLEVER scores both specifications and implementations. It also separates CLEVER’s protocol from our variations, and labels the preliminary setup of Appendix A separately.
  4. Consistent terminology. A short table in Section 2.2 (Table 1) defines the status flags, our scoring terms (valid, accepted, certified, end-to-end), and the entry subsets we evaluate on, and we now use one term for each concept throughout.
  5. Results led by findings. Each results paragraph now opens with its main finding in plain words, before the numbers.
  6. Other improvements. We moved the detailed related work to a new Appendix B, keeping in Section 2.1 a shorter and more focused paragraph on our choice of CLEVER, and made several smaller formatting and layout improvements throughout.

We believe these changes make our work more accessible to a broader ML audience. Formal verification is becoming both a key way to trust AI-generated code and a demanding testbed for AI reasoning, and the techniques developed for it carry over to other areas of AI. Auditing its leading benchmarks therefore matters beyond formal methods, and, as Reviewer or49 notes, parts of our methodology apply directly to related fields such as the broader math AI community.

We would welcome pointers to any passage that remains hard to follow. We hope our responses resolve your concerns and, if so, would be grateful if you would reconsider your presentation and overall scores.

AuthorsOpenReview ↗

Kind reminder to reviewer DgKV

Dear Reviewer DgKV,

We once again thank you for your thoughtful review and feedback. As the discussion period ends in about 24 hours, we hope our responses and the revised PDF have addressed your concerns, and we remain available in case you have any follow-up questions.

Best,
Paper Authors

I thank the authors for their updates, I had a great time reading the paper given the new content. I think a careful audit of benchmarks is an important contribution to the machine learning community. The paper also contains recommendations for best practices for new benchmarks. I have changed my rating to a clear accept.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Rebuttal

We thank the AC and the reviewers. The reviewers agree that CLEVER is a single point of failure for evaluating verified code generation in Lean (or49), that our results on it are consequential and a careful audit of benchmarks an important contribution to the machine learning community (DgKV), and that our per-entry diagnoses and taxonomy, reviewed and accepted by the CLEVER authors, give a practical basis for repairing it and building future benchmarks (yPMT, or49). The main concerns were the accessibility of the exposition (DgKV) and the support for our performance claims (yPMT). The revised PDF marks new and revised text in blue. After the revision, or49 found that it "undoubtedly strengthened the paper", and DgKV, who "had a great time reading the paper given the new content", raised their rating to a clear accept.

Revisions to the paper

  • Accessibility for a broader ML audience [DgKV]: a plain-language introduction to verified code generation and to CLEVER (Sections 1 and 2.1); worked examples of a flawed reference and of a defensible alternative reading, used throughout the paper (Figure 2); a redesigned overview figure (Figure 1); a table of terms (Table 1); results paragraphs that lead with their finding; related work moved to Appendix B.
  • Proof validation [yPMT - Q1]: all 1,415 accepted proofs re-checked with Lean's Comparator (Section 2.2, Appendix C.4).
  • Specification strength [yPMT - Q2]: mechanical evidence on the strength of the generated specifications (Appendix D.1).
  • Scope of performance claims [yPMT - Q3]: the contribution formerly labelled "Saturation" now reads "Residual difficulty", with the distinction made explicit (Section 1).

Responses to the reviewers

  • [DgKV] Presentation. Addressed by the accessibility revisions above.
  • [or49] No weaknesses or questions raised.
  • [yPMT - Q1] Proof validation. Accepted results are scored by CLEVER's own harness, against the original statements. With native_decide permitted, Comparator accepts all 1,415 accepted proofs except four runs affected by test-case discrepancies already reported in the paper; counting the 55 native_decide proofs as failures changes our results only marginally.
  • [yPMT - Q2 and W2] Specification validity. Of Claude's 151 end-to-end successes, 73 are mechanically shown to be at least as strong as the reference (60 proved isomorphic, 13 proved to imply it), and 18 more come with formal counterexamples that exclude degenerate specifications. The others rest on our inspection of the agents' arguments and on cross-task consistency. We agree that the end-to-end rates of our alternative pipeline are conditional on these assessments, which is why the paper reports them separately from CLEVER's own scoring (valid vs. accepted).
  • [yPMT - Q3 and W2] Saturation. We agree, and the paper claims residual difficulty, as a secondary and hedged contribution, rather than saturation. On the 78 entries with sound references, CLEVER's own scoring, independent of our generated specifications, gives 89.7–92.3% implementation certification and 70.5% end-to-end success for the two frontier stacks (94.9% and 79.5% for the best of three). We will re-run our experiments once a repaired release of CLEVER is available.
  • [yPMT - W1] Methodological contribution. The paper claims a diagnosis of isomorphism-based scoring, not a replacement. Beyond its experiments, it contributes an audit protocol, a diagnosis of the method, and concrete recommendations; one of them, a witness check, already flags all four vacuous isomorphisms we observed. A full replacement is future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.