- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper highlights a critical issue in code verification: reference specifications may be incorrect or capture only one of several reasonable interpretations of the requirements, making equivalence to a single reference a potentially misleading criterion for evaluating generated specifications. The authors audit CLEVER, a benchmark for verified code generation in Lean, using three agentic systems to generate specifications, check equivalence, generate implementations, and prove correctness. They report errors in 80 of 161 reference specifications, 13 entries where reasonable interpretations differ from the reference, and four entries with issues in fixed benchmark components, with some overlap between categories. The main contributions are a detailed audit, an error taxonomy, and an analysis of the limitations of equivalence-based scoring. The findings are supported by Lean checks, comparisons across systems and prompt settings, manual inspection, and reported confirmation by the benchmark authors. The results suggest that much of CLEVER’s remaining difficulty stems from problems in its specifications and evaluation criteria.
Critical strengths
- The paper highlights an important problem in evaluating verified code generation. Reference specifications may be incorrect or capture only one reasonable interpretation of the requirements, making equivalence to a single reference a potentially misleading measure of correctness. The audit shows that this concern has substantial consequences in practice. This is a meaningful contribution because it calls into question whether benchmark scores accurately reflect models’ verification capabilities.
- The paper provides actionable findings for improving verification benchmarks. Beyond reporting aggregate error counts, the authors provide per-entry diagnoses and a taxonomy of recurring specification errors, with the findings reportedly reviewed and accepted by the benchmark authors. These results offer a concrete basis for repairing CLEVER and a useful checklist for constructing future benchmarks, giving the work practical value beyond its critique of the evaluation method.
Critical weaknesses
- The methodological contribution is limited. The paper identifies an important problem, but its main contributions are an audit and an error taxonomy. It recommends improvements to evaluation without implementing and validating an alternative. While the findings are useful, the paper leaves open how to reliably evaluate generated specifications when the reference is flawed or the requirements are ambiguous, which limits its contribution toward resolving the problem it highlights.
- The evidence for benchmark saturation is less convincing than the evidence for benchmark defects. The headline 96.2% end-to-end success rate combines two prompt settings and evaluates implementations against the systems’ own specifications, whose fidelity to the requirements relies on agent judgments and author inspection. These results do not directly establish comparable performance on a repaired benchmark with independently validated specifications. Such an evaluation would more convincingly support the claim that CLEVER no longer meaningfully challenges modern agentic systems.
Questions for authors
- Proof validation: Did you use Lean Comparator or an equivalent mechanism to verify that accepted proofs establish the intended theorems under the original definitions and permitted axioms? Were the same validation checks applied to all three systems?
- Specification validity: For the reported 96.2% end-to-end success rate, how did you establish that the generated specifications faithfully capture the requirements? What independent checks ruled out missing constraints or overly weak specifications?
- Benchmark saturation: Do you have results on corrected reference specifications with clarified requirements? If not, what evidence supports the saturation claim beyond success against the systems’ own generated specifications?
3 comments
Response to Reviewer yPMT (Part 2/2)
Part 2/2
Q3 and W2: Residual difficulty
Framing in the paper. Our main contributions are those noted in the reviewer's summary: the audit, the error taxonomy, and the analysis of equivalence-based scoring. The performance results come as a by-product, and the analysis of residual difficulty stands as a secondary contribution. Rather than saturation of CLEVER's score, our experiments were meant to capture the residual difficulty of the underlying problems, independently of the benchmark's infrastructure and references. This is why the paper states the claim in hedged form. The Abstract reads the results as “suggesting that CLEVER no longer stresses modern agentic provers as much as its design intended”, and the Conclusions find that “much of its remaining difficulty is an artifact of its evaluation”. We understand, however, how the imprecise label “Saturation” of our third contribution (Section 1) may have suggested a stronger claim: we have replaced it with “Residual difficulty”, made the distinction explicit in Section 1, and will hedge the TLDR accordingly in the camera-ready.
Measuring saturation. The very issues we identify make saturation hard to measure within CLEVER's own protocol. Flawed references and mismatches hide performance, so CLEVER's score systematically under-reports it. This motivated our distinction between valid and accepted specifications, and the alternative pipeline that yields the 96.2% end-to-end rate. As the reviewer notes, this rate is not directly comparable with CLEVER's original pipeline by construction.
Sound references. The CLEVER authors are preparing a repaired release based on our findings; we will link the upstream issue in the camera-ready and re-run our experiments once the release is available. Meanwhile, CLEVER's original pipeline on its sound portion offers the closest evidence. On , the 78 entries in which neither the CLEVER authors nor we found a flaw, Claude and GPT-5.4 certify implementations for 89.7% and 92.3% of entries. Both also meet CLEVER's own end-to-end criterion on 70.5% of them. Taking the best of the three stacks per entry, these rates rise to 94.9% and 79.5%, respectively. Unlike the 96.2% end-to-end rate, these figures use CLEVER's own scoring and do not depend on our assessment of generated specifications. Our only input is the selection of the 78 entries, which rests on flaw verdicts confirmed by the CLEVER authors. Since sound entries may also be easier than flawed ones, these results hint at saturation rather than establish it. Together, they point to HumanEval-level problems no longer stressing modern agentic provers, and suggest that even once repaired, CLEVER may no longer be informative for these systems, which is why our recommendations end with more challenging tasks (Section 4.4).
W1: Methodological contribution
We agree that the paper diagnoses isomorphism-based scoring rather than replacing it, and this is the scope it claims: our findings "expose structural limitations" of the method and "motivate recommendations" for future benchmarks (Section 1). We see the audit as a necessary first step towards a replacement. Its per-entry findings and taxonomies map the failure modes that a new evaluator of generated specifications must handle, and the confirmed reference flaws provide concrete cases on which to test it.
The audit, however, contributes more than its experimental results:
- Protocol: separating proof failures, benchmark defects, and defensible divergences required status flags that attribute each failure to its source, and task variants run as diagnostic probes;
- Diagnosis of the method itself: isomorphism against a single reference presumes an unambiguous NL description, and CLEVER's theorem form accepts vacuous equivalences between unsatisfiable specifications (Section 5);
- Recommendations that outline how to design new evaluators (Section 4.4): restoring a shared intent for specifications, ruling out vacuous successes, and combining LLM-based and deterministic evaluators.
Moreover, our implementation runs already instantiate one of these recommendations. The witness check of Section 4.4 rules out vacuous isomorphisms. If an implementation can be certified against a specification, that specification is satisfiable, so an isomorphism with it cannot be vacuous. Seven isomorphisms were accepted against flawed references. Four of them, on two entries (67 and 160), are vacuous; since no stack certifies an implementation against either reference, the check flags them. The other three (entries 20, 95, and 97) reproduce a flawed reference that is still satisfiable, so the check rightly passes them. Catching these flaws needs another evaluator, hence our recommendation to combine evaluators.
Implementing and validating a full replacement is a contribution of its own, which we plan to pursue in future work.
Kind reminder to reviewer yPMT
Dear Reviewer yPMT,
We once again thank you for your thoughtful review and feedback. As the discussion period ends in about 24 hours, we hope our responses and the revised PDF have addressed your concerns, and we remain available in case you have any follow-up questions.
Best,
Paper Authors
Response to Reviewer yPMT (Part 1/2)
Part 1/2
We thank the reviewer for the careful review, for recognizing that equivalence to a single reference can be a misleading measure of correctness, and for finding our per-entry diagnoses and taxonomy a practical basis for repairing CLEVER. We answer Q1 and Q2 below, and Q3 and W1 in part 2 of our response. The revised PDF marks new and revised text in blue. We hope our responses resolve your concerns and, if so, would be grateful if you would reconsider your soundness and overall scores.
Q1: Proof validation
CLEVER's own scoring. Every accepted result is scored by CLEVER's evaluation interface: each solution is rebuilt from the benchmark source, keeping only its editable sections (generated specification, implementation, proofs, and helper lemmas), so the theorem statements and reference definitions are the original ones. The rebuilt file is rejected if it does not compile or any
sorryremains. On top of this, we checked that no accepted solution declares an axiom. For all three stacks, agent-reported successes and harness acceptance coincide on every run, up to two benign exceptions for Claude explained in Sec. 2.2 (lines 235–239).Comparator. We thank the reviewer for this suggestion. We have now checked all 1,415 accepted proofs of the three stacks with Comparator, against the original statements and permitted axioms, and report this check in Sec. 2.2 (lines 242–245) and Appendix C.4. 55 proofs (32 by Claude, 17 by GPT-5.4, 6 by Kimi) use
native_decide, a standard Lean tactic accepted by CLEVER but rejected by Comparator by default, since it trusts compiled code through two extra axioms. With Comparator adapted to permit these two axioms besides the standard ones (propext,Quot.sound,Classical.choice), all accepted proofs pass except four of Claude’s runs (entries 95, 50, 118, and 163), already reported in Sec. 2.2. In each, a benchmark test contradicts the specification the implementation is proved against, so the file does not compile; all four pass once that test is removed. Counting thenative_decideproofs as failures would not change our conclusions. Certification against the 78 sound references (Cgt) would drop only slightly (Claude 70 → 66, GPT-5.4 72 → 69, Kimi 53 → 52), as would end-to-end successes on the 157 sound premises (Cgen; 151 → 144, 142 → 138, 118 → 117).Q2 and W2: Specification validity
Prompt settings. With the guided prompt alone, the end-to-end rate is 92.4% (145 of 157 entries); combining both prompts adds six entries, which only the simple chain solves.
Mechanical evidence. Of Claude's 151 end-to-end successes, 73 are mechanically shown (Appendix D.1) to be at least as strong as the reference: 60 are proved isomorphic to it, and for 13 more Claude proved that the generated specification implies it. For 63 of these 73, neither the CLEVER authors nor we found a flaw in the reference. These isomorphisms are not vacuous, since the certified implementation satisfies both specifications. For 18 others, formal counterexamples show that the generated specification constrains behavior the reference leaves open, which excludes degenerate specifications such as True.
Assessment. For the remaining entries, faithfulness is assessed as described in Sec. 2.2. We inspected Claude's solutions together with the agent's reasoning, including the arguments, often counterexamples, with which it defended its own specification and criticized the reference. The CLEVER authors then independently reviewed and accepted the resulting findings on the references.
Cross-task consistency. Results across tasks also support the specifications. In our alternative pipeline, each end-to-end success also requires an implementation, certified against the generated specification, that passes the benchmark's test cases (save for one of the test-case discrepancies above, entry 163). This rules out unsatisfiable specifications and those that reject the tested behavior, though not overly weak ones, hence the evidence above.
Inherent limit. Ultimately, this difficulty is inherent to evaluating specification autoformalization. Whether a weaker specification misses a constraint or legitimately leaves it open (for instance, behavior outside the valid input domain) often depends on how the NL description is read, and no mechanical check against a single reference can settle it. This ambiguity is why we introduced
MISMATCHflags. We therefore agree with the reviewer that our end-to-end rates on the alternative pipeline are conditional on the trustworthiness of these assessments, and the paper scopes them accordingly. Every specification and end-to-end success is scored twice, as valid (its reading assessed as admissible) and as accepted (proved isomorphic to the reference; CLEVER's own criterion), and both counts are reported throughout Sec. 3.