- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies verification hallucinations: formally valid proofs that establish properties of mistranslated programs or weakened specifications. VeriBench-Infinity separates translation, specification, and proof or refutation stages. It evaluates submissions against independently curated Lean references and contracts, combining semantic judgments with proof obligations that relate potentially different data representations and observable behaviors. Evaluation uses 24 tasks drawn equally from programming competitions and repository functions, with correct and buggy variants yielding 48 instances and 144 runs across three agents. Results and case studies expose fidelity errors missed by semantic review and distinguish them from proof difficulty.
Critical strengths
- Well-motivated problem identification. implementation-fidelity (Python-to-Lean translation) is indeed a key issue in LLM-based program verification. The paper does a good job of raising this problem and revealing the potential hallucinations it may introduce.
Critical weaknesses
-
Bad writing. The paper is filled with a large amount of concrete implementation details rather than abstracted methodology, making it read like a technical blog. The examples in the Introduction and Section 3.2 are overly detailed, and the paper repeatedly mentions specific LLM version numbers (e.g., GPT-6 Astra, Claude Fable 5.1). These could all be abstracted to a higher level to improve readability.
-
The correctness of the golden reference package cannot be guaranteed. Section 3.1 states that the reference code of the benchmark is generated using GPT-6. Even after LLM-as-a-judge, the semantic equivalence between these model-generated Lean implementations and the Python implementations cannot be guaranteed—which is insufficient for a formal-verification benchmark. Since the entire benchmark contains only 24 tasks, the authors could ensure semantic equivalence through manual inspection.
Questions for authors
- What is Figure 2 trying to convey? It is too complex to understand.