- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focuses on evaluating statement autoformalization: determining whether a generated formal statement is semantically equal to a reference formal statement. It introduces a principled equivalence relation, SemantEq: whether two formal statements can prove each other by a fixed set of steps. Based on this relation, it implements a Lean metaprogram, SemantEq-Eval for determining such relatin in Lean 4. It also proposes SemantEq-ProofAudit, an auditing policy for externally generated proofs motivated by the relation.
Critical strengths
This paper brings a novel, reasonable, and intelligent idea to the community: for reference-based autoformalization evaluation, restricting the finer-grained steps in elaborated proof terms instead of coarse-grained tactics. They implement this idea in Lean 4 metaprogramming and conduct experiments to validate its effectiveness.
Critical weaknesses
- Absence of implication from ProofAudit to Definition 2. This paper lacks a formal derivation or intuitive explanation that once a proof passes
SemantEq-ProofAudit, there exists chains from P to Q and from Q to P. - Insufficient literature review and comparison. LeanScorer[1] and ASSESS[2] are a closely related work but is not discussed.
[1] Mathesis: Towards Formal Theorem Proving from Natural Languages. ICLR 2026. https://openreview.net/forum?id=CJdX82odge [2] ASSESS: A Semantic and Structural Evaluation Framework for Statement Similarity. ICLR 2026. https://openreview.net/forum?id=avwNGWtiHF
Questions for authors
- Could the authors please explain the connection between the implementation of ProofAudit and the original definition?
- Could LeanScorer[1] and ASSESS[2] be included for comparison?
[1] Mathesis: Towards Formal Theorem Proving from Natural Languages. ICLR 2026. https://openreview.net/forum?id=CJdX82odge [2] ASSESS: A Semantic and Structural Evaluation Framework for Statement Similarity. ICLR 2026. https://openreview.net/forum?id=avwNGWtiHF
Thank you for the response and the revisions to the manuscript.
While some gaps remain, the paper addresses a meaningful problem in the field, provides valuable insights, and presents promising evaluation results. I would therefore raise my score to 3 (weak acceptance).