acceptodds
Withdrawn from ICLR 2027

Beyond LLM Judge: Mechanical Verification for Agentic Autoformalization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We introduce a novel method for verifying Lean formalizations that goes beyond LLM-judged faithfulness. Unlike most verification methods, which rely on LLMs or trained models, our method combines LLM-based semantic review with mechanical numerical and vacuity checks, providing checkable feedback and detecting errors missed by LLM judges. We apply this method, along with our Extract-Verify-Proof autoformalization pipeline, to construct the Formalized Library of Mathematical Functions (FLMF), a partial formalization of the NIST Digital Library of Mathematical Functions (DLMF). On three labeled datasets, our method achieves the lowest accepted-error rate among the evaluated methods, measured as the fraction of incorrect statements accepted by the verification method. Expert review of the results also identifies translation and source defects in existing datasets. We will release these findings, along with the FLMF dataset, with this paper.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
autoformalization, faithfulness evaluation, Lean 4, automated theorem proving, special functions, human-in-the-loop verification

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_3f8EOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a verification method for evaluating autoformalized Lean code. The method combines numerical verification and LLM-based checking for formal statements, along with vacuity checking for proofs. The paper also introduces an autoformalization pipeline that first extracts equations from the source material, then applies the proposed verification method to evaluate agent-generated autoformalizations, and finally employs agents to prove the autoformalized statements. Experiments on 1,810 informal–formal pairs from four datasets demonstrate that the proposed verification method achieves low accepted-error rates.

Critical strengths
  1. The dataset constructed in this paper could serve as a useful benchmark.

  2. The human evaluation resources used in this paper could support further analyses, such as the case studies presented in Section 5 and the error analysis in Table 3.

Critical weaknesses
  1. The verification method is not really novel. It is merely a small exploration of combining tool-based verification and LLM judgments. Also, the verification workflow in Figure 1 does not involve a theorem prover, which is unacceptable for a method claimed to verify autoformalization. The validation of the method is also flawed. The primary metric, accepted-error rate, is basically the precision score (false positives/predicted positives). Recall is not evaluated.

  2. The proposed autoformalization pipeline has been tested experimentally but not evaluated, which makes it difficult to compare with other methods. This seems to be an intuitive application without any supporting evidence.

Questions for authors

I think this paper is far from complete as an ICLR submission, and substantial improvements cannot be achieved through questions and the rebuttal process.

Reviewer_747POpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies mathematical statement autoformalization: given an informal mathematical statement, the goal is to generate a Lean theorem that is both formally valid and semantically faithful to the source.

The authors propose an Extract–Verify–Prove pipeline. First, equations and constraints are extracted from the NIST Digital Library of Mathematical Functions (DLMF). LLM agents then build a dependency DAG, identify and write required definitions, and generate Lean theorem statements. The generated statements are checked semantically using direct LLM comparison and back-translation, and numerically when applicable. After verification, an LLM agent generates proof plans and Lean proofs, which are checked for compilation and absence of `sorry'. A final vacuity check replaces the theorem goal with an unrelated proposition to detect proofs that succeed only because of contradictory assumptions.

The paper makes three main contributions. First, it introduces numerical and vacuity checks as complements to LLM-based semantic verification. Second, it integrates these checks into a gated Extract–Verify–Prove autoformalization pipeline. Third, it applies the method to existing datasets and identifies previously unnoticed formalization and source-level defects.

Critical strengths
  • The proposed pipeline is able to identify previously unnoticed formalization defects in existing datasets, demonstrating practical value beyond benchmark-level evaluation.
Critical weaknesses
  • I am not convinced that proof generation should be part of evaluating formalization faithfulness. A faithful formalization should preserve the meaning of the original mathematical statement even when the statement itself is false or difficult to prove.

  • The inclusion of proof generation and vacuity checking appears to implicitly assume that the source mathematical statement is intended to be provable, which restricts the scope of the proposed notion of faithfulness.

  • More generally, it is not entirely clear whether the paper is evaluating statement-level formalization faithfulness, mathematical validity of the source statement, or proof-level validity, as these notions are partially mixed within the pipeline.

Questions for authors

see weaknesses

Reviewer_HFtoOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a verification framework for agentic autoformalization, combining semantic, numerical, and vacuity checks to improve the reliability of Lean formalizations. The framework is integrated into an Extract–Verify–Prove pipeline and evaluated on multiple datasets.

Critical strengths
  1. Important problem. The paper tries to address the reliability of autoformalization, which is crucial for constructing trustworthy mathematical libraries.
Critical weaknesses

W1. Unclear role of numerical checks in faithfulness verification.

While the paper emphasizes faithfulness verification, numerical checks primarily assess mathematical validity rather than translation faithfulness. As demonstrated in Case Study 2, a faithful formalization may fail numerical checks because the original mathematical statement is incorrect. Conversely, an unfaithful formalization may still pass numerical checks. The authors should clarify how numerical checks contribute to faithfulness verification and distinguish translation errors from mathematical errors in their evaluation.

W2. Limited applicability of numerical verification.

Numerical checking, a key component of the proposed framework, is mainly applicable to numerically evaluable mathematical statements. Its applicability to more general mathematical domains, such as abstract algebra and topology, is limited.

W3. Insufficient discussion of related work.

The paper claims that all existing autoformalization verification methods rely on LLM judgments, which is overstated. Greiner-Petter et al.[1] have already investigated symbolic and numerical verification of DLMF formulas. The authors should better discuss these related approaches and clarify the novelty of their proposed numerical checking method. Moreover, sampling-based approaches to verifying the correctness of propositions have been extensively studied in the SMT related literature[2,3]. The authors should discuss how their work relates to these existing approaches.

W4. Insufficient evaluation of the reliability–coverage trade-off.

The proposed method achieves lower accepted-error rates but also rejects more correct statements than some baselines. For example, on FLMF, it achieves a 4.8% accepted-error rate while flagging 45.1% of correct statements. Although prioritizing reliability is reasonable, the current evaluation does not fully characterize this trade-off.

Minor Issue

The Introduction section heading appears to be missing. The paper proceeds directly from the abstract to introductory text, followed by "1 Related Work."

References

[1] Greiner-Petter, André et al. “Comparative Verification of the Digital Library of Mathematical Functions and Computer Algebra Systems.” International Conference on Tools and Algorithms for Construction and Analysis of Systems (2022).

[2] Griggio, A., Phan, Q.-S., Sebastiani, R., & Tomasi, S. (2011). Stochastic Local Search for SMT: Combining Theory Solvers with WalkSAT.

[3] Cohl, H. S., Greiner-Petter, A., & Schubotz, M. (2018). Automated symbolic and numerical testing of DLMF formulae using computer algebra systems. In Intelligent Computer Mathematics (CICM 2018), LNCS 11006, pp. 39–52.

Questions for authors

How do the authors distinguish translation faithfulness from mathematical validity in their verification framework, given that numerical checks may reject faithful formalizations of incorrect source statements?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.