- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a verification method for evaluating autoformalized Lean code. The method combines numerical verification and LLM-based checking for formal statements, along with vacuity checking for proofs. The paper also introduces an autoformalization pipeline that first extracts equations from the source material, then applies the proposed verification method to evaluate agent-generated autoformalizations, and finally employs agents to prove the autoformalized statements. Experiments on 1,810 informal–formal pairs from four datasets demonstrate that the proposed verification method achieves low accepted-error rates.
Critical strengths
-
The dataset constructed in this paper could serve as a useful benchmark.
-
The human evaluation resources used in this paper could support further analyses, such as the case studies presented in Section 5 and the error analysis in Table 3.
Critical weaknesses
-
The verification method is not really novel. It is merely a small exploration of combining tool-based verification and LLM judgments. Also, the verification workflow in Figure 1 does not involve a theorem prover, which is unacceptable for a method claimed to verify autoformalization. The validation of the method is also flawed. The primary metric, accepted-error rate, is basically the precision score (false positives/predicted positives). Recall is not evaluated.
-
The proposed autoformalization pipeline has been tested experimentally but not evaluated, which makes it difficult to compare with other methods. This seems to be an intuitive application without any supporting evidence.
Questions for authors
I think this paper is far from complete as an ICLR submission, and substantial improvements cannot be achieved through questions and the rebuttal process.