acceptodds
Under review as a conference paper at ICLR 2027

FormalMathArena for Reliable Evaluation from Natural Language Problems to Lean Proofs

Abstract

Recent advances in LLM-based theorem proving have largely been evaluated through proof-only benchmarks or separate assessments of formalization and proof quality. Such evaluations neither provide a unified accurate measure of end-to-end theorem-proving capability nor reliably identify the source of failure. To address these limitations, we introduce FormalMathArena, an agentic workflow for evaluating the entire formal theorem-proving process. FormalMathArena combines semantic alignment, proof-assistant verification, and proof-guided failure diagnosis to accurately and separately measure formalization accuracy, proof accuracy, and overall end-to-end accuracy. We further construct a benchmark of 5,427 mathematical problems and validate FormalMathArena on a dedicated expert-annotated subset, confirming its reliability. Applying the validated framework to representative LLMs at scale reveals a gap of up to 21.78% between proof-only and end-to-end performance, identifying accurate and proof-ready formalization as a major bottleneck. We further repurpose FormalMathArena as an action-verification harness, improving formalization accuracy by up to 15.2% and proof accuracy by up to 14.1% on an additional held-out subset. These results establish FormalMathArena as both a reliable evaluation framework and an effective inference-time verifier for end-to-end formal theorem proving. Our code and data will be released once accepted.

Primary area:
datasets and benchmarks
Keywords:
formal theorem proving, autoformalization, large language models, evaluation, benchmarks, Lean 4

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TTbBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces FormalMathArena, a benchmark and diagnostic framework for evaluating the full pipeline from natural-language mathematics to verifiable Lean proofs. Validated through expert annotation, the SA and PGD modules achieve accuracies of 91.50% and 86.00%, respectively, revealing failure modes obscured by binary compilation checks. Evaluations of 10 frontier LLMs show that losses in the formalization stage constitute the primary practical bottleneck.

Critical strengths
  1. The paper addresses a meaningful evaluation gap: a valid Lean proof of the generated proposition does not necessarily mean that the original problem has been solved. The expert verification provides substantive validation rather than relying solely on agreement among models: SA agrees with experts in 183 of 200 cases, while PGD agrees in 172 of another 200-case set, demonstrating that both modules are reasonably reliable.

  2. The paired revision experiment fixes the initial outputs, generation model, and maximum of three revision rounds. Full feedback achieves a reported success rate of 52.1%, higher than the 49.2% achieved with Lean-only feedback.

Critical weaknesses
  1. The most significant reliability gap is that the final-success branch is not subject to a corresponding independent verification. PGD only checks cases with S=1 and V=0 ; if SA incorrectly passes a proposition that is not faithful to the original problem and Lean subsequently proves that proposition, it is counted as an end-to-end success and never enters PGD. For example, weakening the target can yield a valid proof without actually solving the original problem. Consequently, the overall 91.5% agreement rate does not establish how much of the reported final success is truly faithful to the original problem.
  2. The fast–slow design lacks separate evidence regarding its accuracy–cost trade-off. Compared with an all-slow setup, does it meaningfully reduce token usage or latency while maintaining comparable evaluation accuracy? If the savings are small but the loss in accuracy is substantial, the benefit of this design would be unclear. Please provide a controlled comparison to clarify this trade-off.
Questions for authors

See weaknesses.

Reviewer_o2nDOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper considers the ability of models and agentic systems to solve mathematical problems end to end starting from natural language statements to formal statements and ending with formal proofs. The paper builds a dataset containing the said components for 5,427 problems. The paper goes on to develop its own agentic system that tries to diagnose the issues in problematic proofs and tries to solve them. The paper reports that such diagnosis can lead to about 2.9% accuracy gain. The paper reports accuracy of several models on 4,827 problems from its benchmark and demonstrates that end to end accuracy is lower than individual accuracies of autoformalization and ATP.

Critical strengths
  • The topic is interesting.

  • Paper appears to have the intention to release a dataset of problems in Lean and natural language.

  • The attempt to develop an agentic system that analyzes the errors in generated proofs may lead to useful methods in the future.

Critical weaknesses
  • Computational costs are not reported including the costs of running the models and agentic frameworks as well as the computational costs of the agentic system proposed in the paper. Without such information it is difficult to evaluate the advantage of this method over existing ones.

  • In my view, the agentic framework should be evaluated in combination with various LLMs and on known benchmarks. Evaluating a new method on a new benchmark does not provide adequate insights to connect the results to what is already established in the literature.

  • Accuracy gains are marginal and paper does not provide adequate insights about the additional problems that its agentic system succeeds in autoformalizing and proving.

  • I do not have a clear understanding of the benchmark itself and its difficulty. From Table 1, it appears that accuracies can be high in the range of 90% which may imply that the benchmark is not challenging. The benchmark consists of problems from GSM8K which are not in the same category as some of the other sources. It is not clear to me how the problems from chosen to be included in this benchmark.

  • I found the reporting of the results weak and vague. For example, it appears that accuracy results are not pass 32, yet looking at the tables, it is not mentioned how many attempts were used by the models.

  • The paper has missed the following paper from NeurIPS 2025 which evaluates end to end accuracy of models on benchmarks the same way that this paper is concerned about.

Ospanov, A., Farnia, F. and Yousefzadeh, R., 2026. minif2f-lean revisited: Reviewing limitations and charting a path forward. Advances in Neural Information Processing Systems, 38, pp.79964-80002.

Questions for authors

Please see weaknesses. I'd be happy to see any new results or clarifications.

Reviewer_18GnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces FormalMathArena, an end-to-end benchmark and evaluation framework for measuring an LLM’s ability to formalize and prove mathematical problems in Lean. Unlike most existing benchmarks, which provide a formal Lean statement and evaluate only proof generation, FormalMathArena gives the model a natural-language problem and requires it to first translate the problem into Lean and then construct a proof. The generated statement is evaluated for semantic alignment using five LLM judges, each performing direct comparison and back-translation checks, with a fast screening stage followed by slower deliberation when the judgments disagree. Lean verifies the proof mechanically. When verification fails, an LLM-based diagnostic module determines whether the failure is caused by missing conditions, an overly complex formalization, or an incorrect proof. The paper evaluates ten representative LLMs and compares their formalization, proof-generation, and overall end-to-end performance, showing that formalization is a major performance bottleneck.

Critical strengths

The paper provides thorough evaluations across multiple models, including expert validation, detailed failure analysis, and inference-time revision experiments. Moreover, the critical semantic LLM judge part has gone through human review, which agreed with human experts in 90%+ of 200 reviewed cases.

Critical weaknesses

The benchmark mainly rely on existing open-sourced problems from NuminaMath/Lean Workbook and provides limited analysis stratified by difficulty. It is therefore unclear whether the framework’s conclusions generalize to research-level mathematics or long-horizon proofs requiring substantially more complex formalization and proof search.

Questions for authors
  1. Accurately evaluating formalization correctness is difficult, and I appreciate the authors’ effort to validate semantic alignment against expert annotations. Could the authors provide a qualitative analysis of the cases where FormalMathArena disagrees with human experts? Do these errors exhibit systematic patterns, such as incorrect quantifiers, implicit assumptions, domain mismatches, or vacuous statements?

  2. Does the model evaluation include agentic harness or any iterative draft–review–revision process? A relatively simple self-review setup may correct many autoformalization errors. Evaluating only a single generation could underestimate the models’ practical end-to-end capabilities.

  3. Controlled proof-only comparison. Since Lean Workbook provides reference Lean formalizations, could the authors report model proof accuracy using those trusted statements, at least on the Lean Workbook subset? Comparing this directly with end-to-end accuracy on the same models and problems would provide stronger evidence for the claimed gap between proof-only and end-to-end evaluation.

  4. Formalization failure taxonomy. I would like to see a more detailed analysis of formalization failures, particularly those rejected by the semantic-alignment module.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.