- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces FormalMathArena, a benchmark and diagnostic framework for evaluating the full pipeline from natural-language mathematics to verifiable Lean proofs. Validated through expert annotation, the SA and PGD modules achieve accuracies of 91.50% and 86.00%, respectively, revealing failure modes obscured by binary compilation checks. Evaluations of 10 frontier LLMs show that losses in the formalization stage constitute the primary practical bottleneck.
Critical strengths
-
The paper addresses a meaningful evaluation gap: a valid Lean proof of the generated proposition does not necessarily mean that the original problem has been solved. The expert verification provides substantive validation rather than relying solely on agreement among models: SA agrees with experts in 183 of 200 cases, while PGD agrees in 172 of another 200-case set, demonstrating that both modules are reasonably reliable.
-
The paired revision experiment fixes the initial outputs, generation model, and maximum of three revision rounds. Full feedback achieves a reported success rate of 52.1%, higher than the 49.2% achieved with Lean-only feedback.
Critical weaknesses
- The most significant reliability gap is that the final-success branch is not subject to a corresponding independent verification. PGD only checks cases with S=1 and V=0 ; if SA incorrectly passes a proposition that is not faithful to the original problem and Lean subsequently proves that proposition, it is counted as an end-to-end success and never enters PGD. For example, weakening the target can yield a valid proof without actually solving the original problem. Consequently, the overall 91.5% agreement rate does not establish how much of the reported final success is truly faithful to the original problem.
- The fast–slow design lacks separate evidence regarding its accuracy–cost trade-off. Compared with an all-slow setup, does it meaningfully reduce token usage or latency while maintaining comparable evaluation accuracy? If the savings are small but the loss in accuracy is substantial, the benefit of this design would be unclear. Please provide a controlled comparison to clarify this trade-off.
Questions for authors
See weaknesses.