- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces AutoformBench, a benchmark of advanced mathematical formalization tasks that require constructing supporting definitions and lemmas beyond what is readily available in Mathlib. The authors evaluate several frontier LLMs on the benchmark and analyze their performance under different resource budgets and experimental settings.
One interesting finding is that providing the original textbook proofs does not significantly improve overall formalization success rates.
Critical strengths
The human-validated benchmark is a valuable contribution. The tasks are drawn from advanced mathematical sources and require formalization beyond directly available Mathlib results. The careful construction and verification process makes the benchmark a potentially useful resource for evaluating and improving future formalization models and agents.
Critical weaknesses
My main concern is the limited research insight beyond the benchmark itself. Most of the paper focuses on evaluating existing frontier LLMs, and many of the observations, such as the importance of tool feedback and the need for auxiliary lemmas, are already broadly recognized in automated formalization research.
While the evaluation is extensive, the analysis remains largely descriptive and offers limited insight into why certain formalization tasks are difficult or what fundamentally limits current models.
Questions for authors
Based on the benchmark results, can the authors provide a more systematic analysis of the factors that make certain formalization tasks particularly difficult? For example, can the difficulty be attributed to missing mathematical infrastructure, proof complexity, Mathlib integration, or limitations in the models' mathematical reasoning? Such an analysis would help clarify what new insights the benchmark provides beyond model performance comparisons.
7 comments
We also appreciate the reviewer’s request for a more systematic analysis of model failures. To address this, we have conducted additional analyses examining the proof obligations left unresolved, and testing whether changing the development order helps models complete them.
How models fail
Models have access to the mathematical proof through the textbook we provide but still struggle to complete its formalization. Providing the proof barely changes success, and models follow the book’s argument at similar rates with and without it (see our response to Reviewer qmM5 on the informal-source ablation) so the mathematical reasoning on the informal proof itself is not a bottleneck for the models. In 98.7% of failed runs the final submission compiles but still contains sorry indicating that the failures are not syntactic, so the agents are very well aware of lean syntax and do not fail because of that. Agents also stop by their own decision with most of the budget left (median 14–23% of $200 used). At the end of their failed run they often list the infrastructure they built and name the missing steps, typically a substantial classical result absent from Mathlib (e.g. Schauder theory, a degree argument), and they cite that it is a substantial result that requires a lot of code to be formalized.
To investigate what is missing, we parse the final files and replay the edits of every failed run on AutoformBench-Ext, the set of 250 harder tasks we introduce in our response to Reviewer qmM5. 88% of Claude Fable 5.1's and all of GPT-6 Astra's failed runs leave exactly one unproved declaration, most often the target theorem itself. We also observed that agents tend to fill the gap between Mathlib and the target bottom-up: they write supporting lemmas before the results that use them (~98% of dependencies), almost never introduce a lemma as a placeholder to be proved later, and assemble the final target theorem last (in solved runs, the target is completed after a median 98% of all edits).
To test whether this order is what limits them, we prompted Astra to work explicitly either top-down (prove the target first from helper lemmas left open, then fill them in) or bottom-up (start from Mathlib definitions and work towards the theorem) on two randomly sampled groups of 25 AutoformBench-Ext tasks: tasks Astra previously solved and tasks no model had solved, keeping the rest of the configuration unchanged.
Replaying file edits and parsing the Lean code confirmed that Astra followed both instructions: top-down runs completed the target proof using open helper lemmas after a median 4–5% of edits, while bottom-up runs completed it at the end. Top-down solved more previously solved tasks (22 versus 17), and each approach solved one previously unsolved task. The main difference was where failures occurred: bottom-up runs generally left the target theorem unproved, whereas top-down runs completed its proof conditional on unresolved helper lemmas. In the failed top-down runs on previously unsolved tasks, these helpers were typically substantial classical results absent from Mathlib. These results suggest that Astra can organize the overall argument, but changing the development order alone does not overcome the difficulty of constructing the substantial missing theory required by its key steps.
We will add all the above in the revised paper which we will post soon. Would these analyses address your concern? We would be happy to provide further details.
Thank you for the detailed response and additional analyses. I have three follow-up points.
- Inconsistencies between the rebuttal and the paper.
Several numbers differ between the rebuttal and the revised manuscript: 10 vs. 11 models solving Skolem–Noether, 21.7 definitions on 66 tasks vs. 30.1 on 20, Bézout solved only by Astra vs. also by Fable 5.1. Could you clarify which results are authoritative and ensure consistency?
- Difficulty analysis should use pre-hoc features.
Solution LOC, new definitions, and new lemmas are post-hoc properties of successful solutions. Their correlations with model success are informative but do not establish what independently predicts difficulty, and they exclude the hardest unsolved tasks.
Could you analyze features available before solving, such as textbook proof length and context size? A simple regression or manual categorization of the 52 universally unsolved tasks would help distinguish infrastructure gaps from proof complexity and other bottlenecks.
- Early stopping deserves further investigation.
Appendix M shows that Astra and Fable 5.1 voluntarily stop after spending a median of only 23% and 14% of their budgets on failed runs. Appendix K.5 reports 12, 39, and 41 additional successes after rejecting early surrender for Astra, DeepSeek, and Gemini. However, the 10% minimum-effort threshold does not address many later stops.
Have you tried higher thresholds (25% or 50%), stronger continuation prompts, or forced re-planning? How would success rates and costs change?
We thank the reviewer for recognizing the benchmark’s value and the care taken in its construction and validation. Our primary contribution to the community is the benchmark itself as well as its evaluation protocol (using Harbor and Comparator), in line with our submission to the Datasets and Benchmarks area.
The reviewer raises a useful question about what makes these tasks difficult. Our evaluation provides several insights into this which we clarify below:
What makes a task difficult
Infrastructure demand predicts task difficulty. The paper shows that solution length correlates with the number of models solving a task (ρ = −0.86, Figure 5). Following Reviewer YyFv’s comment, we now measure infrastructure through new definitions and lemmas. Astra introduces an average of 1.5 new definitions on the 114 tasks every model solves, compared with 21.7 on the 66 tasks only it solves. Models also agree on which tasks require more development (response to Reviewer YyFv). A concrete example is Skolem–Noether, whose ingredients are already available in Mathlib and which all ten models solve, versus Bézout’s theorem for plane curves, solved only by Astra after introducing 18 definitions and 80 lemmas (1,247 lines). We will add these analyses to the revision.