acceptodds
Under review as a conference paper at ICLR 2027

AutoformBench: A Lean Formalization Benchmark for Advanced Mathematics

Abstract

AI systems are increasingly capable of producing machine-checked Lean proofs of advanced mathematical results. Real mathematical formalization, however, is not isolated theorem proving: it requires building the definitions and intermediate lemmas that connect a result to a formal library. Existing evaluations often provide isolated theorem statements in settings where much of this supporting theory is already available in Mathlib. We introduce AutoformBench, a benchmark of 341 Lean 4 formalization tasks drawn from 24 graduate and research-level mathematical sources. Lean-experienced mathematicians formalized and audited each target statement, leaving the corresponding Lean proof to the evaluated agent. Each problem is embedded in a broader mathematical development: agents receive the informal proof but must translate it into Lean while constructing the missing definitions, intermediate lemmas, and supporting infrastructure. We compare the performance of frontier models on these problems. We find a largely shared difficulty frontier: success depends less on mathematical domain than on models' ability to construct missing formal infrastructure.

Primary area:
datasets and benchmarks
Keywords:
autoformalization, formal theorem proving, Lean 4, proof assistants, mathematical reasoning, language-model agents, benchmark evaluation.

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ejupOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces AutoformBench, a benchmark of advanced mathematical formalization tasks that require constructing supporting definitions and lemmas beyond what is readily available in Mathlib. The authors evaluate several frontier LLMs on the benchmark and analyze their performance under different resource budgets and experimental settings.

One interesting finding is that providing the original textbook proofs does not significantly improve overall formalization success rates.

Critical strengths

The human-validated benchmark is a valuable contribution. The tasks are drawn from advanced mathematical sources and require formalization beyond directly available Mathlib results. The careful construction and verification process makes the benchmark a potentially useful resource for evaluating and improving future formalization models and agents.

Critical weaknesses

My main concern is the limited research insight beyond the benchmark itself. Most of the paper focuses on evaluating existing frontier LLMs, and many of the observations, such as the importance of tool feedback and the need for auxiliary lemmas, are already broadly recognized in automated formalization research.

While the evaluation is extensive, the analysis remains largely descriptive and offers limited insight into why certain formalization tasks are difficult or what fundamentally limits current models.

Questions for authors

Based on the benchmark results, can the authors provide a more systematic analysis of the factors that make certain formalization tasks particularly difficult? For example, can the difficulty be attributed to missing mathematical infrastructure, proof complexity, Mathlib integration, or limitations in the models' mathematical reasoning? Such an analysis would help clarify what new insights the benchmark provides beyond model performance comparisons.

7 comments

We thank the reviewer for recognizing the benchmark’s value and the care taken in its construction and validation. Our primary contribution to the community is the benchmark itself as well as its evaluation protocol (using Harbor and Comparator), in line with our submission to the Datasets and Benchmarks area.

Based on the benchmark results, can the authors provide a more systematic analysis of the factors that make certain formalization tasks particularly difficult? For example, can the difficulty be attributed to missing mathematical infrastructure, proof complexity, Mathlib integration, or limitations in the models' mathematical reasoning? Such an analysis would help clarify what new insights the benchmark provides beyond model performance comparisons.

The reviewer raises a useful question about what makes these tasks difficult. Our evaluation provides several insights into this which we clarify below:

What makes a task difficult

Infrastructure demand predicts task difficulty. The paper shows that solution length correlates with the number of models solving a task (ρ = −0.86, Figure 5). Following Reviewer YyFv’s comment, we now measure infrastructure through new definitions and lemmas. Astra introduces an average of 1.5 new definitions on the 114 tasks every model solves, compared with 21.7 on the 66 tasks only it solves. Models also agree on which tasks require more development (response to Reviewer YyFv). A concrete example is Skolem–Noether, whose ingredients are already available in Mathlib and which all ten models solve, versus Bézout’s theorem for plane curves, solved only by Astra after introducing 18 definitions and 80 lemmas (1,247 lines). We will add these analyses to the revision.

We also appreciate the reviewer’s request for a more systematic analysis of model failures. To address this, we have conducted additional analyses examining the proof obligations left unresolved, and testing whether changing the development order helps models complete them.

How models fail

Models have access to the mathematical proof through the textbook we provide but still struggle to complete its formalization. Providing the proof barely changes success, and models follow the book’s argument at similar rates with and without it (see our response to Reviewer qmM5 on the informal-source ablation) so the mathematical reasoning on the informal proof itself is not a bottleneck for the models. In 98.7% of failed runs the final submission compiles but still contains sorry indicating that the failures are not syntactic, so the agents are very well aware of lean syntax and do not fail because of that. Agents also stop by their own decision with most of the budget left (median 14–23% of $200 used). At the end of their failed run they often list the infrastructure they built and name the missing steps, typically a substantial classical result absent from Mathlib (e.g. Schauder theory, a degree argument), and they cite that it is a substantial result that requires a lot of code to be formalized.

To investigate what is missing, we parse the final files and replay the edits of every failed run on AutoformBench-Ext, the set of 250 harder tasks we introduce in our response to Reviewer qmM5. 88% of Claude Fable 5.1's and all of GPT-6 Astra's failed runs leave exactly one unproved declaration, most often the target theorem itself. We also observed that agents tend to fill the gap between Mathlib and the target bottom-up: they write supporting lemmas before the results that use them (~98% of dependencies), almost never introduce a lemma as a placeholder to be proved later, and assemble the final target theorem last (in solved runs, the target is completed after a median 98% of all edits).

To test whether this order is what limits them, we prompted Astra to work explicitly either top-down (prove the target first from helper lemmas left open, then fill them in) or bottom-up (start from Mathlib definitions and work towards the theorem) on two randomly sampled groups of 25 AutoformBench-Ext tasks: tasks Astra previously solved and tasks no model had solved, keeping the rest of the configuration unchanged.

Replaying file edits and parsing the Lean code confirmed that Astra followed both instructions: top-down runs completed the target proof using open helper lemmas after a median 4–5% of edits, while bottom-up runs completed it at the end. Top-down solved more previously solved tasks (22 versus 17), and each approach solved one previously unsolved task. The main difference was where failures occurred: bottom-up runs generally left the target theorem unproved, whereas top-down runs completed its proof conditional on unresolved helper lemmas. In the failed top-down runs on previously unsolved tasks, these helpers were typically substantial classical results absent from Mathlib. These results suggest that Astra can organize the overall argument, but changing the development order alone does not overcome the difficulty of constructing the substantial missing theory required by its key steps.

We will add all the above in the revised paper which we will post soon. Would these analyses address your concern? We would be happy to provide further details.

Thank you for the detailed response and additional analyses. I have three follow-up points.

  1. Inconsistencies between the rebuttal and the paper.

Several numbers differ between the rebuttal and the revised manuscript: 10 vs. 11 models solving Skolem–Noether, 21.7 definitions on 66 tasks vs. 30.1 on 20, Bézout solved only by Astra vs. also by Fable 5.1. Could you clarify which results are authoritative and ensure consistency?

  1. Difficulty analysis should use pre-hoc features.

Solution LOC, new definitions, and new lemmas are post-hoc properties of successful solutions. Their correlations with model success are informative but do not establish what independently predicts difficulty, and they exclude the hardest unsolved tasks.

Could you analyze features available before solving, such as textbook proof length and context size? A simple regression or manual categorization of the 52 universally unsolved tasks would help distinguish infrastructure gaps from proof complexity and other bottlenecks.

  1. Early stopping deserves further investigation.

Appendix M shows that Astra and Fable 5.1 voluntarily stop after spending a median of only 23% and 14% of their budgets on failed runs. Appendix K.5 reports 12, 39, and 41 additional successes after rejecting early surrender for Astra, DeepSeek, and Gemini. However, the 10% minimum-effort threshold does not address many later stops.

Have you tried higher thresholds (25% or 50%), stronger continuation prompts, or forced re-planning? How would success rates and costs change?

Reviewer_qmM5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a benchmark consisting of tasks related to completing Lean proofs. The input is a statement of a Lean theorem(s) whose proof bodies are omitted (contain the Lean keyword sorry that indicates that the proof is missing). The goal is to have the model (agent) replace all the 'sorry' by proofs. The model (agent) is given helper definitions. The workspace given to the agent also includes the textbook (from where the formal theorem was taken), surrounding definitions, and informal proof. The goal is to evaluate the agent's ability to (auto)formalize an informal proof.

The benchmark has 341 such proof-completion problems - across diverse mathematical fields - at grad/research level.

The paper has extensive evaluation of several frontier models on the benchmark, and the results show that gpt-6-astra solving the most, followed by Opus 5, gpt-5.6-sol, fable 5, and so on.

Critical strengths
  • The fact that the benchmark can distinguish between frontier models is nice to see.
  • The agentic harness proposed and used for evaluations is sensible for the task of generating Lean proofs.
  • The benchmark appears to have used several human experts in its curation process.
  • The evaluation reveals some interesting insights -- such as, gpt-6-astra uses fewest turns and produces most concise answers, and astra's solution LOC is ( strongly negatively) correlated with the number of models that solve that task.
Critical weaknesses
  • gpt-6-Astra has a 82% success rate. While the other models do not do as well, arguably the benchmark may not remain challenging for too long.

  • The biggest drawback is that the benchmark appears to have not served its original purpose -- the motivation was to put out a benchmark for evaluating models on their ability to take an informal argument and formalize it as a proof in Lean. The "informal-source ablation" experiment shows that removing the informal argument had minimal impact on the model's ability to formalize the proof -- which indicates that the models are working off their built-in knowledge, rather than actually formalizing the given informal proof. More evidence also comes from the ablation of search: removing the ability to search the mathlib library does not have much impact. So, AutoformBench is not necessarily testing models' ability to formalize a proof, rather its ability to just generate or complete a Lean proof. In other words, there is no strong evidence that the informal proof is indeed getting formalized or even if it is formalizable in Lean -- raising questions about the benchmark itself.

+- It was interesting to see Claude Code harness, of course including the Lean-based tools, doing (slightly) better than the paper's harness.

  • The scientific contribution remains unclear to me. The benchmark was generated by: "Human experts (atleast PhD-level) from the annotation company Labelbox (Labelbox,2026) weret asked with formalizing the selected theorems, as well as all the definitions and lemmas needed to state them ,in Lean4 (deMoura&Ullrich,2021)(version4.32). Each resulting formalization was then reviewed by an independent human expert, as well as by the authors assisted by frontier models." This essentially looks like an exercise in manual formalization of certain Math results. What then is the unique aspect of this benchmark compared to all the existing ones?
Questions for authors

The weaknesses section contains some questions whose answers can help me better understand the significance of the work.

5 comments

We thank the reviewer for their careful reading and for recognizing that the benchmark distinguishes between frontier models, that the agentic harness is sensible, that human experts were involved in its curation, and that the evaluation reveals interesting insights. We address each of the reviewer’s concerns below.

gpt-6-Astra has a 82% success rate. While the other models do not do as well, arguably the benchmark may not remain challenging for too long.

We agree that 82% is high, but it is reached only under a generous ceiling of 200 dollars per task. This represents an enormous amount of compute, far beyond the budget allowed on most benchmarks. Furthermore, the fact that most tasks can eventually be solved given enough compute (unlike e.g. mathematical discovery tasks) should be expected, as the underlying maths are known to the models. Pushed to the limit, this is true for any autoformalization benchmark: Claude is able to formalize Fermat's last theorem if given enough compute. The point is not whether models are able to formalize the statements or not but whether they can do it efficiently as pushing autoformalization at scale will only be possible if models have good performances at reasonable cost. At $10 per task, GPT-6 Astra solves only 44% of the benchmark (26% at 5 dollars). As most solutions are a few hundred lines long, an omniscient model would be able to solve the corresponding tasks for well under a dollar at current API prices; our benchmark shows frontier models are still very far from such efficiency, and is consequently far from being saturated. In the revision, we will reframe Figure 1 to show the budget curves of Figure 3 alongside the final solve rates, making clear that the benchmark can be used at a fixed per-task budget. We will also highlight the performances at 10 dollars per task.

To address saturation even more directly, we are adding AutoformBench-Ext, an additional set of 250 harder tasks built by selecting more advanced theorems across 30 areas of mathematics, from algebraic geometry and analytic number theory to PDE and stochastic processes. Solutions are verified with the same clean-room pipeline as the main benchmark. GPT-6 Astra and Claude Fable 5.1 solve only 30% and 29% of these tasks at a $200 budget, and the tasks they do solve require much more new infrastructure (per accepted solution):

Mean / median / maxAutoformBench (341 tasks)AutoformBench-Ext (250 tasks)
Tasks solved: GPT-6 Astra281 (82.4%)75 (30.0%)
Tasks solved: Claude Fable 5.1268 (78.6%)73 (29.2%)
Solved by neither model52 (15.2%)150 (60.0%)
New definitions: GPT-6 Astra8 / 3 / 10621 / 16 / 105
New definitions: Claude Fable 5.18 / 3 / 6725 / 23 / 93
New lemmas: GPT-6 Astra30 / 20 / 191106 / 96 / 267
New lemmas: Claude Fable 5.144 / 26 / 370147 / 136 / 312
Added lines of Lean: GPT-6 Astra471 / 319 / 2,7631,734 / 1,649 / 3,052
Added lines of Lean: Claude Fable 5.1579 / 391 / 3,3982,335 / 2,183 / 4,133

Together, the two models leave 150 of the 250 AutoformBench-Ext tasks (60.0%) unsolved, against 52 of the 341 main tasks (15.2%). The tasks they do solve need about three times as many new definitions and lemmas and about four times as much new Lean code, and the median cost of a solved task rises from 9 to 77 dollars for GPT-6 Astra and from 41 to 99 dollars for Claude Fable 5.1. We will add this set to the paper with these statistics.

We also evaluated Claude Fable 5.1 on all 341 tasks of the benchmark with the same harness and verification for completeness. It solves 268 tasks (78.6%), second only to GPT-6 Astra (281, 82.4%). We will add it to the leaderboard and to all analyses in the paper.

The biggest drawback is that the benchmark appears to have not served its original purpose -- the motivation was to put out a benchmark for evaluating models on their ability to take an informal argument and formalize it as a proof in Lean. […] So, AutoformBench is not necessarily testing models' ability to formalize a proof, rather its ability to just generate or complete a Lean proof.

We thank the reviewer for raising this point. The core of our answer is that the capability that we ultimately care about, and which the benchmark tests, is the capability to produce Lean code from known mathematics (be it a provided informal proof or the model’s internal knowledge) - we call this capability “formalization” in opposition to “proving” (the production of new mathematical knowledge). In particular, it is acceptable for the model to depart from the structure of a provided informal proof, as long as the target theorem is proved, the goal being that the targeted mathematical notions and theorems are formalized. As the word “formalization” is somewhat overloaded, we will make sure to rephrase the introduction to clarify this point.

The "informal-source ablation" experiment shows that removing the informal argument had minimal impact on the model's ability to formalize the proof -- which indicates that the models are working off their built-in knowledge, rather than actually formalizing the given informal proof.

We provide the informal proof to take mathematical discovery out of the task and to ensure that the capability we evaluate is infrastructure building. As frontier models have strong internal knowledge of graduate mathematics, this does not have much of an impact on task difficulty.

More evidence also comes from the ablation of search: removing the ability to search the mathlib library does not have much impact.

The search ablation. Models do not need a structured search tool to search Mathlib. Without Loogle, they fall back on unstructured search: they read the Mathlib source with their file tools (median file reads per run rise from about 7 to 55 for GPT-5.6 Sol and from 20 to 40 for Gemini) and check candidate names directly in the Lean REPL (#check/#print queries rise from 8.5 to 21 and from 4 to 24), while success is essentially unchanged. The paper reports this fallback in Appendix K. However, we want to emphasize that in the experiments the models can still search Mathlib even without Loogle. We will make sure to clarify this in the revision.

In other words, there is no strong evidence that the informal proof is indeed getting formalized or even if it is formalizable in Lean -- raising questions about the benchmark itself.

Do models follow the informal proof? As pointed out above, this is not the main focus of the benchmark, in particular because it cannot be checked deterministically. In particular, we do not order the models to follow the source material, and only present it as an inspiration to remove any potential difficulty coming from mathematical discovery. However, we agree that this is an interesting question. We used an LLM as a judge to measure the similarity between the model’s solution and the informal proof; the similarity was fairly high (~75% of proofs follow the book's argument fully or partly). However, the similarity appears to be the same when we provide the informal proof and when we don’t. In other words, while the models generate “standard proofs” that often resemble the reference material, they do not actively try to follow it. This is not surprising, for the reason mentioned above. If you want, we can provide additional details on the protocol and results of this similarity experiment (which we will add to the appendix).

We will clarify all the points above in the revised version of the paper.

The scientific contribution remains unclear to me. […] This essentially looks like an exercise in manual formalization of certain Math results. What then is the unique aspect of this benchmark compared to all the existing ones?

Our main contribution to the community is that we provide a new, diverse, very challenging formalization benchmark, as well as a detailed analysis of the performances of current frontier models on it.

Existing Lean benchmarks mostly test proving rather than formalization: their statements are written in vocabulary Mathlib already provides, so most never involve missing infrastructure, and their tasks are either saturated or such that the models would also struggle to solve them informally (thus putting emphasis on proving). By comparison, the difficulty of our tasks resides in the task of writing Lean code in itself. The expert formalization of the statements is deliberate: automatically formalized statements are often unfaithful or can be trivialized, and fixed, audited targets are what make fully automatic grading trustworthy. We cite some the other prominent lean benchmarks and compare them to AutoformBench:

  • In PutnamBench, a widely used proving benchmark of formalized problems, DeepSeek V4.1 Flash’s solutions can be long but remain largely monolithic, and the published reference proofs of ProofNet add almost no infrastructure. On AutoformBench, the same model introduces new definitions and develops numerous auxiliary lemmas (Table B). These statistics cover only the tasks solved by DeepSeek V4.1 Flash, a considerably smaller set than those solved by Astra or Fable 5.1 on AutoformBench since it is the model for which we have PutnamBench solutions and therefore capture only part of the formal development undertaken by stronger models.

Table B. Structure of accepted solutions (mean / median / max per solution). The ProofNet row reports the published reference proofs.

Model, benchmarkSolvedNew definitionsNew lemmasAdded LoC
DeepSeek V4.1 Flash, PutnamBench89.4%0.6 / 0 / 124.1 / 0 / 45373 / 283 / 2,043
DeepSeek V4.1 Flash, AutoformBench52.8%6.5 / 4 / 6146.1 / 32 / 209633 / 521 / 2,247
Reference proofs, ProofNet (ProofNet-Verified)–0.2 / 0 / 81.5 / 0 / 29113 / 55 / 1,271
  • ProofBench ( https://www.vals.ai/benchmarks/proof_bench ) is the closest comparison to AutoformBench (graduate-level, expert-formalized statements and the same agentic tools), the same models that saturate it are far from saturating AutoformBench, at about 20–25× the cost per task (Table A). This is in part because its statements must be expressible in current Mathlib and helper lemmas are not allowed.

Table A. ProofBench vs. AutoformBench and AutoformBench-Ext, same models. Cost per task: mean / median / max (the ProofBench leaderboard reports only the mean).

ModelProofBench accuracyProofBench $ per taskAutoformBench accuracyAutoformBench $ per taskAutoformBench-Ext accuracyAutoformBench-Ext $ per task
GPT-6 Astra99%$1.71 / – / –82%$ 43 / $ 15 / $ 19530%$ 63 / $ 53 / $ 185
Claude Fable 5.1100%$2.70 / – / –79%$ 52 / $ 36 / $ 20029%$ 57 / $ 33 / $ 200
  • Another proving benchmark FATE-X supplies its non-Mathlib definitions in the task file (algebra only); in AutoformBench the solver must build what is missing.

We will add these comparisons in the paper to clarify the contribution better.

Reviewer_YyFvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces AutoformBench, comprising 341 Lean proof-completion tasks from 24 advanced mathematical sources. It evaluates ten models using an agentic environment and independent proof verification, with analyses of tool access, cost, and supporting infrastructure.

Critical strengths

The expert-audited tasks extend evaluation toward advanced proofs requiring substantial supporting development, making the benchmark a useful resource. Clean-room verification with Comparator and an independent kernel provides strong safeguards against invalid submissions. The tool and scaffold ablations are also informative.

Critical weaknesses

Generated code length is useful evidence, but does not fully establish that missing infrastructure is the main cause of difficulty; it may also reflect model-specific proof choices. The source-enabled ablation scores differ from the headline results without a clear explanation, making the experiments harder to compare.

Questions for authors

Why do the source-enabled ablation scores differ from the headline scores—for example, 136 versus 179 for Gemini? Which configurations or run-selection rules differ?

1 comment

We thank the reviewer for the positive assessment of the benchmark and for the precise questions which we address below.

Why do the source-enabled ablation scores differ from the headline scores—for example, 136 versus 179 for Gemini? Which configurations or run-selection rules differ?

The source ablation was run below the headline reasoning-effort settings (Gemini at medium, Opus 5 at high). Appendix K.2 (Reasoning-Effort Sensitivity) reports these lower-effort runs: Gemini solves 136/341 (179 at its headline setting) and Opus 5 solves 204/341 (211 at its headline setting). The source ablation reuses these runs as its source-enabled arm; without the source, the same configurations solve 133 and 200. Thank you for raising this point, this was ambiguous in the paper and misleading, we will fix this in the revision.

Generated code length is useful evidence, but does not fully establish that missing infrastructure is the main cause of difficulty; […]

We agree that code length alone does not establish that missing infrastructure drives difficulty, and this is a very good point. Following your comment, we have run the following additional analyses, which we will add to the revision:

  1. A direct measure of infrastructure: new definitions and lemmas. Beyond code length, we now count the new definitions (def, abbrev, structure, class, inductive) and lemmas in each accepted solution. On the 114 tasks every model solves, GPT-6 Astra adds 1.5 / 0 / 17 new definitions (mean / median / max); on the 66 tasks only it solves, it adds 21.7 / 17.5 / 106 illustrating that these problems require more infrastructure, at least for Astra. Moreover, on AutoformBench-Ext, the set of 250 harder tasks we introduce in our response to Reviewer qmM5, accepted solutions contain about three times as many new definitions and lemmas as on the main benchmark (GPT-6 Astra: 21 / 16 / 105 new definitions, against 8 / 3 / 106 for the main benchmark). We will add this to the revision and add a figure analogous to Figure 5, plotting new definitions against the number of models solving each task along with the statistics.
  2. Comparison with other existing benchmarks. To your point, solutions to proving benchmarks often contain almost no new infrastructure: on PutnamBench, DeepSeek V4.1 Flash produces long but monolithic proofs with almost no new definitions or lemmas (see our response to Reviewer qmM5). Similarly, the published reference proofs of ProofNet add 0.2 new definitions and 1.5 lemmas on average (ProofNet-Verified, 260 complete proofs). Our benchmark therefore provides a more direct measure of models’ ability to build the formal infrastructure needed to formalize mathematical arguments.

[…] it may also reflect model-specific proof choices.

We agree that both measures, code length and our new measure, the number of new definitions and lemmas, are computed from the solutions the models write, so they depend on each model's style since some models consistently write longer developments than others (Appendix E).

Despite differences in coding style (on the same task, some models write more lines than others), the models show substantial agreement in the relative formalization demands of individual tasks. Tasks that require more code, lemmas, or definitions from one model generally require more from the others. Across model pairs, the median rank correlations are 0.94 for code length, 0.85 for lemmas, and 0.75 for definitions. This suggests that these measures capture shared demands of the tasks beyond individual models’ styles.

Astra is consistently the most concise: on tasks it shares with another model, it writes the shorter solution in 96% of cases, including 98% on hard tasks solved by only 2–5 models. Its solutions are therefore the closest available estimate of the minimal development each task requires.

We will add all the above in the revised paper which we will post soon. Would these analyses address your concern? We would be happy to provide further details.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Revision Summary

We thank the reviewers for their comments. We have uploaded the revised paper. The supplementary material now also includes revision_diff.pdf (all changes against the submitted version: additions in blue, deletions in red), the 250 AutoformBench-Ext workspaces, and per-task results for the new runs.

Reviewer YyFv

  • Why do the source-ablation scores differ from the headline scores? The ablation reuses lower-effort runs. We now state this in §7 (Informal-Source Ablation), and App K.2 reports these runs.
  • Code length alone does not show that missing infrastructure drives difficulty. We added a direct measure of infrastructure: the new definitions and lemmas in every accepted solution (§6, Figure 5b, App G.4). GPT-6 Astra adds 1.5 definitions on average on tasks every model solves, against 30.1 on tasks only it solves. We also added a comparison with PutnamBench and ProofNet, whose solutions add almost no new infrastructure (App N).
  • These measures may reflect model-specific proof choices. We added a cross-model agreement analysis (§6, App G.4): rankings of tasks by code length, lemmas and definitions agree across model pairs (median Spearman 0.94 / 0.85 / 0.75), so the measures reflect the tasks rather than model styles.

Reviewer qmM5

  • The benchmark may not stay challenging (82% for GPT-6 Astra). We reframed Figure 1 to show the dollar-budget curves next to the final solve rates, with the $10 per-task level marked. We also added AutoformBench-Ext (§8, App L): 250 harder tasks with the same construction and evaluation protocol. Because these evaluations are expensive, we ran only GPT-6 Astra and Claude Fable 5.1, the two strongest models on AutoformBench; they solve only 30% and 29% of its tasks. The analysis and conclusions remain unchanged.
  • Models may not use the informal proof, and removing search changes little. The informal proof serves as guidance and removes the need for mathematical discovery. Any proof of the target is accepted, whether or not it follows the textbook argument. Since whether models follow it remains an interesting question, we added App K.7, which measures whether formal proofs follow the textbook argument, with and without the source. We also clarified in §7 and App K.3 that without Loogle, models still search Mathlib by reading source files and querying the REPL.
  • Contribution relative to existing benchmarks. We added App N (pointer in §2), which compares AutoformBench with existing benchmarks in the field to highlight how AutoformBench challenges frontier models on mathematical infrastructure construction in Lean.

Reviewer ejup

  • "Can the difficulty be attributed to missing mathematical infrastructure, proof complexity, Mathlib integration, or limitations in the models' mathematical reasoning?" We added an analysis of the infrastructure each task requires to §6 "What makes a problem difficult?" (Figure 5, App G.4). We also added App K.7, which shows that models follow the textbook argument at similar rates with and without the source, so reasoning on the informal proof is not the bottleneck.
  • "What fundamentally limits current models?" We added "How Agents Fail" (§6, App M). Failed runs compile but leave the main theorem with a sorry. Agents stop on their own and name a missing classical result. They also build bottom-up.
  • Additional ablation: we added a "Development Order" ablation experiment (§7, App M) that tests whether this bottom-up order is what limits models. Top-down prompting improves planning, but it does not overcome building the missing theory.

Additional changes

  • We added Claude Fable 5.1 to the leaderboard and to all analyses (eleven models; 268/341, 78.6%).
  • With Fable 5.1 in the coverage analysis, a few numbers differ from our responses: 20 tasks (previously 66) are solved only by GPT-6 Astra, with 30.1 new definitions on average (previously 21.7) on those tasks. Bézout's theorem is solved by Astra and Fable 5.1; Skolem–Noether is solved by all eleven models. The conclusions are unchanged.

Your feedback has made the paper clearer and the analysis more thorough. We hope these clarifications and changes address your concerns and can be reflected in your final assessment. We are happy to clarify anything that remains.

AuthorsOpenReview ↗

Revision Summary: update (18 Nov)

We have uploaded an updated revision following Reviewer ejup's follow-up comments. The supplementary revision_diff.pdf has been updated accordingly (additions in blue, deletions in red). We added the following elements.

  • Pre-hoc difficulty features. We added App F, "Pre-hoc difficulty predictors" (pointer in §6, "What makes a problem difficult?"). It correlates features available before solving with the number of models that solve each task on all 341 tasks. Formal-context features predict difficulty best (provided Lean context: Spearman ρ = −0.48; provided definitions: −0.43; graded holes and Mathlib imports: −0.41).
  • Categorization of the 52 universally unsolved tasks. We added "Failure causes on the unsolved tasks" to App M (summary in §6, "How Agents Fail"). An LLM judge classified each of the 572 failed runs on these tasks as missing infrastructure, proof complexity, or Lean/Mathlib engineering. Missing infrastructure is the primary cause in 99% of runs and the majority cause on 51 of the 52 tasks.
  • Higher thresholds and stronger continuation prompts. We added this ablation to App K.5, including a full stronger continuation message. We reran from scratch the 94 tasks on which GPT-6 Astra or Claude Fable 5.1 gave up between 10% and 50% of the budget. Any surrender before 50% of the budget was rejected, and the agents received a stronger continuation message. Raising the threshold from 10% to 50% solves only two additional tasks, both by GPT-6 Astra, for 2.1k (GPT-6 Astra) and 1.3k (Claude Fable 5.1) of additional spending.

We thank all the reviewers for their helpful feedback that we integrated into the paper to the best that we can. We are happy to respond to any questions in the remaining time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.