acceptodds
Under review as a conference paper at ICLR 2027

Beyond the Library: An Agentic Framework for Autoformalizing Research Mathematics

Abstract

Research mathematics often introduces concepts and dependencies beyond the coverage of existing formal libraries. Autoformalizing such work requires constructing suitable definitions, checking their correspondence with the source, and developing proofs over them. We introduce Theo, an agentic framework powered by general coding LLMs that integrates these tasks in a definition-first workflow. To select among candidate formalizations of new definitions, Theo generates auxiliary lemmas and attempts to prove them under each candidate. It favors candidates that make these proofs easier, using proof difficulty as a proxy for formalization quality. An orchestrator then builds on the selected definitions through recursive proof development and semantic review, revising them when downstream issues arise. We evaluate proof generation on 32 randomly sampled PutnamBench problems and obtain checked proofs for all 32. We also formalize selected main results from nine research papers, including five STOC papers, Babai’s graph isomorphism result and the Strong Perfect Graph Theorem of Chudnovsky et al. The case studies expose a gap in a STOC proof, three incorrect intermediate statements in the Strong Perfect Graph Theorem proof, and three gaps in Babai’s graph isomorphism manuscript. For the latter two developments, we prove repaired statements and steps that preserve the main results. These findings demonstrate how formalization can complement mathematical proof review.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Autoformalization, Agentic Pipeline, faithfulness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_etbPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work presents Theo, a multi-agent framework built on top of Claude Code for automatic mathematical formalization in Lean 4. It uses a two-pipeline design. The formalization pipeline extracts mathematical statements, constructs missing definitions, and evaluates candidate formalizations using auxiliary lemmas and semantic checks. The proving pipeline generates natural-language proofs, recursively decomposes them into lemmas, and produces machine-checked Lean 4 proofs. Both pipelines consist of multiple agents, each with a specific responsibility. The authors evaluate the approach on a 32-problem subset of PutnamBench and selected main results from nine research papers.

Critical strengths
  • Impressive experimental results. The framework successfully proves all 32 sampled PutnamBench problems and formalizes selected main results from nine research papers with a lower cost per question.
  • Practical significance. The authors report identifying incorrect intermediate statements and proof gaps in existing mathematical research, demonstrating the potential of automated formalization for mathematical proof review.
  • Easy integration. The framework builds on Claude Code, potentially making it easy to integrate into existing workflows.
Critical weaknesses
  • Limited methodological novelty. Most components, including multi-agent orchestration, dependency planning, recursive lemma decomposition, and semantic consistency checking, are established techniques. The paper does not sufficiently distinguish its contributions from existing agentic autoformalization frameworks.

  • No ablation studies. The proposed approach is built on top of Claude Code but does not use vanilla Claude Code as a baseline. It is unclear whether the improvements come from the proposed pipelines or the underlying Claude Code system.

  • Limited evaluation. The evaluation only uses 32 of the 672 PutnamBench problems.

Questions for authors
  1. The introduction mentions that the orchestrator can backtrack to a previous pipeline state to avoid failures common to rigid pipelines, but the details of this backtracking mechanism are not sufficiently discussed in the main paper or the appendix.

  2. The Faithfulness Judge and Proof Critic seem to be heavily relied upon by both pipelines. Have the authors evaluated or tested their performance separately, perhaps using incorrectly generated formalizations or proofs?

  3. Could the authors provide a comparison against vanilla Claude Code with equivalent inference budgets to demonstrate the benefit of the proposed multi-agent orchestration?

2 comments

Thank you for the careful review. We appreciate your recognition of the research-paper formalizations and their value for identifying incorrect statements and proof gaps. We address your concerns below.

Weakness 1: Methodological novelty

Our main contribution is a complete system for formalizing research mathematics, including constructing definitions missing from existing libraries and developing proofs over those definitions. We agree that multi-agent orchestration, recursive lemma decomposition, and semantic checking have precedents. Theo combines these techniques with auxiliary-lemma testing to evaluate new definitions before using them in the main proof.

Auxiliary-lemma testing checks a candidate definition by attempting to prove properties expected of the intended concept. A definition can compile while omitting a condition or making its expected properties unnecessarily difficult to prove. Theo generates these auxiliary properties, attempts to prove them under candidate Lean definitions, and uses the results alongside semantic checks to select a representation. A failed proof can reflect a problem with the definition.

The formalizations themselves are also a contribution, alongside the method. Theo produced formal developments of selected main results from nine research papers, including the definitions and supporting lemmas needed for their proofs. For SPGT, whose published proof spans 179 pages. It also identified and repaired three incorrect intermediate statements while preserving the main results. The completed SPGT development uses only Lean’s foundational axioms.

Weakness 2: vanilla Claude Code baseline

We tested vanilla Claude Code with Opus 4.8 on A Generalized Trace Reconstruction Problem: Recovering a String of Probabilities, one of our research-paper cases. Both systems were given the same inference budget. Claude Code stopped before completing the formalization and before exhausting its budget, whereas Theo completed the selected lower-bound result under the assumptions reported in Table 2.

We will add this comparison to the paper. Thank you for suggesting this evaluation.

Weakness 3: PutnamBench problems

We could not evaluate all 672 PutnamBench problems within our computational budget. We randomly sampled 32 problems using the fixed seed 0, and Theo solved all 32. Under the stated random-sampling assumptions, the exact one-sided 95% hypergeometric lower confidence bound is approximately 91.3%.

PutnamBench is a supplementary experiment evaluating the proof pipeline on supplied formal statements. Our main results are the formalizations of selected main results from nine research papers, including the missing definitions and supporting lemmas needed to prove them.

Question 1: How does backtracking work?

The main agent can invoke the subagent responsible for any stage in whatever order it judges appropriate. Backtracking is therefore possible from any stage to any other stage. The main agent’s prompt instructs it to investigate why a stage failed and return to the stage that can address the cause.

For example, a failed proof may reveal that an intermediate lemma is too weak, prompting the main agent to call the formalization agent again. If a required concept is missing or incorrectly represented, it can return to definition construction. If the argument needs another intermediate result, it can call the breakdown agent to add that lemma. It can then resume the affected proof task using the revised output.

Theo also attempts the parent proof using the proposed child statements as temporary premises before proving the children. This lets the main agent check whether the proposed lemmas support the parent and revise the decomposition when needed. The Claim Check tool requires each proof attempt to preserve its assigned statement, so a prover cannot silently weaken its goal. We will clarify this orchestration mechanism in the main method description.

Question 2: Faithfulness Judge and Proof Critic

We reviewed the statements accepted by the Faithfulness Judge and the proofs accepted by the Proof Critic, specifically looking for unfaithful statements and missed proof errors. We found no such errors in the outputs we examined which is directly related to precision of that approach.

Question 3: vanilla Claude Code with equivalent inference budget

Our vanilla Claude Code experiment on A Generalized Trace Reconstruction Problem: Recovering a String of Probabilities did not yield a matched-budget comparison because Claude Code stopped before completing the formalization. Theo completed the selected lower-bound result under the assumptions in Table 2.

Thanks for the clarification. However, the single research-paper comparison and the mentioned result "did not yield a matched-budget comparison because Claude code stopped before completing" are not very convincing and provide limited evidence of the proposed workflow's benefit over vanilla Claude Code. It remains unclear how much the proposed pipeline contributes to the reported high success rate and lower cost per problem.

Reviewer_rjpmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors present Theo, an agentic framework for autoformalization. They evaluate Theo on 32 randomly sampled problems from PutnamBench and obtain proofs for all 32, and formalize 9 research papers to demonstrate Theo's efficacy.

Critical strengths

It is clear that Theo is an effective framework for autoformalizations, and the authors highlight this by formalizing several research papers. The authors ensure this formalization by human validation, which is critical to ensuring faithfulness.

Critical weaknesses
  1. PutnamBench results are not comparable across different works. The base model is different across results, so it is unclear what gains are made from the Theo framework or from more powerful LLMs. An ablation on Theo or a comparison across projects using the same base LLM would make this a more clear and fair comparison.
  2. Similarly, on the research papers, there is no evidence provided that Theo is better in any way than prior work. Better in this case doesn't necessarily mean more potent at autoformalization, but by any desirable metric, for example, efficiency.

While not essential for publication, there was no analysis on which parts of Theo were the most effective at formalization. It is understandable that compute limitations would make this difficult at such a large scale, but perhaps a small scale experiment analyzing the contributions of the different components of Theo would be informative for future researchers. This is a minor criticism and doesn't affect my overall score.

Similarly, I hesitate to accept the term "formalization" when axioms exist for prior work. While it's true MerLean does the same thing, that is not (yet) a published paper and therefore I don't think it's reasonable to cite their decision as "following prior work." The authors don't hide this weakness necessarily, but I do think it should be clearly stated in the abstract, or wherever there is mention of "9 formalized research papers", as there is a huge difference in cost, etc.

Questions for authors

Did you discover anything interesting as to where the efficiency gains in particular were coming from? I am very curious, as personal experience has in some ways gone against what you claim in this paper. Like you I have found that spawning subagents is the easy trick to save lots of compute, but in my experience, this is not enough to make the massive efficiency gains as your experiments with putnambench seem to indicate.

Similarly, the formalizations of the research papers took 5 months. Is that of continuous formalization? If so, that seems astronomical. Do you have any breakdown of what specifically took so much time, or perhaps even an educated guess?

2 comments

Thank you for the thoughtful review and for recognizing Theo's research-paper formalizations and the role of human validation.

Weakness 1: PutnamBench comparisons

Table 1 places Theo's proving performance alongside existing systems in their reported configurations. A common-backbone comparison is not straightforward: several baselines are specialized trained provers, so replacing their model would change the method being evaluated. Other systems, such as Aleph, do not expose a way to substitute our backbone through their public interface. We will clarify that this is a comparison of complete systems.

Also PutnamBench only provides a supplementary evaluation of our proving pipeline. Our main contribution is the formal developments of nine research papers and the resulting artifacts, including the Strong Perfect Graph Theorem (SPGT), whose published proof spans 179 pages.

Weakness 2: Advantage over prior research formalizations

Beyond cost, Theo produces compact formal developments and identifies errors in the source proofs. Code size provides a concrete comparison with prior formalizations.

For the planar unit-distance result, Theo produces 14,831 lines of Lean, compared with approximately 33,000 in the independent Aleph Prover development: about 55% fewer lines. Both retain the same two external inputs, the Golod–Shafarevich filtration input and Shafarevich's relation-rank estimate. We will move this comparison from Appendix C.7 into the main evaluation.

For Cycle Double Cover, Theo formalizes the manuscript's argument in 5,713 lines, compared with 7,156 in OpenAI's development; Although we made different assumptions.

For SPGT, Theo's development has approximately 350,000 lines; the Trellis project reports approximately 530,000. Both formalize SPGT and its structural decomposition theorem using only foundational axioms. Theo additionally identified and repaired three incorrect intermediate statements while preserving the main results; these errors are not reported in Trellis's project description.

Additional axioms

Thank you for pointing out this distinction. We agree that “nine formalized research papers” needs an explicit qualification. Three developments use only foundational axioms; five additionally assume cited external theorems; and the Pham development remains conditional on a lemma with wrong proof in the source paper. By treating cited theorems as external inputs, our approach makes formalization more efficient and practical, much as journal peer review assesses new arguments while relying on established results. We will state these distinctions in the abstract and wherever we summarize the nine-paper result.

Question 1: Where do the efficiency gains come from?

Our experience points to how proof work and context are organized. We found it particularly helpful to assign each concrete proof task to a fresh subagent. The main agent retains a small coordination context, while each subagent receives only the definitions, statements, and dependencies needed for its task. Local proof attempts and compiler feedback therefore do not accumulate in the main agent's context. This is an empirical observation from our runs.

Subscription pricing also contributes to the reported dollar savings; Appendix B reports API-equivalent costs alongside subscription expenditure.

Question 2: Did formalization run continuously for months?

The six months refer to the total subscription period, which included PutnamBench, research-paper formalization, and other system work. Each individual formalization required less than three days of active execution, excluding pauses caused by weekly usage limits. Those pauses sometimes stretched the elapsed time.

W1. Thank you for responding to my concerns. I understand that not all comparisons are possible due to limitations of various methods' APIs and the high cost. However, I do believe that at least a small comparable set is necessary for this claim to be made:

Notably, this approach outperforms state-of-the-art specialized models using only a standard $200 software subscription, averaging an operational cost of approximately \$5 per problem. Otherwise, it's entirely possible that Theo made no difference and it was entirely the backbone model being superior.

W2. Generated LOC is an interesting metric. I'm not entirely convinced that lower LOC is better - perhaps the longer one is more faithful to the original proof, or in some way more human readable, etc. (I don't know if either of those are true in this case or in general, but I would recommend to be careful about your claims regarding LOC.) However, it is at least useful in one way, being smaller artifact sizes. I think you should explicitly state this improvement and why you believe it is worthwhile. I will consider this adequately addressed if the authors update their paper accordingly.

Additionally, as mentioned in W1, I don't believe your paper has sufficiently proven that Theo is more cost effective. For most of the prior systems mentioned in the paper, the key contribution is the agentic harness or methodology rather than the backbone model itself. Therefore, to make the claim that your system is cheaper to run, you would need to compare equivalent backbones in my view. I understand this is not always possible, but it is in at least some cases, and you can tune your claims accordingly.

Again, I would emphasize that I don't necessarily disbelieve the effectiveness of your harness, but the paper as-is is not providing complete proof for all your claims.

W3. Wonderful. I think change this would address my concerns with the phrasing. Please include it in the rebuttal revision, as ICLR allows you to update the PDF (I don't see the change reflected in the abstract yet).

Q1/Q2 Thank you for answering my questions. These are both insightful answers.

Overall, I believe you have addressed some concerns, but I am still unwilling to fully accept Theo's superiority in efficiency over prior methods without some sort of direct comparison with equivalent settings. I think comparing methods on a small (e.g., <50) randomly sampled set from PutnamBench, or just one of the research papers may be good evidence, depending on the results.

Reviewer_jgQyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Theo, an agentic system for translating definitions, theorem statements, and proofs from research-level mathematics into Lean 4. Its distinctive feature is to firstly construct definitions missing from existing libraries and evaluate candidate definitions through attempts to prove generated auxiliary lemmas. In the evaluation section, authors focus on proof generation on 32 sampled PutnamBench problems and formalizations of nine research papers. However, the former evaluates only the proof pipeline, while the latter consists of case studies that may involve reproductivity issues. Neither provides a controlled evaluation of the effectiveness of the proposed method as a whole, which may limit the scope of the proposed method.

Critical strengths
  1. Good problem setting. Constructing concepts missing from existing formal libraries and faithfully translating definitions and theorem statements are important challenges in the autoformalization of research mathematics.
  2. Testing definitions through auxiliary lemmas. Evaluating candidate definitions based on whether the expected properties are proved or not, rather than the compilation success alone, is a useful idea.
  3. Large-scale formalization results including real math papers. Beyond formalizing major results from research articles, authors report identifying and correcting problems in intermediate statements and proofs in the original papers. These results demonstrate the practical value of the proposed method of formalization for mathematical verification.
Critical weaknesses
  1. Limited evaluation in Table 1. Theo is tested on only 32/672 problems, without justification for this sample size. Its confidence lower bound is compared to other methods’ full-benchmark results. Evaluation on the full benchmark or the same subset is recommended.
  2. Uncontrolled experimental conditions over configurations. Table 1 compares different models, agent configurations, and computational budgets. Sections 3.1 and 3.2 also use different models. Without matched baselines, improvements cannot be sorely attributed to the proposed architecture rather than the base model or computation.
  3. The main novelty is not directly evaluated. PutnamBench provides formal statements, so it does not test the construction of missing definitions. No ablations are performed on the contribution of auxiliary-lemma testing or semantic checking.
  4. Section 3.2 demonstrates feasibility but provides limited evidence of general effectiveness. Although the overall motivation and findings are interesting, paper selection criteria, success rates across all attempted targets, and evaluation budgets are unclear. Formalization scope and retained assumptions vary, making comparison and generalization difficult.
  5. The appendix section of "B.1 RESEARCH-PAPER COSTS": the authors used Claude Code instead of Claude API for cost reduction. The authors require further clarification and justification for this use of Claude Code for this purpose from the perspective of the license and terms of service.
Questions for authors
  1. Can you clarify there are any human effort on results in Section 3.2? If so, can the extent of automation and the system’s contribution independently of human assistance be assessed?
  2. In Appendix B.1, the authors report an API-equivalent cost of more than 13,000 USD but an actual subscription cost of only 1,200 USD. Is such large-scale automated use permitted under the terms of service?
3 comments

Thank you for the careful review. We appreciate your recognition of auxiliary-lemma testing and the research-paper formalizations. We address your concerns below.

Weakness 1: PutnamBench sample size and comparison with full-benchmark results

We could not evaluate all 672 PutnamBench problems within our computational budget. We therefore sampled 32 problems randomly, using the fixed seed 0, and accounted for sampling uncertainty. Theo solved all 32. Rather than extrapolating 100% accuracy to the full benchmark, we reported an exact one-sided 95% hypergeometric confidence lower bound of approximately 91.3%. This bound accounts for sampling without replacement from the 672 problems.

PutnamBench is a supplementary experiment evaluating the proof pipeline. Our main results are the formalizations of selected main results from nine research papers, including the missing definitions and supporting lemmas needed to prove them.

Weakness 2: Uncontrolled models and budgets

To address this concern, we tested vanilla Claude Code with Opus 4.8 on A Generalized Trace Reconstruction Problem: Recovering a String of Probabilities, one of our research-paper cases. Both runs had similar available budgets. Vanilla Claude Code stopped voluntarily after reporting difficulty with the problem, leaving part of its budget unused. Theo completed the selected lower-bound result under the assumptions reported in Table 2.

We will add this comparison and the settings of our research-paper baseline experiment to the paper. Thank you for suggesting this evaluation.

Weakness 3: Evaluation of definition construction and auxiliary-lemma testing

Our main contribution is an end-to-end system for formalizing research mathematics, including constructing definitions missing from existing libraries and developing proofs over those definitions. The nine research-paper case studies directly exercise this capability: Theo generated the definitions, theorem statements, and Lean proofs for the selected results, under the assumptions recorded in Table 2. Human experts validated the statements without supplying corrections or proof hints.

We agree that isolating the contributions of auxiliary-lemma testing and semantic checking requires component ablations. Our current experiments establish the capabilities of the combined system. We will make this distinction explicit in the paper. PutnamBench serves as a supplementary evaluation of the proof pipeline, while the research-paper formalizations provide the evidence for the full workflow.

Weakness 4: Paper selection, success rates, scope, and budgets

Our nine case studies cover subjects ranging from communication complexity and learning theory to structural graph theory, with substantial variation in scale. The SPGT paper spans 179 published pages, demonstrating that Theo can handle long mathematical developments. We formalize selected main results and their supporting developments, and Table 2 records the retained assumptions for each case. These are case studies; we do not claim a general paper-level success rate.

Appendix A describes the limited coverage of algorithmic results. Our total expenditure was approximately $1,200 over six months, covering all model usage across PutnamBench and all tested research papers.

Weakness 5: Claude Code subscriptions and terms of service

A high API-equivalent cost is not itself a terms-of-service violation. Anthropic’s published subscription rules do not impose a separate dollar cap on API-equivalent usage. We used Claude Code within the subscription’s usage limits throughout the evaluation.

Question 1: How much human effort was involved?

Human experts checked that the Lean statements matched the intended mathematics in the source papers. They did not provide corrections, proof hints, or other mathematical input to the system. Theo constructed the definitions and statements and generated the Lean proofs. The human contribution was validation of the statements only.

Question 2: Was the reported scale of subscription use permitted?

Yes. We used the subscription’s included Claude Code access and complied with its usage limits. The API-equivalent figure estimates what the logged work would cost under metered billing; it is not a separate charge incurred through the subscription.

Reviewer_jgQyOpenReview ↗

Please make it clear whether it is justified or not to use of Claude Code for this purpose.

Indeed, if authors use Claude Code instead of Claude APIs, there are two major concerns: the first concern is the reproductivity as Claude Code may access past dialog histories and even other directories, which may cause the execution of Claude models (Opus or others) ad-hoc and difficult to reproduce. The second concern is the Claude Code subscriptions and terms of service: the authors state that they performed within the subscription’s usage limits, while the limitation of the usage limit (or simply tokens) is irrelevant to the legal allowance of the purpose permitted by the Claude Code term of use.

For specifically, subscription-based Claude Code is under Anthoropic Consumer Terms while Claude API is under Anthoropic Commercial Terms. (Needless to say, different terms are applied to Claude Code and APIs! See Legal and compliance doc of Claude Code Doc.) Anthoropic Consumer Terms state in Article 3-7 that

Except when you are accessing our Services via an Anthropic API Key or where we otherwise explicitly permit it, to access the Services through automated or non-human means, whether through a bot, script, or otherwise.

In this work, authors constructed an automated system that uses Claude Code for math proofs without human efforts (Q1), which may be contrary to this term of Anthoropic Consumer Terms.

Thank you for clarifying these concerns. We address reproducibility and the interaction with Claude Code separately below.

Regarding the terms of service, we launched Claude Code through its standard interactive terminal interface and manually entered the starting prompts; no external script initiated the conversations. Claude Code then performed the formalization work within those sessions. Anthropic explicitly documents autonomous task execution as part of its intended operation. This interaction model fully complies with the applicable agreement.

Our answer to Q1 concerned human mathematical assistance. Researchers initiated the tasks by entering the starting prompt, but did not construct proofs or supply proof hints or corrections.

For reproducibility, auto memory was disabled, and each session ran in a separate project directory. We found no evidence that the runs used files outside their respective repositories. Reproducibility limitations are already discussed in the paper, and we will expand that discussion to specify these settings and the files made available to each run.

Finally, the formalizations themselves are one of our main contributions. The nine research-paper developments include the 179-page Strong Perfect Graph Theorem paper, alongside results across several other areas of mathematics. These artifacts remain available for independent Lean verification. We ask that the assessment weigh these completed developments and their mathematical verification value.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.