- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work presents Theo, a multi-agent framework built on top of Claude Code for automatic mathematical formalization in Lean 4. It uses a two-pipeline design. The formalization pipeline extracts mathematical statements, constructs missing definitions, and evaluates candidate formalizations using auxiliary lemmas and semantic checks. The proving pipeline generates natural-language proofs, recursively decomposes them into lemmas, and produces machine-checked Lean 4 proofs. Both pipelines consist of multiple agents, each with a specific responsibility. The authors evaluate the approach on a 32-problem subset of PutnamBench and selected main results from nine research papers.
Critical strengths
- Impressive experimental results. The framework successfully proves all 32 sampled PutnamBench problems and formalizes selected main results from nine research papers with a lower cost per question.
- Practical significance. The authors report identifying incorrect intermediate statements and proof gaps in existing mathematical research, demonstrating the potential of automated formalization for mathematical proof review.
- Easy integration. The framework builds on Claude Code, potentially making it easy to integrate into existing workflows.
Critical weaknesses
-
Limited methodological novelty. Most components, including multi-agent orchestration, dependency planning, recursive lemma decomposition, and semantic consistency checking, are established techniques. The paper does not sufficiently distinguish its contributions from existing agentic autoformalization frameworks.
-
No ablation studies. The proposed approach is built on top of Claude Code but does not use vanilla Claude Code as a baseline. It is unclear whether the improvements come from the proposed pipelines or the underlying Claude Code system.
-
Limited evaluation. The evaluation only uses 32 of the 672 PutnamBench problems.
Questions for authors
-
The introduction mentions that the orchestrator can backtrack to a previous pipeline state to avoid failures common to rigid pipelines, but the details of this backtracking mechanism are not sufficiently discussed in the main paper or the appendix.
-
The Faithfulness Judge and Proof Critic seem to be heavily relied upon by both pipelines. Have the authors evaluated or tested their performance separately, perhaps using incorrectly generated formalizations or proofs?
-
Could the authors provide a comparison against vanilla Claude Code with equivalent inference budgets to demonstrate the benefit of the proposed multi-agent orchestration?
2 comments
Thanks for the clarification. However, the single research-paper comparison and the mentioned result "did not yield a matched-budget comparison because Claude code stopped before completing" are not very convincing and provide limited evidence of the proposed workflow's benefit over vanilla Claude Code. It remains unclear how much the proposed pipeline contributes to the reported high success rate and lower cost per problem.
Thank you for the careful review. We appreciate your recognition of the research-paper formalizations and their value for identifying incorrect statements and proof gaps. We address your concerns below.
Weakness 1: Methodological novelty
Our main contribution is a complete system for formalizing research mathematics, including constructing definitions missing from existing libraries and developing proofs over those definitions. We agree that multi-agent orchestration, recursive lemma decomposition, and semantic checking have precedents. Theo combines these techniques with auxiliary-lemma testing to evaluate new definitions before using them in the main proof.
Auxiliary-lemma testing checks a candidate definition by attempting to prove properties expected of the intended concept. A definition can compile while omitting a condition or making its expected properties unnecessarily difficult to prove. Theo generates these auxiliary properties, attempts to prove them under candidate Lean definitions, and uses the results alongside semantic checks to select a representation. A failed proof can reflect a problem with the definition.
The formalizations themselves are also a contribution, alongside the method. Theo produced formal developments of selected main results from nine research papers, including the definitions and supporting lemmas needed for their proofs. For SPGT, whose published proof spans 179 pages. It also identified and repaired three incorrect intermediate statements while preserving the main results. The completed SPGT development uses only Lean’s foundational axioms.
Weakness 2: vanilla Claude Code baseline
We tested vanilla Claude Code with Opus 4.8 on A Generalized Trace Reconstruction Problem: Recovering a String of Probabilities, one of our research-paper cases. Both systems were given the same inference budget. Claude Code stopped before completing the formalization and before exhausting its budget, whereas Theo completed the selected lower-bound result under the assumptions reported in Table 2.
We will add this comparison to the paper. Thank you for suggesting this evaluation.
Weakness 3: PutnamBench problems
We could not evaluate all 672 PutnamBench problems within our computational budget. We randomly sampled 32 problems using the fixed seed 0, and Theo solved all 32. Under the stated random-sampling assumptions, the exact one-sided 95% hypergeometric lower confidence bound is approximately 91.3%.
PutnamBench is a supplementary experiment evaluating the proof pipeline on supplied formal statements. Our main results are the formalizations of selected main results from nine research papers, including the missing definitions and supporting lemmas needed to prove them.
Question 1: How does backtracking work?
The main agent can invoke the subagent responsible for any stage in whatever order it judges appropriate. Backtracking is therefore possible from any stage to any other stage. The main agent’s prompt instructs it to investigate why a stage failed and return to the stage that can address the cause.
For example, a failed proof may reveal that an intermediate lemma is too weak, prompting the main agent to call the formalization agent again. If a required concept is missing or incorrectly represented, it can return to definition construction. If the argument needs another intermediate result, it can call the breakdown agent to add that lemma. It can then resume the affected proof task using the revised output.
Theo also attempts the parent proof using the proposed child statements as temporary premises before proving the children. This lets the main agent check whether the proposed lemmas support the parent and revise the decomposition when needed. The Claim Check tool requires each proof attempt to preserve its assigned statement, so a prover cannot silently weaken its goal. We will clarify this orchestration mechanism in the main method description.
Question 2: Faithfulness Judge and Proof Critic
We reviewed the statements accepted by the Faithfulness Judge and the proofs accepted by the Proof Critic, specifically looking for unfaithful statements and missed proof errors. We found no such errors in the outputs we examined which is directly related to precision of that approach.
Question 3: vanilla Claude Code with equivalent inference budget
Our vanilla Claude Code experiment on A Generalized Trace Reconstruction Problem: Recovering a String of Probabilities did not yield a matched-budget comparison because Claude Code stopped before completing the formalization. Theo completed the selected lower-bound result under the assumptions in Table 2.