acceptodds
Under review as a conference paper at ICLR 2027

AIProver: Agentic Auto-Formalization of Mathematical Research via Certificate-Driven Evolving Harness

46 views

Abstract

Proof auto-formalization translates natural-language (NL) theorems and proofs into a formal language (FL) such as Lean, enabling mechanical verification. Despite rapid progress, research-level proofs often depend on concepts missing from leading proof assistant libraries (e.g., Lean's Mathlib), and successful compilation does not guarantee that a translation preserves the theorem's meaning or the proof's reasoning. Furthermore, aligned NL-FL training data are scarce, and leading agents often rely on costly frontier models and manually engineered harnesses. To address the above issues, we present AIProver, an agentic framework for autonomous proof auto-formalization and proof synthesis (AFPS) that jointly post-trains a 119B open-weight language model and evolves its agentic, tool-calling harness with HarnessEvolve. Verifiers assess type correctness, proof completeness, and semantic correctness, returning rewards and diagnostic certificates that drive model fine-tuning and alternating reinforcement learning via symbolic feedback and HarnessEvolve, a certificate-driven evolutionary search over the whole harness control flow that re-tailors the harness to the updated model. HarnessEvolve retains and generalizes rejected designs as negative evidence to steer later mutations away from unproductive parts of the search. For research-level training and evaluation, we introduce LoCoBench, 58.9k instances from Mathlib, CSLib, Mizar Math Library, and a bounded-arithmetic textbook, combining 18.8k labelled NL-Lean pairs and 40.1k unlabelled NL-Mizar pairs with a 771-instance validation split whose theorem-proof pairs have no public Lean formalization. Against 39 frameworks spanning AFPS agents, frontier LLMs, and coding agents, AIProver lifts pass@4 semantic correctness over its Leanstral-1.5 base from 15.7% to 36.7% and outperforms every other open-weight system and Aristotle. As a Claude Code and Codex skill, it lifts their semantic correctness from 41.9% and 34.1% to 79.8% and 62.4%, respectively, outperforming Numina-Lean-Agent (72.5% and 44.5%) and OpenGauss (54.3% and 41.9%) in that role. Further, it is also 24% cheaper than Numina-Lean-Agent, pushing the accuracy-cost frontier of research-level AFPS.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
agentic proof auto-formalization, Lean 4 interactive theorem prover, automated harness discovery, certificate-driven evolutionary search, reinforcement learning from symbolic feedback

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7kX9OpenReview ↗
Rating
2
Confidence
3(+1)
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper presents AIProver, a framework for research-level proof auto-formalization in Lean. It combines contrastive semantic alignment training with alternating verifier-guided reinforcement learning and automated harness evolution, using a shared graded reward to improve the model and its execution workflow. The paper also introduces LoCoBench, with a 771-instance evaluation split drawn from Mizar and a bounded-arithmetic textbook. It reports improved performance over the base model and several existing systems, both as a standalone agent and as a Lean-specialist sub-agent for coding agents.

Critical strengths

Substantial system-level gains on challenging auto-formalization tasks. The paper combines data construction, model post-training, and harness optimization into an end-to-end system for research-level proof auto-formalization. On LoCoBench-Val, the reported pass@4 under the complete-proof TC+SC criterion rises from 15.7% for the base model in the seed harness to 36.7% for AIProver. Its integration with coding agents yields further gains, demonstrating practical value both as a standalone system and as a specialist component.

A promising use of verification diagnostics for agent adaptation. The framework distinguishes type errors, incomplete proofs, and semantic mismatches, using graded rewards for model training and diagnostic certificates to guide harness evolution. This provides more informative feedback than compilation success alone and offers a concrete way to adapt an agent’s execution workflow to failures encountered in formalization.

Critical weaknesses

The central research contribution remains insufficiently established. The paper assembles an extensive pipeline and reports improvements in the complete system, but it is difficult to identify the new capability or reusable insight established by the combination. The ablations show that harness evolution improves performance, but do not distinguish the benefit of adapting the harness to the updated model from the benefit of improving the model and harness individually.

Limited independent validation. All end-to-end results are obtained on the newly introduced LoCoBench, and the central semantic-correctness metric relies on LLM judges. The paper reports neither evaluation on an established external benchmark nor an expert-validated audit of the judges’ decisions on these research-level examples. Consequently, the extensive baseline comparisons establish relative performance within this evaluation setup, but provide limited independent evidence that the reported gains translate into more reliable mathematical formalization.

Questions for authors

Appendix F.2 describes the changes from the seed to the evolved harness. Which of these changes specifically responded to behavior changes after model updates, rather than failures already present in the base model?

Can the authors provide an expert audit of a sample of accepted and rejected outputs from AIProver and the strongest baselines, reporting false acceptances and false rejections? This would help establish whether the reported semantic-correctness gains are robust to judging errors.

11 comments
AuthorsOpenReview ↗

Authors' response to Reviewer 7kX9 (Part 1/6)

We sincerely thank the reviewer for the careful reading and the insightful questions, and for recognizing AIProver's substantial system-level gains, its practical value as a standalone system and as a specialist component, and our use of verification diagnostics for agent adaptation. The reviewer's concerns led us to sharpen the paper's central contribution and to substantially expand its evaluation. The revised paper now establishes three complementary results. (1) Controlled ablations show the benefit of adapting the model and its harness together (Table 4). (2) Evaluations on three established research-level benchmarks show that the gains transfer beyond LoCoBench (Figure 5a). (3) A blind human expert audit of the semantic-correctness judges (Table 6) and formalizations of three research papers, one reviewed by its own author (Table 5, Figure 6), give evidence about the faithfulness of the resulting formalizations. We also identify the harness changes that address the post-trained model's failures (Appendix G.2).

Below, we respond to each point under the reviewer's Weaknesses (W) and Questions (Q), with corresponding revisions highlighted in blue in the revised PDF. When a weakness and a question ask the same thing, we answer them together.


[R3-W1-partA] The central research contribution remains insufficiently established … it is difficult to identify the new capability or reusable insight established by the combination.

We agree that simply combining model post-training with harness evolution would not, by itself, be a research contribution. Our contribution is demonstrating that independently improving a model and its harness is insufficient: the harness must also be re-adapted as the model's behavior changes. A harness evolved for Leanstral helps an updated model, SAM, compile twice as often (TC 30.6% → 60.4%), but SAM then produces correct, complete proofs on only 5.1% of problems. Adapting the two together raises this to 36.7% (Table 4, rows g–i). This gives a concrete design principle for verified agents: re-adapt the harness as the model's behavior changes, rather than treat it as a fixed wrapper around an improving model. In Section 5.3, we now state the findings of the new Table 4, which builds up AIProver one component per row (pass@4, %, on LoCoBench-Val; TC = type-correct, Full = TC+SC with complete proofs):

  • (a) Not just a better harness. The seed harness and HarnessEvolve lift Leanstral's TC from 27.4% to 84.4%, but its full proofs only from 12.6% to 22.6% (b→d). A harness mostly fixes compilation, not faithfulness.

  • (b) Not just a better model. Without tools, SAM lowers full proofs (1.4% vs. 3.0% for Leanstral without tools, f vs. a). Yet co-adapted with its harness, it reaches 36.7%, above the 29.3% from co-adapting Leanstral (i vs. e). As we show in Appendix F, SAM aligns NL and Lean in the hidden states, unlike SFT on the same data (matched-pair cosine 0.11 → 0.75, still 0.64 after RLSF; Figure 9).

  • (c) Co-adaptation helps even from the same model. From Leanstral in its evolved harness, interleaving RLSF with HarnessEvolve lifts full proofs from 22.6% to 29.3% (d→e).

  • (d) A new capability: an open-weight Lean specialist for frontier coding agents (Table 3, Figure 4). As a skill, the host plans and AIProver writes Lean and searches for proofs (Appendix B). AIProver lifts Claude Code (CC) from 41.9% to 79.8% and Codex from 34.1% to 62.4%. With either host, it outperforms Numina-Lean-Agent (NLA) and OpenGauss (OG), which reach 72.5% and 54.3% with CC and 44.5% and 41.9% with Codex, at 15–26% lower cost.

The contribution is therefore not an aggregation of post-training and harness engineering. The two are linked: post-training changes how the model fails, and the harness must change what it checks in response (see [R3-Q1]). The resulting specialist also transfers beyond LoCoBench and improves frontier coding agents as a reusable component.

[continued in next comment box 2/6...]

AuthorsOpenReview ↗

Authors' response to Reviewer 7kX9 (Part 2/6)

[...continued from previous comment box 1/6]

[R3-W1-partB + R3-Q1] The ablations … do not distinguish the benefit of adapting the harness to the updated model from the benefit of improving the model and harness individually. … Appendix F.2 describes the changes from the seed to the evolved harness. Which of these changes specifically responded to behavior changes after model updates, rather than failures already present in the base model?

We thank the reviewer for these insightful questions, which we answer together. A harness evolved for one model helps an updated model compile, but not produce correct proofs. To show this, we make the ablation more fine-grained in the revised paper. The new Table 4 builds up the system one component at a time, from Leanstral (rows a–e) and from SAM-tuned Leanstral (rows f–i, SAM for short), so that SAM is evaluated both in a harness evolved for Leanstral and in a harness re-evolved for SAM itself. In the rows marked RLSF ⇄ HarnessEvolve, we alternate RLSF on the model with re-evolving the harness for the updated model (Section 4). Selected rows (Full = correct, complete proofs):

RowModelHarness / trainingTCFull
(c)Leanstralseed H057.615.7
(d)LeanstralHarnessEvolve for Leanstral84.422.6
(e)LeanstralRLSF ⇄ HarnessEvolve91.329.3
(g)SAM-tuned Leanstralseed H030.63.4
(h)SAM-tuned Leanstralharness of (d), evolved for Leanstral60.45.1
(i)SAM-tuned LeanstralRLSF ⇄ HarnessEvolve (AIProver)94.836.7
  • (a) A harness evolved for Leanstral helps SAM compile, but rarely produce correct proofs. In this harness, SAM's TC nearly doubles (30.6% → 60.4%), but it produces correct, complete proofs on only 5.1% of problems (g→h). This is far below Leanstral's 22.6% in the same harness (d). As we explain in Appendix F.2.4, this harness was searched with Leanstral as the model.

  • (b) Re-evolving the harness with the model unlocks the gain. Alternating RLSF with harness re-evolution lifts SAM from 5.1% to 36.7% (h→i), and Leanstral from 22.6% to 29.3% (d→e). Table 4 evaluates the joint procedure as a whole. The key finding is that pairing the post-trained model with a harness evolved for the original model yields only 5.1% full proofs, whereas the joint adaptation procedure reaches 36.7%.

On R3-Q1. In a new paragraph at the start of Appendix G.2, we now state which harness changes respond to the post-trained model.

  • (c) The base model mainly fails to compile; the post-trained model mainly states the wrong theorem. In the seed harness, Leanstral's output compiles on only 57.6% of problems (Table 4, c). The write-through, search-gate, budget, and compile-and-repair hooks, and the 60 → 90 turn cap, address this base-model failure. After post-training, AIProver compiles on 94.8% of problems but states the correct theorem on only 39.0% (i). On 509 hard training problems, its file compiles with a complete proof of the wrong theorem in 173 cases (Appendix G.2).

  • (d) Five changes target the post-trained model's failure, so that it proves the theorem it was given (Appendix G.2).

    • Clause-by-clause review. Once the file compiles, the harness shows the informal theorem beside the model's Lean statement. The model must show where each object, hypothesis, quantifier, and the conclusion appears, in both directions.
    • Rejecting statements that say nothing. If a fixed tactic sequence proves the statement on its own, the harness rejects it and explains that the claim was likely narrowed or lost a hypothesis.
    • No way around that check. The model may not unfold, rename, or substitute the theorem's definitions to pass it.
    • What to give up first. A faithful statement with one unfinished step ranks above a fully proved weaker one. The last repair request asks the model to keep the statement exactly and leave each failing step as sorry.
    • Library definitions for named concepts. For "e is idempotent (i.e., e² = e)", the model must use the library's definition of idempotent, after checking that it matches, rather than transcribe the parenthetical.

Thus, the harness does not merely accumulate fixes for failures already present in the base model. As the post-trained model's failure shifts from not compiling to stating the wrong theorem, HarnessEvolve finds a different set of safeguards, using the verifiers' certificates to re-tailor the harness automatically (Section 4, Appendix G.2).

[continued in next comment box 3/6...]

AuthorsOpenReview ↗

Authors' response to Reviewer 7kX9 (Part 3/6)

[...continued from previous comment box 2/6]

[R3-W2-partA] Limited independent validation. All end-to-end results are obtained on the newly introduced LoCoBench … The paper reports neither evaluation on an established external benchmark …

We agree that results on established benchmarks are essential to show that a new agentic system generalizes beyond its own benchmark. We introduced LoCoBench because no existing research-level dataset offers NL theorem–proof pairs at the scale needed for training, and its Val instances have no Lean formalization online to retrieve or imitate (Section 1). Following the reviewer's suggestion, we now also evaluate AIProver out of distribution on three established research-level test sets (Sections 5.2 and 5.4, Figure 5a): ShadowBench, with postgraduate- to research-level problems in eight areas; MathAtlas-Test, drawn from 72 graduate textbooks; and MathAtlas-Hard, with the theorems that have the deepest dependency trees. With CC as the host, AIProver+CC is the most accurate on all three, at 20–31% lower cost than NLA+CC.

The three test sets cover inputs with and without NL proofs (Section 5.2, Figure 5a):

  • With NL proofs (proof auto-formalization). From ShadowBench, we use the 98 problems with NL proofs. From MathAtlas-Test, we sample up to two theorem–proof pairs per textbook (142 in total).
  • Without NL proofs (auto-formalization and proof synthesis, AFPS). MathAtlas-Hard has no proofs, so the agent must also find the proof. We use all 328 of its theorems, from 33 textbooks.

AIProver's gains over NLA+CC are in parentheses:

TC+SC (%) / USD per attemptShadowBenchMathAtlas-TestMathAtlas-Hard
CC32.7 / 1.4430.3 / 0.615.8 / 1.06
OG+CC76.5 / 1.6548.6 / 1.0920.7 / 2.55
NLA+CC76.5 / 1.9462.0 / 2.4831.7 / 4.73
AIProver+CC82.7 (+6.2%) / 1.5568.3 (+6.3%) / 1.9434.5 (+2.8%) / 3.25

We also scale attempts from pass@1 to pass@32 on the 20 most intricate MathAtlas-Hard theorems (Section 5.4, Figure 5b). AIProver+CC leads at every k. At k = 32, it reaches 35.0%, while NLA+CC reaches 30.0% and OG+CC 25.0%.

Thus, AIProver's gains transfer beyond LoCoBench, including to MathAtlas-Hard, where the system must find the proof as well as formalize it. We also formalize three complete mathematical research papers and compare AIProver with the other SoTA frameworks on them, as we describe in [R3-W2-partC].

[continued in next comment box 4/6...]

Reviewer_YzWjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AIPROVER is an agentic framework designed for autonomous proof auto-formalization and proof synthesis (AFPS) at the research mathematics level. It addresses the limitations of existing systems by jointly post-training a 119B open-weight language model (Leanstral-1.5) and evolving its tool-calling harness via a method called HarnessEvolve. The system utilizes a Semantic Alignment Model (SAM) fine-tuning stage to treat natural language (NL) and formal language (FL) theorem-proof pairs as identical mathematical objects within the model's representations. To train and evaluate the framework, the authors introduce LoCoBench, a dataset of 58.9k instances sourced from Mathlib, CSLib, the Mizar Mathematical Library, and a bounded-arithmetic textbook.

Critical strengths

Model-Harness Co-Evolution: Alternating reinforcement learning (Agentic RLSF) and HarnessEvolve optimizes both the model and its control flow simultaneously, uniquely utilizing rejected harness designs as negative evidence to steer future mutations away from unproductive search paths.

Fine-Grained Verification: The framework employs a graded reward ladder based on sound verifiers (scoring type-correctness, completeness, semantic correctness, and length fidelity), enabling the model to learn from partial successes rather than relying solely on binary compile verdicts.

Cost Efficiency: AIPROVER pushes the accuracy-cost Pareto frontier, delivering higher accuracy at a lower cost per attempt compared to prior open-weight systems and frontier coding agents

Critical weaknesses

Maybe it is needed to find the hardest problems that the model can formalize/solve in the eval set and then do some human spot-check to make sure that Claude/GPT are good for the semantic correctness judge. Although the whole pipeline make sense and the models are already strong and there is already the "test-time" judge (review many times), it is still very easy to get wrong during the formalization and probably need some stats/results for human expert audit.

Questions for authors

See weakness, thank you.

6 comments
AuthorsOpenReview ↗

Authors' response to Reviewer YzWj (Part 1/3)

We sincerely thank the reviewer for the encouraging review and for recognizing three core strengths of AIProver: model–harness co-evolution that learns from rejected harness designs, a graded reward ladder that rewards partial successes, and a better accuracy–cost Pareto frontier. We are especially grateful for the suggestion to examine the hardest cases and to audit the semantic-correctness judges. We have carried out both analyses in the revision. We identify the hardest LoCoBench-Val instances, have a human expert audit 120 candidates from them, and report both false-acceptance and false-rejection rates. We also examine concrete successes and failures on the hardest problems (Section 5.4, Appendix C). The audit shows that our semantic-correctness metric rarely errs, because a candidate counts as semantically correct only if four separate checks, by two judges from different model families, all accept it.

Below, we address the reviewer's concern and summarize the additional results, with corresponding revisions highlighted in blue in the revised PDF.


[R2-W1] Maybe it is needed to find the hardest problems that the model can formalize/solve in the eval set and then do some human spot-check to make sure that Claude/GPT are good for the semantic correctness judge. … there is already the "test-time" judge (review many times), it is still very easy to get wrong during the formalization and probably need some stats/results for human expert audit.

We thank the reviewer for this valuable suggestion, and we agree that a formalization is easy to get wrong even with test-time review. Following it, we now report a blind audit of the semantic-correctness (SC) judge on the hardest LoCoBench-Val instances (Section 5.4, Appendix C). In the audited sample, the judge wrongly accepts only 3 of 60 candidates (5.0%) and wrongly rejects only 4 of 60 (6.7%) (Table 6).

  • (a) The SC metric being audited (Appendix A.4.1). Our SC metric uses two judges from different model families, Claude Opus 4.8 and GPT-5.5. Each compares the candidate with the Mizar gold (FL↔FL′) and the candidate's informalization with the NL input (NL↔NL′). A candidate is SC only if all four checks accept.

  • (b) The human audit (Appendix C). We ask a human expert familiar with Lean and the mathematics to audit the SC metric's verdicts. The audit covers Claude Code (CC) alone and CC with AIProver, Numina-Lean-Agent (NLA), or OpenGauss (OG), on 124 hard LoCoBench-Val instances. For each system, we draw 15 SC-accepted and 15 SC-rejected type-correct candidates at random, at most one per instance, for 120 candidates in total. The expert compares each Lean file with the NL input and with the Mizar gold, a formalization written by mathematicians, together with its MML definitions. The expert does not know which system produced the file or how the judges ruled, as files are shuffled, renamed, and stripped of comments. The expert judges faithfulness by criteria fixed before the audit. A formalization is unfaithful if it weakens, strengthens, or changes a claim, adds a hypothesis that does real work, uses a vacuous definition, or omits a claim. Renamed variables and equivalent reformulations count as faithful.

  • (c) Results (Table 6).

    SystemFalse acceptancesFalse rejections
    AIProver+CC1 / 150 / 15
    NLA+CC2 / 151 / 15
    OG+CC0 / 153 / 15
    CC0 / 150 / 15
    Total3 / 60 (5.0%)4 / 60 (6.7%)

    Errors are rare for every system, at most 3 of its 30 audited candidates. The audit thus supports the reliability of the SC results in Table 3 (Appendix C).

[continued in next comment box 2/3...]

AuthorsOpenReview ↗

Authors' response to Reviewer YzWj (Part 2/3)

[...continued from previous comment box 1/3]

  • (d) Whole research papers, reviewed by a mathematician (Section 5.4, Table 5, Appendix D).

    • Proof faithfulness. We formalize three research papers: one on groups associated with the Monster Lie algebra, the 3D Jacobian conjecture counterexample, and Tarski's high school algebra problem. We compare a single CC session with three systems in the Archon agent orchestration framework. All twelve formalizations build without sorry or new axioms, and all Archon systems state every main result faithfully (Tables 7–9). The difference is in the proofs. Only AIProver's proofs follow each paper's argument through its key results, such as the countermodel theorem of the Tarski paper. Archon+CC+AIProver also costs 2.9–5.7× less than the other Archon systems (Table 5). The others bypass some of these results. Their proofs remain valid, but a mathematician must reconstruct the paper's argument to follow them (Appendix D). On the Monster Lie algebra paper, AIProver's proofs reuse every lemma that the paper's own proofs rely on, with 25% fewer declarations than CC's (Table 7).

    • The mathematician's verdict. The author of the Monster Lie algebra paper, a mathematician working in the area, reviewed all four formalizations. Below is the paper author's assessment of all four versions, reproduced verbatim with permission (Figure 6, Appendix D):

    I have reviewed the outputs from Claude Code, Archon + Claude Code, Archon + Claude Code + Numina-Lean-Agent, and Archon + Claude Code + AIProver.

    I can confirm that the output of Archon + Claude Code + AIProver is my choice for a mathematician reading the argument. It is the best first read among the four. Its intermediate lemmas make it easy to see which hypothesis is doing the work and which argument is being reused. It makes the dependencies in the hardest proof easiest to see. It also is the best at explaining how the argument is organized in Lean. Its explanations keep the main mathematical steps visible with fewer intervening declarations. Among all four outputs, this one makes it easier to reconstruct the mathematical argument from the formal development.

    The Claude Code output, while correct, is dense and hard to read. The generator representations make individual relators hard to trace.

    Following the Archon + Claude Code output requires learning auxiliary notation, which is cumbersome.

    The Archon + Claude Code + Numina-Lean-Agent has many small helper declarations which make a continuous read slower.

[continued in next comment box 3/3...]

AuthorsOpenReview ↗

Authors' response to Reviewer YzWj (Part 3/3)

[...continued from previous comment box 2/3]

  • (e) On two of the longest hard LoCoBench-Val instances, AIProver+CC produces the most faithful formalization (Appendix C):

    • Models of ZF from a sequence of domains (six claims, 15k characters). The proof needs ZF syntax and satisfaction, Grothendieck universes, and a Löwenheim–Skolem-type reflection theorem, none of which is in Mathlib. AIProver+CC builds all of it in 1,405 lines and proves the reflection theorem. NLA+CC never proves it. Instead, it assumes it as a hypothesis, so its main result is conditional on the very lemma it should establish. CC formalizes only the first claim, and OG+CC does not compile.

    • A measure on LTL tableaux (24 theorems, 23k characters). AIProver+CC formalizes all 24 theorems over full LTL. NLA+CC and CC negate only atoms, so they cover only a fragment of LTL.

  • (f) The SC judge also catches AIProver's own failures (Appendix C). We also report the two longest hard instances on which AIProver+CC fails but another system succeeds. On the first, AIProver+CC assumes lemmas that it should prove. On the second, it formalizes a flawed NL input literally. The judge correctly rejects both, so these failures do not inflate our results. On two more ambiguous inputs, the judge wrongly rejects AIProver+CC, so its errors work against our system.

Thus, the audit, the case studies, and the mathematician's review together support both the reliability of our SC metric and the faithfulness of AIProver's formalizations, while also showing where AIProver fails.


The revised paper also adds the following new results:

  1. A detailed ablation study (Section 5.3, Table 4). It builds up AIProver one component per row. A harness alone mostly fixes compilation: the seed harness and HarnessEvolve lift Leanstral's TC from 27.4% to 84.4%, but full proofs only from 12.6% to 22.6% (b→d). Co-adapting the model with its harness lifts full proofs to 29.3% (e), and to 36.7% when starting from SAM (i).
  2. Experiments on three established benchmarks (Section 5.4, Figure 5a). AIProver+CC is the most accurate on ShadowBench, MathAtlas-Test, and MathAtlas-Hard, which also requires proof synthesis, at 20–31% lower cost than NLA+CC.
  3. A study from pass@1 to pass@32 (Section 5.4, Figure 5b). AIProver+CC leads at every k, at about half the cost of NLA+CC.
  4. Cost next to accuracy in every comparison (Section 5.3, Figure 4), together with the one-time fine-tuning cost.

We are sincerely grateful for the reviewer's constructive suggestion. It led us to go beyond aggregate benchmark results and to examine the reliability of our semantic-correctness metric through a human expert audit, including both successful and unsuccessful formalizations. We believe these analyses directly address the concern, and that the new ablation, benchmarks, and cost analysis further strengthen the contributions that the reviewer recognized.

If the reviewer feels that these additions address the concern, we would be very grateful if they would consider reflecting this in their final assessment. We would be happy to clarify any remaining questions, and we thank the reviewer again for the thoughtful feedback and the encouraging assessment of our work.

Reviewer_g9usOpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper studies the topic of autoformalization of mathematical proofs in which a proof is available in natural language and the task is to autoformalize the proof. It develops an agentic framework that trains a 119B LLM. The system also consists of HarnessEvolve which learns from its experiences of tool calling. The work also presents LoCoBench which is dataset of 58.9k instances taken from Mathlib and other sources (pairs of NL and formal). The paper goes on to evaluate a list of models on its benchmark and concludes that its agentic system works better than existing LLMs and agentic systems.

Critical strengths

The topic is interesting.

The paper considers many different models in its evaluation although the benchmarks considered in the paper seem inadequate.

Critical weaknesses
  • Computational costs are not reported. It is not clear whether the marginal improvements reported in the paper are obtained at what computational cost especially in comparison with existing methods.

  • The method requires training a 119B LLM along the way which comes with a huge computational cost not quantified and not reported in the paper. This makes the comparison with other competing methods difficult.

  • It seems to me that most results are only pass 4. I did not understand why pass 4 is reported instead of the common practices in the literature. It is not clear to me whether more attempts by the models, e.g., pass 32, could diminish the marginal advantages reported in the paper.

  • It seems to me that the paper only experiments on LoCoBench in its Table 1. I find that inadequate. To evaluate a new agentic system, I'd need to see its performance on existing benchmarks rather than a new benchmark.


Considering the authors' promise to reviewer 7kX9 to expand their human audit in the final version of the paper, I raised my score by 2 and I am not opposed to acceptance.

Questions for authors

Please see weaknesses. I'd be happy to see any new results or clarifications.

10 comments
AuthorsOpenReview ↗

Authors' response to Reviewer g9us (Part 1/6)

We sincerely thank the reviewer for the thoughtful review, for finding the topic interesting, and for noting our extensive evaluation across many models. We have carefully addressed each of the reviewer's concerns with new experiments and corresponding revisions to the paper. The reviewer's feedback motivated us to substantially strengthen our evidence.

Below, we respond to each of the reviewer's Weaknesses (W), with corresponding revisions highlighted in blue in the revised PDF. We hope the new results address the reviewer's concerns and demonstrate the practical value of AIProver.


[R1-W1] Computational costs are not reported. It is not clear whether the marginal improvements reported in the paper are obtained at what computational cost especially in comparison with existing methods.

We appreciate the reviewer raising the question of computational cost, but we respectfully disagree that the improvements are marginal. In the revised paper, we substantially expand the evaluation. Beyond LoCoBench-Val, we evaluate AIProver on three established research-level benchmarks (ShadowBench, MathAtlas-Test, and MathAtlas-Hard), scale the number of attempts up to pass@32, and report cost next to accuracy in every comparison. We believe this makes our cost analysis one of the most extensive in proof auto-formalization. We also add a blind human audit of the hardest instances and formalize three research papers. For one of them, its own author, a mathematician, reviewed every system's formalization. Across these evaluations, AIProver achieves substantial gains at lower cost than Numina-Lean-Agent, the strongest competing Lean harness. These results show that AIProver makes research-level auto-formalization more reliable and affordable, and thus more accessible to mathematicians.

  • (a) On LoCoBench-Val, AIProver gives the largest gains, at lower cost (Section 5.3, Table 3, Figure 4).

    • Performance. Standalone, AIProver lifts its Leanstral-1.5 base from 15.7% to 36.7% TC+SC (full proofs). It outperforms OpenGauss with Leanstral-1.5 (21.5%), Aristotle (18.4%), GPT-5.6-Sol (25.2%), and even Codex (34.1%). As a skill, AIProver delivers the largest improvement for both hosts, Claude Code and Codex. It lifts Claude Code from 41.9% to 79.8% (+37.9%), while Numina-Lean-Agent adds +30.6% and OpenGauss +12.4%. It lifts Codex from 34.1% to 62.4% (+28.3%), while Numina-Lean-Agent and OpenGauss add +10.4% and +7.8% (Table 3).

    • Cost. As a skill, AIProver's cost includes the host's frontier tokens and its own GPU time: two GPUs reserved for three days per evaluation run, at USD 3.11 per GPU-hour (USD 0.15 per attempt; Section 5.3, Appendix A.1). Even so, AIProver configurations form the Pareto frontier in Figure 4: for every other system, there is an AIProver configuration that is more accurate at no greater cost. With Claude Code as the host, AIProver outperforms Numina-Lean-Agent at lower cost (79.8% vs. 72.5%, USD 1.38 vs. 1.85 per attempt). With Codex as the host, the gap is even larger (62.4% vs. 44.5%, USD 0.67 vs. 0.90). From Table 3 and Figure 4:

    HostLean agentTC+SCUSD/attempt
    noneOpenGauss21.5%0.15
    noneAIProver36.7%0.15
    Codexnone34.1%0.53
    CodexOpenGauss41.9%0.87
    CodexNumina-Lean-Agent44.5%0.90
    CodexAIProver62.4%0.67
    Claude Codenone41.9%0.81
    Claude CodeOpenGauss54.3%1.63
    Claude CodeNumina-Lean-Agent72.5%1.85
    Claude CodeAIProver79.8%1.38
  • (b) On three established benchmarks, AIProver+Claude Code is the most accurate (Section 5.4, Figure 5a).

    • Performance. We also test out-of-distribution generalization on ShadowBench, MathAtlas-Test, and MathAtlas-Hard, which contain postgraduate- and research-level problems. MathAtlas-Hard gives no NL proofs, so it tests proof synthesis along with auto-formalization. With Claude Code as the host, AIProver reaches TC+SC of 82.7% on ShadowBench, 68.3% on MathAtlas-Test, and 34.5% on MathAtlas-Hard. This is +6.2%, +6.3%, and +2.8% over Numina-Lean-Agent+Claude Code, and +50.0%, +38.0%, and +28.7% over Claude Code alone.

    • Cost. In Figure 5a, AIProver+Claude Code costs USD 1.55–3.25 per attempt, 20–31% less than Numina-Lean-Agent+Claude Code (USD 1.94–4.73). The full results, from Figure 5a:

    TC / TC+SC (%) / USD per attemptShadowBenchMathAtlas-TestMathAtlas-Hard
    Claude Code68.4 / 32.7 / 1.4488.7 / 30.3 / 0.6195.1 / 5.8 / 1.06
    OpenGauss+Claude Code100 / 76.5 / 1.65100 / 48.6 / 1.09100 / 20.7 / 2.55
    Numina-Lean-Agent+Claude Code99.0 / 76.5 / 1.94100 / 62.0 / 2.4896.0 / 31.7 / 4.73
    AIProver+Claude Code100 / 82.7 / 1.55100 / 68.3 / 1.9499.4 / 34.5 / 3.25

[continued in next comment box 2/6...]

AuthorsOpenReview ↗

Authors' response to Reviewer g9us (Part 2/6)

...continued from previous comment box 1/6]

  • (c) We evaluate pass@k from k = 1 to 32, and AIProver leads at every k (Section 5.4, Figure 5b).

    • Performance. In Section 5.4, we run Claude Code alone and with AIProver, Numina-Lean-Agent, or OpenGauss on the 20 most intricate MathAtlas-Hard theorems, with up to 32 attempts per theorem. AIProver+Claude Code leads at every k, and the ranking never changes. At k = 32, it reaches 35.0%, which is +5.0% over Numina-Lean-Agent+Claude Code (30.0%), +10.0% over OpenGauss+Claude Code (25.0%), and +35.0% over Claude Code alone (0%; see [R1-W3]).

    • Cost. In Figure 5b, AIProver+Claude Code at k = 16 (31.9%, USD 38 per theorem) already beats Numina-Lean-Agent+Claude Code at k = 32 (30.0%, USD 149). At k = 32, AIProver+Claude Code (USD 76 per theorem) is 49% cheaper than Numina-Lean-Agent+Claude Code (USD 149 per theorem).

  • (d) A human expert audits the formalizations on the hardest instances (Section 5.4, Appendix C). We audit AIProver, Numina-Lean-Agent, and OpenGauss, each with Claude Code, and Claude Code alone, on 124 hard LoCoBench-Val instances. The review checks each formalization against the NL input and the Mizar gold, blind to the system that produced it. On two of the longest inputs, AIProver+Claude Code produces the most faithful formalization:

    • Models of ZF (six claims, 15k characters). The proof needs ZF syntax and satisfaction, Grothendieck universes, and a Löwenheim–Skolem-type reflection theorem, none of which is in Mathlib. AIProver+Claude Code builds all of it in 1,405 lines and proves the reflection theorem. Numina-Lean-Agent+Claude Code never proves it. Instead, it assumes it as a hypothesis, so its main result is conditional on the very lemma it should establish. Claude Code formalizes only the first claim, and OpenGauss+Claude Code does not compile.

    • LTL tableaux (24 theorems, 23k characters). AIProver+Claude Code formalizes all 24 theorems over full LTL. Numina-Lean-Agent+Claude Code and Claude Code negate only atoms, so they cover only a fragment of LTL. Claude Code also replaces satisfaction by an arbitrary predicate, and OpenGauss+Claude Code builds the conclusion into its own construction.

[continued in next comment box 3/6...]

AuthorsOpenReview ↗

Authors' response to Reviewer g9us (Part 3/6)

[...continued from previous comment box 2/6]

  • (e) On three research papers, AIProver gives the most faithful proofs, and one paper's author, a mathematician, calls its formalization "the best first read among the four" (Section 5.4, Table 5, Appendix D).

    • Performance. We formalize three research papers: one on groups associated with the Monster Lie algebra, the 3D Jacobian conjecture counterexample, and Tarski's high school algebra problem. We compare a single Claude Code session with three systems in the Archon agent orchestration framework. All twelve formalizations build without sorry or new axioms, and all Archon systems state every main result faithfully (Tables 7–9). The difference is in the proofs. Only AIProver's proofs follow each paper's argument through its key results, such as the countermodel theorem of the Tarski paper. The others bypass some of these results. Their proofs remain valid, but a mathematician must reconstruct the paper's argument to follow them (Appendix D).

    • The mathematician's verdict. The author of the Monster Lie algebra paper, a mathematician working in the area, reviewed all four formalizations. Below is the paper author's assessment of all four versions, reproduced verbatim with permission (Figure 6, Appendix D):

    I have reviewed the outputs from Claude Code, Archon + Claude Code, Archon + Claude Code + Numina-Lean-Agent, and Archon + Claude Code + AIProver.

    I can confirm that the output of Archon + Claude Code + AIProver is my choice for a mathematician reading the argument. It is the best first read among the four. Its intermediate lemmas make it easy to see which hypothesis is doing the work and which argument is being reused. It makes the dependencies in the hardest proof easiest to see. It also is the best at explaining how the argument is organized in Lean. Its explanations keep the main mathematical steps visible with fewer intervening declarations. Among all four outputs, this one makes it easier to reconstruct the mathematical argument from the formal development.

    The Claude Code output, while correct, is dense and hard to read. The generator representations make individual relators hard to trace.

    Following the Archon + Claude Code output requires learning auxiliary notation, which is cumbersome.

    The Archon + Claude Code + Numina-Lean-Agent has many small helper declarations which make a continuous read slower.

    • Cost. Orchestration is expensive: without AIProver, the Archon systems cost 5.6–30.5× as much as a single Claude Code session (Tables 7–9). AIProver makes it affordable, as Archon+Claude Code+AIProver is 2.9–5.7× cheaper than both (Table 5):
    USD per paper (Table 5)GroupsJacobianTarski
    Claude Code, single session7.02.30.9
    Archon+Claude Code38.924.125.9
    Archon+Claude Code+Numina-Lean-Agent65.933.319.3
    Archon+Claude Code+AIProver11.58.35.3
    • Usage limits (Appendix D). We would also like to point out a practical limitation of formalizing with Claude Code on a subscription. We ran Claude on Claude Max, which caps usage in five-hour windows. The other two Archon systems repeatedly hit this cap and needed four to six windows per formalization, with manual resumptions. AIProver removes this bottleneck. In Archon+Claude Code+AIProver, Claude only plans and AIProver does all proving and review. Formalizing a whole paper thus uses a single window of the subscription, rather than four to six.

Thus, AIProver's improvements are substantial rather than marginal, and they come at lower computational cost than Numina-Lean-Agent.

[continued in next comment box 4/6...]

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overall Response by Authors (Part 1/2)

We sincerely thank all reviewers for their thoughtful and constructive feedback, which has substantially strengthened our work. Below, R1, R2, and R3 refer to Reviewers g9us, YzWj, and 7kX9. We are grateful that the reviewers recognized several strengths of AIProver: model–harness co-evolution "uniquely utilizing rejected harness designs as negative evidence" (R2), a graded reward ladder that lets the model "learn from partial successes rather than relying solely on binary compile verdicts" (R2), "substantial system-level gains on challenging auto-formalization tasks" (R3), and "practical value both as a standalone system and as a specialist component" (R3). The reviewers also noted that AIProver "pushes the accuracy-cost Pareto frontier" (R2), that "the topic is interesting", and that the paper "considers many different models in its evaluation" (R1).

We have responded to every weakness and question raised by the three reviewers through additional experiments, analyses, and corresponding revisions, with the main changes highlighted in blue in the revised PDF. The reviewers' main requests concerned (i) evaluation on established benchmarks and with more attempts, (ii) human validation of the semantic-correctness (SC) judge, (iii) a comprehensive account of inference and training costs, and (iv) a more fine-grained ablation of model adaptation and harness evolution.

The revised paper now provides:

  1. Three established research-level benchmarks (Sections 5.2 and 5.4, Figure 5a). ShadowBench (98 problems) and MathAtlas-Test (142 theorem–proof pairs) give NL proofs and test proof auto-formalization. MathAtlas-Hard (328 theorems) gives no proofs, so it also tests proof synthesis.

  2. A study from pass@1 to pass@32 (Section 5.4, Figure 5b), on the 20 most intricate MathAtlas-Hard theorems.

  3. A blind human expert audit of the SC judge (Section 5.4, Appendix C, Table 6). On 120 candidates from hard LoCoBench-Val instances, sampled equally from accepted and rejected outputs, we report false acceptances and false rejections per system, with case studies and failure cases.

  4. Formalizations of three complete research papers (Section 5.4, Table 5, Appendix D). The author of one paper, a mathematician working in the area, reviewed all four formalizations of it (Figure 6).

  5. Inference and one-time costs (Section 5.3, Figure 4). We report inference cost next to accuracy in every comparison. We also add the one-time cost of LoRA fine-tuning and harness evolution, USD 6.3k (~1.8k GPU-hours), or USD 7.4k including synthetic data generation. The 119B base itself stays frozen.

  6. A more fine-grained ablation (Section 5.3, Table 4), which builds up AIProver one component per row, from Leanstral and from SAM-tuned Leanstral.


Headline results. TC+SC · cost per attempt, with AIProver's gain over NLA+CC in parentheses. For pass@32, the cost is per theorem.

BenchmarkCC aloneOG+CCNLA+CCAIProver+CC
LoCoBench-Val, full proofs (Table 3, Figure 4)41.9% · $0.8154.3% · $1.6372.5% · $1.8579.8% (+7.3%) · $1.38
ShadowBench (Figure 5a)32.7% · $1.4476.5% · $1.6576.5% · $1.9482.7% (+6.2%) · $1.55
MathAtlas-Test (Figure 5a)30.3% · $0.6148.6% · $1.0962.0% · $2.4868.3% (+6.3%) · $1.94
MathAtlas-Hard, AFPS (Figure 5a)5.8% · $1.0620.7% · $2.5531.7% · $4.7334.5% (+2.8%) · $3.25
20 most intricate MathAtlas-Hard, pass@32 (Figure 5b)0.0% · $2625.0% · $4530.0% · $14935.0% (+5.0%) · $76

Here, CC is Claude Code, OG is OpenGauss, NLA is Numina-Lean-Agent, and AFPS is auto-formalization and proof synthesis. Used on its own, without a frontier coding agent, AIProver raises its Leanstral-1.5 base from 15.7% to 36.7% at USD 0.15 per attempt, exceeding both OpenGauss with Leanstral-1.5 (21.5%) and Codex (34.1%) (Table 3, Figure 4).

[continued in next comment box 2/2...]

AuthorsOpenReview ↗

Overall Response by Authors (Part 2/2)

[...continued from previous comment box 1/2]

Key findings from the revision:

  1. Improving a model and its harness separately is not enough. They must be adapted together (Section 5.3, Table 4, Appendix G.2). A harness alone mostly fixes compilation, not faithfulness. A harness evolved for Leanstral helps SAM-tuned Leanstral compile nearly twice as often (TC 30.6% → 60.4%), but it yields correct, complete proofs on only 5.1% of problems. Jointly re-adapting the harness as the model changes raises this to 36.7%. As the model's failures shift from not compiling to stating the wrong theorem, HarnessEvolve automatically finds new safeguards that target them.

  2. AIProver forms the accuracy–cost Pareto frontier (Section 5.3, Figure 4). As a skill, it lifts Claude Code from 41.9% to 79.8% and Codex from 34.1% to 62.4%, more than Numina-Lean-Agent and OpenGauss on either host, at 15–26% lower cost. Its cost includes its own GPU time.

  3. The gains transfer beyond LoCoBench (Section 5.4, Figure 5a). Although AIProver is post-trained only on LoCoBench-Train, AIProver+CC is the most accurate on ShadowBench, MathAtlas-Test, and MathAtlas-Hard, at 20–31% lower cost than NLA+CC. On MathAtlas-Hard, it must also find the proof.

  4. AIProver leads at every k from 1 to 32 (Section 5.4, Figure 5b). At every k, AIProver+CC costs about half as much as NLA+CC. At k = 16, it already beats NLA+CC at k = 32 (31.9% vs. 30.0%, at USD 38 vs. 149 per theorem).

  5. The blind audit supports the reliability of the SC judge, even on hard instances (Table 6, Appendix C). Of the 60 audited acceptances, only 3 (5.0%) are false. Of the 60 audited rejections, only 4 (6.7%) are false. On two of the longest hard inputs, the human review also finds that only AIProver+CC's formalization is faithful.

  6. AIProver most faithfully preserves the arguments of the three research papers, at a fraction of the cost (Section 5.4, Table 5, Appendix D). Its proofs follow each paper's argument through its key results, whereas the other systems bypass some of them, leaving a mathematician to reconstruct parts of the argument. Archon+CC+AIProver also costs 2.9–5.7× less than the other Archon systems, and its Claude usage fits within one five-hour Claude Max window rather than four to six. The author of the Monster Lie algebra paper chose its formalization as "the best first read among the four" (Figure 6).


Every request and where we addressed it. The table covers every weakness (W) and question (Q). Each reviewer's thread gives the detailed, point-by-point response.

RequestRaised inWhere addressed
Inference cost vs. existing methodsR1-W1Section 5.3, Figure 4
Training costR1-W2Section 5.3
Why pass@4, and pass@32R1-W3Section 5.4, Figure 5b
Results on established benchmarksR1-W4, R3-W2-partASections 5.2 and 5.4, Figure 5a
Hardest instances that AIProver solvesR2-W1Section 5.4, Appendix C
Human audit of the SC judgeR2-W1, R3-W2-partB, R3-Q2Section 5.4, Appendix C, Table 6
Central contribution and reusable insightR3-W1-partASection 5.3, Table 4
Harness adapted to the updated model, and harness changesR3-W1-partB, R3-Q1Section 4, Table 4, Appendices F.2.4 and G.2
Reliable formalization beyond relative scoresR3-W2-partCSection 5.4, Table 5, Appendices C and D, Figure 6
PresentationR3-PresentationSections 5.2–5.4, Table 4, Figure 4

We are grateful to all reviewers for their time and constructive feedback, which led to a substantially stronger paper. We would be happy to clarify any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.