- Rating
- 2
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper presents AIProver, a framework for research-level proof auto-formalization in Lean. It combines contrastive semantic alignment training with alternating verifier-guided reinforcement learning and automated harness evolution, using a shared graded reward to improve the model and its execution workflow. The paper also introduces LoCoBench, with a 771-instance evaluation split drawn from Mizar and a bounded-arithmetic textbook. It reports improved performance over the base model and several existing systems, both as a standalone agent and as a Lean-specialist sub-agent for coding agents.
Critical strengths
Substantial system-level gains on challenging auto-formalization tasks. The paper combines data construction, model post-training, and harness optimization into an end-to-end system for research-level proof auto-formalization. On LoCoBench-Val, the reported pass@4 under the complete-proof TC+SC criterion rises from 15.7% for the base model in the seed harness to 36.7% for AIProver. Its integration with coding agents yields further gains, demonstrating practical value both as a standalone system and as a specialist component.
A promising use of verification diagnostics for agent adaptation. The framework distinguishes type errors, incomplete proofs, and semantic mismatches, using graded rewards for model training and diagnostic certificates to guide harness evolution. This provides more informative feedback than compilation success alone and offers a concrete way to adapt an agent’s execution workflow to failures encountered in formalization.
Critical weaknesses
The central research contribution remains insufficiently established. The paper assembles an extensive pipeline and reports improvements in the complete system, but it is difficult to identify the new capability or reusable insight established by the combination. The ablations show that harness evolution improves performance, but do not distinguish the benefit of adapting the harness to the updated model from the benefit of improving the model and harness individually.
Limited independent validation. All end-to-end results are obtained on the newly introduced LoCoBench, and the central semantic-correctness metric relies on LLM judges. The paper reports neither evaluation on an established external benchmark nor an expert-validated audit of the judges’ decisions on these research-level examples. Consequently, the extensive baseline comparisons establish relative performance within this evaluation setup, but provide limited independent evidence that the reported gains translate into more reliable mathematical formalization.
Questions for authors
Appendix F.2 describes the changes from the seed to the evolved harness. Which of these changes specifically responded to behavior changes after model updates, rather than failures already present in the base model?
Can the authors provide an expert audit of a sample of accepted and rejected outputs from AIProver and the strongest baselines, reporting false acceptances and false rejections? This would help establish whether the reported semantic-correctness gains are robust to judging errors.
11 comments
Authors' response to Reviewer 7kX9 (Part 2/6)
[...continued from previous comment box 1/6]
[R3-W1-partB + R3-Q1] The ablations … do not distinguish the benefit of adapting the harness to the updated model from the benefit of improving the model and harness individually. … Appendix F.2 describes the changes from the seed to the evolved harness. Which of these changes specifically responded to behavior changes after model updates, rather than failures already present in the base model?
We thank the reviewer for these insightful questions, which we answer together. A harness evolved for one model helps an updated model compile, but not produce correct proofs. To show this, we make the ablation more fine-grained in the revised paper. The new Table 4 builds up the system one component at a time, from Leanstral (rows a–e) and from SAM-tuned Leanstral (rows f–i, SAM for short), so that SAM is evaluated both in a harness evolved for Leanstral and in a harness re-evolved for SAM itself. In the rows marked RLSF ⇄ HarnessEvolve, we alternate RLSF on the model with re-evolving the harness for the updated model (Section 4). Selected rows (Full = correct, complete proofs):
| Row | Model | Harness / training | TC | Full |
|---|---|---|---|---|
| (c) | Leanstral | seed H0 | 57.6 | 15.7 |
| (d) | Leanstral | HarnessEvolve for Leanstral | 84.4 | 22.6 |
| (e) | Leanstral | RLSF ⇄ HarnessEvolve | 91.3 | 29.3 |
| (g) | SAM-tuned Leanstral | seed H0 | 30.6 | 3.4 |
| (h) | SAM-tuned Leanstral | harness of (d), evolved for Leanstral | 60.4 | 5.1 |
| (i) | SAM-tuned Leanstral | RLSF ⇄ HarnessEvolve (AIProver) | 94.8 | 36.7 |
-
(a) A harness evolved for Leanstral helps SAM compile, but rarely produce correct proofs. In this harness, SAM's TC nearly doubles (30.6% → 60.4%), but it produces correct, complete proofs on only 5.1% of problems (g→h). This is far below Leanstral's 22.6% in the same harness (d). As we explain in Appendix F.2.4, this harness was searched with Leanstral as the model.
-
(b) Re-evolving the harness with the model unlocks the gain. Alternating RLSF with harness re-evolution lifts SAM from 5.1% to 36.7% (h→i), and Leanstral from 22.6% to 29.3% (d→e). Table 4 evaluates the joint procedure as a whole. The key finding is that pairing the post-trained model with a harness evolved for the original model yields only 5.1% full proofs, whereas the joint adaptation procedure reaches 36.7%.
On R3-Q1. In a new paragraph at the start of Appendix G.2, we now state which harness changes respond to the post-trained model.
-
(c) The base model mainly fails to compile; the post-trained model mainly states the wrong theorem. In the seed harness, Leanstral's output compiles on only 57.6% of problems (Table 4, c). The write-through, search-gate, budget, and compile-and-repair hooks, and the 60 → 90 turn cap, address this base-model failure. After post-training, AIProver compiles on 94.8% of problems but states the correct theorem on only 39.0% (i). On 509 hard training problems, its file compiles with a complete proof of the wrong theorem in 173 cases (Appendix G.2).
-
(d) Five changes target the post-trained model's failure, so that it proves the theorem it was given (Appendix G.2).
- Clause-by-clause review. Once the file compiles, the harness shows the informal theorem beside the model's Lean statement. The model must show where each object, hypothesis, quantifier, and the conclusion appears, in both directions.
- Rejecting statements that say nothing. If a fixed tactic sequence proves the statement on its own, the harness rejects it and explains that the claim was likely narrowed or lost a hypothesis.
- No way around that check. The model may not unfold, rename, or substitute the theorem's definitions to pass it.
- What to give up first. A faithful statement with one unfinished step ranks above a fully proved weaker one. The last repair request asks the model to keep the statement exactly and leave each failing step as
sorry. - Library definitions for named concepts. For "e is idempotent (i.e., e² = e)", the model must use the library's definition of idempotent, after checking that it matches, rather than transcribe the parenthetical.
Thus, the harness does not merely accumulate fixes for failures already present in the base model. As the post-trained model's failure shifts from not compiling to stating the wrong theorem, HarnessEvolve finds a different set of safeguards, using the verifiers' certificates to re-tailor the harness automatically (Section 4, Appendix G.2).
[continued in next comment box 3/6...]
Authors' response to Reviewer 7kX9 (Part 3/6)
[...continued from previous comment box 2/6]
[R3-W2-partA] Limited independent validation. All end-to-end results are obtained on the newly introduced LoCoBench … The paper reports neither evaluation on an established external benchmark …
We agree that results on established benchmarks are essential to show that a new agentic system generalizes beyond its own benchmark. We introduced LoCoBench because no existing research-level dataset offers NL theorem–proof pairs at the scale needed for training, and its Val instances have no Lean formalization online to retrieve or imitate (Section 1). Following the reviewer's suggestion, we now also evaluate AIProver out of distribution on three established research-level test sets (Sections 5.2 and 5.4, Figure 5a): ShadowBench, with postgraduate- to research-level problems in eight areas; MathAtlas-Test, drawn from 72 graduate textbooks; and MathAtlas-Hard, with the theorems that have the deepest dependency trees. With CC as the host, AIProver+CC is the most accurate on all three, at 20–31% lower cost than NLA+CC.
The three test sets cover inputs with and without NL proofs (Section 5.2, Figure 5a):
- With NL proofs (proof auto-formalization). From ShadowBench, we use the 98 problems with NL proofs. From MathAtlas-Test, we sample up to two theorem–proof pairs per textbook (142 in total).
- Without NL proofs (auto-formalization and proof synthesis, AFPS). MathAtlas-Hard has no proofs, so the agent must also find the proof. We use all 328 of its theorems, from 33 textbooks.
AIProver's gains over NLA+CC are in parentheses:
| TC+SC (%) / USD per attempt | ShadowBench | MathAtlas-Test | MathAtlas-Hard |
|---|---|---|---|
| CC | 32.7 / 1.44 | 30.3 / 0.61 | 5.8 / 1.06 |
| OG+CC | 76.5 / 1.65 | 48.6 / 1.09 | 20.7 / 2.55 |
| NLA+CC | 76.5 / 1.94 | 62.0 / 2.48 | 31.7 / 4.73 |
| AIProver+CC | 82.7 (+6.2%) / 1.55 | 68.3 (+6.3%) / 1.94 | 34.5 (+2.8%) / 3.25 |
We also scale attempts from pass@1 to pass@32 on the 20 most intricate MathAtlas-Hard theorems (Section 5.4, Figure 5b). AIProver+CC leads at every k. At k = 32, it reaches 35.0%, while NLA+CC reaches 30.0% and OG+CC 25.0%.
Thus, AIProver's gains transfer beyond LoCoBench, including to MathAtlas-Hard, where the system must find the proof as well as formalize it. We also formalize three complete mathematical research papers and compare AIProver with the other SoTA frameworks on them, as we describe in [R3-W2-partC].
[continued in next comment box 4/6...]
Authors' response to Reviewer 7kX9 (Part 1/6)
We sincerely thank the reviewer for the careful reading and the insightful questions, and for recognizing AIProver's substantial system-level gains, its practical value as a standalone system and as a specialist component, and our use of verification diagnostics for agent adaptation. The reviewer's concerns led us to sharpen the paper's central contribution and to substantially expand its evaluation. The revised paper now establishes three complementary results. (1) Controlled ablations show the benefit of adapting the model and its harness together (Table 4). (2) Evaluations on three established research-level benchmarks show that the gains transfer beyond LoCoBench (Figure 5a). (3) A blind human expert audit of the semantic-correctness judges (Table 6) and formalizations of three research papers, one reviewed by its own author (Table 5, Figure 6), give evidence about the faithfulness of the resulting formalizations. We also identify the harness changes that address the post-trained model's failures (Appendix G.2).
Below, we respond to each point under the reviewer's Weaknesses (W) and Questions (Q), with corresponding revisions highlighted in blue in the revised PDF. When a weakness and a question ask the same thing, we answer them together.
[R3-W1-partA] The central research contribution remains insufficiently established … it is difficult to identify the new capability or reusable insight established by the combination.
We agree that simply combining model post-training with harness evolution would not, by itself, be a research contribution. Our contribution is demonstrating that independently improving a model and its harness is insufficient: the harness must also be re-adapted as the model's behavior changes. A harness evolved for Leanstral helps an updated model, SAM, compile twice as often (TC 30.6% → 60.4%), but SAM then produces correct, complete proofs on only 5.1% of problems. Adapting the two together raises this to 36.7% (Table 4, rows g–i). This gives a concrete design principle for verified agents: re-adapt the harness as the model's behavior changes, rather than treat it as a fixed wrapper around an improving model. In Section 5.3, we now state the findings of the new Table 4, which builds up AIProver one component per row (pass@4, %, on LoCoBench-Val; TC = type-correct, Full = TC+SC with complete proofs):
(a) Not just a better harness. The seed harness and HarnessEvolve lift Leanstral's TC from 27.4% to 84.4%, but its full proofs only from 12.6% to 22.6% (b→d). A harness mostly fixes compilation, not faithfulness.
(b) Not just a better model. Without tools, SAM lowers full proofs (1.4% vs. 3.0% for Leanstral without tools, f vs. a). Yet co-adapted with its harness, it reaches 36.7%, above the 29.3% from co-adapting Leanstral (i vs. e). As we show in Appendix F, SAM aligns NL and Lean in the hidden states, unlike SFT on the same data (matched-pair cosine 0.11 → 0.75, still 0.64 after RLSF; Figure 9).
(c) Co-adaptation helps even from the same model. From Leanstral in its evolved harness, interleaving RLSF with HarnessEvolve lifts full proofs from 22.6% to 29.3% (d→e).
(d) A new capability: an open-weight Lean specialist for frontier coding agents (Table 3, Figure 4). As a skill, the host plans and AIProver writes Lean and searches for proofs (Appendix B). AIProver lifts Claude Code (CC) from 41.9% to 79.8% and Codex from 34.1% to 62.4%. With either host, it outperforms Numina-Lean-Agent (NLA) and OpenGauss (OG), which reach 72.5% and 54.3% with CC and 44.5% and 41.9% with Codex, at 15–26% lower cost.
The contribution is therefore not an aggregation of post-training and harness engineering. The two are linked: post-training changes how the model fails, and the harness must change what it checks in response (see [R3-Q1]). The resulting specialist also transfers beyond LoCoBench and improves frontier coding agents as a reusable component.
[continued in next comment box 2/6...]