acceptodds
Withdrawn from ICLR 2027

Paper2Lean: Agentic Autoformalization of Mathematical Research Papers at Scale

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

12 views

Abstract

Formalizing a mathematical research paper requires more than translating isolated theorems: statements and proofs must remain semantically faithful to the source and globally consistent across long-range dependencies. We introduce Paper2Lean, an end-to-end agentic framework that converts mathematical research papers into integrated Lean projects. The framework extracts mathematical content into a dependency graph and coordinates specialized agents for statement formalization, semantic checking and repair, proof construction, and project integration. We evaluate Paper2Lean on 20 mathematical optimization research papers, using mathlib and a shared library of supporting results. All 20 automatically generated projects compile successfully without proof placeholders. Following human review and semantic repair, we introduce OptPaper-20, a corpus of 20 corrected paper-level formalizations comprising 494 Lean files and 110,910 lines of Lean code. All released projects pass clean builds, with no unresolved discrepancies identified in the audit. These results demonstrate the feasibility of agentic paper-level formalization while highlighting semantic fidelity as a central challenge beyond proof verification. The source code and dataset are publicly available at https://anonymous.4open.science/r/Paper2Lean-5A6D/.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Autoformalization, Formal Reasoning, Automated Theorem Proving, LLM Agents, Lean

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_tSepOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Paper2Lean, an end-to-end agentic framework that translates mathematical papers into Lean. The system organizes the task using a dependency graph and uses multiple specialized agents for statement formalization, semantic checking, and proof generation. The system is evaluated on 20 mathematical optimization papers. The authors report that all automatically generated projects compile without proof placeholders. However, expert review gives a source-fidelity rate of 95.7%, identifying cases that compile but are unfaithful to the original papers. After correcting the identified errors, the authors release OptPaper-20, a corpus of 20 paper-level formalizations reviewed by experts.

Critical strengths
  1. The authors show the system's ability to handle interdependent mathematical content in 20 mathematical optimization papers and release the formalization corpus OptPaper-20 following human review and semantic repair. The corpus includes Lean code and corresponding natural language descriptions, offering reusable resources for developing and evaluating paper-level autoformalization systems.

  2. With both compilation checks and expert review, the authors identify cases that compile but are unfaithful to the original papers. These analyses help readers understand the system's specific failure modes and provide useful insights for improving semantic fidelity.

Critical weaknesses
  1. The novelty over M2F is not clear. M2F already supports document-level formalization, dependency management, and separate statement and proof generation. Without a direct comparison or ablation studies, it is difficult to judge what improvements come from Paper2Lean's additional designs.

  2. The criteria for semantic fidelity need clarification. The pipeline allows assumptions to be added during proof generation without requiring another automated semantic check. It is also unclear how expert review distinguishes recovering existing assumptions from adding conditions that weaken the original statement, as illustrated by Appendix D.1. This affects the interpretation of the reported fidelity rate.

Questions for authors
  1. What are the key additions in Paper2Lean compared with M2F? Could the authors explain what additional capabilities or improvements they bring, and whether there are existing comparison or ablation results supporting them?

  2. In Appendix D.1, is the added convergence condition already part of the original theorem's assumptions, or can it be derived from them? Please provide the relevant source text or derivation. How does the final expert review track these statement changes and distinguish recovering existing assumptions from adding conditions that weaken the original statement?

Reviewer_xd7aOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Paper2Lean, an agentic framework for converting mathematical research papers into Lean projects. It extracts statements and dependencies from PDFs, formalizes the statements, checks their meaning against the source, and constructs proofs. A shared Foundation library supports reuse across papers. The evaluation covers 20 optimization papers and includes expert checks of semantic faithfulness and a comparison of two base models on three papers. The authors also release OptPaper-20, a human-refined corpus containing 494 Lean files and 110,910 lines of code.

Critical strengths
  1. Formalization at the paper level. Paper2Lean handles shared definitions and dependencies between results within a paper. This is useful for research papers, where formalizing one theorem often requires several earlier results. Completing Lean projects for 20 research papers is a useful result, and the released corpus provides material for further evaluation.
  2. Evaluation of semantic faithfulness. The expert checks and case studies examine whether the Lean statements preserve the intended mathematics. Figure 5 provides a clear example of why this matters: the generated theorem repeats its assumption and is proved by id, while the original convergence claim is missing. Reporting these failures helps readers understand what successful compilation does and does not establish.
Critical weaknesses
  1. Missing comparisons with essential baselines. Tables 2–3 compare GPT-5.6-Terra and DeepSeek-V4.1-Flash within Paper2Lean, using identical JSON inputs or fixed Lean statements. These results do not establish how the full workflow performs against other methods. M2F is used to construct Foundation, yet is not evaluated as a baseline. The framework runs through Codex CLI, yet direct use of a native coding agent is not evaluated. At minimum, the authors should compare against M2F and direct use of Codex or Claude Code on the same papers.
  2. Limited comparison with other formalization agents. Beyond native coding-agent harnesses, there is a substantial body of work on specialized formalization agents. The authors should identify relevant systems and select representative methods for experimental comparison, so that Paper2Lean’s performance can be assessed against existing approaches. Comparisons with a broader range of formalization agents would strengthen the experimental evidence and could support a higher score. The evaluation should compare final formalization quality alongside token consumption.
  3. Unquantified manual refinement. Section 4.4 distinguishes the automatic outputs from the human-refined OptPaper-20 release, but does not quantify the manual work required. The examples illustrate both semantic corrections and changes intended to improve readability or reuse, without reporting their frequency across the corpus. It is therefore difficult to estimate how much expert effort is needed to turn the automatic output into the released dataset.
  4. Incomplete accounting of Foundation’s construction cost. Appendix B provides useful information about Foundation’s size, organization and reuse. However, the resources required to construct and prepare this supporting library are not separately reported. The per-paper token figures alone do not indicate the initial investment needed before formalizing the target papers. Separating this preparation cost from the cost of processing additional papers would make the practical resource requirements clearer.
  5. Limited analysis of retries and computational bottlenecks. The case studies and average round counts provide some insight into the workflow, but do not show which difficulties account for most of the repair effort across the 20 papers. A summary of the main causes of retries and the token consumption of individual stages would help explain the reported costs. This would also make it easier to identify the parts of the workflow that remain difficult in practice.
Questions for authors
  1. Minimum baseline comparisons. Could the authors add M2F and direct use of Codex or Claude Code as baselines, at least on the three papers already used in the model comparison? These comparisons are needed to assess the benefit of the Paper2Lean workflow.
  2. Additional formalization-agent comparisons. Could the authors identify relevant formalization agents and include additional comparisons under comparable conditions? This would help establish Paper2Lean’s relative performance and could improve my assessment.
  3. Comparable settings and outcome measures. Could all methods receive the same source papers, access to the same frozen Foundation and Lean tools, and comparable budgets? Please match the base model where possible and clearly identify model differences, especially for Claude Code, so that model effects can be distinguished from workflow effects. Please report proof completion and final source faithfulness, together with input/output/cache tokens and wall-clock time. The direct-agent baseline should start from the source documents so that the comparison covers the complete task.
  4. Manual refinement effort. How many statements and proofs were modified when producing OptPaper-20, and approximately how much expert time did this require? Please distinguish corrections to mathematical content from interface refactoring and stylistic changes, so that readers can understand the work needed after automatic formalization.
  5. Foundation preparation costs. Could the authors separately report the computational and human effort involved in preparing Foundation, and clarify whether any of these costs are included in Table 1? It would be useful to distinguish the initial library preparation from the additional cost of formalizing each paper.
  6. Retry patterns and stage-level costs. Could the authors summarize the existing execution logs across the 20 papers, including the main causes of retries and repairs and the token consumption of the major stages? Which stages account for most of the computation, and which intermediate failures most often require repeated attempts?
Reviewer_PKUwOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

Paper2Lean is an agentic system for translating mathematical research papers into integrated Lean projects. For a given paper, Paper2Lean extracts its mathematical items and their dependencies into a directed acyclic graph (DAG), then coordinates agents for statement formalization, semantic checking and repair of the formalized statements, proof construction, proof simplification, and library integration.

The authors evaluate the framework on 20 optimization papers using mathlib and a shared Foundation library. They report that all generated projects compile without proof placeholders. Each paper is audited by two expert reviewers, reporting cumulative statement fidelity produced by Paper2Lean of around 95.7% (although Table 1 contains an inconsistency discussed below). The expert-refined OptPaper-20 corpus contains ~500 Lean files and ~110k lines of code. A comparison of three papers finds that both tested models complete the proof tasks, with substantially different token consumption.

The principal contributions are the integrated pipeline, the library integration idea, the formalization corpus, and the analysis of semantic failures that survive Lean verification.

Critical strengths
  1. A strong demonstration of agentic paper formalization: The work addresses the practical difficulty of maintaining compatible definitions, statements, proofs, and imports across a complete development and demonstrates the capability of the Paper2Lean agent for scalable autoformalization on 20 real-world research papers (beyond isolated and over-saturated theorem benchmarks). This makes the work potentially useful as both a formal mathematics resource and an evaluation corpus.
  2. The evaluation explicitly examines semantic fidelity beyond compilation: Each paper receives two expert reviews, and the manuscript openly analyzes outputs that pass automated semantic checking and Lean verification while failing to express the intended claim. Figure 5 is an excellent example of the importance of (human-)semantic checking: the generated theorem repeats an assumed property while omitting its consequence for the source trajectory. This audit and failure analysis substantially strengthen the paper’s empirical value.
Critical weaknesses
  1. The contribution of the proposed architecture is not clear relative to existing autoformalization agentic systems and abolation studies seem week: Effectevely every part of the workflow has already been discussed and introduced in the literature, and general autoformalization capabilities of real-world mathematical projects by agentic systems run by SOTA closed-source models like those from OpenAI have been "long" established (SpherePacking, Feb 2026), so it is unclear what the actual contribution of the paper is (it would be significant to see if this harness would help some smaller cheaper model do better in autoformalization task?), beyond an additional 20 formalized papers. M2F already addresses project-scale formalization using dependency-aware statement construction, fixed interfaces, and verifier-guided proof repair. Goedel-architect has already introduced DAG for orchestrated theorem proving. LeanFlow has already discussed orchestrated formalization and a gated semantic checker that reviews and repairs statement translations from NL to Lean. The paper doesn't seem to have done a proper literature review. Consequently, architectural designs of Paper2Lean need clearer evidence. The only controlled experiment changes the base model within the same pipeline; it does not test whether DAG scheduling, specialized roles, or semantic repair improves source fidelity, completion, or resource efficiency. Appendix E acknowledges the absence of alternative-pipeline comparisons; however, the work should have at least compared the performance of the chosen models using the base Codex agent (I suspect that GPT Sol or Astra models could have matched the reported performance using plain Codex and a simple prompt). This matters particularly because the reported token counts sum to approximately 2.22 billion, excluding proof golfing: the results demonstrate feasibility with substantial resources, but leave the architecture’s benefit at a comparable budget unclear.

  2. The evidence for complete, source-faithful formalization needs stronger accounting and justification: The reported fidelity rate concerns generated items, but the paper does not explicitly establish extraction coverage against an independently checked inventory of the source papers’ mathematical claims. Moreover, the human-repaired theorem in Figure 8 introduces ContDiff ℝ 2 f and ContDiff ℝ 2 z; Appendix D.2 acknowledges that the displayed statements are not unconditionally equivalent. The manuscript should establish that these assumptions are present in, or follow from, the source context before treating the repair as preserving the original result’s scope. There is also a concrete reporting inconsistency: the Faithful column in Table 1 sums to 473/497, implying 24 discrepancies, whereas Section 4.2 reports 473/494 and 21 discrepancies.

Questions for authors
  1. Architectural contribution: What is the actual formalization method novelty that the paper introduced? What specific capability distinguishes Paper2Lean from M2F, Goedel-Architect, OpenGauss, LeanFlow, etc., and can you provide a comparison or focused ablation on a representative subset using the same model, Foundation snapshot, and comparable resource budget? Source fidelity, completion, and cost would be the most informative outcomes.
  2. Comparison with general coding agents: I suspect that when we move to the level of SOTA LLM models, like Astra or Sol6.1 from OpenAI or Fable5.1 and Opus5.5, many of these techniques matter very little. Can you show evidence that you are better than just asking Codex or the Claude Code model (in regimes when they can also launch multiple subagents) "Please formalize this paper in Lean4"?
  3. Scope of human repairs: For Figure 8, where are the (C^2) assumptions on the potential and trajectory stated or derived in the source paper? Were all human-repaired statements audited again against the original source, including any newly introduced assumptions? If some repaired results cover a restricted setting, how is that reflected in the corpus’s fidelity claims?
  4. Evaluation denominator and coverage: Please reconcile the 473/494 aggregate with the 473/497 total obtained from Table 1. Did the expert audit also check for mathematical claims omitted during extraction, or only assess the generated declarations? A source-to-formalization inventory, with principal theorems distinguished from definitions and supporting items, would materially clarify the paper-level completion claim.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.