acceptodds
Under review as a conference paper at ICLR 2027

Paper Formalization for Efficient and Rigorous Proof Review

Abstract

Reviewing mathematical proofs is demanding. In our survey of NeurIPS, ICML, and ICLR in 2025, reviewers report checking at least part of a proof for only 28.4% of theoretical publications with proofs. Although missing reports do not establish that proofs go unchecked, this limited reported coverage motivates additional support for rigorous proof review. Proof assistants such as Lean could reduce this burden by automatically checking reasoning, but applying them to frontier proofs presents two challenges. First, complete formalization can unnecessarily require formalizing many results outside the paper. Second, assessing faithfulness, the semantic alignment between the formalization and the original mathematics, can require inspecting a large formal body. To address these challenges, we define Paper Formalization, propose four metrics for this task, and develop Paper Formalizer as a solution. Paper Formalization makes internal arguments machine-checkable while exposing external results as explicit dependencies, allowing each correct paper argument to be formalized without recursively formalizing all dependencies. Paper Formalizer solves the Paper Formalization task while minimizing the review burden. Across theoretical papers spanning multiple topics, Paper Formalizer with GPT-6-Astra uses approximately 90% fewer custom definitions and 64% fewer explicit axioms than the purely agentic baseline. Beyond these reductions, it faithfully formalizes more goals than purely agentic baselines with both GPT-6-Astra and GPT-5.6-Sol. With GPT-6-Astra, it achieves 100% faithfulness, confirmed by two independent human reviewers. Furthermore, in collaborations with authors of three additional publications, Paper Formalizer helps identify and correct theorem-statement typos, ambiguous proof arguments, and incorrect results. After these corrections, it faithfully formalizes all goals in these publications without further human intervention. These results support Paper Formalization as a practical supplement to proof review and Paper Formalizer as a useful tool for authors to improve their proofs, with the potential to make proof review more efficient and rigorous.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
proof review, formal math, lean

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_M4qpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Paper Formalization, which checks proofs in Lean while treating external results as dependencies. The authors develop Paper Formalizer and 4 evaluation metrics. Experiments on 9 papers show improved faithfulness and reduced review complexity. The system also helps authors find and fix errors in 3 additional papers.

Critical strengths
  1. The task separates internal proofs from external dependencies, avoiding unnecessary formalization and making the approach more practical for research papers.

  2. The paper provides a structured workflow and multiple evaluation metrics, with experiments on 9 papers and 3 additional author collaborations.

Critical weaknesses
  1. GPT-6-Astra accepts only 40%. The 100% faithfulness applies only to these goals, and external dependencies still require validation.

  2. The paper lacks evaluation of actual review time and error detection. With GPT-6-Astra, cost increases from $373.8 to $522.7, while faithfully formalized goals increase only from 55 to 60.

Questions for authors
  1. How is 100% faithfulness verified by humans? Why 90 goals are not accepted, and how is the correctness of external dependencies ensured?

  2. Can the authors evaluate actual review time and error detection rates to show whether the benefits justify the additional cost?

1 comment
AuthorsOpenReview ↗

Response to Reviewer

We thank Reviewer \Rm\Rm for recognizing the practical task formulation and structured workflow, which improve faithfulness and reduce review complexity. We also appreciate their positive assessment of the evaluation across research papers and collaborations with proof authors. We address all the questions below.

Q1. Why does Paper Formalizer with GPT-6-Astra accept only 40% of eligible goals in the end-to-end evaluation? Is the low goal acceptance in end-to-end mode with GPT-6-Astra a weakness of this work?

As defined in Section 3, a goal is accepted if the system identifies no known mathematical issue during the informal assessment, allowing formalization to proceed. When a concern remains unresolved, end-to-end mode avoids silently changing or bypassing the questionable argument in a formalization. Acceptance therefore has no absolute target, as Section 3 explicitly clarifies: a stronger model may identify concerns that a weaker model misses and accept fewer goals. Figure 5 illustrates this pattern: GPT-6-Astra identifies more concerns than GPT-5.6-Sol, particularly assumption gaps, resulting in 40% acceptance. In our author-collaboration evaluation, all three papers reach 100% acceptance after the authors resolve the concerns (Section 5).

Q2. How are the faithfulness and correctness of Paper Formalizer’s output ensured?

Lean checks the formal proofs relative to their definitions and declared assumptions, while human review checks their faithfulness to the original math. Two independent human reviewers examine Paper Formalizer’s Astra outputs, checking that the goal statements, definitions and declared external dependencies correspond faithfully to the original papers. This review confirmed 100% faithfulness for the formalized goals. For cited external results, the reviewers check that they are faithfully stated and appropriately referenced; they do not independently establish the correctness of those results. Section 5 of the revised manuscript clarifies this review procedure and its scope.

Q3. Do the benefits of Paper Formalizer justify its additional cost?

With Astra, Paper Formalizer improves faithfulness to 100% with approximately 30% higher cost per formalized goal. Faithfulness makes review easier by allowing direct comparison of the formal statements and proof steps with the original argument. Without this correspondence, reviewers must work out how the formalization differs before the affected parts can be rewritten, substantially increasing cognitive load and review effort. Paper Formalizer further reduces this burden by limiting the material to inspect: across the nine papers, custom definitions decrease by approximately 90%, duplicate definitions by 97% and explicit axioms by 64%. Its structured layout also makes the relevant content easy to locate. As a result, two independent reviewers report spending 5–20 minutes per goal, mostly gathering details from the original paper. Reviewing baseline outputs requires additional effort to navigate their chaotic layout. One of the same reviewers used AI assistance to navigate the baseline formalizations and manually reviewed several goals, reporting approximately three times the time needed to complete the same review for Paper Formalizer outputs. The additional computation thus yields both higher faithfulness and substantial savings in human review effort.

Reviewer_SBKxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a formalization task for mathematical paper review that explicitly separates the internal reasoning of a paper from its external dependencies. This allows the paper's own arguments to be machine-checked without requiring the recursive formalization of all external dependencies. The authors further introduce four evaluation metrics and develop Paper Formalizer, a system designed to improve formalization faithfulness and reduce the burden of manual verification. Experimental results show that the system improves acceptance rates and formalization faithfulness on the evaluated papers while reducing the number of custom definitions and explicit axioms.

Critical strengths
  1. A practically motivated formalization task. The paper introduces a formalization task for reviewing mathematical proofs that separates a paper's internal arguments from its external dependencies. This allows reviewers to focus on verifying the faithfulness of the formalization and the reliability of its dependencies. The proposed task addresses a clear practical need and has promising real-world applications.

  2. Improvements in formalization quality. Paper Formalizer effectively improves the Acceptance and Faithfulness metrics while substantially reducing the number of custom definitions. These results demonstrate the effectiveness of the proposed method under the paper's experimental setting.

  3. Potential for assisting mathematical paper review. In the human-in-the-loop evaluation, Paper Formalizer helps identify and correct typographical errors in theorem statements, ambiguous or incorrect proof arguments, and erroneous theorem statements. These findings demonstrate the potential of the proposed approach to assist in reviewing and improving mathematical papers.

Critical weaknesses

Weakness 1: The experimental evaluation is limited in scope.

The end-to-end evaluation covers only nine mathematical papers. Although these papers span multiple mathematical topics, the authors do not sufficiently explain the selection criteria or establish how representative the selected papers are of broader mathematical research.

Furthermore, the paper lacks a systematic analysis of performance across different levels of proof complexity, scales of external dependencies, and mathematical domains. Since these factors can substantially affect both the difficulty of formalization and the performance of the proposed system, the current experimental evidence is insufficient to establish that the approach generalizes to a broader range of research papers.

Weakness 2: The contributions of prompt design and workflow architecture are not sufficiently disentangled.

Paper Formalizer substantially reduces the number of custom definitions and explicit axioms compared with the baseline. However, the current experiments do not sufficiently distinguish how much of this improvement comes from the overall system architecture versus the prompts, output constraints, or result-filtering mechanisms employed at different stages.

Although Appendix B provides the baseline instructions, the baseline does not explicitly aim to minimize redundant definitions. As a result, it remains unclear whether some of the reported improvements could be achieved through relatively simple prompt modifications rather than the proposed multi-stage workflow.

The authors should provide more details about the key instructions and constraints used at each stage of Paper Formalizer. In addition, a stronger prompt-enhanced baseline that explicitly encourages reusing existing definitions and minimizing redundant declarations would help establish the actual benefits of the full workflow over simpler prompt-engineering strategies.

Weakness 3: The lack of ablation studies on key components.

The paper does not provide sufficient ablation studies to evaluate the individual contributions of the key components of Paper Formalizer. Without systematically removing or modifying these components, it is difficult to determine which design choices are responsible for the observed improvements in acceptance, faithfulness, and the reduction of custom definitions and explicit axioms.

A more comprehensive ablation study would help clarify the necessity of each component and provide stronger evidence for the effectiveness of the overall system design.

Questions for authors

Seeing weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer

We thank Reviewer \Rs\Rs for recognizing the practical value of the task and the improved acceptance and faithfulness with substantially fewer custom definitions. We also appreciate their recognition of the potential to help authors identify and correct errors in their proofs. We address all the questions below.

Q1. Can you clarify the paper selection criteria on evaluated papers?

We selected papers with significant theoretical contributions across core machine-learning topics, requiring that no public Lean formalization was known at the time of evaluation. The nine papers cover different topics and range from 2 to 48 eligible goals each, allowing us to examine the system across different subject areas and formalization scales. Table 1 reports the topic and number of eligible goals for each paper.

Q2. Can you provide a systematic analysis of performance on different levels of proof complexity, scales of external dependencies, and mathematical domains?

A systematic comparison by proof complexity is difficult because there is no well-defined measure that applies across mathematical domains. Our original benchmark spans nine topics and includes formalizations with up to five custom axioms. To extend both domain coverage and the scale of external dependencies, we have applied Paper Formalizer to the recent unit-distance breakthrough in discrete geometry, faithfully formalizing theoretical results using 11 custom axioms. Compared to the baseline formalization, Paper Formalizer produces more compact formalizations, with fewer custom axioms (1.00 vs. 2.50), definitions (2.64 vs. 8.58), and duplicate families (0.91 vs. 1.67) per formalized goal (Table 4). Section 5 and Appendix E of the revised manuscript present this application and additionally compares our formalization with the three existing public formalizations, each of which we found to be unfaithful.

Q3. How much does prompt design contribute to performance on the evaluated metrics?

We tested whether explicit instructions to minimize formalization verbosity could reproduce Paper Formalizer’s improvements. We added a prompt-enhanced GPT-6-Astra baseline that encourages reuse of existing definitions and theorems, sharing across goals and agents, and avoidance of redundant declarations.

Metric per formalized goalOriginal baselinePrompt-enhanced baselinePaper Formalizer
Defs11.8010.081.13
Axioms0.200.290.07
Dup.0.610.210.02
MetricOriginal baselinePrompt-enhanced baselinePaper Formalizer
Faithfulness98.2%96.8%100%

The added prompt reduces custom definitions and duplication, but both remain substantially higher than with Paper Formalizer. It also increases explicit axioms and lowers faithfulness from 98.2% to 96.8%, compared with 100% for Paper Formalizer. Thus, explicitly requesting a smaller formalization does not recover the system’s combination of compactness and faithfulness.

Paper Formalizer achieves these improvements without explicitly prompting agents to minimize verbosity. Instead, its central contract and the workflow that consistently enforces it organize the formalization around shared definitions and the original arguments. The revised manuscript provides the full role and review instructions in Appendix F and the added baseline results and prompt in Section 5 and Appendix C.

Q4. How much does the constrained output structure contribute to the performance on the evaluated metrics?

We tested this the same way as Q3. We added a structure-targeted GPT-6-Astra baseline that tells the agent to use Paper Formalizer's file organization: a shared definitions file, a shared axioms file, one result file per goal, and separate helper modules.

Metric per formalized goalOriginal baselineStructure-targeted baselinePaper Formalizer
Defs11.805.581.13
Axioms0.200.190.07
Dup.0.610.250.02

Requiring this file structure roughly halves custom definitions and duplicate families. However, the baseline still has about 5 times more definitions, about 12 times more duplicates and over 2.5 times more axioms per goal than Paper Formalizer. These results suggest that Paper Formalizer's gains come mainly from the work flow design, not from the output layout alone.

Q5. What are the individual contributions of the system’s components and design choices?

Paper Formalizer is an integrated system whose core components depend on one another, so a complete component-removal ablation would prevent the system from functioning. The dedicated review stages are an exception: formalization can proceed without them. To examine their specific contribution, we analyzed the unfaithfulness they caught across the nine evaluated papers.

Review stageSolAstra
Source262
Definition421
Scaffold370
Final proof160
Total reports1213

With GPT-5.6-Sol, every review stage catches unfaithfulness, yielding 121 reports in total. With GPT-6-Astra, the reviews catch three instances missed during formalization, all at the source and definition stages, and the final formalizations achieve 100% faithfulness. These findings demonstrate the value of dedicated reviews in detecting unfaithfulness left by the formalization stages.

I appreciate the author’s comprehensive response. Although the dataset—consisting of only 9 mathematics papers—remains quite small in scale, the additional experiments (including prompt comparisons and ablation studies) have addressed the other issues raised. Therefore, I raise my score to 3

Reviewer_qKoUOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a new tool for efficient and rigorous proof review. The novelty and impact rest on the proposal to accept external facts and sources as axioms and restrict formalization to results derived in a given text. The authors demonstrate significant improvement and efficiency of the proposed approach over competition

Critical strengths

The paper allows to reduce review burden and automate processing of large volumes of papers via automated and rigorous verification of proofs. External sources and results are considered as axioms, and the formalisation effort focuses largely on the text being assessed.

Critical weaknesses

The paper's weakness is its marginal novelty. Outsourcing formalisation (or replacing external refs and sources with axioms) has been reported elsewhere. Yet, I fully realise the importance of a general framework implementing all relevant ideas.

Questions for authors

The tool was tested on 9 papers. Some of them are classical, and I wonder if and how the Authors mitigated risks that these works might have already been used in training models used as a part of the pipeline?

2 comments
AuthorsOpenReview ↗

Response to Reviewer

We thank Reviewer \Rq\Rq for recognizing the significant improvements and efficiency gains over competing approaches. We also appreciate their recognition that focusing formalization on a paper’s own arguments can reduce proof-review burden. We address all the questions below.

Q1. Does this paper have marginal novelty?

As clarified in the revised Related Work, our task differs from prior work in what may be assumed rather than proved. Ren et al. (2026), for example, allow unresolved subgoals to become assumptions after a fixed number of attempts, leaving parts of the paper’s internal proofs unformalized. Our Paper Formalization task requires those internal arguments to be formalized, while treating cited external results as dependencies. This focuses verification on the paper’s own proofs and avoids recursively formalizing external results. Further, we implement this task in Paper Formalizer through a structured workflow designed to make the resulting formalizations easy to review. Our evaluation demonstrates higher faithfulness, fewer axiom dependencies and lower review complexity than purely agentic baselines.

Q2. Do you mitigate the risk that the evaluated papers were already included in model training?

Although the models may have encountered the evaluated papers during training, to our knowledge no public Lean formalizations of these papers were available at the time of evaluation. Even if the models have seen the informal proofs, they still need to translate them into faithful Lean statements and proofs. This removes the risk that the models are simply recalling Lean proofs from their training data. In addition, Paper Formalizer and the baseline agents use the same underlying models, so their comparison evaluates our system’s added value under the same prior knowledge.

Dear Authors, many thanks for the thorough response. I confirm that all my concerns and queries have been addressed in full.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

\newcommand{\Rq}{\textcolor{green}{qKoU}} \newcommand{\Rs}{\textcolor{blue}{SBKx}} \newcommand{\Rm}{\textcolor{fuchsia}{M4qp}}

We thank the reviewers for their thoughtful reviews. We appreciate their recognition of the practical value of separating a paper’s own arguments from external dependencies (Reviewers \Rq\Rq, \Rs\Rs and \Rm\Rm). The reviewers acknowledge significant improvements and efficiency gains over competing approaches (Reviewer \Rq\Rq), including higher faithfulness with fewer custom definitions (Reviewer \Rs\Rs) and reduced review complexity (Reviewer \Rm\Rm). They highlight the value of these improvements for reducing proof-review burden (Reviewer \Rq\Rq) and helping authors identify and correct errors in theorem statements and proof arguments (Reviewer \Rs\Rs).

In response to their constructive feedback, we have expanded the evaluation with minimality-targeted and structure-targeted baselines (Section 5 and Appendix C) and a faithful formalization of the recent unit-distance breakthrough in discrete geometry (Section 5 and Appendix E). We provide the full system instructions in Appendix F and analyze the issues detected at each review stage in our response to Reviewer \Rs\Rs. We have also clarified the human review procedure in Section 5 and revised Related Work in Section 2 to distinguish our task from prior formalization approaches. The manuscript revisions are marked in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.