- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Paper Formalization, which checks proofs in Lean while treating external results as dependencies. The authors develop Paper Formalizer and 4 evaluation metrics. Experiments on 9 papers show improved faithfulness and reduced review complexity. The system also helps authors find and fix errors in 3 additional papers.
Critical strengths
-
The task separates internal proofs from external dependencies, avoiding unnecessary formalization and making the approach more practical for research papers.
-
The paper provides a structured workflow and multiple evaluation metrics, with experiments on 9 papers and 3 additional author collaborations.
Critical weaknesses
-
GPT-6-Astra accepts only 40%. The 100% faithfulness applies only to these goals, and external dependencies still require validation.
-
The paper lacks evaluation of actual review time and error detection. With GPT-6-Astra, cost increases from $373.8 to $522.7, while faithfully formalized goals increase only from 55 to 60.
Questions for authors
-
How is 100% faithfulness verified by humans? Why 90 goals are not accepted, and how is the correctness of external dependencies ensured?
-
Can the authors evaluate actual review time and error detection rates to show whether the benefits justify the additional cost?
Response to Reviewer \Rm
We thank Reviewer \Rm for recognizing the practical task formulation and structured workflow, which improve faithfulness and reduce review complexity. We also appreciate their positive assessment of the evaluation across research papers and collaborations with proof authors. We address all the questions below.
Q1. Why does Paper Formalizer with GPT-6-Astra accept only 40% of eligible goals in the end-to-end evaluation? Is the low goal acceptance in end-to-end mode with GPT-6-Astra a weakness of this work?
As defined in Section 3, a goal is accepted if the system identifies no known mathematical issue during the informal assessment, allowing formalization to proceed. When a concern remains unresolved, end-to-end mode avoids silently changing or bypassing the questionable argument in a formalization. Acceptance therefore has no absolute target, as Section 3 explicitly clarifies: a stronger model may identify concerns that a weaker model misses and accept fewer goals. Figure 5 illustrates this pattern: GPT-6-Astra identifies more concerns than GPT-5.6-Sol, particularly assumption gaps, resulting in 40% acceptance. In our author-collaboration evaluation, all three papers reach 100% acceptance after the authors resolve the concerns (Section 5).
Q2. How are the faithfulness and correctness of Paper Formalizer’s output ensured?
Lean checks the formal proofs relative to their definitions and declared assumptions, while human review checks their faithfulness to the original math. Two independent human reviewers examine Paper Formalizer’s Astra outputs, checking that the goal statements, definitions and declared external dependencies correspond faithfully to the original papers. This review confirmed 100% faithfulness for the formalized goals. For cited external results, the reviewers check that they are faithfully stated and appropriately referenced; they do not independently establish the correctness of those results. Section 5 of the revised manuscript clarifies this review procedure and its scope.
Q3. Do the benefits of Paper Formalizer justify its additional cost?
With Astra, Paper Formalizer improves faithfulness to 100% with approximately 30% higher cost per formalized goal. Faithfulness makes review easier by allowing direct comparison of the formal statements and proof steps with the original argument. Without this correspondence, reviewers must work out how the formalization differs before the affected parts can be rewritten, substantially increasing cognitive load and review effort. Paper Formalizer further reduces this burden by limiting the material to inspect: across the nine papers, custom definitions decrease by approximately 90%, duplicate definitions by 97% and explicit axioms by 64%. Its structured layout also makes the relevant content easy to locate. As a result, two independent reviewers report spending 5–20 minutes per goal, mostly gathering details from the original paper. Reviewing baseline outputs requires additional effort to navigate their chaotic layout. One of the same reviewers used AI assistance to navigate the baseline formalizations and manually reviewed several goals, reporting approximately three times the time needed to complete the same review for Paper Formalizer outputs. The additional computation thus yields both higher faithfulness and substantial savings in human review effort.