- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents partial soundness, a theorem-proving framework that bridges natural-language proof generation and Lean4 verification of proof structure. The framework first decomposes a theorem into intermediate subgoals and then formalizes them in Lean. Then it uses Lean to verify whether the decomposed subgoals can prove the theorem. This results in a partial soundness guarantee: the logical dependency structure is machine-verified, while leaf-level reasoning must be verified manually. The work also introduces the InformalRA dataset, which contains 114 undergraduate analysis theorems with human-reviewed Lean formalizations. The experiment indicates that partial soundness achieves performance comparable to direct generation.
Critical strengths
- Interesting problem setup: The proposed partial soundness framework offers a middle ground between unconstrained informal proving and fully formalized proofs. It has the potential for LLMs for math with partial formal guarantees
- Human evaluation effort: The author team performs extensive human-in-the-loop evaluation for both dataset construction and evaluation, which is worth acknowledging, since evaluating informal proofs remains difficult to automate reliably.
Critical weaknesses
- Small and limited dataset: Despite the Lean formalizations being human-reviewed, the InformalRA dataset contains only 114 problems and covers only basic topics in classical analysis from a single textbook. Each problem also includes only the natural language statement and the Lean4 theorem statement. It has no aligned gold subgoal decomposition in either natural language or Lean, which has the closest connection to the paper's partial soundness contribution of the paper.
- Unclear positioning of experiments:
- The experimental section mainly compares the overall correctness of partial soundness (PS) and direct generation (DG), which are nearly identical. It does not clearly demonstrate why the method is working or in what sense it is superior.
- Both methods are saturated on all the evaluated datasets, with manual evaluation accuracy above 97% for both PS and DG. Comparing on these datasets leaves little room to differentiate the two methods, which weakens the experimental evidence.
Questions for authors
- Can the authors clarify the intended use of InformalRA? In particular, does it differ from existing benchmarks, given that both PS and DG achieve similar, saturated performance on them?
- Can the authors compare PS and DG on harder benchmarks, or with weaker or open-source models instead of only GPT-5.5? Settings where DG is far from saturated would better reveal whether and when the verified decomposition helps.
- Can the authors provide a more detailed case analysis? The appendix presents several examples but does not systematically analyze which aspects of the framework they reflect or how representative they are.
2 comments
Thanks a lot for your response. Most of my concerns remain unresolved, especially regarding the small, saturated benchmark and the paper's unclear positioning. Thus, I have decided to maintain my current score.
Response to Reviewer krQm
Thank you for your valuable feedback. We respond to the concerns you have raised below.
Response to Weaknesses 1 and 2.2:
InformalRA is not meant as a difficulty benchmark for frontier models; as you note, GPT-5.5 saturates it. Its intended uses are:
(i) evaluating proofs under fixed textbook definitions (we release the definitions used, since textbook definitions may differ from those in formal libraries) and
(ii) paired natural-language and human-reviewed Lean statements with one objective per problem, which enable studies of formalization fidelity.
We do not prescribe gold decompositions, since a theorem may admit multiple valid proof structures. InformalRA is intended to support the generation and verification of such decompositions rather than comparison against a single reference outline.
Response to Weakness 2.1:
We tried to point out a qualitative difference between PS and DG. Section 4 pinpoints some characteristics of PS, while Table 2 (Section 7.2) and Table 3 (Section 7.3) compare additional qualitative features, such as Completeness, Correctness, Clarity, and proof length.
Response to Question 1:
For the intended use of InformalRA, see the above response to weaknesses 1 and 2.2. InformalRA overlaps in spirit with the analysis portion of ProofNet; the differences are course-centered coverage of one textbook and the accompanying definition file.
Response to Question 2:
We agree that this is a worthy point of comparison and is potentially enlightening. However, it may be difficult to present these comparisons within the discussion period – we will try to integrate such studies in updates/edits.
Response to Question 3:
Below, we clarify what some of the examples in the appendix illustrate and how representative they are.
(i) Explicit stepwise structure (App. B.2, Fig. 3, Sec. 4.2): PS proofs state their intermediate claims as labeled steps or cases (on average 6.6–8.5 per proof, and at least three in 91–100% of proofs in every dataset and trial), whereas no DG proof contains labeled intermediate claims. This structure follows from the framework’s design, so the example is representative of PS outputs in this respect. The examples of errors in monolithic proofs (App. B.3) illustrate a related issue: the difficulty of locating errors without such a structure. These errors are rare with GPT-5.5 (7 of 1,456 DG proofs were judged incorrect), so the examples illustrate how difficult an error can be to locate when one occurs, rather than how often errors occur.
(ii) Effect of Lean verification (App. B.4, Fig. 4, Sec. 4.3): Fig. 4 presents a controlled example illustrating the effect of Lean verification. To contextualize this example, Lean rejected the first candidate composition for 11–38% of problems in our GPT-5.5 runs. In a separate LLM-judged rerun on Putnam, about 90% of the failed attempts were attributed to Lean-level errors and the rest were indeterminate; none were attributed to logically insufficient decompositions.