- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper looks maths proofs in Lean but constructs a dataset, MathAdv, with 4 variantes: standard formalised statement to be proved, multiple choice question based on known results needed for the proof, informal reasoning step, and an alternate question formulation. The methodology to generate the 321 question dataset (298 formalised statement) across 13 domains is clearly explained with multiple human verification to ensure the quality of the output. Various models which include specific provers and general-purpose models are evaluated on all of these variations. The paper shows that models often perform better at the direct-answer and multiple choice questions than the Lean proofs. They also show that transformed problems perform worse than the original problems.
Critical strengths
- The dataset generated goes through careful evaluation both automatic and human evaluation to ensure that the final questions are of high-quality.
- MathAdv is tested on 18 different models which tests a lot of different types of models (proof-step generation models, whole-proof generation models and general-purpose models).
- Good examples are shown to illustrate the dataset.
Critical weaknesses
- The dataset does not have parallel comparison. For example, 81 direct-answers, 30 transformed problems, 293 Multiple-choice question. This means that the results presented in section 4.3 are not comparable and therefore the analyse is not valid. This raises questions about the soundness of the work.
- The dataset presented test different capabilities of the models but some in formal language against natural language. This is firstly unfair as some models are specialised model in Lean proving as mentioned in Line 357/358 "Goedel-Prover remains an exception, consistent with its stronger specialization toward Lean proof search than general answer selection". Moreover, I feel like some of these have already been tested in fairer settings. For example, direct answers is tested by PutnamBench https://proceedings.neurips.cc/paper_files/paper/2024/file/1582eaf9e0cf349e1e5a6ee453100aa1-Paper-Datasets_and_Benchmarks_Track.pdf), CombiBench (https://openreview.net/pdf?id=qeZStJVYnc), ConjectureBench (https://arxiv.org/pdf/2510.11986), but they do so in the formal language setting as opposed to testing it on one hand in natural language and on the other hand in formal language.
- Some of the models that are presented in the Table 2 and 3, despite performing the best, are not mentioned in the analysis. I think they have been forgotten because line 322/323 "Goedel-Prover-V2 achieves the best performance at 21.88%, followed by DeepSeek-Prover-V1.5RL + RMaxTS" directly contradicts the results shown in Table 1.
- Section 5.1 and 5.2 are entirely written with no statistics or results to show that what is claimed is observed in the data.
Questions for authors
Suggestions and questions:
- Check the referencing you use in your latex version, \citep{, \citet{, \cite{