- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes a data synthesis method for automated theorem proving, enabling the generation of exploration data for fine-tuning LLMs for ATP. By incorporating an adaptive beam size strategy, the proposed method balances exploration and exploitation during tree search. The approach contributes to improving the proving capabilities of LLM-based ATP systems, particularly under limited computational budgets. Specifically, it achieves an average pass rate of 60.74% on MiniF2F under the Pass@1 setting. Given the relative scarcity of training data in formal theorem proving, the proposed data synthesis method also offers practical value.
Critical strengths
-
The proposed data synthesis method has practical significance for improving LLM-based ATP, as the amount of available training data for automated theorem proving is considerably smaller than that for other programming languages.
-
The proposed method achieves a promising proof success rate under the limited inference budget of Pass@1. This result provides valuable insights into balancing inference costs and proof success rates in ATP.
Critical weaknesses
-
The experimental evaluation lacks sufficient benchmark diversity and, more importantly, comparisons with more recent LLM-based ATP methods, such as Goedel-Prover (https://openreview.net/forum?id=j4C0nALrgK) and APOLLO (https://arxiv.org/abs/2505.05758). While the proposed method does not necessarily need to outperform all existing approaches, the current baseline comparisons are insufficient, especially considering the large number of LLM-based ATP methods already evaluated on MiniF2F and ProofNet. Similarly, the coverage of tree-search-based approaches in the related work section is not comprehensive enough. Have more recent and effective tree-search methods emerged in 2025 and 2026? The paper lacks sufficient citations and comparisons with these recent approaches.
-
In Sections 3.1 and 3.2, several parameter choices appear to be largely based on empirical observations. While such design choices are reasonable, they raise concerns about the generalizability of the proposed method.
Questions for authors
1.The introduction states that the proposed method uses Lean 4 as its underlying ITP but can be easily applied to other proof assistants. However, has the method been evaluated on other proof assistants, such as Isabelle or Coq, to demonstrate its generalizability?
2.In the methodology section, the paper claims that a carefully selected set of commonly used tactics is sufficient to solve the vast majority of problems in practice. However, formal theorem proving extends beyond mathematical proofs to software and systems verification, such as the seL4 project (https://dl.acm.org/doi/abs/10.1145/1629575.1629596). I believe that commonly used tactics alone may not be sufficient for such systems-level verification tasks, as the proofs often rely heavily on project-specific contexts. Has the proposed method been evaluated in these more complex verification scenarios?
6 comments
Response to W2
W2
We thank the reviewer for raising this issue. We understand the reviewer's concerns about hyperparameter selection and the generalizability of the method. Some parameters in this work are indeed set primarily based on empirical observations across multiple experiments, but their design also reflects the corresponding search mechanisms and practical considerations. Specifically, the parameters in Section 3.1 primarily control the trade-off between exploration diversity and computational efficiency during data synthesis, whereas those in Section 3.2 dynamically adjust the beam size at different search stages to balance exploration and exploitation. We further discuss the roles and selection rationale of these parameters in Appendices A.9.4 and A.10 and provide results under different parameter configurations in Tables 8 and 12. These results show that the method's performance is not limited to a single parameter configuration. In addition, Appendix A.5.2 (Table 4) demonstrates the applicability of the adaptive beam size strategy across multiple datasets under the same parameter configuration. These results provide further support for the effectiveness of the method in different settings. In the revised manuscript, we will further clarify the rationale behind the hyperparameter design. In future work, we will conduct more systematic sensitivity analyses and evaluations across scenarios to further examine the method's robustness and generalizability.
Response to Q1
Q1
We thank the reviewer for the question. Although all current experiments are based on Lean 4, our method does not depend on proof syntax specific to Lean 4. Coq and Isabelle also support proof-state-based interactive reasoning, and their commonly used proof commands have similar structures. For example, apply H and rewrite H in Coq, as well as apply (rule lemma) and apply (simp add: lemma) in Isabelle, can all be separated into proof operations and their corresponding arguments. Therefore, our core idea of exploring different proof operations through constrained decoding, completing their arguments with a policy model, and verifying state transitions with a proof assistant is, in principle, also applicable to these proof systems.
Of course, proof languages differ in syntax, proof state representation, and verification interfaces, so practical application still requires corresponding adaptations. We also agree that methodological similarities alone are insufficient to establish cross-language generalizability, and direct experimental validation on Coq and Isabelle is currently lacking. We thank the reviewer for the suggestion. In the revised manuscript, we will adjust the relevant wording to clarify the transferability of the method and the scope of the current experiments, and further validate its effectiveness on other proof assistants in future work.
Response to W1
W1
We thank the reviewer for the suggestions regarding experimental coverage and recent related work. Our experiments primarily examine the effectiveness of data synthesis and search strategies in step-level proving settings. We therefore select InternLM2.5-StepProver and BFS-Prover as the main representative baselines under a unified evaluation environment and search budget to make the comparison as fair as possible. In addition, Appendix A.5.2 (Table 4) includes experiments on four datasets, miniF2F_v2s, miniF2F_v2c, Lean-Workbook, and NuminaMath-LEAN, to further examine the applicability of the adaptive beam size strategy across different problem distributions.
We consider the recent works mentioned by the reviewer to provide important research directions for ATP as well. Goedel-Prover-V2 primarily improves whole-proof generation through multi-stage training and self-correction, while APOLLO focuses on proof repair using Lean feedback. In addition, recent tree-search methods, including MPS-Prover, 3D-Prover, and BFS-Prover-V2, improve search performance through multi-perspective search guidance, tactic diversity selection, and multi-agent collaboration, respectively. In comparison, our work primarily focuses on obtaining high-quality training data at a lower data synthesis cost through a single round of proof state exploration and improving search performance through a lightweight adaptive beam size strategy. These methods differ from ours in research objectives and technical approaches and offer complementary directions for further ATP development. We thank the reviewer for the suggestion. In the revised manuscript, we will further introduce and compare these recent representative methods and, together with the existing experiments, more clearly discuss the differences in training pipelines, search mechanisms, and inference budgets to present the positioning and contributions of our work more comprehensively.