acceptodds
Under review as a conference paper at ICLR 2027

Leanify: Verified Proof Compression and Reusable Lemma Discovery

12 views

Abstract

Language models can prove difficult mathematical statements, but their proofs still need to be organized for understanding and reuse. It has been argued that mathematics advances by compression: concepts and theorems turn long deductions into reusable arguments. We formulate proof compression as a verifiable task: shorten Lean proof collections while preserving their original statements. Our agents compress the Lean proofs of the permanent lower bound and of the counterexamples to the compactness conjecture from OpenAI's Ten Advances in Mathematics and Theoretical Computer Science by 94.2% and up to 99.2%, respectively, under our proof-term size metric. For the permanent proof, compression also reduces the number of required intermediate theorems and shortens model-generated mathematical explanations. Using a lemma-discovery pipeline, we observe two scaling trends: later compression stages yield a larger share of lemmas judged non-trivial by a language model, and compressing more related Fermat's Last Theorem results together yields wider lemma reuse. These trends suggest that scaling compression may uncover more reusable mathematical arguments. We therefore release Leanify as a benchmark to test this possibility at scale.

Primary area:
datasets and benchmarks
Keywords:
Proof compression, benchmark

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_dP79OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Leanify, a framework and benchmark for compressing existing Lean proofs and discovering reusable lemmas. The authors formulate proof compression as an optimization problem, measuring proof size by counting nodes in the abstract syntax trees (ASTs) of kernel-checked proof terms and their dependencies. They propose an LLM-based pipeline that discovers candidate lemmas, verifies them in Lean, and rewrites existing proofs to reduce their overall size.

Experiments on several formal mathematics benchmarks demonstrate substantial proof compression, with reductions exceeding 90% in some cases. The authors further investigate whether compression can reveal mathematically meaningful abstractions, reporting that deeper compression tends to produce more non-trivial lemmas, while larger collections of related proofs enable broader lemma reuse.

Critical strengths

The formulation of proof compression using AST-based proof-term size is rigorous and well motivated. Unlike token-based or tactic-based metrics, this objective accounts for the actual proof terms checked by the Lean kernel, providing a more principled measure of compression.

More importantly, I find the broader research direction promising. Beyond simply shortening existing proofs, this work explores whether AI systems can identify meaningful intermediate results and reusable mathematical abstractions. This may encourage research into what constitutes a good lemma or theorem, potentially moving AI-assisted mathematics beyond proof generation toward a deeper understanding and organization of mathematical reasoning.

Critical weaknesses

My main concern is the evaluation of the mathematical significance of the discovered lemmas. Although the compression ratios are objectively measured and verified in Lean, the claim that deeper compression leads to more meaningful mathematical insights relies primarily on judgments from another LLM, with limited human evaluation.

Mathematical significance is inherently difficult to assess, and an LLM-based evaluation may not reliably distinguish genuinely insightful abstractions from technically non-trivial but conceptually uninteresting reformulations. Since discovering meaningful and reusable mathematical ideas is a central motivation of the paper, I believe stronger validation from human mathematicians would substantially improve the evidence supporting this claim.

Questions for authors

To what extent are the impressive compression ratios attributable to redundancy or inefficiencies in the original proofs, rather than the discovery of fundamentally simpler mathematical arguments? Could the authors provide a more systematic analysis distinguishing these two sources of improvement?

Reviewer_Aq2mOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes mathematical proof compression as a challenge, and proposes a workflow for discovering common and general mathematical abstractions by the reorganizing the intermediate lemmas and proof structures. The authors introduce a new metric for abstraction discovery as the rate of minimizing the size of Lean kernel proof terms. They release the corresponding evaluation workflow as a benchmark for comparing agents’ ability to compress mathematical proofs. They also report empirical findings in the compression tasks, including that longer searches lead to higher compression rate and that later stages of compression promote the discovery of “modest, identifiable mathematical observations.”

Critical strengths
  1. A new metric and optimization objective for proof compression. The paper proposed the proof size as a metric for measuring the compression rate of proofs. Together with fixed target statements and kernel verification, this provides an optimization objective for reorganizing proof collections and a consistent evaluation framework for comparing agents’ compression performance.

  2. Empirical insights into how compression reorganizes mathematical proofs. The experiments show that longer searches can achieve greater compression. And proof compression can reduce intermediate dependencies and simplify mathematical arguments. They also suggest that later compression stages favor helper lemmas rated as non-trivial by the LLM judge (scoring at least 3), and considering larger collections of related theorems together leads to wider lemma reuse. These findings help characterize how compression reorganizes proofs and supports the discovery of reusable mathematical abstractions.

Critical weaknesses
  1. Availability of benchmark and experimental artifacts. Although the authors describe Leanify as a released benchmark, the manuscript does not provide any access to its implementation sources. Code and experimental artifacts needed to reproduce the reported results are also unavailable in the supplied materials. This limits independent verification and reproducibility. Making these resources available would strengthen the reliability and practical value of this work.

  2. The interpretation of “deeper” part in Section 4.2. is not complete. It emphasizes an increasing proportion of helpers scoring at least 3. However, Figure 5a shows no comparable trend for helpers scoring at least 5; their proportion at the final checkpoints is below that of the original checkpoint. These results support an increase in modest mathematical observations, but do not establish that deeper compression produces more substantive insights. Appendix E.8 acknowledges this limitation, and the main text should make this distinction clearer when interpreting the results.

Questions for authors

How can the proposed benchmark and empirical findings benefit related research or the broader field of automated mathematical reasoning? In particular, which findings are expected to generalize beyond the specific tasks studied in these experiments, and what evidence supports their broader applicability?

Reviewer_ZyZNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a proof-compression benchmark that uses the number of AST nodes in Lean kernel proof terms as its optimization objective. It evaluates the compression capabilities of several state-of-the-art open-source and closed-source models. The authors further investigate whether deeper compression and larger sets of jointly optimized targets can facilitate the discovery of auxiliary lemmas that are more mathematically meaningful and broadly reusable.

Critical strengths
  1. A different perspective on formal proof compression. Compared with conventional metrics such as source-code length or token count, the number of AST nodes in Lean kernel proof terms offers a different perspective on formal proof compression. This metric is worth investigating, although its relationship to practical proving costs and lemma reusability requires further validation.

  2. Detailed reproducibility materials. The comprehensive appendix provides detailed instructions for reproducing the experiments, facilitating future replication and verification of the results.

  3. Broad model coverage. The benchmark evaluates a diverse range of state-of-the-art open-source and closed-source models, providing a relatively comprehensive assessment of their proof-compression capabilities.

Critical weaknesses

Weakness 1: The practical significance of the proposed optimization objective is insufficiently established.

The paper uses the number of AST nodes as its primary optimization objective. However, it remains unclear what practical advantages this metric provides over conventional source-code or token-length metrics. Although proof-term size captures a different aspect of formal proofs, the paper does not establish that reducing it leads to lower kernel-checking costs, improved proof readability, reduced agent context requirements, or better downstream proving performance. Similarly, the paper does not demonstrate that AST-based compression is more effective than source-level compression at discovering mathematically meaningful and reusable lemmas. A direct comparison between these objectives would be important to establish the practical value of the proposed benchmark.

Weakness 2: The evaluation of cross-target lemma reuse does not adequately control for target-set size.

Figure 5(b) reports the number of distinct targets helped by an accepted auxiliary lemma as the joint target-set size increases. However, larger target sets naturally provide more opportunities for a lemma to help additional targets. Consequently, the observed increase in the absolute number of targets helped does not necessarily demonstrate improved abstraction discovery or reuse efficiency. The evaluation would be more convincing if it reported normalized reuse rates and net AST-node savings, and compared joint compression against independently compressing the same targets. More importantly, even if joint compression improves lemma reuse, this result alone does not establish an advantage of AST-node-based optimization over source-level or token-level compression.

Weakness 3: The lack of comparisons with existing proof-compression methods makes it difficult to assess the contribution of the proposed objective

The paper evaluates several models on its proposed benchmark but does not adequately compare its approach with existing proof-optimization and compression systems, such as ImProver and ProofOptimizer. Although these methods primarily target source-level or token-level metrics, they could potentially be adapted to optimize or at least evaluate kernel proof-term AST size. Without such comparisons, it is difficult to determine whether the proposed objective or compression procedure offers meaningful advantages over existing approaches. In addition, the paper does not sufficiently characterize the relationship between its benchmark and those used in prior work, including the distributions of initial proof sizes, target difficulty, and evaluation budgets. Controlled comparisons under a common verification protocol would substantially strengthen the paper.

Questions for authors

Seeing weaknesses

1 comment

I appreciate the author for providing a comprehensive response and a thorough comparative analysis of the two previously listed methods, along with additional controlled experiments and analyses. These have effectively addressed my concerns; therefore, I am raising my score to 3

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.