- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Leanify, a framework and benchmark for compressing existing Lean proofs and discovering reusable lemmas. The authors formulate proof compression as an optimization problem, measuring proof size by counting nodes in the abstract syntax trees (ASTs) of kernel-checked proof terms and their dependencies. They propose an LLM-based pipeline that discovers candidate lemmas, verifies them in Lean, and rewrites existing proofs to reduce their overall size.
Experiments on several formal mathematics benchmarks demonstrate substantial proof compression, with reductions exceeding 90% in some cases. The authors further investigate whether compression can reveal mathematically meaningful abstractions, reporting that deeper compression tends to produce more non-trivial lemmas, while larger collections of related proofs enable broader lemma reuse.
Critical strengths
The formulation of proof compression using AST-based proof-term size is rigorous and well motivated. Unlike token-based or tactic-based metrics, this objective accounts for the actual proof terms checked by the Lean kernel, providing a more principled measure of compression.
More importantly, I find the broader research direction promising. Beyond simply shortening existing proofs, this work explores whether AI systems can identify meaningful intermediate results and reusable mathematical abstractions. This may encourage research into what constitutes a good lemma or theorem, potentially moving AI-assisted mathematics beyond proof generation toward a deeper understanding and organization of mathematical reasoning.
Critical weaknesses
My main concern is the evaluation of the mathematical significance of the discovered lemmas. Although the compression ratios are objectively measured and verified in Lean, the claim that deeper compression leads to more meaningful mathematical insights relies primarily on judgments from another LLM, with limited human evaluation.
Mathematical significance is inherently difficult to assess, and an LLM-based evaluation may not reliably distinguish genuinely insightful abstractions from technically non-trivial but conceptually uninteresting reformulations. Since discovering meaningful and reusable mathematical ideas is a central motivation of the paper, I believe stronger validation from human mathematicians would substantially improve the evidence supporting this claim.
Questions for authors
To what extent are the impressive compression ratios attributable to redundancy or inefficiencies in the original proofs, rather than the discovery of fundamentally simpler mathematical arguments? Could the authors provide a more systematic analysis distinguishing these two sources of improvement?
I appreciate the author for providing a comprehensive response and a thorough comparative analysis of the two previously listed methods, along with additional controlled experiments and analyses. These have effectively addressed my concerns; therefore, I am raising my score to 3