- Rating
- 3
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces a benchmark motivated by the engineering challenges arising from the rapid advances in LLMs and agents for Lean-based formal reasoning. The benchmark evaluates agents' ability to simplify and compress Lean repositories.
The experimental results suggest that current closed-source agents, such as Claude Code and Codex, still have considerable room for improvement, demonstrating the challenging nature of the benchmark.
Critical strengths
The paper presents solid engineering efforts tailored to the characteristics of Lean. It also provides comprehensive experiments comparing agent capabilities from multiple perspectives, including time and cost budgets.
Additionally, the paper analyzes agents' behaviors and compression strategies. Figure 8 provides an intuitive example illustrating how Opus 5 successfully compresses a repository.
Critical weaknesses
The paper could benefit from further analysis of the benchmark's intrinsic difficulty and the potential compression space of its repositories.
A low compression score may result from limited model capabilities or from a repository that is already relatively concise. Without such analysis, it may be difficult to accurately assess model capabilities and benchmark difficulty.
Questions for authors
-
How can the potential compression space of each repository be evaluated? When a model fails to achieve further compression, how can we distinguish between limitations in model capability and a repository that is already close to optimal? Have the authors considered any methods to estimate the achievable compression?
-
What concrete benefits could repository compression bring to future Lean-based formal reasoning? The authors mention reusability and the potential of compression as a reinforcement learning objective. However, is there reason to believe that improved compression capabilities could transfer to theorem proving or proof refactoring? Could this connection be further validated experimentally?
2 comments
Thank you to the authors for their response. I would like to maintain my rating of 3 while increasing my confidence from 2 to 3.
We thank the reviewer for their thorough review of the paper. We are happy that the reviewer appreciates the engineering effort put into the curation of the benchmark, and that they found our analysis useful and our examples intuitive. In this rebuttal, we address their remaining questions and concerns.
Q1. How can the maximum score on the benchmark be evaluated?
As LeanLean is an open ended benchmark, it is impossible to know the maximum theoretical amount of compression that can be made on a repository. Benchmarks are frequently designed like this, e.g. code optimization benchmarks such as SWE-fficiency [1] and AlgoTune [2] have the same property, where a predetermined expert patch can be surpassed. Even with standard metrics, the best achievable score on many benchmarks is unclear due to noise. Even well-known benchmarks, like HLE and FrontierMath are affected by this issue [3].
However, we do have concrete evidence that models have currently not yet achieved the best possible performance: newer models evaluated such as Opus 5.5, GPT Sol 6.1, and Gemini 4 Argon show meaningful improvements on the compression scores on the benchmark. Opus 5.5 leads the benchmark with 64.3% score for $36.2 per instance, significantly better and cheaper than Opus 5.
[1] Jeffrey Jian Ma et al. “SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads?” arXiv:2511.06090v3, 2026
[2] Ori Press et al. “AlgoTune: Can Language Models Speed Up General-Purpose Numerical Programs?” Advances in Neural Information Processing Systems 38 (NeurIPS 2025), Datasets and Benchmarks Track.
[3] Epoch AI. “Benchmark Reviews Documentation.” Accessed November 13, 2026. Documentation
Q2. What insights into model capabilities can we obtain from compression scores?
Compression is a difficult, open-ended task like algorithmic optimization. In those settings a meaningful insight can be obtained from compression scores regardless of their absolute magnitude: by ranking the achieved scores of different methods. For this ranking, it is irrelevant to which degree the repository was previously already compressed and how much further compression is possible. Since we still observe that newer models and higher effort achieves higher scores, we believe that this ranking is meaningful and that the benchmark is difficult for current models.
Q3. Do compression capabilities transfer to theorem proving or proof refactoring?
Great question. We observe a general correlation between model scores on our benchmark and model scores on popular benchmarks such as coding. For example, we find a 0.98 Pearson correlation between our benchmark scores and the scores of 10 models (including more recently evaluated models) we have in common with TerminalBench 4.0 [1].
To observe a direct transfer from training on our tasks, we suggest that our tasks could be used in future work as training tasks for model posttraining. We expect that such effects can only be observed on models with non-negligible scores on our benchmark, which includes only very large and closed models.
[1] Terminal-Bench 4.0 leaderboard, archived 4 Nov. 2026.