acceptodds
Under review as a conference paper at ICLR 2027

LeanLean: Benchmarking Repository-Scale Lean Proof Compression

10 views

Abstract

As large language models (LLMs) formalize increasingly advanced mathematics, their proofs can span millions of lines of code. Producing more concise formalizations requires models to discover simpler arguments and extract reusable lemmas across large codebases. To measure this important capability, we introduce LeanLean, a benchmark consisting of 64 large-scale, real-world Lean repositories containing up to 174 thousand lines of code. Within a 12-hour time limit, models are tasked with reducing repository size while preserving the mathematical validity of their main theorems. To achieve maximal compression, models need to optimize at three levels of abstraction: (1) simplifying the underlying mathematical arguments, (2) restructuring proof dependencies by deleting and adding new declarations, and (3) applying syntactic optimizations. The best model, Opus 5, achieves a very high average compression score of 48.3%, while smaller models such as Gemini 3.8 Flash and GPT-5.6 Luna reach less than 25%. We find that most compression comes from restructuring proof dependencies by making previously used declarations unnecessary, while Opus 5 is the only model to achieve substantial gains through syntactic optimizations. LeanLean provides a strong testbed for evaluating models across a combination of capabilities, including long-context handling, mathematical theorem proving, and coding proficiency in Lean.

Primary area:
datasets and benchmarks
Keywords:
Lean, Proof Compression, Formal Mathematics, LLM Agents, Benchmark, Automated Theorem Proving

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ad7JOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a benchmark motivated by the engineering challenges arising from the rapid advances in LLMs and agents for Lean-based formal reasoning. The benchmark evaluates agents' ability to simplify and compress Lean repositories.

The experimental results suggest that current closed-source agents, such as Claude Code and Codex, still have considerable room for improvement, demonstrating the challenging nature of the benchmark.

Critical strengths

The paper presents solid engineering efforts tailored to the characteristics of Lean. It also provides comprehensive experiments comparing agent capabilities from multiple perspectives, including time and cost budgets.

Additionally, the paper analyzes agents' behaviors and compression strategies. Figure 8 provides an intuitive example illustrating how Opus 5 successfully compresses a repository.

Critical weaknesses

The paper could benefit from further analysis of the benchmark's intrinsic difficulty and the potential compression space of its repositories.

A low compression score may result from limited model capabilities or from a repository that is already relatively concise. Without such analysis, it may be difficult to accurately assess model capabilities and benchmark difficulty.

Questions for authors
  1. How can the potential compression space of each repository be evaluated? When a model fails to achieve further compression, how can we distinguish between limitations in model capability and a repository that is already close to optimal? Have the authors considered any methods to estimate the achievable compression?

  2. What concrete benefits could repository compression bring to future Lean-based formal reasoning? The authors mention reusability and the potential of compression as a reinforcement learning objective. However, is there reason to believe that improved compression capabilities could transfer to theorem proving or proof refactoring? Could this connection be further validated experimentally?

2 comments

We thank the reviewer for their thorough review of the paper. We are happy that the reviewer appreciates the engineering effort put into the curation of the benchmark, and that they found our analysis useful and our examples intuitive. In this rebuttal, we address their remaining questions and concerns.

Q1. How can the maximum score on the benchmark be evaluated?

As LeanLean is an open ended benchmark, it is impossible to know the maximum theoretical amount of compression that can be made on a repository. Benchmarks are frequently designed like this, e.g. code optimization benchmarks such as SWE-fficiency [1] and AlgoTune [2] have the same property, where a predetermined expert patch can be surpassed. Even with standard metrics, the best achievable score on many benchmarks is unclear due to noise. Even well-known benchmarks, like HLE and FrontierMath are affected by this issue [3].

However, we do have concrete evidence that models have currently not yet achieved the best possible performance: newer models evaluated such as Opus 5.5, GPT Sol 6.1, and Gemini 4 Argon show meaningful improvements on the compression scores on the benchmark. Opus 5.5 leads the benchmark with 64.3% score for $36.2 per instance, significantly better and cheaper than Opus 5.

[1] Jeffrey Jian Ma et al. “SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads?” arXiv:2511.06090v3, 2026

[2] Ori Press et al. “AlgoTune: Can Language Models Speed Up General-Purpose Numerical Programs?” Advances in Neural Information Processing Systems 38 (NeurIPS 2025), Datasets and Benchmarks Track.

[3] Epoch AI. “Benchmark Reviews Documentation.” Accessed November 13, 2026. Documentation

Q2. What insights into model capabilities can we obtain from compression scores?

Compression is a difficult, open-ended task like algorithmic optimization. In those settings a meaningful insight can be obtained from compression scores regardless of their absolute magnitude: by ranking the achieved scores of different methods. For this ranking, it is irrelevant to which degree the repository was previously already compressed and how much further compression is possible. Since we still observe that newer models and higher effort achieves higher scores, we believe that this ranking is meaningful and that the benchmark is difficult for current models.

Q3. Do compression capabilities transfer to theorem proving or proof refactoring?

Great question. We observe a general correlation between model scores on our benchmark and model scores on popular benchmarks such as coding. For example, we find a 0.98 Pearson correlation between our benchmark scores and the scores of 10 models (including more recently evaluated models) we have in common with TerminalBench 4.0 [1].

To observe a direct transfer from training on our tasks, we suggest that our tasks could be used in future work as training tasks for model posttraining. We expect that such effects can only be observed on models with non-negligible scores on our benchmark, which includes only very large and closed models.

[1] Terminal-Bench 4.0 leaderboard, archived 4 Nov. 2026.

Thank you to the authors for their response. I would like to maintain my rating of 3 while increasing my confidence from 2 to 3.

Reviewer_YEbgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces LeanLean, a benchmark for evaluating the ability of frontier LLMs to compress Lean code repositories while preserving their mathematical validity. The benchmark consists of 64 real-world Lean repositories extracted from the Palomar registry. Models are given 12 hours to compress each repository, with the goal of reducing the number of lines of code while maintaining equivalence to the original implementation, as verified by a Lean comparator. The authors argue that code compression can serve as a proxy for code quality, reflecting improvements in conciseness, simplification of tactics and arguments, and the use of better abstractions and reusable code. The benchmark compares several frontier models, including Claude 5 Opus, Gemini 3.8 Flash, and GPT-5.6 Luna. The best-performing model, Claude 5 Opus, achieves a compression score of 48%, while smaller models achieve scores below 25%. The authors also observe that compression scores tend to decrease with increasing repository size. The paper positions Lean code compression as a complementary benchmark to existing theorem-proving evaluations, arguing that it assesses models' understanding of Lean code and their ability to reason over long contexts.

Critical strengths
  • The paper introduces a relevant benchmark that complements existing theorem-proving benchmarks by incorporating code quality and compression as additional dimensions for assessing language models.
  • The paper provides a useful breakdown of how different models reduce the number of lines of Lean code. The examples visualized in Figure 8 are particularly helpful for understanding the types of modifications made by the models.
Critical weaknesses
  • While code compression is a relevant metric for assessing code quality, optimizing primarily for the number of lines of code risks reducing the evaluation to a software engineering task rather than assessing improvements in mathematical reasoning or formalization. Although compression and better abstractions can improve code readability and maintainability, shorter code is not necessarily easier to understand or extend. The benchmark would benefit from incorporating human comprehensibility, code extensibility and build time as additional evaluation criteria.
  • The evaluation lacks a human assessment of the compressed repositories, i.e. by the original library authors or mathematical domain experts. Without such an evaluation, it is difficult to determine whether the models primarily reduce code length or introduce meaningful abstractions that improve the structure, readability and reusability of the repositories.
  • Some compression strategies, such as replacing explicit proofs with automation tactics like aesop, grind or simp_all, as illustrated in Appendix C.3, may reduce code length at the expense of readability and compilation time. The benchmark scoring should include compilation performance in its evaluation.
Questions for authors
  • Why is unused code removed during preprocessing? Identifying and removing unused definitions or code snippets could itself be a relevant challenge for evaluating the models understanding of larger Lean repositories.
  • What does the asterisk “*” in Figure 2 indicate? Its meaning is not explained in the figure caption or a footnote.
  • Why do compression scores decrease with increasing repository size? This seems somewhat counterintuitive, as larger repositories might offer more opportunities for identifying shared abstractions and eliminating redundant code. Could the lower compression scores be explained by larger repositories being already optimized by their authors or communities?
  • How frequently do models produce compressed repositories that fail to typecheck or are not mathematically equivalent to the original repositories?
3 comments

We thank the reviewer for their thorough review of the paper. We are happy to see that they find our benchmark relevant and the analysis of the compression origin and visualizations useful. In this rebuttal, we address their questions and concerns.

Q1. Can you clarify your choice of metric and why you think it is appropriate?

See Q2 in the general response.

Q2. Compression could produce less readable or computationally expensive formalizations. Is this a problem?

See Q1 in the general response.

Q3. Does a human review of the resulting compressions find interesting patterns?

See Q3 in the general response.

Q4. Why is unused code removed during preprocessing?

Removing dead code does not inherently require mathematical insights or very detailed knowledge of Lean - in fact our automated tool can achieve this removal deterministically using simple rules. When developing the benchmark, we noticed that the repositories in the Palomar registry often contained code irrelevant to the results. In initial development evaluations, we noticed that models would spend the first turns of their evaluation simply performing the same automatable removal that our tool provides. In our benchmark, we want to focus on the difficult, non-automatable parts of code compression.

Q5. Why do compression scores decrease with increasing repository size?

In absolute terms, all models remove more tokens for larger repositories. The compression scores go down with larger repositories since the score is relative to the total repository size. The table below illustrates this, and a visualization can be seen on Fig. 18 (Appendix D, p21).

We suspect that models further fail to apply the same degree of compression on such large repositories as on small code bases because they require much better handling of long context input and reasoning over very large codebases.

ModelCompactStandardLargeMassive
Opus 54.2k (68%)18k (66%)45k (39%)151k (21%)
GPT-5.6 Sol3.4k (55%)12k (41%)40k (34%)96k (15%)
Gemini 3.8 Flash1.8k (28%)9k (32%)20k (17%)60k (9%)
Muse Spark 1.31.5k (26%)6k (20%)13k (11%)37k (5%)
GPT-5.6 Luna1.5k (25%)5k (18%)10k (8%)13k (2%)

Q6. How often do the models generate outputs that fail to typecheck in Lean or fail the equivalence test?

As outlined in Table 8 (Appendix D, p22), most models generate valid, typechecking and equivalent outputs, with the main exception being Leanstral which produces 11 submissions whose build fails. Gemini produces two outputs that do not build, Sol two, Muse four, Luna one, and Opus none.

ModelTimeoutsBuild failedVerify failedEquivalence failureVerified
Opus 5000064
GPT-5.6 Sol010162
Gemini 3.8 Flash830061
Muse Spark 1.3030061
GPT-5.6 Luna010063
Leanstral 1.50110053

Q7. What does the asterisk in Fig. 2 indicate?

The asterisk was a leftover from a stale remark on the model configuration. This has been resolved since, and the asterisk is thus removed.

Thank you for addressing my comments. I believe that lean code compression is an interesting and very relevant topic. However, the manuscript does not discuss and reflect on the potential issues of automated compression. I believe the evidence included in the general response should be expanded in a revised study. In the current state, my evaluation remains unchanged.

We thank the reviewer for the follow up. We agree that potential issues of automated compression should be discussed, and we have revised the “Approximate metric” paragraph in §5 with this in mind, highlighting that Opus 5 increases the heartbeat count on average. As suggested, we also added an author review in Appendix E.1, based on the first qualitative example in the general response. The second expert review is not part of the benchmark and is therefore out of scope for the paper.

Reviewer_t7TROpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work builds LeanLean, a benchmark for repository-scale proof compression in Lean. It has 64 real-world repositories spanning diverse math domains, with up to 1.7 M Lean tokens. Agents are given 12 hours to minimize repository size while preserving designated theorems, verified through Lean Comparator.

They develop a dependency-aware preprocessing pipeline to remove unnecessary declarations and evaluate six coding agents on the full benchmark. Additional experiments check performance across repository sizes, computational budgets, prompting strategies, and compression mechanisms. The main finding is that removing previously necessary declarations through proof restructuring accounts for most compression, though syntactic optimizations also help.

Critical strengths

Unlike existing benchmarks focused on individual theorems, LeanLean evaluates repository-wide optimization involving cross-file dependencies and reusable abstractions. The diverse collection of 64 repositories, dependency-aware preprocessing, and automated theorem verification make this a valuable evaluation setting.

The analyses in addition to the full evaluations are valuable. For example, the finding that substantial improvements arise from changing proof dependencies rather than merely shortening individual proofs is inspiring.

Critical weaknesses

Maximizing the benchmark score could produce less readable or computationally expensive formalizations. So the relationship between the reported scores and the intended capabilities of mathematical reasoning, abstraction, and proof simplification remains insufficiently established. Evaluating compression under alternative metrics would strengthen the evaluation validity.

The models use different native agent harnesses, reasoning configurations, and effective computational budgets. Although realistic, these differences confound model-level comparisons. The configuration error affecting Muse Spark is particularly illustrative: correcting it substantially improves performance and changes its relative ranking on the evaluated subset. Repeated-run variability, for example, could be reported to better understand the stability. Some simple comparisons, such as with simple automated compression baselines, could also be valuable to include. Generally, more controls can help determine how much of the measured improvement requires sophisticated repository-level reasoning, or how stable the rankings are across execution configurations.

Questions for authors

Clarifications / additional analysis that resolve some of the main concerns above would be helpful. For example, some questions worth answering might be: How would the model rankings change if syntactic compression through macros, abbreviations, and encoded strings were excluded or evaluated using an expanded representation? How sensitive are the main rankings to agent harnesses, model configurations, and stochastic variation? Could there be any uncertainty estimates or repeated evaluations on a representative subset, together with a simple automated compression baseline?

5 comments

We thank the reviewer for their thorough review of the paper. We appreciate that they find the benchmark valuable due to the size and diversity of the curated repositories, and find our analysis of the results enlightening. Below, we address their questions and concerns.

Q1. Are capabilities at mathematical reasoning, abstraction, and proof simplification correlated to model scores in your benchmark?

Yes. Mathematical reasoning, abstraction and simplification are definitely required to achieve a high score on our benchmark. Beyond that, models need capabilities at low-level code optimization. However, we observe empirically in Fig. 7 that low-level code optimizations have a marginal effect on achieved scores and most improvements stem from high-level mathematical work, such as proof restructuring. We provide more details and reasoning behind our score design and additional insights obtained from manual output analysis in the answers to Q1 and Q3 in the general response.

Q2. Have you considered alternative metrics?

Yes, but they suffer from disadvantages as outlined in Section 5 (p9) of the revised manuscript:

  • Robustness: Many metrics, such as build time and readability, are nondeterministic or subjective.
  • Reward hacking: Patching metrics to avoid syntax optimization or other behavior can introduce opportunities for subtle reward hacking. Lean tokens are transparent by design.
  • Compute cost: Metrics such as heartbeats are expensive to compute for large repositories.

A concern about the metric could be that models achieve proof shortening merely through increased automation, which would increase build time. We have computed heartbeats for our compressed repositories (see Table 7, p22). Many optimizations result in decreased heartbeats, showing that proof rewrites are not merely applying additional automations. We acknowledge that a reduction of heartbeats is a conceivable alternative metric target for future work, but highlight that optimizing for heartbeats is less connected to a natural desire for concise proofs, and even frequently rewards the opposite.

Q3. Would the model rankings change if syntactic compression were evaluated using an expanded representation?

No, the rankings would remain the same. As shown in Fig. 7, Opus 5 is the only model to apply significant syntactic optimization, with 7.2 percentage points out of its 48.3% compression stemming from syntactic optimizations, and it leads the rankings even when excluding those entirely. Our paper’s definition of syntax optimization matches the reviewers suggestion of expanding macros, notation, and definitions, additionally accounting for known syntax shortcuts.

Q4. Can the authors assess how sensitive the rankings are to stochastic noise?

Yes, we measured variance and found that our rankings are robust. We ran our models four times on a 12 repository subset and created 95% confidence intervals on the full benchmark using residual bootstrapping. The results are now shown in Section 4.2.5 and Appendix D. The full 64 repository benchmark preserves the ranking of all five models 99.8% of the time.

Q5. Does the fact that the Muse rerun changed the score mean that there is large variance in the results?

No. The only change we made to the initial rerun was changing tiers to a cheaper tier. The model providers' indication that these tiers are equivalent, yet among several reruns, we observe a statistically significant difference between the tiers, with the cheaper tier achieving the observed lower scores. As the tier seems to dramatically affect the scores of Muse in the benchmark, we reran all experiments concerning Muse with the same tier, constituting the results presented here and in the revised manuscript.

Q6. Are the evaluations a fair comparison between underlying models?

Yes. We run the models using the settings that model providers recommend. The ideal settings can vary significantly per model, and providers usually have the best intuition about the ideal conditions for their own model. From a practical perspective, this configuration matches typical user experience, which also follows these settings.

Q7: Are the main rankings sensitive to the agent harness?

No. We investigate model sensitivity by running five models on a subset of 12 problems using the PiCode harness. This analysis is outlined in Appendix D of the revised paper (p21). Running the models through PiCode maintained the model rankings.

Q8. How does reasoning effort affect the benchmark scores?

Great question. We have run the benchmark with various reasoning efforts for GPT Sol 6.1 and Opus 5.5. The scores follow a log-linear trend with the cost, showing no disproportional trends. The table below summarizes these results.

Sol 6.1Opus 5.5
EffortScoreCostScoreCost
low25%$334%$5
medium32%$360%$24
high39%$564%$36
xhigh48%$11——
max57%$18——

Q9. Can you compare against a simple, automated compression baseline?

Yes. Using the linter.unusedTactic as an automated baseline as done in [1] for a 12 repository subsample of our benchmark found 8 unused tactics in total and saved 86 tokens in total. For comparison, the evaluated models remove 916 - 426K tokens from the same subset.

The dead code removal performed during preprocessing could also have been included as a baseline. However, the focus of our benchmark is to investigate how much models can compress repositories beyond automated methods, since automated methods for Lean compression are highly limited.

[1] Alex Gu et al. “ProofOptimizer: Training Language Models to Simplify Proofs without Human Demonstrations.” International Conference on Learning Representations (ICLR 2026).

Q10. Does a human review of the resulting compressions reveal interesting patterns?

See Q3 in the general response.

Thank you for the detailed rebuttal and substantial additional experiments. I appreciate the authors' efforts to address my concerns.

The repeated-run analysis, harness sensitivity experiments, and automated baseline comparison are helpful. The qualitative examples also provide more convincing evidence that the benchmark captures meaningful mathematical reasoning and abstraction. I love seeing that model rankings remain stable after excluding syntactic optimizations and under the alternative harness.

My main remaining concern is evaluation validity beyond token compression. In particular, Table 7 shows that Opus 5, despite achieving the highest compression score, increases heartbeat counts by 16.8% on average. This illustrates that token compression can diverge substantially from computational efficiency. I agree that token count is a reasonable, transparent primary metric, but would need more cautious claims about its relationship to broader proof quality.

There are also some remaining limitations: the variance estimates extrapolate from a 12-repository subset, harness choice produces measurable score differences for some models, and the automated baseline is relatively narrow. Nevertheless, these limitations are now much better characterized. Overall, the rebuttal resolves most of my original concerns. Given the benchmark's novelty and the strengthened experimental evidence, I am inclined to increase my overall rating from 2 to 3. Thank you again to the authors!

Comments on the paper

1 comment

We thank the reviewers for the feedback. We are happy the reviewers appreciate the relevance of this benchmark (YEbg19), the effort put into curating it (t7TR, ad7J) and our extensive analysis of the results (t7TR, YEbg19, ad7J).

Here, we address common reviewer questions and summarize the main changes made to the paper. Changes are marked in blue.

  • Model reruns (Table 1, p5): We ran a clean re-run of Muse and Leanstral on the whole benchmark. Both reruns had minimal effect on the model scores.
  • Cost accounting fixes (Table 1 on p5, Table 2 on p6, App. B) In our run with GPT-5.6, input tokens were not always cached accurately, which led to inflated cost measurements. This has been fixed and has no effect on model rankings.
  • Benchmark variance estimate (§4.2.5 and App. D): We run models on a subset of our benchmark and use residual bootstrapping to create variance estimates for the full benchmark. Our benchmark separates all five models 99.8% of the time.
  • Sensitivity of harness choice (App. D): We run the models through the PiCode harness, which has a marginal effect on model scores. The ranking of the models is preserved.
  • Lake Comparator false positive (App. D): On one instance, GPT 5.6 Sol exploited the definition_names mechanism in the Lake Comparator. We have patched the exploitation and scored the instance as 0.
  • Qualitative analysis (App. E): We added more qualitative explanations to our case studies.

Q1. Compression could produce less readable or computationally expensive formalizations. Do these factors affect results significantly?

No. These changes account for only a small portion of the model edits (see Fig. 7, p7), and disregarding them would not alter the ranking between the models. Most of the compression comes from restructuring the proofs by deleting previously used declarations and adding new ones, i.e., by revising the proof structure itself. To maximize the Lean token metric, meaningful simplifications are more valuable than minor syntactical edits.

Notably, we are not the first to propose compression as a capability central to artificial intelligence. The popular Hutter Prize [1] applies a similar idea to natural language. They identify compression and utility maximization as key concepts in a formal definition of intelligence.

[1] Marcus Hutter. Human Knowledge Compression Contest: Frequently Asked Questions & Answers.

Q2. Is your metric appropriate to measure performance on the task?

Yes, we believe it is the best metric for the task, as it is simple, robust, and deterministic. By keeping our metric simple, we ensure interpretability and by allowing all changes that improve it, we avoid reward hacking. Reviewer YEbg suggested alternative metrics like “readability”, “reusability” or “maintainability”. As of current research, no robust and deterministic methods are known to compute these and they would likely involve significant complexity. Alternative compression metrics could include heuristics to penalize some forms of syntactic optimizations, but such heuristics are typically not robust, especially against RL-trained AI agents. Meanwhile, honest attempts that don’t exploit flaws would be disadvantaged.

Q3. Does a human review of the resulting compressions find interesting patterns?

Yes, two doctoral-level researchers in the field of mathematics and computer science reviewed the compressions and found interesting abstractions. To maintain anonymity, we will not mention the specific repositories here, but highlight their key findings.

We collaborated with an author of a repository in the Massive category.

  • The repository contains theorems of 13 properties. Sol 6.1 was the only model to realize that 10 of the properties can be reconstructed from a set of 3 minimal properties, and used this to significantly reduce the remaining 10 proofs. It achieves a significant compression score of 23.5%.
  • Other models score less than 10% by mainly applying syntactic optimization and increased automation. Our compression origin metrics agree with the reviewer: 3.9% out of Opus 5’s 7.2% score is attributed to syntax optimization and automation.

Additionally, we evaluated selected models on a repository based on a paper from a mathematician. The reviewer highlighted two discoveries:

  • Mathematical insight: An intermediate result was stronger than necessary to prove the main result. Both Opus 5.5 and Sol 6.1 realized this and found a shorter proof for the weaker intermediate result.
  • Better machinery: Opus 5.5 replaces a proof that is argued by symmetry in the paper but was implemented using separate proofs in the original Lean proof by a clever abstraction to prove both cases together.

As model capabilities increase over time, we can expect an increasing amount of interesting concepts to arise, especially when compressing Lean proofs of larger size covering even more results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.