acceptodds
Under review as a conference paper at ICLR 2027

Learning to Discover Interesting Mathematics

Abstract

Recently, Large Language Models (LLMs) have been increasingly able to solve advanced mathematical problems, including many that have been open for decades. This opens the door to expansion of mathematical knowledge at unprecedented scale. Yet, while LLMs may be able to conjecture and prove more and more theorems, it remains open whether this new mathematical knowledge is interesting or useful. We define intrinsic interestingness of a theorem as the ratio between the length of its proof and the length of its statement. We show that this correlates strongly with an extrinsic measure of the downstream utility of a theorem. We identify the difficulty of a proof conditioned on a set of premises as a useful primitive for computing these metrics, and train a 27B model that predicts proof difficulty more accurately than frontier general-purpose models. Optimizing for our metric creates a model which is capable of producing more interesting theorems, while also reducing substantial or full overlap with Mathlib from 91.9% to 30.6%, showcasing the creation of more out-of-distribution math. We show that our system can generate candidate theorems, select the most interesting among them, and iteratively build on a self-expanding mathematical library. These metrics provide a practical and quantifiable signal for ranking conjectures and guiding proof search within formal mathematical libraries. Our framework provides a path towards self-expanding, machine-verified mathematical libraries that can choose worthwhile statements without relying on human-supplied targets.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Lean 4, Automated Discovery, Formal Mathematics, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Jy3xOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a computable, library-grounded measure of mathematical interestingness to guide automated discovery in Lean 4. The core primitive is the conditional proof difficulty V(T | P), the number of proof lines needed to derive a theorem T from premises P. The authors learn it with a 27B model trained via GRPO on Mathlib data enriched by “premise expansion,” using rewards built from grounding, monotonicity, and Bellman-style composition axioms, and the estimator beats GPT-5.5 and Claude Opus 4.6 in MAE and Spearman ρ. Interestingness is defined as proof difficulty divided by premise-relative description length, and it correlates with a downstream-utility measure on Mathlib (ρ = 0.756). Used as an RL reward, it raises the verified interestingness of generated conjectures from 1.76 to 7.58 and reduces substantial-or-full Mathlib containment from 91.9% to 30.6%. Used as a promotion rule in an iterative discovery loop, it beats no-pruning, random, and proof-length baselines under blinded LLM judging.

Critical strengths
  • A simple, computable, well-motivated metric. Measuring interestingness as proof difficulty normalized by description length is intuitive, echoes compression and Pólya-style views of elegance, and needs no human labels. The normalization is well argued: conjoining unrelated theorems grows difficulty linearly but leaves interestingness O(1). The resulting ordering of Mathlib, with trivial identities at the bottom and small cases of FLT at the top, is a convincing sanity check.
  • Conditional proof difficulty as a reusable primitive. Framing V(T | P) as a premise-conditioned value function with explicit axioms is clean, and turning those axioms directly into GRPO reward terms is a nice design choice. Premise expansion is a practical answer to Mathlib’s atomized structure, where the median proof is 3 lines. The resulting estimator is clearly better calibrated than frontier models (MAE 20.2 vs. 31.0/34.3).
  • Connecting intrinsic and extrinsic value. Relating a quantity available when a theorem is proposed (interestingness) to one observable only afterward (how much a theorem compresses downstream proofs) is the right question for autonomous discovery. It gives a principled reason to use the metric as a proposal-time surrogate.
  • End-to-end demonstration with an informative ablation. The metric works both as a training reward and as an inference-time selection rule. Including proof-length pruning as a baseline is especially valuable. Interestingness pruning wins the blind ranking 60% vs. 19%, which suggests the gain comes from the ratio itself rather than from favoring long proofs.
Critical weaknesses
  • The discovered mathematics remains shallow. Promoted theorems in the discovery loop average 7.4 proof lines (Table 1), and verified candidates average 3.7–5.5. Many showcased statements look like artificial packaging rather than discoveries: 1) biconditionals with an unrelated side condition (n > 0 ↔ … Chebyshev nodes); 2) a conjunction of two unrelated facts (modular inversion); 3) near-tautologies, such as g⁻¹ acting on g·z restoring Im z. With no evaluation by human mathematicians, it is unclear whether gains in the metric reflect gains in genuine interestingness.
  • Evaluation relies on one LLM family and may favor the trained model. Claude Opus 4.6 serves as baseline conjecturer, prover, repair agent, semantic filter, containment judge, and quality, diversity, and ranking judge. The containment judge has no retrieval over Mathlib, so the 91.9% → 30.6% drop may partly reflect unnatural phrasings that nobody would add to Mathlib, and the novelty is not valuable. The verified cohort is also built by sampling until 20 proofs exist per area, with a repair pass allowed, yet per-model prove, repair, and validity rates are not reported.
Questions for authors
  • Novelty and scope of the definition. Footnote 2 says I0 corresponds to the “interest” of Aksenov et al. (2026); what is new beyond the conditional version and the learned estimator? And since L(T|P) charges for definitional vocabulary, would the metric penalize discoveries that introduce new definitions?
1 comment
AuthorsOpenReview ↗

Response to Reviewer Jy3x

Thank you for the thoughtful review and constructive suggestions, which have helped us improve the paper. We would like to acknowledge the mentioned weaknesses and questions here:

With no evaluation by human mathematicians, it is unclear whether gains in the metric reflect gains in genuine interestingness.

We agree that whether gains in our metric translate into gains that mathematicians would find meaningful remains open. On competition math problems, prior work found that LLM ratings often broadly agreed with human ratings but missed much of the variation in which problems people found interesting and why (Mishra et al., 2025, A Matter of Interest).

A human study of generated Lean theorems would address several questions. What makes a theorem interesting to mathematicians, and how do their judgments vary across fields and levels of expertise? How do those judgments change when mathematicians see the surrounding theory and downstream uses of a theorem? How well do our structural score and blinded LLM rankings track expert judgments? Together, these questions would form the basis of a full follow-up paper.

Evaluation relies on one LLM family and may favor the trained model.

Appendix E.4 adds GPT-5.5 as a second judge. Across the same 100 policy-blinded comparisons, interestingness pruning ranked first in 57% with GPT-5.5 and 60% with Claude, with mean ranks of 1.86 and 1.80, respectively (Table 8). In this experiment, Section 3.4 holds the Claude conjecturer, semantic filter, and proving agent fixed across promotion rules. In the separate conjecturing experiment, we compute interestingness from verified proofs, and the trained model’s mean remains 5.03× Claude’s on statements proved as written (Appendix E.3, Table 7). The conjecturer comparison remains conditional on the statements the common proving pipeline verifies.

The containment judge has no retrieval over Mathlib

The original Figure 5 protocol supplies the statement, verified proof, and referenced Mathlib declarations. In the additional evaluation in Appendix E.4, GPT-5.5 had access to the top 20 BM25-retrieved Mathlib declarations for each of the 480 verified statements from the base, trained, and Claude conjecturers. It judged 31% of the trained model’s statements to be rephrasings of existing declarations, compared with 84% for the base model and 68% for Claude (Table 9).

The verified cohort is also built by sampling until 20 proofs exist per area, with a repair pass allowed, yet per-model prove, repair, and validity rates are not reported.

Section 3.3 and Appendix E.2 now report the per-model audit. The base, trained, and Claude models needed 240, 880, and 280 sampled prompts, respectively, to fill their balanced cohorts. Their syntactic filter pass rates were 99%, 55%, and 85%. Their exact proof rates were 58.0%, 7.4%, and 70.7%, and repair succeeded on 59.0%, 35.5%, and 44.3% of exact proof failures. Overall validity under the pipeline, including repair, was 82.8%, 40.2%, and 83.7%. Of the 160 retained statements per model, 48 base, 131 trained, and 24 Claude statements were repaired (Table 5). The trained model therefore has a substantial verification disadvantage, which we will report clearly in the main text of the paper. However, among statements proved as written, its mean interestingness still exceeds the base model’s by 5.13× and Claude’s by 5.03× (Table 7).

what is new beyond the conditional version and the learned estimator?

The introduction now cites Aksenov et al. (2026) more explicitly for the underlying theoretical compression principle. Our work introduces a conditional variant of this principle and a learned difficulty estimator, then develops these ideas into an end-to-end discovery framework. Specifically, we train a conjecturer with the resulting signal, and run iterative theorem-proving experiments in which verified theorems are selected for further exploration. To our knowledge, all of these contributions are novel and not mentioned in prior works.

would the metric penalize discoveries that introduce new definitions?

Yes. Charging for new vocabulary can disadvantage a statement whose new definition becomes useful only later. Section 4 identifies definition invention as future work, since the current setup doesn’t allow models to create new definitions alongside new theorems. New ideas as to how to measure compression may be needed in future work to better handle definition generation.

Reviewer_yKuGOpenReview ↗
Rating
1
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This work proposes a quantitative notion of mathematical interestingness to facilitate LLM-driven discovery of novel mathematical statements and/or proofs. The authors first define a notion of conditional proof difficulty V(T∣P)V(T\mid P), the number of proof lines needed to derive a theorem TT from premises PP, and train a model to predict difficulty using GRPO.They used this notion to define "Interestingness" as the difficulty relative to the combined description-statement length. The evaluation shows this is correlated to a notion of theorem utility, and increases the Mathlib "novelty" when trained for.

Critical strengths
  1. The presentation and explanation of the paper seem quite strong to me. The authors have clearly described their design choices, and the motivation behind most if not every one of them. This clearly presents the thought process behind the methodology's design and makes the paper easy to follow. Despite their irregular nature, I quite enjoyed the philosophical-style citations to further articulate the authors' points.

  2. The problem the paper investigates is clearly important, and the authors try to address it in a pretty explainable fashion. If generalisable beyond simple Mathlib, the methodology can be used as a building block for creating new conjectures, or library building.

Critical weaknesses

Overall, while I believe that the problem at hand is important and the approach is sensible conditioned on the authors' beliefs, the instantiations of difficulty and interestingness, along the accompanied evaluation metrics and observations seem to be misaligned by definition or biased without external support. I have attempted to summarise my points as briefly below. I am happy to discuss with the authors during the rebuttal period.

  1. The Difficulty Measurement: I have several notes on the definition of difficulty VV:

    • (Major) The idealistic view of V (axioms A2,3) are motivated ofr the shortest proof, however the labels are unlikely to fit this, as they were human-written, and were designed for library usage, which usually requires better readability, potentially at the cost of verbosity/length. For example, some concurrent work measures that many Lean repositories can be compressed [1] significantly, implying that taking unfiltered data as ground truth might violate the expected requirements. As VV is primarily used as a component of II, it would be informative to observe if any shortening during the training of VV impacts any of the following conclusions.
    • (Medium) Measuring difficulty as lines can be relatively fragile, as they can depend on tactic usage (in actual reporsitories, maybe less likely for Mathlib), and different syntax styles can produce different numbers of lines with equivalent Lean code.
  2. The Interestingness Metric: While the definition of the interestingness metric makes intuitive sense, I believe its definition is open to being hacked, and the evaluation methodology leaves some gaps:

    • (Major, if confirmed) When evaluating interestigness, the utility correlation may be built into its definition. Since U0=∣D(T)∣⋅VU_0 = |D(T)|\cdot V, I0=100 V/LI_0 = 100\,V/L, log⁡U0=log⁡∣D∣+log⁡V\log U_0 = \log|D| + \log V and log⁡I0=log⁡V−log⁡L+const\log I_0 = \log V - \log L + \text{const}, there is a risk of log⁡V\log V dominating in terms of variance, which may cause high correlations. In particular, since VV may span several orders of magnitude, a high ρ\rho can potentially be expected. A principled analysis of where the variance comes from can address this concern.

    • (Major) The metric can be structurally hacked in multiple ways:

      • Converse appending. Adding the converse of a statement is very cheap statement wise (  ⟺  \iff instead of ⇒\Rightarrow), but expensive proof-wise, increasing interestingess significantly, without necessarily being a lot more interesting.
      • Conjunction. The O(1)O(1) argument for T^n\hat T_n does not transfer generally. For example, if a statement requires some additional definitions/assumptions that carry across multiple statements generally, combining 2 of them may produce a statement that is shorter than the combination of the 2, while the proof might be closer to the sum of the proof lengths.
      • Statement compression. The conjecturer may learn to minimize S(T)S(T), through shortening variable names, type templates and notation.
      • Proof padding. Observed proof length is unbounded above and can be padded unlimitedly by fluff.
    • (Medium) Interestingness is inherently subjective, and a fully objective measurement might be poorly aligned on average with human expectation. Not having a small-scale human study leaves this question open in a sense.

    • (Medium?) The Opus validation for Mathlib novelty may be somewhat biased, as this might be a mismatch to the style of Mathlib (i.e. the model deciding "this doesnt read like anything that belongs in a generalist library"), rather than genuine novelty.

  3. (Medium) Transferrability to natural language -> Ideally, the interestingness of a theorem should also carry over to its natural language equivalent. However, it remains unclear whether this definition of interestingness and difficulty can transfer to a natural language proof, which, by definition should have carry similar structure and assumptions.

I am happy to further discuss these points, especially if the authors have objective evidence against these points. If addressed, I would be happy to raise my score (currently I stand around a score of 1.5 rather than a hard 1).

References

[1] Rognvaldsson, K., Mündler-Sasahara, N., Dekoninck, J., & Vechev, M.. (2026). LeanLean: Benchmarking Repository-Scale Lean Proof Compression. (https://leanleanbench.com/)

Questions for authors
  1. Most aspects of the paper seem reasonably clear, so can you please address the comments from the Weaknesses section?
  2. Minor typo in L174: VV should be VθV_\theta, am I correct?
6 comments
AuthorsOpenReview ↗

Response to Reviewer yKuG

Thank you for the thoughtful review and constructive suggestions, which have helped us improve the paper. We would like to acknowledge the mentioned weaknesses and questions here:

the labels are unlikely to fit this, as they were human-written, and were designed for library usage, which usually requires better readability, potentially at the cost of verbosity/length.

Section 2 distinguishes the ideal shortest-proof setting of A2–A3 from the observed proof lengths used as supervision. Mathlib proofs are highly atomized, with a median of three lines and a median premise-use count of one. Premise expansion varies the available premises and adds the derivation cost of intermediate lemmas when they are removed. To test sensitivity to proof compression, Appendix E.3 (Table 7) uses GPT-5.5 to shorten proofs, verifies the candidates in Lean, and keeps the shortest verified proof for each item. The trained model’s mean interestingness remains 3.09× the base model’s and 2.74× Claude’s. This tests sensitivity of the downstream comparison to shorter evaluation proofs. Training the difficulty predictor on compressed labels would address the separate question of how compression affects learning.

Measuring difficulty as lines can be relatively fragile, as they can depend on tactic usage (in actual reporsitories, maybe less likely for Mathlib), and different syntax styles can produce different numbers of lines with equivalent Lean code.

Appendix B.1 specifies that Mathlib extraction counts nonblank physical lines in the proof span. Appendix E.3 (Table 7) checks alternatives to line count in the conjecturer comparison. The trained-to-base mean interestingness ratio is 3.75× with proof characters, 4.92× with proof tokens, and 4.73× with elaborated proof-term size. The corresponding ratios against Claude are 3.31×, 4.17×, and 4.82×. Section 4 also discusses the tradeoff between shorter scripts and longer runtime.

When evaluating interestigness, the utility correlation may be built into its definition.

Section 3.2 now acknowledges that I₀ = 100V₀/L₀ and U₀ = dV₀ share the expanded proof length V₀ and points to the new stratified analysis in Appendix E.1. Figure 8 reports Spearman correlations between I₀ and U₀ in 14 multiplicative bands of V₀. The estimate is positive in 13 of the 14 bands. Comparing theorems within bands limits variation in proof length, and the remaining variation can still contribute to the association.

The metric can be structurally hacked in multiple ways:

  • Converse appending. Adding the converse of a statement is very cheap statement wise […] but expensive proof-wise […].
  • Conjunction. […] combining 2 of them may produce a statement that is shorter than the combination of the 2, while the proof might be closer to the sum of the proof lengths.
  • Statement compression. […] shortening variable names, type templates and notation.
  • Proof padding. Observed proof length is unbounded above and can be padded unlimitedly by fluff.

The trained conjecturer outputs only a standalone Lean proposition (§3.3 and Appendix C.1). It can choose how to state the theorem, but it cannot introduce or rewrite library definitions to shorten its statement. A separate Claude prover supplies the proofs used for evaluation. We train the conjecturer alone not the prover, so it cannot pad those proofs, and the prover receives no reward for making them longer.

The conjecturer can still use shorter variable names or package claims into conjunctions and biconditionals. Appendix E.3 (Table 7) tests whether these choices explain the gains. Canonicalizing variable names leaves mean verified interestingness as 4.23× the base model’s and 4.03× Claude’s.

To check whether verbosity in the prover’s output explains the gains, we recompute the score using elaborated proof-term size and, separately, the number of distinct library lemmas used. We also ask GPT-5.5 to shorten the proofs and keep the shortest verified proof for each item. After that check, the trained model’s mean verified interestingness remains 3.09× the base model’s and 2.74× Claude’s (Appendix E.3, Table 7).

AuthorsOpenReview ↗

Response to Reviewer yKuG (cont.)

Not having a small-scale human study leaves this question open in a sense.

We agree that alignment with human judgments remains an open question in our setting. Prior work has studied this question for competition math problems and found that LLM ratings often broadly agree with human ratings while missing much of the variation in what people find interesting (Mishra et al., 2025, A Matter of Interest).

A human study of generated Lean theorems would address several questions. What makes a theorem interesting to mathematicians, and how do their judgments vary across fields and levels of expertise? How do those judgments change when mathematicians see the surrounding theory and downstream uses of a theorem? How well do our structural score and blinded LLM rankings track expert judgments? Together, these questions would form the basis of a full follow-up paper.

this might be a mismatch to the style of Mathlib

Unusual phrasing could affect judged Mathlib containment. In the original evaluation, Claude saw the verified proof and referenced Mathlib declarations alongside the statement. The rubric counted trivial reformulations and changes in notation as fully contained when the judge could identify a concrete matching declaration (Appendix C.2.2–C.2.3).

In the additional evaluation, GPT-5.5 received the retrieved Mathlib declarations for each of the verified statements. Using a rephrasing rubric, it judged 31% of the trained model’s statements to be rephrasings of existing declarations, compared with 84% for the base model and 68% for Claude (Appendix E.4, Table 9). The trained model thus had the lowest judged overlap under both evaluations.

it remains unclear whether this definition of interestingness and difficulty can transfer to a natural language proof

The intuition of a concise statement requiring substantial reasoning can apply to informal mathematics. Applying the metric there would require specifying how to represent implicit premises, omitted proof steps, and choices of notation and language. Answering these questions, although interesting, remain out of scope for this paper.

Minor typo in L174:

Thank you for pointing this out. We have corrected the typo at line 174 of the reviewed version.

Reviewer_yKuGOpenReview ↗

Response to Author Rebuttal

I thank the authors for their clarification. Below I first list all points I consider resolved and will then ask for further clarifications on the remaining points.

Resolved:

  • Alternative of metrics at evaluation time: The authors provided modified evaluations on when specific metric definitions were substituted into the GT interestingness evaluation, which I appreciate.
  • Metric Hacking: I thank the reviewers for the clarifications, and agree on the points regarding proof padding.
  • Lack of human study: While I explicitly did say that I would prefer if a small-scale human annotation was provided, I understand that this preference could conflict with automability and scalability. I thus consider this partially resolved, as I have some troubles with the interpretation of the cited sources.
  • Typo

Remaining Concerns:

On the length concern

While the authors argue for evaluating on shortened proofs, which does in fact lower the interestingness, though there is still a 3x advantage over the base model, that still leaves my original concern for training on longer proofs. Furthermore, when looking at the referenced Table 7, its caption mentions that " For proof golfing, 274 of 368 golfed proofs compiled.". Can the authors explain the implications of this, including on the interestingness ratio of the non-compiled proofs, what was counted in the metric, and why they believe the results are comparable given some proofs do not compile? Also where does the 368 figure come from, I do not believe seeing it referenced in the paper?

On measuring difficulty through no. lines

I appreciate the effort, but my primary concern with no.lines is that the downstream training of the conjecturer is also impacted by this design choice, and the artifacts of this paradigm are unclear. Can you give examples of conjectures that are considered difficult or easy, with outputs of all three models on the same premises, with proofs and line counts?

On the utility correlation

While the authors state that "the estimate is positive in 13 of the 14 bands" with respect to the correlation, these values now range between approx. 0.05 and 0.2, much lower than the reported >0.7, meaning the confound I pointed out in the review is significant and heavily impacts the uncerlying conclusion. I do appreciate the results, but this mostly confirms my concerns rather than arguing against them.

On the Metric Hacking

The response and Appendix E.3 both do not really tell me how all these modifications were achieved on a technical level, i.e. what it means to standaradize variable names, how many theorems are left that do not have top-level conjunctions or equivalences, what "top-level" means, and whether these procedures were also applied to the base/Claude generations. Furthermore, even if all unclarities were resolved, the reply shows the patterns are not the whole reason for the gain, and does not tell me that the metric is safe from them. For me, a large part of the contribution of the work is the metrics defined, and if not resolved, it mostly leaves the conjecturer training pipeline, which is likely minor given that it is not clear this pipeline can be broadly applied and scaled, given its evaluation and training are centered on Mathlib.

On the Human Study

While such a study is likely to be time-prohibitive, the cited paper, as the authors note, found that LLMs do not reliably match what humans find interesting, which weakens the LLM judge results of Fig. 7.

On the LLM-as-a-judge style mismatch concern

Before I judge on this, I believe providing some examples of different styles of conjectures here (from all 3 evaluated models, on the same premises, as requested above), would be quite valuable.

Overall, a lot of the rebuttal focuses on the end conjecturer provided and its evaluation, and while helpful, I believe my core concerns with the metrics, which are a very significant part of the contribution, remain. For now I will keep my score, and hope to continue the discussion further.

Reviewer_XFKAOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a method for automating the generation of new mathematical theories using an interestingness metric based on estimated proof difficulty compared to theorem statement conciseness. For estimating proof difficulty, a model is fine-tuned and shown to outperform frontier models. Once the metric is established by comparison to a utility metric, experiments are run on training a conjecturer to output statements with a higher interestingness metric, as well as iterative theory creation using interestingness-based pruning, with LLM-judge evaluations.

Critical strengths
  1. The method of training a model to predict proof difficulty, which is then used as a proxy of proof length, in order to evaluate the interestingness metric of proof length / theorem conciseness, is itself an interesting methodology. The trained proof difficulty predictor looks good and the interestingness metrics on Mathlib look reasonable. The training of a conjecturer model on these interestingness metrics follows from the previous steps in the paper.
Critical weaknesses
  1. Interestingness in math is a difficult thing to measure and I appreciate the thought that went into this paper's exposition of the topic, but I still have notable concerns--especially considering how strongly some of the study's implications are stated.

    a) A major justification for the stated measure of interestingness is the correlation between interestingness and utility shown in Fig. 3 together with the Spearman correlation. I question how surprising that is considering that U0(T)U_0(T) and I0(T)I_0(T) share the factor V(T∣∅)V(T \mid \varnothing). Acknowledgement and some small tests for reconciliation of this fact would be helpful (moreso than the Thurston quote!).

    b) Sections 3.1-2 include examples of well-known Mathlib statements which give confidence in this interestingness metric. However 3.3 states that reduced overlap with Mathlib is more novel and therefore more interesting/indicative of future success in developing theories, without giving similar intuition for what has been generated. Not enough justification is given that this reduced overlap is actually expressing math that people care about. Some example generated Lean statements are given in C.3 and the informalization shows that they are interpretable math, but this is orthogonal to the Mathlib containment that is stressed in the main paper.

    c) In terms of presentation I am concerned that Aksenov et al. gets a small footnote in the main paper in comparison to more verbose reflections on philosophy of mathematics. Many of their ideas appear here and although much new has been added, it can appear from a quick skim that these interestingness ideas were first presented here. Perhaps citing them in the intro interestingness paragraph would help.

    d) Section 3.4 aims to demonstrate theory creation, however the LLM-judge based evaluation is difficult to interpret absent some example LLM judgements and example theories generated.

    e) As mentioned in the discussion, some of the Lean automation tactics may shorten proofs significantly at a runtime cost, thereby obscuring mathematical depth. Some additional small evaluation would be helpful to understand the impact this has on the interestingness metric.

Questions for authors
  1. Can GPT performance be added to the ground-truth interestingness and Mathlib overlap evals in Figs. 4 and 5? It's clear in general and even in Fig. 1 that GPT and Claude performance on advanced math are quite different, sometimes in unexpected ways.

  2. Section 3.4 seems most ripe for follow-up experiments, even before adding in definition writing. Are there any ideas that can be added to the discussion on this point?

  3. A major concern in mathematics is the outpouring of AI-generated results much faster than human mathematicians can parse through poorly-written proofs to determine which results are valid and how to interpret the proofs for human understanding. What methods for screening quality of automatically generated theories can be adopted to avoid this problem from snowballing even further? I ask this just because the current manuscript is using LLM-judges for the eval and I wonder how scalable this is and what the authors have thought about this question.

3 comments
AuthorsOpenReview ↗

Response to Reviewer XFKA

Thank you for the thoughtful review and constructive suggestions, which have helped us improve the paper. We would like to acknowledge the mentioned weaknesses and questions here:

A major justification for the stated measure of interestingness is the correlation between interestingness and utility shown in Fig. 3 together with the Spearman correlation.

Section 3.2 now acknowledges that I₀ = 100V₀/L₀ and U₀ = dV₀ share the expanded proof length V₀. To examine the association over narrower ranges of V₀, Figure 8 (Appendix E.1) reports the Spearman correlation between I₀ and U₀ within 14 multiplicative proof-length bands. The estimate is positive in 13 of the 14 bands. Stratification limits variation in V₀ within each band, and the remaining variation can still contribute to the association.

Not enough justification is given that this reduced overlap is actually expressing math that people care about.

Figure 5 measures judged overlap with Mathlib, but overlap is not our only evidence that the score identifies worthwhile mathematics. In the separate experiment (§3.4), we hold the conjecturer, semantic filter, and proving agent fixed, and vary only the promotion rule. Blinded judges therefore compare theories produced under the same generation and verification pipeline. Claude rated interestingness-based selection higher than proof-length selection on both quality (3.4/5 vs. 2.6/5) and diversity (2.1/5 vs. 1.3/5) (Table 4). GPT-5.5 independently ranked interestingness-based selection first in 57% of the same 100 blinded comparisons, compared with 60% for Claude (Appendix E.4, Table 8). Together, these results provide evidence beyond Mathlib overlap that the score can select statements judged more mathematically valuable than those selected by proof length alone. A systematic comparison with ratings from mathematicians remains valuable future work outside the scope of this paper.

Perhaps citing them in the intro interestingness paragraph would help.

We now cite Aksenov et al. (2026) in the introduction’s discussion of intrinsic interestingness. Our contribution develops this theoretical principle into a concrete framework in which it can be estimated, and then optimized. These aspects, along with the learned difficulty estimator and the conjecturer training are all novel contributions of this work.

some of the Lean automation tactics may shorten proofs significantly at a runtime cost, thereby obscuring mathematical depth.

Section 4 discusses how tactics such as aesop and grind can trade shorter Lean scripts for longer runtime. To test sensitivity to proof representation and shortening, we added Appendix E.3 (Table 7). The trained conjecturer’s mean interestingness remains 4.73× the base model’s and 4.82× Claude’s when using elaborated proof-term size. After GPT-5.5 attempts to shorten the proofs and we keep the shortest verified proof for each item, the ratios are 3.09× and 2.74×. These checks concern different forms of proof representation and compression. Future work may want to utilize related metrics, such as the Levin complexity, which account for both the description length as well as runtime.

Can GPT performance be added to the ground-truth interestingness and Mathlib overlap evals in Figs. 4 and 5?

Appendix E.5 adds GPT-5.5 as a conjecturer under the same filtering, deduplication, and Claude Code proving protocol. Its cohort contains 160 verified statements, 20 per area. Mean verified interestingness is 2.27 for GPT-5.5 and 7.58 for the trained model, compared with 1.76 for the base model and 1.84 for Claude (Table 11). GPT-5.5 verifies substantially more often in the reported audit, with a 92.5% exact proof rate and 98.7% overall validity (Table 10). Appendix E.4 also uses GPT-5.5 as a retrieval-assisted containment judge on the original data.

Section 3.4 seems most ripe for follow-up experiments, even before adding in definition writing. Are there any ideas that can be added to the discussion on this point?

Section 4 discusses updating the difficulty estimator online as the premise library grows and asks whether the new theorems help prove future results. A concrete follow-up would hold out independently specified targets and compare proving success with and without access to the generated lemmas. Longer runs could also track which promoted lemmas are reused across rounds, and try to develop methods that result in a maximally reusable codebase.

AuthorsOpenReview ↗

Response to Reviewer XFKA (cont.)

What methods for screening quality of automatically generated theories can be adopted to avoid this problem from snowballing even further?

We share the concern that AI could generate valid results faster than mathematicians can read and assess them. Lean can check correctness automatically, but deciding which results merit attention and making their proofs understandable remain separate challenges. This makes it urgent to develop quantitative measures of mathematical interestingness that can screen results at scale without relying on an LLM judges. Our work is a first step in that direction. In §3.4, the promotion rule scores candidate theorems using their description length and verified proof length, and seems to show early signs of being successful. We use blinded LLM judges only to evaluate that selection rule, as a proxy for the human evaluation we could not conduct in this study. A systematic comparison with mathematicians’ judgments remains an open area for future work.

I'm satisfied with the follow-ups and raised my score. Overall, I do think the paper in places overstates the applicability of the method and I do not think mathematical interestingness is at all close to solved as the title seems to imply, but the method is interesting enough and conference-worthy in my opinion. I also think the theory creation section in 3.4 would need to be expanded much more to make this line of study complete, but this could be enough material for a follow-up paper and the authors had reasonable ideas in response to my question.

A non-urgent suggestion for the authors that I forgot in my initial review: the convex hull of math terminology in the intro is in fashion now but I don't think it will be legible for readers of the paper in a few years. It would help the prose's longevity to properly explain this concept. Or even just saying unknown or new without referencing convex hull could be good enough.

I may post some other follow-up before the review period is over but nothing else urgent.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.