- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes a computable, library-grounded measure of mathematical interestingness to guide automated discovery in Lean 4. The core primitive is the conditional proof difficulty V(T | P), the number of proof lines needed to derive a theorem T from premises P. The authors learn it with a 27B model trained via GRPO on Mathlib data enriched by “premise expansion,” using rewards built from grounding, monotonicity, and Bellman-style composition axioms, and the estimator beats GPT-5.5 and Claude Opus 4.6 in MAE and Spearman ρ. Interestingness is defined as proof difficulty divided by premise-relative description length, and it correlates with a downstream-utility measure on Mathlib (ρ = 0.756). Used as an RL reward, it raises the verified interestingness of generated conjectures from 1.76 to 7.58 and reduces substantial-or-full Mathlib containment from 91.9% to 30.6%. Used as a promotion rule in an iterative discovery loop, it beats no-pruning, random, and proof-length baselines under blinded LLM judging.
Critical strengths
- A simple, computable, well-motivated metric. Measuring interestingness as proof difficulty normalized by description length is intuitive, echoes compression and Pólya-style views of elegance, and needs no human labels. The normalization is well argued: conjoining unrelated theorems grows difficulty linearly but leaves interestingness O(1). The resulting ordering of Mathlib, with trivial identities at the bottom and small cases of FLT at the top, is a convincing sanity check.
- Conditional proof difficulty as a reusable primitive. Framing V(T | P) as a premise-conditioned value function with explicit axioms is clean, and turning those axioms directly into GRPO reward terms is a nice design choice. Premise expansion is a practical answer to Mathlib’s atomized structure, where the median proof is 3 lines. The resulting estimator is clearly better calibrated than frontier models (MAE 20.2 vs. 31.0/34.3).
- Connecting intrinsic and extrinsic value. Relating a quantity available when a theorem is proposed (interestingness) to one observable only afterward (how much a theorem compresses downstream proofs) is the right question for autonomous discovery. It gives a principled reason to use the metric as a proposal-time surrogate.
- End-to-end demonstration with an informative ablation. The metric works both as a training reward and as an inference-time selection rule. Including proof-length pruning as a baseline is especially valuable. Interestingness pruning wins the blind ranking 60% vs. 19%, which suggests the gain comes from the ratio itself rather than from favoring long proofs.
Critical weaknesses
- The discovered mathematics remains shallow. Promoted theorems in the discovery loop average 7.4 proof lines (Table 1), and verified candidates average 3.7–5.5. Many showcased statements look like artificial packaging rather than discoveries: 1) biconditionals with an unrelated side condition (n > 0 ↔ … Chebyshev nodes); 2) a conjunction of two unrelated facts (modular inversion); 3) near-tautologies, such as g⁻¹ acting on g·z restoring Im z. With no evaluation by human mathematicians, it is unclear whether gains in the metric reflect gains in genuine interestingness.
- Evaluation relies on one LLM family and may favor the trained model. Claude Opus 4.6 serves as baseline conjecturer, prover, repair agent, semantic filter, containment judge, and quality, diversity, and ranking judge. The containment judge has no retrieval over Mathlib, so the 91.9% → 30.6% drop may partly reflect unnatural phrasings that nobody would add to Mathlib, and the novelty is not valuable. The verified cohort is also built by sampling until 20 proofs exist per area, with a repair pass allowed, yet per-model prove, repair, and validity rates are not reported.
Questions for authors
- Novelty and scope of the definition. Footnote 2 says I0 corresponds to the “interest” of Aksenov et al. (2026); what is new beyond the conditional version and the learned estimator? And since L(T|P) charges for definitional vocabulary, would the metric penalize discoveries that introduce new definitions?
Response to Reviewer Jy3x
Thank you for the thoughtful review and constructive suggestions, which have helped us improve the paper. We would like to acknowledge the mentioned weaknesses and questions here:
We agree that whether gains in our metric translate into gains that mathematicians would find meaningful remains open. On competition math problems, prior work found that LLM ratings often broadly agreed with human ratings but missed much of the variation in which problems people found interesting and why (Mishra et al., 2025, A Matter of Interest).
A human study of generated Lean theorems would address several questions. What makes a theorem interesting to mathematicians, and how do their judgments vary across fields and levels of expertise? How do those judgments change when mathematicians see the surrounding theory and downstream uses of a theorem? How well do our structural score and blinded LLM rankings track expert judgments? Together, these questions would form the basis of a full follow-up paper.
Appendix E.4 adds GPT-5.5 as a second judge. Across the same 100 policy-blinded comparisons, interestingness pruning ranked first in 57% with GPT-5.5 and 60% with Claude, with mean ranks of 1.86 and 1.80, respectively (Table 8). In this experiment, Section 3.4 holds the Claude conjecturer, semantic filter, and proving agent fixed across promotion rules. In the separate conjecturing experiment, we compute interestingness from verified proofs, and the trained model’s mean remains 5.03× Claude’s on statements proved as written (Appendix E.3, Table 7). The conjecturer comparison remains conditional on the statements the common proving pipeline verifies.
The original Figure 5 protocol supplies the statement, verified proof, and referenced Mathlib declarations. In the additional evaluation in Appendix E.4, GPT-5.5 had access to the top 20 BM25-retrieved Mathlib declarations for each of the 480 verified statements from the base, trained, and Claude conjecturers. It judged 31% of the trained model’s statements to be rephrasings of existing declarations, compared with 84% for the base model and 68% for Claude (Table 9).
Section 3.3 and Appendix E.2 now report the per-model audit. The base, trained, and Claude models needed 240, 880, and 280 sampled prompts, respectively, to fill their balanced cohorts. Their syntactic filter pass rates were 99%, 55%, and 85%. Their exact proof rates were 58.0%, 7.4%, and 70.7%, and repair succeeded on 59.0%, 35.5%, and 44.3% of exact proof failures. Overall validity under the pipeline, including repair, was 82.8%, 40.2%, and 83.7%. Of the 160 retained statements per model, 48 base, 131 trained, and 24 Claude statements were repaired (Table 5). The trained model therefore has a substantial verification disadvantage, which we will report clearly in the main text of the paper. However, among statements proved as written, its mean interestingness still exceeds the base model’s by 5.13× and Claude’s by 5.03× (Table 7).
The introduction now cites Aksenov et al. (2026) more explicitly for the underlying theoretical compression principle. Our work introduces a conditional variant of this principle and a learned difficulty estimator, then develops these ideas into an end-to-end discovery framework. Specifically, we train a conjecturer with the resulting signal, and run iterative theorem-proving experiments in which verified theorems are selected for further exploration. To our knowledge, all of these contributions are novel and not mentioned in prior works.
Yes. Charging for new vocabulary can disadvantage a statement whose new definition becomes useful only later. Section 4 identifies definition invention as future work, since the current setup doesn’t allow models to create new definitions alongside new theorems. New ideas as to how to measure compression may be needed in future work to better handle definition generation.