- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces TheoremGraph, a statement-level graph connecting informal mathematical literature with formal Lean libraries. It combines 11.7 million statements extracted from arXiv with LeanGraph, which extracts 388,105 declarations and typed dependencies across 25 Lean projects. Cross-formality links are constructed through LLM-generated natural-language summaries, embedding-based retrieval, and LLM judgments, yielding 47,952 pairs labeled as exact or inexact matches. Downstream retrieval and autoformalization experiments show the application and benefit of TheoremGraph.
Critical strengths
- Useful infrastructure at substantial scale. The 11.7 million informal statements, multi-project Lean graph, typed edges, and extractor provenance could support valuable research.
- A promising retrieval result. Recall@10 improves from 0.780 to 0.883 after reranking. Reproducing LeanSearch v2’s baseline with the authors’ own code strengthens the comparison.
- Thoughtful evaluation choices and disclosures. Blueprint correspondences provide author-annotated examples, the bootstrap accounts for queries sharing a target declaration, and the paper openly reports negative transfer and limitations of its judges.
Critical weaknesses
-
Clarity and specificity problems make the details hard to follow. The manuscript moves between graph construction, cross-formality matching, proof auditing, and retrieval experiments without clearly explaining how the components connect within each evaluated pipeline. Several consequential implementation choices are inadequately specified: “wider shortlist” is introduced without its candidate counts or selection procedure; the “Search” component bundles multiple interventions; and the trained query head in §5.1 appears without a substantive description of its architecture or training protocol. Some key concepts are not clearly defined, like "slogan", "FLT". These are more than presentation imperfections: they make it difficult to reconstruct the evaluated methods, interpret the ablations, and determine which components account for the reported gains.
-
The semantic correctness of the cross-formality bridge is insufficiently established. Section 3 defines matches as statements expressing the same mathematical result under the same assumptions, yet the evaluation counts both exact and inexact pairs—including generalizations, special cases, and one direction of an equivalence—as successful matches. These relations are not interchangeable for formalization, so the reported 47,952 matches do not establish the scale of faithful informal–formal correspondence. Moreover, expert calibration covers only ten pairs (Appendix E). The reported κ = 0.86 measures the judge’s consistency across repeated runs, rather than its correctness, while the 87% acceptance rate for high-similarity candidates reflects LLM judgments rather than independently verified precision. These limitations are particularly consequential because Lean’s kernel validates proofs of formal statements; it does not certify that those statements faithfully represent the original informal mathematics. The authors should distinguish exact correspondence from directional or weaker relationships and conduct a substantially larger, stratified expert audit using full mathematical context, reporting precision and confidence intervals separately for each relation type.
Questions for authors
- What is your core contributions? Can you specify the effort you made to construct the TheoremGraph and ensure its correctness?
- Can you summerize the application of TheoremGraph? Can you explain and compare more about why some components drop the performance (e.g., Graph) ?
- Considering recent frontier models like Opus5.5 getting more powerful in math reasoning, what is your strengths in combining such prover?
2 comments
Response to Reviewer xnB1 (2/2)
(Part 2 of 2)
Q1. Core contributions, and the effort to ensure correctness. Our core contribution is a statement-level graph connecting informal and formal mathematics and supporting search across both. It has three parts:
- An informal graph: 11.7M theorem-like statements from arXiv (theorems, lemmas, corollaries, and others), with 18.3M dependency edges. Three extractors identify candidate edges: explicit LaTeX references (
\ref,\cite), textual heuristics such as "by Theorem 3.2", and an LLM-based extractor that links notation to its definitions. Each edge carries the name of its extractor. - A formal graph: an extractor that runs inside Lean and reads elaborated declarations. It gives 388,105 declarations and 11.3M edges across 25 projects. Each edge has a type that separates dependencies in signatures, proofs, definitions, and other roles.
- A bridge: an LLM writes a short summary ("slogan") of every statement and declaration. All slogans are embedded in one space. Pairs with cosine similarity 0.8 or more are candidates. A GPT-5.4 judge labels each candidate as exact, inexact (a generalization, a special case, or one direction of an equivalence), or wrong.
Correctness: the explicit-reference edges reach 98.8% precision against an independent judge on 500 papers. LeanGraph reads the formal edges from compiled Lean, so every formal edge is a dependency that Lean recorded. The audit in W2 checks the match labels.
Q2. Applications, and why some components lower performance. TheoremGraph supports three uses:
- Finding the Lean counterpart of an informal result, and the reverse.
- Retrieval for autoformalization.
- Finding repeated statements within a formalization project.
Graph expansion adds the parents of each of the top 50 results. Configurations B and C differ only in graph expansion (Table 10, page 18). It improves retrieval for queries targeting classes: Recall@10 goes from 0.575 to 0.903. It hurts theorems: Recall@10 goes from 0.749 to 0.543. A theorem's parents are the definitions that it mentions, and they push the theorem down the list. Our headline system does not use graph expansion.
Q3. Value alongside strong provers. We tested this for the rebuttal on 48 theorems added to Mathlib after the training cutoff of Opus 5.5 (Appendix H.1, page 27). Opus 5.5 formalizes 36 of 48 with no help, 39 with our retrieval, and 36 with its own library search. Retrieval reaches the same accuracy at about half the cost of the model's own search.
The larger benefit is reuse. In the Fermat's Last Theorem formalization, 67 pairs of theorems state the same thing under different names, and each was proved separately (Section 4, page 6). With TheoremGraph, an agent finds the existing result before it proves the result again.
Reuse matters most to the people who read these proofs. Agents now write formal proofs faster than mathematicians can read them. A new result must be linked to the literature that it restates, specializes, and depends on. Only then can the community interpret it, credit it, and build on it. Stronger provers make this link more necessary, and as the size of formalization projects grows, their interdependence with existing formal statements becomes increasingly important to the broader mathematical community for deriving insight and understanding of results.
Response to Reviewer xnB1 (1/2)
Thank you for the review. We address each weakness and question below.
W1. Clarity and specificity. We made the following changes.
Table 9. Ablation results on MathlibQR fair-810, with each intervention added independently to the baseline:
HyDE accounts for most of the improvement, while the wider shortlist leaves Recall@10 and nDCG@10 unchanged.
Query head: a single linear map on the query embedding (4096 × 4096, no bias, initialized to the identity). We train it on 300,000 declarations that have at least two premises, split by file. The loss is softmax cross-entropy at temperature 0.05, with random and popular premises as negatives. Training runs six epochs of AdamW at learning rate 1e-4. On 3,000 held-out declarations, Recall@100 rises from 0.16 to 0.54. A popularity-based baseline that returns the most frequently used premises reaches 0.38.
W2. Semantic correctness of the bridge. We agree that generalizations and special cases should not be treated as faithful formalizations, and that reporting only the aggregate count of 47,952 obscured this distinction.
Both models call about 90% of the exact pairs a match. They keep the
exactlabel itself for 75% and 70% of these pairs, and move most of the others toinexact. Both reject about 30% of the inexact pairs. We report these results explicitly as model judgments rather than human validation. Our human evidence is one expert and twenty pairs (Tables 11 and 12). The rating sheet is in the supplement so that experts can run the audit.We agree on Lean's kernel. TheoremGraph does not certify that a formal statement means what the informal statement says. A match is a labeled link between two statements.
(Continued in part 2 of 2.)