acceptodds
Under review as a conference paper at ICLR 2027

TheoremGraph: Bridging Formal and Informal Mathematics

Abstract

Mathematical knowledge is organized around statements and their dependencies, but this structure is exposed unevenly: informal papers cite mostly at the document level, while formal libraries record fine-grained dependencies over a much smaller body of mathematics. We introduce TheoremGraph, a unified statement-level dependency graph spanning both: 11.7M theorem-like environments parsed from mathematics arXiv with 18.3M candidate directed dependencies, each labeled by the extractor that proposed it, and LeanGraph, a Lean 4 elaborator-level extractor producing 388,105 declaration nodes and 11.3M typed edges across 25 Lean projects. We bridge the two graphs by embedding generated natural-language slogans into a shared semantic space; an LLM judge affirms 47,952 (informal, formal) matches among the candidates above a 0.8 cosine-similarity floor (48% of those judged), with the acceptance rate rising to 87% for scores ≥ 0.9. On the MathlibQR fair-810 retrieval benchmark, our name-and-signature retriever, reranked with the same Qwen3-Reranker-8B protocol as LeanSearch v2, reaches 0.883 Recall@10 / 0.739 nDCG@10, +10.3pp / +11.6pp over its reranked system, a comparison we verify by reproducing its reported numbers with our code. A recall-tuned variant of our retriever comes within 0.5pp of that reranked recall with no reranker at all. On 24 held-out Mathlib theorems, retrieval raises the number of correctly autoformalized statements from 5 to 8, using a quarter of the tokens of library search. We release the dataset, extractors, HTTP API, and MCP interface as infrastructure for mathematical search, attribution, and agentic formalization.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
formal mathematics, Lean, autoformalization, mathematical information retrieval, dependency graphs, semantic search, dataset

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xnB1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces TheoremGraph, a statement-level graph connecting informal mathematical literature with formal Lean libraries. It combines 11.7 million statements extracted from arXiv with LeanGraph, which extracts 388,105 declarations and typed dependencies across 25 Lean projects. Cross-formality links are constructed through LLM-generated natural-language summaries, embedding-based retrieval, and LLM judgments, yielding 47,952 pairs labeled as exact or inexact matches. Downstream retrieval and autoformalization experiments show the application and benefit of TheoremGraph.

Critical strengths
  1. Useful infrastructure at substantial scale. The 11.7 million informal statements, multi-project Lean graph, typed edges, and extractor provenance could support valuable research.
  2. A promising retrieval result. Recall@10 improves from 0.780 to 0.883 after reranking. Reproducing LeanSearch v2’s baseline with the authors’ own code strengthens the comparison.
  3. Thoughtful evaluation choices and disclosures. Blueprint correspondences provide author-annotated examples, the bootstrap accounts for queries sharing a target declaration, and the paper openly reports negative transfer and limitations of its judges.
Critical weaknesses
  1. Clarity and specificity problems make the details hard to follow. The manuscript moves between graph construction, cross-formality matching, proof auditing, and retrieval experiments without clearly explaining how the components connect within each evaluated pipeline. Several consequential implementation choices are inadequately specified: “wider shortlist” is introduced without its candidate counts or selection procedure; the “Search” component bundles multiple interventions; and the trained query head in §5.1 appears without a substantive description of its architecture or training protocol. Some key concepts are not clearly defined, like "slogan", "FLT". These are more than presentation imperfections: they make it difficult to reconstruct the evaluated methods, interpret the ablations, and determine which components account for the reported gains.

  2. The semantic correctness of the cross-formality bridge is insufficiently established. Section 3 defines matches as statements expressing the same mathematical result under the same assumptions, yet the evaluation counts both exact and inexact pairs—including generalizations, special cases, and one direction of an equivalence—as successful matches. These relations are not interchangeable for formalization, so the reported 47,952 matches do not establish the scale of faithful informal–formal correspondence. Moreover, expert calibration covers only ten pairs (Appendix E). The reported κ = 0.86 measures the judge’s consistency across repeated runs, rather than its correctness, while the 87% acceptance rate for high-similarity candidates reflects LLM judgments rather than independently verified precision. These limitations are particularly consequential because Lean’s kernel validates proofs of formal statements; it does not certify that those statements faithfully represent the original informal mathematics. The authors should distinguish exact correspondence from directional or weaker relationships and conduct a substantially larger, stratified expert audit using full mathematical context, reporting precision and confidence intervals separately for each relation type.

Questions for authors
  1. What is your core contributions? Can you specify the effort you made to construct the TheoremGraph and ensure its correctness?
  2. Can you summerize the application of TheoremGraph? Can you explain and compare more about why some components drop the performance (e.g., Graph) ?
  3. Considering recent frontier models like Opus5.5 getting more powerful in math reasoning, what is your strengths in combining such prover?
2 comments
AuthorsOpenReview ↗

Response to Reviewer xnB1 (1/2)

Thank you for the review. We address each weakness and question below.

W1. Clarity and specificity. We made the following changes.

  1. Components are not connected. The introduction now identifies the four components of the graph (Section 1, page 2). Table 1 specifies which slogans, dependency edges, and judge are used in each evaluation (page 2).
  2. "Wider shortlist" was previously underspecified. Before exact re-scoring, the approximate index returns 1,000 neighbors per query instead of 200. For each declaration, we retain only the highest-scoring passage (Section 5.2, page 8).
  3. "Search" bundles several interventions. Section 5.2 describes the three changes and specifies their settings (paragraph "Closing the gap", page 8). Table 9 reports the effect of adding each change independently to the baseline (Appendix D, page 17).
  4. Query head has no description. Appendix H describes the architecture, training data, loss function, training procedure, and evaluation results (paragraph "Query head", page 26).
  5. "Slogan" and "FLT" are not defined. Both are now defined at first use (abstract and Section 1), and the appendix opens with a table of terms (page 14).

Table 9. Ablation results on MathlibQR fair-810, with each intervention added independently to the baseline:

nDCG@1nDCG@10Recall@10
Baseline0.2010.3800.586
+ name index0.2230.3950.593
+ wider shortlist0.1990.3800.586
+ HyDE0.2430.4520.673
All three0.2620.4670.680

HyDE accounts for most of the improvement, while the wider shortlist leaves Recall@10 and nDCG@10 unchanged.

Query head: a single linear map on the query embedding (4096 × 4096, no bias, initialized to the identity). We train it on 300,000 declarations that have at least two premises, split by file. The loss is softmax cross-entropy at temperature 0.05, with random and popular premises as negatives. Training runs six epochs of AdamW at learning rate 1e-4. On 3,000 held-out declarations, Recall@100 rises from 0.16 to 0.54. A popularity-based baseline that returns the most frequently used premises reaches 0.38.

W2. Semantic correctness of the bridge. We agree that generalizations and special cases should not be treated as faithful formalizations, and that reporting only the aggregate count of 47,952 obscured this distinction.

  1. Distinguishing two relations. Section 3 now defines exact and inexact matches separately and states what each is for (paragraph "Use of each relation", page 4). An exact match is a formal counterpart. An inexact match is a related result useful for retrieval, not a formalization.
  2. Split counts. We now report 4,765 exact matches and 43,187 inexact matches separately. Table 14 (page 22) gives the split for each similarity band, and each released match carries its label.
  3. Kappa. We clarify in Section 3 that κ = 0.86 measures the judge's consistency across repeated runs, not the correctness of the resulting labels.
  4. Audit. We prepared an audit along the lines suggested: 200 judged candidates, stratified by label and by similarity band. We could not recruit mathematicians within the discussion period. Instead, two models from a different developer rated the sheet without seeing any verdict (Appendix E.3, Table 13, page 21; 95% Wilson intervals). Both models marked the same pair as impossible to judge, so the table has 199 pairs:
GPT-5.4 labelnOpus 5.5: is a matchFable 5.2: is a match
exact6991.3% [82.3, 96.0]89.9% [80.5, 95.0]
inexact8071.3% [60.5, 80.0]67.5% [56.6, 76.8]
wrong506.0% [2.1, 16.2]8.0% [3.2, 18.8]

Both models call about 90% of the exact pairs a match. They keep the exact label itself for 75% and 70% of these pairs, and move most of the others to inexact. Both reject about 30% of the inexact pairs. We report these results explicitly as model judgments rather than human validation. Our human evidence is one expert and twenty pairs (Tables 11 and 12). The rating sheet is in the supplement so that experts can run the audit.

We agree on Lean's kernel. TheoremGraph does not certify that a formal statement means what the informal statement says. A match is a labeled link between two statements.

(Continued in part 2 of 2.)

AuthorsOpenReview ↗

Response to Reviewer xnB1 (2/2)

(Part 2 of 2)

Q1. Core contributions, and the effort to ensure correctness. Our core contribution is a statement-level graph connecting informal and formal mathematics and supporting search across both. It has three parts:

  • An informal graph: 11.7M theorem-like statements from arXiv (theorems, lemmas, corollaries, and others), with 18.3M dependency edges. Three extractors identify candidate edges: explicit LaTeX references (\ref, \cite), textual heuristics such as "by Theorem 3.2", and an LLM-based extractor that links notation to its definitions. Each edge carries the name of its extractor.
  • A formal graph: an extractor that runs inside Lean and reads elaborated declarations. It gives 388,105 declarations and 11.3M edges across 25 projects. Each edge has a type that separates dependencies in signatures, proofs, definitions, and other roles.
  • A bridge: an LLM writes a short summary ("slogan") of every statement and declaration. All slogans are embedded in one space. Pairs with cosine similarity 0.8 or more are candidates. A GPT-5.4 judge labels each candidate as exact, inexact (a generalization, a special case, or one direction of an equivalence), or wrong.

Correctness: the explicit-reference edges reach 98.8% precision against an independent judge on 500 papers. LeanGraph reads the formal edges from compiled Lean, so every formal edge is a dependency that Lean recorded. The audit in W2 checks the match labels.

Q2. Applications, and why some components lower performance. TheoremGraph supports three uses:

  1. Finding the Lean counterpart of an informal result, and the reverse.
  2. Retrieval for autoformalization.
  3. Finding repeated statements within a formalization project.

Graph expansion adds the parents of each of the top 50 results. Configurations B and C differ only in graph expansion (Table 10, page 18). It improves retrieval for queries targeting classes: Recall@10 goes from 0.575 to 0.903. It hurts theorems: Recall@10 goes from 0.749 to 0.543. A theorem's parents are the definitions that it mentions, and they push the theorem down the list. Our headline system does not use graph expansion.

Q3. Value alongside strong provers. We tested this for the rebuttal on 48 theorems added to Mathlib after the training cutoff of Opus 5.5 (Appendix H.1, page 27). Opus 5.5 formalizes 36 of 48 with no help, 39 with our retrieval, and 36 with its own library search. Retrieval reaches the same accuracy at about half the cost of the model's own search.

The larger benefit is reuse. In the Fermat's Last Theorem formalization, 67 pairs of theorems state the same thing under different names, and each was proved separately (Section 4, page 6). With TheoremGraph, an agent finds the existing result before it proves the result again.

Reuse matters most to the people who read these proofs. Agents now write formal proofs faster than mathematicians can read them. A new result must be linked to the literature that it restates, specializes, and depends on. Only then can the community interpret it, credit it, and build on it. Stronger provers make this link more necessary, and as the size of formalization projects grows, their interdependence with existing formal statements becomes increasingly important to the broader mathematical community for deriving insight and understanding of results.

Reviewer_tGwbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper addresses the mismatch between informal mathematical knowledge presented in research papers and the formal mathematical statements available in Lean libraries. The authors argue that connecting these two representations could improve access to existing formalisations and support the automated formalisation of mathematical statements. The authors introduce TheoremGraph, which connects informal mathematical statements extracted from arXiv papers with formal statements from Lean libraries. The arXiv corpus is constructed using metadata from a Kaggle dataset, filtered for mathematics categories. The authors extract mathematical statements from both sources and construct dependency graphs. LLMs are used to generate informal descriptions of the statement nodes, and embedding similarity is used to establish connections between the informal and formal graphs. The resulting graph serves as a retrieval resource for automated formalisation, allowing models to identify relevant formal statements and definitions. The authors evaluate TheoremGraph on a retrieval and a formalisation benchmark based on theorems added between Mathlib v4.29 and v4.30 and report improvements in formalisation performance. The work is inspired by Prove2Me and its related project Formalpedia.

Critical strengths
  • The paper addresses a relevant challenge in formal mathematics by connecting informal mathematical knowledge from research literature with formal statements and definitions in existing Lean libraries.
  • The reported improvements in retrieval and formalisation performance are promising and suggest that connecting informal mathematical statements with existing formal libraries can benefit formalisation.
Critical weaknesses
  • The writing is convoluted and difficult to follow. The terminology is not consistently explained. This makes it difficult to understand the methodology.
  • The reported quality control results raise concerns about the reliability of the extracted mathematical statements and their relationships. The authors report an LLM-as-a-judge score of 68.1% across the extraction methods, suggesting potential errors in the dataset.
  • A more detailed analysis of these errors and their impact on downstream retrieval and formalisation performance would help assessing the quality of the study.
  • The evaluation lacks systematic validation by mathematical domain experts. The authors do not report agreement between human experts and the LLM-based evaluator.
  • The paper lacks descriptive statistics about the structure of the informal and formal dependency graphs, such as the number of nodes and edges, average in-degree and out-degree etc.
  • The paper does not provide sufficient qualitative examples of matched informal and formal mathematical statements. For a resource of this kind, a searchable interface, similar to LeanExplore, would also make the dataset easier to inspect and evaluate.
  • The formalisation evaluation, based on theorems added between Mathlib v4.29 and v4.30, appears arbitrary and provides limited evidence of the generalizability of the reported improvements. I’d suggest evaluating on held-out examples from established formalisation benchmarks, such as Putnam, ProofNet or FATE.
  • The paper provides insufficient details about the extensibility and maintenance of the proposed pipeline. It remains unclear how new research papers, mathematical definitions or Lean libraries could be incorporated into the existing graph and how previously established relationships would be updated.
  • The paper does not sufficiently distinguish TheoremGraph from closely related efforts, particularly Prove2Me's Formalpedia.
Questions for authors
  • What are the main differences between Prove2Me's Formalpedia and TheoremGraph?
  • How do you ensure that the generated descriptions of the formal and informal mathematical statements faithfully capture the underlying statements?
  • What is the agreement with the human expert and the LLM-judge (line 200)? What is the sample size?
  • Why does the proposed method underperform on theorems compared with other types of mathematical statements?
  • How do the authors plan to maintain and extend TheoremGraph as new mathematical papers and Lean libraries become available?
3 comments
AuthorsOpenReview ↗

Response to Reviewer tGwb (1/3)

Thank you for the detailed review. We address each weakness and question below.

W1. Writing and terminology. We agree that the submission was too compressed. "Slogan" and "FLT" are now defined at first use. The introduction lists the four parts of the graph, and Table 1 states which parts each evaluation uses. Section 3 defines the two match relations before it uses them. The appendix opens with a table of terms.

W2, W3. The error rate of dependency extraction, and its effect downstream. The 68.1% reported in Table 2 measures judge agreement on dependency edges between informal statements. It pools three extractors of very different precision:

ExtractorJudge agreement
Deterministic (resolves \ref and \cite)98.8%
Heuristic (nearby references, prose mentions)76.6%
Notation (an LLM links notation to a prior definition)42.7%
Any, pooled68.1%

Each released edge carries the name of its extractor, so users who require higher precision can use the deterministic edges only. The errors are of known kinds (examples in Appendix G). Heuristic edges link neighboring results that are parallel, not dependent. Notation edges link statements that only share a symbol.

Effect on our experiments: no main result uses these edges. Matching, concept retrieval, and autoformalization use slogans and the formal graph. The formal edges come from compiled Lean and are exact. The informal edges are used only in two appendix analyses: the graph-navigation condition of Appendix B and the distance analysis of Appendix C. They are also optional context for at most 5.3% of informal slogans. Table 1 states this for each evaluation.

W4, Q3. Expert validation, and agreement between the expert and the LLM judge. The expert evidence is small: one domain expert and twenty pairs, in two rounds.

  • Round 1 (Table 11): ten matches labeled by our first judge, Opus 4.8. The expert judged five of these matches to be overly permissive. We therefore changed to the stricter GPT-5.4. On these ten pairs, GPT-5.4 agrees with the expert on three.
  • Round 2 (Table 12): ten further candidates, graded by the expert before seeing GPT-5.4's verdict. GPT-5.4 agrees with the expert on eight of the ten candidates. Both disagreements are between inexact and wrong. Over both rounds, GPT-5.4 agrees with the expert on 11 of 20 pairs.

The kappa of 0.86 at line 200 is the judge's agreement with itself over two runs (93.2% of 500 pairs). It measures the judge's consistency, and Section 3 now states this.

We could not recruit more mathematicians during the discussion period. As a weaker check, two models from a different developer relabeled a stratified sample of 200 judged candidates. They saw the Lean declaration and the informal statement, but no verdict and no similarity score. Both models marked the same single pair as impossible to judge, leaving 199 evaluable pairs. The table reports these 199 (Table 13, 95% intervals):

GPT-5.4 labelnOpus 5.5: is a matchFable 5.2: is a match
exact6991% [82, 96]90% [81, 95]
inexact8071% [61, 80]68% [57, 77]
wrong506% [2, 16]8% [3, 19]

These are model ratings. The two models keep the exact label itself for 75% and 70% of these pairs, and move most of the others to inexact. The rating sheet for the same 200 pairs is in the supplement, so that experts can repeat the audit.

W5. Graph statistics. The abstract and Section 2 report the graph sizes: 11.7M informal statements with 18.3M edges, and 388,105 Lean declarations with 11.3M edges. The revision adds the breakdown by Lean library (Table 19) and degree statistics (Table 18):

InformalFormal
Out-degree: mean / median / max1.55 / 1 / 14629.2 / 20 / 816
In-degree: mean / median / max1.27 / 0 / 2,06229.2 / 1 / 184,165
No incoming edge51.3%33.3%
No edge at all22.6%0.02%

(Continued in part 2 of 3.)

AuthorsOpenReview ↗

Response to Reviewer tGwb (2/3)

(Part 2 of 3)

W6. Qualitative examples and an interface. The revision adds example matches for each verdict and at three similarity levels (Tables 15 and 16), twenty pairs with the expert's notes (Tables 11 and 12), and rejected and missed dependency edges for each extractor (Appendix G). A public search interface exists. We cannot link it here without identifying ourselves. The supplement includes a browsable sample of 384 judged pairs, with filters for verdict and similarity (match_sample.html).

W7. Why theorems added between v4.29 and v4.30, and not ProofNet, Putnam, or FATE? The split is a control for memorization. Models may have encountered ProofNet, Putnam, and much of Mathlib during pretraining, so they can succeed there without retrieval. Theorems added after the retrieval corpus was built are not in the index.

For the rebuttal we repeated this control at twice the size (Appendix H.1). The new set has 48 theorems added to Mathlib after July 7, 2026, and it covers the newest models. We additionally probe for memorization: each model gets only the name of a declaration and must write its signature (Table 25). The results are in our response to Reviewer nXFt.

We also have ProofNet results (Table 24). Retrieval lifts a small model from 13.6% to 31.8% (n=22). It does not help a strong model that knows the material: 83% without retrieval and 75% with retrieval (n=12). These samples are small, and we present Section 5.1 as a case study. We have not run Putnam or FATE.

W8, Q5. Maintenance and extension. The pipeline is incremental: each stage processes only new or unfinished work (Appendix I). When a newer arXiv snapshot arrives, we add only the papers that we have not seen. Parsing, dependency extraction, slogan generation, and embedding run on those papers alone. A separate pass resolves citations from older papers to the new ones. Adding a new Lean project requires one extractor run, configured by its repository, revision, and toolchain. We extract each Mathlib version separately and credit each declaration to the earliest version that contains it. A new statement can be searched against the other side of the graph as soon as it is embedded. To extend the judged matches, we judge only the new candidates above the similarity threshold. We currently run these stages by hand. All matches in the paper come from a single sweep that we have not yet repeated.

W9, Q1. Differences from Prove2Me's Formalpedia. TheoremGraph was built independently of Prove2Me. We now discuss the differences in Appendix A and summarize them here:

FormalpediaTheoremGraph
Contentsstatements proved on the Prove2Me platform, accumulated across its missions11.7M arXiv statements and 25 existing Lean projects, including Mathlib
Informal mathematicsa statement can link to its source; the informal statements are not part of the corpus11.7M informal statements, linked to formal ones
Dependenciesthe theorems that a proof importsread from compiled Lean, with six edge types; informal edges carry the name of their extractor
Searchan API over natural-language descriptions; no retrieval evaluation reportedevaluated on MathlibQR fair-810 against LeanSearch v2

The key distinction: Formalpedia grows from formalizations submitted to its platform. TheoremGraph parses the existing formal and informal corpora and connects them. It keeps both kinds of statement, including statements that have no match yet. The two are complementary: a platform like Prove2Me could use TheoremGraph as its search layer.

AuthorsOpenReview ↗

Response to Reviewer tGwb (3/3)

(Part 3 of 3)

Q2. How do you ensure that slogans are faithful? We do not guarantee it. The limitations section says that a slogan loses information. Three measures reduce the risk.

  1. Context in stages. Qwen3-235B writes each slogan. For an informal statement it starts from the statement alone. It receives more context (the proof, then neighboring statements and dependencies) only when it answers that its context is insufficient. The statement alone is enough for 70.3% of statements. The proof and local context bring this to 94.7%.
  2. The judge reads the statements. A slogan proposes a candidate. The judge then reads the Lean signature and the LaTeX statement together with the slogans. Appendix E.1 shows that it can still lean on a slogan, which is one reason we report the audit in W4.
  3. An end-to-end test. On 1,308 blueprint statements annotated by their authors, search with the informal slogan ranks the author's formal partner first 42.6% of the time and in the top ten 71.7% of the time. BM25 in the same direction gets 31.3% and 61.0% (Table 17).

Q4. Why does the method do worse on theorems? The apparent drop on theorems comes from graph expansion, not from the underlying retrieval method. The figure in the submission was misleading, and we thank you for raising it. The figure compared the baseline with configuration E, which uses graph expansion. Graph expansion adds the parents of each result to the list. This helps queries for structures, classes, and definitions, where the query often describes a parent. It hurts theorems: a theorem's parents are the definitions that it mentions, and the added parents displace the theorem in the ranked list. Configurations B and C differ only in graph expansion: theorems fall from 0.749 to 0.543, and classes rise from 0.575 to 0.903. Recall@10 by kind (Table 10):

structureclassdeftheoreminstance
A, baseline0.4050.4480.5140.7040.794
E, with graph expansion0.8300.9250.9430.6250.722
F, no graph expansion0.6210.6940.6380.7730.897
F + reranker (headline)0.8430.8430.8570.9040.938

The headline system does not use graph expansion. It reaches a Recall@10 of 0.904 on theorems and is above the baseline on every kind.

Reviewer_nXFtOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes TheoremGraph. The motivation of this is that, while formal math proof like lean4 has good dependency graphs, informal math proof area does not. The paper build such graphs with pipelines and matches the two graph. This is useful for theorem retrieval and autoformalization. The authors also claim that the retrieval helps Mathlib theorems autoformalize better with lower cost. Overall I think this is a decent paper, but I am not familiar with the development of formal math after 2026 so I cannot accurately assess the novelty.

Critical strengths
  1. The presentation is nice. The authors made clear about what they did, how they did it, and why this is important for formal math.

  2. It seems like the code is provided.

  3. In my opinion such dependency graph is useful for formal math, and I actually thought it would be nice to have such graphs before.

Critical weaknesses

No major weakness actually, but it would be nice if frontier models are tested for autoformalization and see whether the retrieval still helps.

Questions for authors

Have you provided some results of an agentic search baseline? Perhaps I missed it.

1 comment
AuthorsOpenReview ↗

Response to Reviewer nXFt

Thank you for the review.

Do frontier models still benefit from retrieval? We ran this experiment for the rebuttal (Appendix H.1).

Setup. The 24 original targets were added to Mathlib before the training cutoff of the newest models. So we built a new set of 48 theorems that were not in Mathlib as of July 7, 2026 (four from each of 12 areas). None of them is in the retrieval corpus. Everything else follows Section 5.1: the prompt, tools, conditions and typecheck limit. A judge that does not know the model or condition grades each output against the matching Mathlib declaration.

Results. Correct outputs out of 48 (Table 26):

NoneRetrievalLibrary searchBoth
Sonnet 4.6, correct16212224
Sonnet 4.6, cost per target$0.025$0.026$0.168$0.107
Opus 5.5, correct36393636
Opus 5.5, cost per target$0.036$0.040$0.074$0.066

Opus 5.5 formalizes three quarters of the new statements with no help. For both models, no pairwise difference between conditions is statistically significant, so we do not claim that retrieval improves accuracy. Our claim is that, on this set, retrieval matches library search at lower cost: about one-sixth of the cost for Sonnet 4.6 and half for Opus 5.5.

This does not hold everywhere. On a Lean project that Sonnet had not seen (a formalization of Brownian motion), it formalizes 0 of 18 statements correctly with no help, 9 of 18 with retrieval, and 14 of 18 with library search (Table 24). Library search is better than retrieval there. Both make a task possible that the model cannot do alone. The sample is small and one person graded it.

Limitation. In Section 5.1, a small model wrote the informal descriptions, and a blinded evaluation judged only 9 of 24 faithful. For the new experiment, a larger model wrote them, and 34 of 48 were judged faithful. On those 34 alone, the correct counts (None / Retrieval / Library / Both) are 15 / 17 / 17 / 19 for Sonnet 4.6 and 26 / 29 / 25 / 26 for Opus 5.5. Retrieval again performs about as well as library search.

Is there an agentic search baseline? Yes, in two places.

  • The Library condition of Section 5.1. The model uses a tool to search the Mathlib declaration listing. The table above reports it for both models.
  • Appendix B, an agentic literature-search task. We took 149 passages from mathematics arXiv papers released after February 2026 and removed one citation from each. The model must find the cited statement. Graph navigation beats dense retrieval alone on exact match for every model (Table 6). Neither beats unrestricted web search. However, that comparison is not controlled, because the model can find the source paper by searching for a quote from the passage. That is why the experiment is in the appendix.

Code and data. The original supplement has the code and raw traces for the LeanSearch v2 comparison and the FLT audit. The dataset, both extractors and the API are public, but linking them here would reveal who we are. The updated supplement adds:

  • a browsable sample of 384 judged pairs (match_sample.html)
  • the targets, outputs, labels and scripts for the new autoformalization experiment
  • the 200-pair relabeling and its rating sheet
  • per-query results for the new retrieval ablation

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thank you, and a summary of changes

We thank all three reviewers for the careful reading. The reviews pointed at real problems, and the paper is better for them. We have uploaded a revised PDF and a larger supplement. Replies to each review are below. Here is what changed.

The match count was too generous. The submission reported one total of 47,952 matches. Only 4,765 of these are exact, and we now lead with that number. We present the other 43,187 as related results that help retrieval, not as formalizations.

We checked the judge more carefully. Our expert graded ten more candidates before seeing GPT-5.4's verdict, and the two agree on eight (Table 12). Two models from a different developer relabeled a stratified sample of 200 judged pairs (Appendix E.3, Table 13). We could not recruit more mathematicians in the discussion period, and the paper is clear that this is a check by models.

We ran the experiment that two of you asked for. Appendix H.1 tests Sonnet 4.6 and Opus 5.5 on 48 theorems added to Mathlib after the training cutoff of Opus 5.5. Retrieval is as accurate as the model's own library search and costs less. It does not make either model significantly more accurate, and we say so.

The paper is easier to follow. "Slogan" and "FLT" are defined at first use. Table 1 states which parts of the graph each evaluation uses. The appendix opens with a table of terms, and Appendix H describes the query head.

Smaller additions:

  • each "Search" change tested alone (Table 9)
  • results by declaration kind for every retrieval configuration (Table 10)
  • material restored from our longer draft: the expert's notes, example matches, the corpus by Lean library, rejected dependency edges, and how the graph is updated

We are happy to answer follow-up questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.