acceptodds
Under review as a conference paper at ICLR 2027

MathAtlas: A Benchmark for Autoformalization in the Wild

Abstract

Prior autoformalization benchmarks are largely focused on olympiad or undergraduate mathematics, while graduate and research-level mathematics remains under-studied. In this paper, we introduce MathAtlas, the first large-scale autoformalization benchmark of in-the-wild graduate-level mathematics, containing theorems, definitions, exercises, examples, and proofs extracted from 103 graduate mathematics textbooks. MathAtlas is enriched with a mathematical dependency graph containing relations, and is the first autoformalization benchmark to include such relations, facilitating evaluation and development of dependency-aware autoformalization systems. Our extensive experiments reveal several gaps in current autoformalization systems. First, we find naive, single-pass systems are insufficient for high-level mathematics. Second, we find feedback-aware formalization loops are an improvement, but primarily only help compilation rate, not semantic faithfulness. Third, we find frontier agentic systems, though effective, are extremely expensive and inefficient, thereby limiting at-scale autoformalization and making efficiency a key challenge. Furthermore, we find system performance degrades substantially with dependency depth. To study performance on “deep” mathematics, we release MA-Hard, a split of 700 entities with the deepest dependency trees. On MA-Hard, our strongest agentic system achieves 64% correctness, but does so inefficiently. We release MathAtlas to the community as a benchmark set for large-scale autoformalization of graduate-level mathematics in the wild. Finally, we introduce MA-Align, a semantic-faithfulness benchmark containing 200 labeled graduate-level examples covering both statements and definitions. Existing metrics perform significantly worse on MA-Align than on prior undergraduate-level benchmarks, indicating a need for further work in this area.

Primary area:
datasets and benchmarks
Keywords:
mathematics, autoformalization, dataset, evaluation, llm

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_zugVOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MathAtlas, a large-scale benchmark for autoformalization in the wild, containing 52,052 theorems, definitions, exercises, examples, and proofs. The benchmark also contains a mathematical dependency graph with 178k relations. Experiments on this benchmark reveal several issues with current autoformalization systems, highlighting the difficulty of the benchmark.

Critical strengths

MathAtlas, with its dependency graphs, is a good benchmark for studying various autoformalization tasks.

Critical weaknesses
  1. Some details about the datasets are missing from the paper, which could affect the integrity of the benchmark and the validity of the claims.
  2. Some claims might be overstated because certain experiments are missing from the paper.
Questions for authors
  1. In Section 2.1, the statistics are reported approximately, but I think the exact values could be reported. Could the authors provide the exact numbers here? This would influence my judgment of the integrity of the benchmark.

  2. The authors state that MA-Align is annotated with binary labels. Do the annotations come from human experts or LLMs? If they are from LLMs, the claims about LLM judges could be biased. If the annotations are from humans, what are the annotation criteria? How many annotators were involved? What is the level of agreement between annotators?

  3. The claim in the abstract that "naive, single-pass systems are insufficient for high-level mathematics" is only established on open-source models. Since these open-source models are mostly 8B, and the largest one is gpt-oss-120b, their performance on single-pass correctness would be expected to be lower compared to larger models such as Claude or GPT. However, this does not support the general claim that single-pass systems are insufficient. Conducting experiments on a small set of examples with frontier models could resolve this issue.

  4. Since the benchmark is sourced from textbooks, it is very likely that entities such as definitions or theorems have already been formalized in existing formal libraries such as Mathlib. Testing the autoformalization performance of models on MathAtlas could introduce a data leakage issue. Could the authors estimate how many items in the benchmark have already been formalized in Mathlib?

I would be willing to change my scores if the authors could adequately address my questions.

Reviewer_PtYeOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MathAtlas collects 52,052 mathematical entities from 103 graduate textbooks with approximately 178k dependency relations, and introduces MA-Hard and MA-Align. It evaluates statement and definition autoformalization, reporting 64.1% correctness for its strongest agent on MA-Hard, where correctness combines compilation with LLM-judged semantic faithfulness.

Critical strengths
  • The broad textbook coverage, inclusion of graduate-level definitions, and explicit contextual dependencies provide a potentially useful resource for studying autoformalization beyond isolated statements.

  • Separating compilation from semantic faithfulness, and comparing single-pass, iterative, and agentic systems alongside costs, provides informative diagnostics.

Critical weaknesses
  • The reliability of the headline correctness scores needs stronger validation. The faithfulness judge achieves 77+% balanced accuracy on 200 selectively sampled, compiling outputs from a single generator, gpt-oss-120b. Its reliability on the strongest agents’ outputs is unclear, and no independent expert audit of their final specifications and synthesized prerequisite definitions is reported. Although the authors acknowledge possible score inflation, its magnitude and effect on system rankings remain uncertain. An expert audit and a small reviewed formal-reference subset for evaluating BEq/BEq+ would strengthen the evidence.

  • The strongest-system evaluation and dependency-based difficulty measure require better reproducibility and validation. The strongest agent is identified as Claude Code (Sonnet), but its exact model snapshot and agent version are not specified. A version-pinned stronger baseline, such as Opus, under matched tools and explicit budgets would help establish how sensitive the reported difficulty is to model choice. Separately, although the paper reports relation-extraction F1, the audit establishes reference labels using only ten retrieved candidates per reference, assuming no target exists when none is suitable. This procedure cannot identify valid targets outside the candidate set, so the reported F1 does not independently establish end-to-end retrieval coverage. The effect of potentially missing relations on dependency depths and MA-Hard selection therefore remains insufficiently validated.

  • The paper’s novelty and positioning relative to recent progress need clarification. Although the paper discusses prior advanced-mathematics autoformalization, it does not compare with AutoformBot, which reports dependency-aware formalization across 26 undergraduate/graduate textbooks. The authors should clarify the differences in corpus coverage, evaluation protocols, and the additional value of MathAtlas’s informal dependency annotations. Anthropic’s FLT formalization further challenges the benchmark's capability to disclose the gap against state-of-the-art methods (such as Prove2me).

Questions for authors

Semantic evaluation reliability. Can you provide an expert audit of the strongest agents’ outputs, covering both final specifications and synthesized prerequisite definitions? How closely does the automatic judge agree with these assessments, and do the estimated correctness scores and system rankings remain stable? Could a small independently reviewed formal-reference subset support complementary BEq/BEq+ evaluation?

Model reproducibility and baseline strength. What exact Sonnet model snapshot, Claude Code version, tools, and inference budgets were used? Can you provide a version-pinned stronger baseline, such as Opus, under matched tools and explicit budgets, to assess sensitivity to model choice?

Dependency coverage and MA-Hard robustness. How does the relation audit account for valid targets outside the ten retrieved candidates? Can you independently audit a sample without this restriction and assess whether correcting missing or erroneous relations changes dependency depths or MA-Hard membership?

Distinct contribution and frontier relevance. How does MathAtlas differ from AutoformBot in corpus coverage and overlap, evaluation protocol, and the role of dependency annotations? What evidence shows that it reveals unresolved limitations of state-of-the-art autoformalization systems (e.g., Prove2me), rather than limitations specific to the evaluated models or budgets?

3 comments
AuthorsOpenReview ↗

Response to Reviewer PtYe (part 1/4)

We thank the reviewer for their feedback and thoughts on how to strengthen our paper. We address your concerns here in three parts:

  1. We agree that there is a possibility that our chosen judge (Qwen 3.8 27B) isn't performant across systems, which may affect the conclusions drawn from our reported numbers.

    a. To address this concern, we run additional experiments to validate the quality of our faithfulness judge in several settings.

    First, we test the quality of a much more powerful judge, Claude Opus 5.5, on MA-Align, with “medium” reasoning effort. We find that this judge is very strong on statements (97.0% balanced accuracy) and on definitions (83.1%). (We reiterate that, despite being strong, we do not use expensive frontier models as our primary metric, due to their inaccessibility and potential for inconsistency across versions.)

    Next, we want to confirm that Opus 5.5 is a reliable judge on outputs from all systems. To do so, we have an expert annotate 30 random compiling outputs from each of the systems in the table below. We find that Opus 5.5 agrees with the expert on 93.3% of examples, confirming that Opus 5.5 is reliable across systems.

    Finally, we re-scored the compiling MA-Hard outputs of ReForm 8B, gpt-oss-120b (iterative, k=5k=5), Claude Sonnet (iterative, k=5k=5), and Claude Code (dependency-aware) with Opus 5.5, using the same judge prompts.

    SystemFaithful (Qwen)Faithful (Opus)Correct (Qwen)Correct (Opus)
    ReForm 8B38.7%12.9%1.9%0.6%
    gpt-oss-120b (k=5k=5)22.7%10.7%7.9%3.7%
    Claude Sonnet (k=5k=5)50.9%42.8%50.7%42.6%
    Claude Code (dependency-aware)65.8%60.1%64.1%58.5%

    When the two judges disagree, it is almost always Qwen accepting a formalization that Opus rejects (95% of cases), which is consistent with Qwen over-accepting. This answers the question of magnitude: the inflation is larger for weaker systems, with Qwen’s correctness scores about 3x Opus’s for ReForm 8B but only 1.1x for Claude Code. As expected, this inflation is larger for weaker systems, but notably, the gaps between weak and strong systems are, if anything, larger with a stronger judge, further strengthening our core conclusions in the paper.

    To conclude, despite the imperfections of our chosen judge (Qwen 3.8 27B), it is nonetheless an adequate metric for this benchmark, as its errors (1) only overstate performance, so the benchmark is at least as difficult as reported, and (2) preserve the ranking of systems, therefore giving a reliable way to compare performance and measure progress.

    We strongly believe your critique and our response here have strengthened our paper considerably. We will add these experiments and analyses to Section 3 of the paper.

  2. We also address your concern that synthesized prerequisite definitions may also silently be unfaithful. In our annotation below, we require that auxiliary definitions also maintain faithfulness. This rule invalidated only 3 otherwise faithful formalizations from Claude Code (dependency-aware).

  3. Regarding BEq/BEq+. We understand the potential gains from the proposed experiment, but we respectfully disagree that it is the best use of resources. While BEq(+) is an effective metric in certain settings, it has two limitations for MathAtlas. Firstly, BEq(+) compares theorem statements, so it does not apply to definitions, which make up ~13k of the entities in MathAtlas. Secondly, many MathAtlas statements depend on prerequisite definitions that do not exist in Mathlib. A gold reference would therefore require gold versions of those definitions as well, and BEq(+) cannot compare two statements built on different auxiliary definitions, which is exactly what many system outputs contain. Writing such references is also expensive, requiring both graduate mathematics and formalization expertise. We believe the expert evaluation above provides supporting evidence similar to gold references while also covering definitions. We therefore chose to study, develop, and benchmark LLM judges, as they are a more generally applicable metric in the diverse setting of MathAtlas.

AuthorsOpenReview ↗

Response (part 2/4)

W2

Response

  1. Reproducibility. We agree that these reproducability details should be in the paper. We used Sonnet 5 and Claude Code v2.1.281, with the tools listed in our setup (web search, a Lean compiler via lean-lsp-mcp, and LeanSearch for Mathlib search). On MA-Hard it used on average 5.4 calls and 164.6k tokens per item (6.9 calls and 235.0k tokens for the dependency-aware variant). We will add these details to the paper (or appendix if necessary).

  2. Opus experiments. Unfortunately due to budgetary constraints, we were not able to add an Opus agent baseline. Our experiments ran on a limited compute budget, and Opus is considerably more expensive than Sonnet. We do not discount the fact that a stronger model (e.g., Opus) is likely to perform better. However, we note that cost and efficiency is one of our findings. We note that our strongest system costs over $130 on just MA-Hard, and Opus is likely to be even more given it is 2x more expensive per token.

    Nonetheless, we agree that ablating on model performance is a worthwhile experiment, and an important direction for future research. We hope to be able to study this line of work, and encourage the community to as well. We will add this to the conclusion as future work, and address this in the limitations section.

  3. Relation extraction F1. Yes, we assume our retriever is capable of high recall@10 because we use an off-the-shelf one, used and proven effective in prior work (LeanSearch). However, your reservations are understandable and we therefore evaluate our system, following the recommendations from Reviewer ALCR.

    To address your concerns, we first measured the retrieval model's recall@10. On an annotated set of 154 examples, we find the system has a high recall@10, with 89.5% for object references and 94.1% for entity references (91.6% overall). We also measure end-to-end relation extraction performance, and find an end-to-end F1 score of 91.3%. The discrepancy in reported score (94.8%) and this end-to-end score is primarily from the suspected case, where the correct link is absent from the 10 retrieved candidates (but exists somewhere in the corpus).

    We will update our evaluation protocol description in the paper, and report these new, corrected numbers.

  4. Effect of edge noise on MA-Hard selection. We agree that errors in the extracted relations can affect dependency depths and MA-Hard selection, and we thank the reviewer for raising this.

    To measure the effect of relation errors, we re-linked a random set of 90,000 object reference links with a stronger system (Claude Sonnet) and recomputed depth on the resulting graph. We find that depth is well-preserved (Spearman 0.8550.855). Furthermore, under this graph, MA-Hard remains "deep," with the median item being at the 94th percentile of depth.

    To account for the possible error, we will adjust our phrasing of MA-Hard from being items with "the deepest dependency trees", to being "the set of items with a measured depth of ≥72\geq 72". In this way, we are more precise while conveying the same fundamental idea.

    Furthermore, we note that MA-Hard’s usefulness does not depend on every relation necessarily being correct, since its difficulty is observed directly: for example, ReForm 8B drops from 19.0% compilation and 9.8% correctness on all statements to 5.0% and 1.9% on MA-Hard. Furthermore, the dependency-aware setting improves agentic system performance, indicating these relations are providing useful information despite possible noise.

    We will add these findings to the paper to strengthen the quality analysis of MathAtlas's graph.

AuthorsOpenReview ↗

Response (part 3/4)

W3

Response

  1. autoform-bot and ATLAS Thank you for pointing out the missing citation - we will add and discuss this in the related works.

    However, we would like to point out that autoform-bot is available only as an arXiv preprint and code release, and ICLR's reviewer guidelines state that authors are not required to compare to papers solely on arXiv, and that the lack of such a comparison cannot be a basis for rejection (see guide here).

    Nonetheless, there are several differences in our work.

    1. Autoform-bot is a multi-agent autoformalization system that produces a Lean library from textbooks (ATLAS), while MathAtlas is a benchmark, designed to test the performance of autoformalization systems. Our work also tests several baselines across different classes of systems to evaluate performance, whereas autoform-bot is a singular presentation of a new system.
    2. Importantly, MathAtlas links dependencies across documents. Relations are prioritized to be in-book, but will search across books if needed. This matters for advanced textbooks that do not redefine basic concepts (for example, an algebraic geometry text assumes basic algebra). This makes MathAtlas a unified dataset, rather than a collection of unrelated subgraphs. In contrast, in ATLAS, "each textbook was formalized in isolation" (Sec. 6).
    3. Autoform-bot, by default, uses expensive frontier models such as GPT and Claude for construction. We construct MathAtlas using much cheaper, open-source models. The authors even note this dependency as a limitation in their paper.
    4. MathAtlas is much larger, containing 100 textbooks across 87 fields. ATLAS v1 contains only 26 books, and a much narrower subset of math.
    5. Finally, we would like to emphasize the fact that autoform-bot/ATLAS (and its companion paper) are not peer reviewed works, and take several liberties in their. While they present their methodology and results, their intention is a demonstration rather than a rigorous study of the setting. For example, the authors use LLM-judges, but report no quantitative testing as to the effectiveness of the judge (in contrast to our work). Similarly, they do not evaluate the integrity of their data extraction or relation extraction systems (again, in contrast to our work).

    For these reasons, in addition to the ICLR reviewer guidelines, we do not believe a direct comparison to autoform-bot/ATLAS is warranted. However, we do believe noting the work as related work is prudent, and we will make this change to the paper.

  2. We do not claim to evaluate state-of-the-art systems, but rather we restrict our claims to the systems we ran (e.g. "our strongest agentic system"). We are happy to cite relevant prior work, but please note that under ICLR guidelines, a lack of comparison or ciation of non-peer reviewed and contemporaneous work (i.e., work since July 17th) "cannot be a basis for rejection".

    Furthermore, we believe these results complement MathAtlas, as they address a different setting. The FLT formalization is a very targeted “attack” on a single, high-profile theorem, carried out by agents on a collaborative platform (Prove2Me), following a simplified version of Wiles’ proof. Lean guarantees that the final theorem is proved, and the post reports that its statement was checked against Mathlib’s. However, the post does not report whether the thousands of synthetic auxiliary definitions and statements were faithfully to the original proof.

    On the other hand, MathAtlas serves as a benchmark to evaluate essentially the same setting, i.e., whether systems can faithfully formalize a wide variety of (graduate) mathematics. Our results suggest this is not solved for the scale we evaluate. Claude Sonnet (k=5k=5) compiles 99.7% of MA-Hard, but only 50.9% of those outputs are judged faithful.

    We also would like to reiterate the difference in budget. FLT, for example, was formalized using dozens of agents over 11 days, and about 6 billion output tokens from an internal-only frontier model. Using Fable 5.1 as an estimate, this would cost 300,000 dollars (plus input token costs). In contrast, our strongest system cost 133 dollars in total on MA-Hard (still potentially cost prohibitive for some settings), and reached only 64.1% correctness. This leaves significant room for improvement in both performance and efficiency, which MathAtlas can help to measure.

Reviewer_ALCROpenReview ↗
Rating
3
Confidence
3(−1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MathAtlas, a large-scale benchmark for autoformalization of graduate-level mathematics, containing approximately 52k mathematical entities extracted from 103 textbooks and 178k dependency relations. The authors additionally introduce MA-Hard, a subset of approximately 700 entities with deep dependency structures, and MA-Align, a human-annotated benchmark of 200 examples for evaluating semantic faithfulness. The experiments evaluate fine-tuned, prompted, feedback-aware, and agentic autoformalization systems. The results suggest that existing systems struggle with advanced mathematics, compiler feedback primarily improves compilation rates, and dependency-aware agents achieve better performance at substantially higher computational costs.

Critical strengths

S1. A valuable large-scale benchmark for advanced mathematics.

MathAtlas substantially expands the scale and mathematical coverage of existing autoformalization benchmarks. Its inclusion of definitions, theorems, examples, exercises, and proofs is particularly valuable for studying autoformalization beyond isolated theorem statements.

S2. Dependency information is a useful addition.

The dependency graph, together with local contextual information and Mathlib grounding, provides a promising foundation for studying dependency-aware autoformalization. This is especially relevant for advanced mathematics, where formalizing unfamiliar concepts may require substantial prerequisite theory.

S3. Comprehensive baseline evaluations.

The paper evaluates a diverse range of autoformalization systems and separately reports compilation, semantic faithfulness, and computational costs. This provides useful empirical insights into the limitations of existing approaches.

Critical weaknesses

W1. The evaluation of reference matching quality is limited by the retrieval model's coverage.

The evaluation of reference matching relies on the retrieval model's top-10 candidates, which may introduce bias and does not rigorously reflect the accuracy of matching extracted references to their true target entities. Specifically, the authors identify ground-truth targets only among the top-10 retrieved candidates and assume that no valid target exists if none of these candidates is correct. However, a valid target may exist in the dataset but fall outside the top-10 candidates. In such cases, the evaluation may fail to identify a missed reference match, potentially inflating the reported recall and F1 score. Therefore, the reported 94.8% F1 does not necessarily reflect the end-to-end reference matching accuracy. I recommend evaluating the retrieval model's Recall@10 against independently established ground-truth targets, or conducting a human evaluation in which annotators identify the correct reference targets directly from the source material without being restricted to the retrieved candidates.

W2. The reliability of the central correctness metric is insufficiently established.

The paper relies on Qwen3.8 27B as its primary semantic faithfulness evaluator. However, Table 2 shows that this judge achieves only 77.3% balanced accuracy on definitions and 77.8% on statements in MA-Align. Such evaluation errors raise concerns about the reliability of the reported correctness scores.

Questions for authors

The authors state that they release the complete dataset construction code(line 276 saying "we release code to generate MathAtlas in full"), but I could not find a corresponding code repository or supplementary material in the submission. Could the authors clarify where the implementation is available and whether the complete dataset construction pipeline can be reproduced using the released resources? Given that MathAtlas is primarily a benchmark contribution, the complete dataset construction pipeline constitutes a substantial part of the paper's contribution, and its reproducibility is particularly important. If such a repository is made available, I would consider raising my rating, as this would substantially strengthen both the paper's reproducibility and its practical value to the community.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Request for further feedback and comments on our rebuttals

We thank the reviewers for their time and expertise in critiquing and suggesting improvements for our work. Your feedback has strengthened our work considerably, and we are eager to discuss our improvements further. In general, we have addressed all weaknesses, and clarified some points in the paper which were poorly communicated.

If the reviewers have any further comments on our responses, we are happy to discuss further and address any remaining concerns with our work.

Otherwise, if we have addressed your concerns, please consider adjusting your overall score of our work.

Again, thank you to the reviewers and chair(s) for your time and energy spent reviewing our work.

AuthorsOpenReview ↗

Follow-up to reviewer response (part 1/1)

Thank you for remaining engaged during the review discussion phase, we appreciate your continued feedback.

  • We have addressed the remaining statistical/numerical concerns brought up by Reviewer zugV. These stemmed from mistaking files from prior experimental (unreleased) versions of the dataset. We greatly apologize for any confusion, and hope we have fully clarified it now.

  • Please see our comments (part 1) and (part 2) for our corrected statistics and other responses to Reviewer zugV.

  • Furthermore, we have now incorporated all feedback and promised edits into the rebuttal PDF draft.

Again, thank you for your time in reviewing our paper, and please let us know if our new experiments, evidence, and analysis have addressed your remaining concerns.

I thank the authors for their clarifications, and I believe that most of my concerns have been addressed. However, I share the concern raised by Reviewer PtYe: “The repeated inconsistencies in the reported numbers, attributed to different experimental versions of the dataset, raised concerns about dataset management and the reliability of the reported results.” Therefore, I will maintain my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.