- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces MathAtlas, a large-scale benchmark for autoformalization in the wild, containing 52,052 theorems, definitions, exercises, examples, and proofs. The benchmark also contains a mathematical dependency graph with 178k relations. Experiments on this benchmark reveal several issues with current autoformalization systems, highlighting the difficulty of the benchmark.
Critical strengths
MathAtlas, with its dependency graphs, is a good benchmark for studying various autoformalization tasks.
Critical weaknesses
- Some details about the datasets are missing from the paper, which could affect the integrity of the benchmark and the validity of the claims.
- Some claims might be overstated because certain experiments are missing from the paper.
Questions for authors
-
In Section 2.1, the statistics are reported approximately, but I think the exact values could be reported. Could the authors provide the exact numbers here? This would influence my judgment of the integrity of the benchmark.
-
The authors state that MA-Align is annotated with binary labels. Do the annotations come from human experts or LLMs? If they are from LLMs, the claims about LLM judges could be biased. If the annotations are from humans, what are the annotation criteria? How many annotators were involved? What is the level of agreement between annotators?
-
The claim in the abstract that "naive, single-pass systems are insufficient for high-level mathematics" is only established on open-source models. Since these open-source models are mostly 8B, and the largest one is gpt-oss-120b, their performance on single-pass correctness would be expected to be lower compared to larger models such as Claude or GPT. However, this does not support the general claim that single-pass systems are insufficient. Conducting experiments on a small set of examples with frontier models could resolve this issue.
-
Since the benchmark is sourced from textbooks, it is very likely that entities such as definitions or theorems have already been formalized in existing formal libraries such as Mathlib. Testing the autoformalization performance of models on MathAtlas could introduce a data leakage issue. Could the authors estimate how many items in the benchmark have already been formalized in Mathlib?
I would be willing to change my scores if the authors could adequately address my questions.
Response to Reviewer PtYe (part 1/4)
We thank the reviewer for their feedback and thoughts on how to strengthen our paper. We address your concerns here in three parts:
We agree that there is a possibility that our chosen judge (Qwen 3.8 27B) isn't performant across systems, which may affect the conclusions drawn from our reported numbers.
a. To address this concern, we run additional experiments to validate the quality of our faithfulness judge in several settings.
First, we test the quality of a much more powerful judge, Claude Opus 5.5, on MA-Align, with “medium” reasoning effort. We find that this judge is very strong on statements (97.0% balanced accuracy) and on definitions (83.1%). (We reiterate that, despite being strong, we do not use expensive frontier models as our primary metric, due to their inaccessibility and potential for inconsistency across versions.)
Next, we want to confirm that Opus 5.5 is a reliable judge on outputs from all systems. To do so, we have an expert annotate 30 random compiling outputs from each of the systems in the table below. We find that Opus 5.5 agrees with the expert on 93.3% of examples, confirming that Opus 5.5 is reliable across systems.
Finally, we re-scored the compiling MA-Hard outputs of ReForm 8B, gpt-oss-120b (iterative, k=5), Claude Sonnet (iterative, k=5), and Claude Code (dependency-aware) with Opus 5.5, using the same judge prompts.
When the two judges disagree, it is almost always Qwen accepting a formalization that Opus rejects (95% of cases), which is consistent with Qwen over-accepting. This answers the question of magnitude: the inflation is larger for weaker systems, with Qwen’s correctness scores about 3x Opus’s for ReForm 8B but only 1.1x for Claude Code. As expected, this inflation is larger for weaker systems, but notably, the gaps between weak and strong systems are, if anything, larger with a stronger judge, further strengthening our core conclusions in the paper.
To conclude, despite the imperfections of our chosen judge (Qwen 3.8 27B), it is nonetheless an adequate metric for this benchmark, as its errors (1) only overstate performance, so the benchmark is at least as difficult as reported, and (2) preserve the ranking of systems, therefore giving a reliable way to compare performance and measure progress.
We strongly believe your critique and our response here have strengthened our paper considerably. We will add these experiments and analyses to Section 3 of the paper.
We also address your concern that synthesized prerequisite definitions may also silently be unfaithful. In our annotation below, we require that auxiliary definitions also maintain faithfulness. This rule invalidated only 3 otherwise faithful formalizations from Claude Code (dependency-aware).
Regarding BEq/BEq+. We understand the potential gains from the proposed experiment, but we respectfully disagree that it is the best use of resources. While BEq(+) is an effective metric in certain settings, it has two limitations for MathAtlas. Firstly, BEq(+) compares theorem statements, so it does not apply to definitions, which make up ~13k of the entities in MathAtlas. Secondly, many MathAtlas statements depend on prerequisite definitions that do not exist in Mathlib. A gold reference would therefore require gold versions of those definitions as well, and BEq(+) cannot compare two statements built on different auxiliary definitions, which is exactly what many system outputs contain. Writing such references is also expensive, requiring both graduate mathematics and formalization expertise. We believe the expert evaluation above provides supporting evidence similar to gold references while also covering definitions. We therefore chose to study, develop, and benchmark LLM judges, as they are a more generally applicable metric in the diverse setting of MathAtlas.