- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies an important problem: how to correctly formalize an informal mathematical statement. The interest is only in formalizing the statement of the theorem, and not the full proof.
There is a lot of work already on Autoformalization. This paper proposes the idea of using an "adversarial" agent to propose alternate formalizations - each alternate option must either be proved equivalent (using Lean) or justified (as a valid formalization) by quoting a sentence from the source. Apart from adversarial agents, the paper also uses tools, such as Lean compiler and library search, that are important; as well as, the idea of "chain of thought" whereby the agent tasked with proposing the initial formalization needs to explicitly state the meaning of every Lean concept. The judge also discards formalizations with trivial proofs.
If we remove the "proposer" step that proposes a formalization, and only use the "adversarial agent" and judge, the proposed pipeline can also be used to verify an existing formalization. The proposed autoformalization pipeline was used in this manner to find 5 defects in published formalizations.
Critical strengths
- The autoformalization pipeline being proposed is quite thorough. It includes the drafting agent, the attacking agents, and the judge, which are served by degenerate proof checking and equivalence checking.
- The use of 'chain of thought' reasoning being employed before generating the formalization is also interesting.
- Finding statements with trivial proofs is also a good way to find incorrect formalizations.
- There is a lot of knowledge about common mistakes to look for in the prompts, indicating that some nontrivial amount of effort was put in the building this autoformalization pipeline.
- Finding 5 flaws in published autoformalization is also a good validation.
Critical weaknesses
- It is not clear if there was a separation between 'train' and 'test' here -- what examples were used to build the prompts, pipeline, and the harness -- and what tests were held out and used only for evaluation.
- At places it feels like certain steps/prompts were put in place to catch certain kinds of observed defects. For example, lines 374-377 describe "asking the judge about convention" explicitly, but it reads as if that change was introduced to catch the March p25 error.
- Lean is one language to formalize; would the findings generalize to other formalization languages?
- While "adversarial mutation" sounds something new, it is just a critic agent; and critic agents are often used in LLM-based multiagent systems.
- The single call of a frontier model baseline is actually not bad (as described on Lines 421-424). Presumably it is much cheaper than what is proposed here. Cost implications of using the complex pipeline is also something that is missing here. +- "Adversarial mutation" has the same motivation as something like "diverse solution generation" -- it was done for autoformalization in this work- https://arxiv.org/pdf/2512.00997, which did diverse generation using different models and then used a judge to generate one. It lacked some insights that are present in this paper, such as, linking changes to source text.
Questions for authors
The first concern on leakage is the major one: the most compelling evidence is that the pipeline detected 5 issues in published formalizations, but some knowledge of the defects seems to have gone into the prompts and pipeline design.
4 comments
Response pt 2
The single call of a frontier model baseline is actually not bad (as described on Lines 421-424). Presumably it is much cheaper than what is proposed here. Cost implications of using the complex pipeline is also something that is missing here.
We now report cost and tokens. One call of GPT-5.6 Sol with the paper costs \0.82 per statement (\$93 for the 89 references) and uses about 1.5M input tokens (mostly cached agent context) and 0.2M output tokens.
A single call is a useful first check, but it does not find the hardest defects. We have now run nine models with one call each. They find between 0 and 3 of the 5 defects, and none finds June p1 or June p23 (Table A). Spending more on single calls does not change this, as the ensemble of all nine costs \$0.47 per statement and still finds at most 3 of the 5. Also worth noting is, as run, a single call also only returns a verdict, and does not write a repaired statement.
If cost is the concern, one way to lower the cost of our pipeline is to run only a single attacker, skipping the kernel check and the judge stage. On the five defective references this repairs 4.9 ± 0.1 at \3.83 for the full pipeline (Table B). We would also note that the cost of a misformalization grows downstream, as any compute spent proving the wrong statement is wasted. Having more options on the Pareto frontier is genuinely useful for mathematicians/practitioners.
"Adversarial mutation" has the same motivation as something like "diverse solution generation" -- it was done for autoformalization in this work- https://arxiv.org/pdf/2512.00997 [...]
Thank you for the reference, we have added it to our working draft. In IndiMathBench, twelve models each formalize an olympiad problem, GPT-5 ranks the candidates, and two human annotators choose, edit or merge them. The diversity there comes from independent drafts, and humans make the final judgement.
Our attackers do not write independent drafts. Each proposes a single change to a given statement and, as you note, has to link that change to the source text. We think this matters at the research level for two reasons. First, independent drafts are hard to compare. For 88 claims drafted by two different models, the kernel proved only 34 of the pairs equivalent, and a judge called 45 more the same statement. Second, diverse drafts can share the same misreading. None of the nine models flags ContDiff ℝ ⊤ in June p1 with a single call (Table A), and in three of the 32 arXiv papers of Table C, Codex CLI with GPT-6 Astra itself wrote ContDiff ℝ ⊤ where the paper means smooth.
The first concern on leakage is the major one: the most compelling evidence is that the pipeline detected 5 issues in published formalizations, but some knowledge of the defects seems to have gone into the prompts and pipeline design.
We apologize again for the confusion around this, and your concern is reasonable. To summarize our two answers above: In the judge prompt, there are two examples that were related to errors we had found in ArXivLean, but they were not originally part of our pipeline, and added after the fact as a list of examples in attempts to formalize separate papers. We can confirm however that a) removing the examples from the judge prompt does not affect the judging stage’s performance even on ArXivLean, b) removing the judge stage entirely, and simply using the attacker finds all the errors reliably, and c) our pipeline can reliably find misformalizations/produce better formalizations than other methods on examples outside the ArXivLean (cf. comparisons to Theo, and the ArXivMath results in Table C).
acknowledging author's response
Thank you for your detailed response. One more clarification question: you state in your response - "If cost is the concern, one way to lower the cost of our pipeline is to run only a single attacker, skipping the kernel check and the judge stage. On the five defective references this repairs 4.9 ± 0.1 at $0.33 per statement, against 4.8 ± 0.2 at $3.83 for the full pipeline (Table B)." -- So, does this mean the leaner pipeline is not only more cost efficient, but also performs better on average than the full pipeline?
Reply
Thank you once more for your engagement and consideration.
So, does this mean the leaner pipeline is not only more cost efficient, but also performs better on average than the full pipeline?
It is cheaper, but we would not say it performs better. Over eight runs on the five defects, the single attacker repairs 39 of 40 and the full pipeline 38 of 40. This is a difference of one repair, which is within the variation between runs.
What Table B shows is that the repairs come from the attacker's lookup procedure, and that the later stages do not add repairs on these five references. What the later stages can add is a degree of confidence about the change, namely that no change is made without a Lean proof or a quoted sentence of the paper behind it. This isn't strictly necessary, and practitioners could simply opt to run our attack strategy to get a faithful formalization.
For reference, we also ran both of these settings on 20 other references, drawn at random. The single attacker changed 3 of the 20 in each of its two runs, and the full pipeline changed 1 and 1. However, none of these changes is a "false positive" as the statements were proved equivalent to the reference statement.
So the single attacker is definitely a reasonable option for lower cost, and the full pipeline just adds confidence/grounding in a Lean proof or quote from the paper. The full pipeline is also the setting we have tested most, with no harmful change in three runs over all 89 references (Table D).
Are there any other questions we can clarify? We do truly appreciate your continued engagement with our work.
Response pt 1
We are very grateful for your comments, careful reading, and your time spent reviewing.
Thank you very much for bringing up this point. We initially developed the pipeline trying to formalize frontier natural language proofs from early in the summer (for example the unit distance lower bound disproof). We found strategies that seemed to work avoiding errors, and then tested these strategies on ArXivLean, where we found 5 potential misformalizations that we verified were indeed problematic. We adjusted prompts in minor ways after finding these references, and all were originally found by a procedure developed on other natural language proofs. We also test on other formalizations from competing systems (Theo), and have added 32 new papers from ArXivMath to our comparison (cf. Table C in the general response).
This is an oversight on our part, and we would like to clarify. We initially found it useful to ask about conventions, and specific meanings before testing on ArXivLean, and, as you can inspect, there are no specific references in the attacker prompts. We did experiment after finding defects in the ArXivLean benchmark with further papers, and when doing this, added examples we had found from ArXivLean to the judge prompt, which is what you see in our manuscript. But this is a was not originally present when we found the defects, and we have since confirmed that a) these are not necessary for the judging stage, and removing them does not change judging success and b) an attacker only ever sees general prompts about looking up definitions, etc., and the attacker is actually solely responsible for identifying the errors. For example in Table B in the general comment, we find that we can remove the prompts/stages you identify entirely without affecting the number of defects our pipeline catches. We apologize for this confusion and have clarified this in our working draft which we will upload once ongoing rebuttal experiments are finished/added.
We have not run experiments outside Lean and Mathlib, so we cannot claim this yet, and we will state it as a limitation in the revision. However, we do expect the approach to carry over. The kernel step only needs a proof assistant that can check an implication between two statements, which Isabelle and Rocq also provide. The lookup procedure is tied to the library rather than to the language. Two of the five defects turn on what a Mathlib notion means:
ContDiff ℝ ⊤is analytic and not smooth (June p1), andSylow pis defined for every natural number p (March p25). Any large formal library has conventions of this sort, so we would expect the same kind of misformalization there, but this remains to be tested.Our attackers are similar critic agents, and our working draft now identifies similarity to critic agents in existing work. What we add is what the critic is required to do. It has to look up the definition of every Mathlib notion and the range of every hypothesis and quantifier before it concludes. It cannot object in prose, but has to write a compiling alternative statement. And its change is accepted only with a Lean proof or a sentence quoted from the paper.
These requirements are what find the hardest defects. With a simple prompt to check a statement against the reference paper and repair it, an off-the-shelf coding agent repairs the same 3 of the 5 defects in each of eight runs, and never repairs June p1 or June p23. With our lookup procedure, the same agent repairs those two in 12 of 16 tries (Table B). Critics that only return a verdict do no better. An ensemble of nine single LLM calls finds at most 3 of the 5 defects while flagging 15 of the 82 other, ostensibly clean, references (Table A).