acceptodds
Under review as a conference paper at ICLR 2027

Goedel-Scribe: Toward Faithful Statement Autoformalization via Adversarial Mutation

Abstract

AI systems now possess the capability to produce Annals-level mathematical content autonomously. This capability introduces an asymmetry in the review process wherein the pace of generation of mathematical content far exceeds the (human) review capacity of the field. One way forward for the community is to more heavily rely on *formalized* proofs in languages like Lean. A pivot to formalization also has benefits for autonomous AI generation as models can iteratively improve proof candidates without the need for human intervention until a final compiling proof is produced. However, this shifts the importance of verifying proofs to verifying *faithfulness of statements* in Lean. A compiling proof matters only insofar as it corresponds precisely to a natural language mathematical statement. Complicating this matter is the reality that most human mathematicians are not currently fluent in Lean, meaning reliable AI assistance in formalizing statements, or whole papers, is necessary. In this work, we outline reasons for shifting focus to (auto)formalization of mathematical statements, we describe several desiderata of a formalization process, and we present a lightweight and flexible pipeline for AI driven autoformalization of mathematical statements. Our pipeline outperforms competing AI autoformalization approaches by relying on kernel certified repairs to mis-formalizations, and, when tested, autonomously finds several mis-formalizations in existing Lean code, including code used in benchmarks, and code certified by human mathematicians.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Lean, AI for math, autoformalization, faithfulness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yTCSOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies an important problem: how to correctly formalize an informal mathematical statement. The interest is only in formalizing the statement of the theorem, and not the full proof.

There is a lot of work already on Autoformalization. This paper proposes the idea of using an "adversarial" agent to propose alternate formalizations - each alternate option must either be proved equivalent (using Lean) or justified (as a valid formalization) by quoting a sentence from the source. Apart from adversarial agents, the paper also uses tools, such as Lean compiler and library search, that are important; as well as, the idea of "chain of thought" whereby the agent tasked with proposing the initial formalization needs to explicitly state the meaning of every Lean concept. The judge also discards formalizations with trivial proofs.

If we remove the "proposer" step that proposes a formalization, and only use the "adversarial agent" and judge, the proposed pipeline can also be used to verify an existing formalization. The proposed autoformalization pipeline was used in this manner to find 5 defects in published formalizations.

Critical strengths
  • The autoformalization pipeline being proposed is quite thorough. It includes the drafting agent, the attacking agents, and the judge, which are served by degenerate proof checking and equivalence checking.
  • The use of 'chain of thought' reasoning being employed before generating the formalization is also interesting.
  • Finding statements with trivial proofs is also a good way to find incorrect formalizations.
  • There is a lot of knowledge about common mistakes to look for in the prompts, indicating that some nontrivial amount of effort was put in the building this autoformalization pipeline.
  • Finding 5 flaws in published autoformalization is also a good validation.
Critical weaknesses
  • It is not clear if there was a separation between 'train' and 'test' here -- what examples were used to build the prompts, pipeline, and the harness -- and what tests were held out and used only for evaluation.
  • At places it feels like certain steps/prompts were put in place to catch certain kinds of observed defects. For example, lines 374-377 describe "asking the judge about convention" explicitly, but it reads as if that change was introduced to catch the March p25 error.
  • Lean is one language to formalize; would the findings generalize to other formalization languages?
  • While "adversarial mutation" sounds something new, it is just a critic agent; and critic agents are often used in LLM-based multiagent systems.
  • The single call of a frontier model baseline is actually not bad (as described on Lines 421-424). Presumably it is much cheaper than what is proposed here. Cost implications of using the complex pipeline is also something that is missing here. +- "Adversarial mutation" has the same motivation as something like "diverse solution generation" -- it was done for autoformalization in this work- https://arxiv.org/pdf/2512.00997, which did diverse generation using different models and then used a judge to generate one. It lacked some insights that are present in this paper, such as, linking changes to source text.
Questions for authors

The first concern on leakage is the major one: the most compelling evidence is that the pipeline detected 5 issues in published formalizations, but some knowledge of the defects seems to have gone into the prompts and pipeline design.

4 comments
AuthorsOpenReview ↗

Response pt 1

We are very grateful for your comments, careful reading, and your time spent reviewing.

It is not clear if there was a separation between 'train' and 'test' here -- what examples were used to build the prompts, pipeline, and the harness -- and what tests were held out and used only for evaluation.

Thank you very much for bringing up this point. We initially developed the pipeline trying to formalize frontier natural language proofs from early in the summer (for example the unit distance lower bound disproof). We found strategies that seemed to work avoiding errors, and then tested these strategies on ArXivLean, where we found 5 potential misformalizations that we verified were indeed problematic. We adjusted prompts in minor ways after finding these references, and all were originally found by a procedure developed on other natural language proofs. We also test on other formalizations from competing systems (Theo), and have added 32 new papers from ArXivMath to our comparison (cf. Table C in the general response).

At places it feels like certain steps/prompts were put in place to catch certain kinds of observed defects. For example, lines 374-377 describe "asking the judge about convention" explicitly, but it reads as if that change was introduced to catch the March p25 error.

This is an oversight on our part, and we would like to clarify. We initially found it useful to ask about conventions, and specific meanings before testing on ArXivLean, and, as you can inspect, there are no specific references in the attacker prompts. We did experiment after finding defects in the ArXivLean benchmark with further papers, and when doing this, added examples we had found from ArXivLean to the judge prompt, which is what you see in our manuscript. But this is a was not originally present when we found the defects, and we have since confirmed that a) these are not necessary for the judging stage, and removing them does not change judging success and b) an attacker only ever sees general prompts about looking up definitions, etc., and the attacker is actually solely responsible for identifying the errors. For example in Table B in the general comment, we find that we can remove the prompts/stages you identify entirely without affecting the number of defects our pipeline catches. We apologize for this confusion and have clarified this in our working draft which we will upload once ongoing rebuttal experiments are finished/added.

Lean is one language to formalize; would the findings generalize to other formalization languages?

We have not run experiments outside Lean and Mathlib, so we cannot claim this yet, and we will state it as a limitation in the revision. However, we do expect the approach to carry over. The kernel step only needs a proof assistant that can check an implication between two statements, which Isabelle and Rocq also provide. The lookup procedure is tied to the library rather than to the language. Two of the five defects turn on what a Mathlib notion means: ContDiff ℝ ⊤ is analytic and not smooth (June p1), and Sylow p is defined for every natural number p (March p25). Any large formal library has conventions of this sort, so we would expect the same kind of misformalization there, but this remains to be tested.

While "adversarial mutation" sounds something new, it is just a critic agent; and critic agents are often used in LLM-based multiagent systems.

Our attackers are similar critic agents, and our working draft now identifies similarity to critic agents in existing work. What we add is what the critic is required to do. It has to look up the definition of every Mathlib notion and the range of every hypothesis and quantifier before it concludes. It cannot object in prose, but has to write a compiling alternative statement. And its change is accepted only with a Lean proof or a sentence quoted from the paper.

These requirements are what find the hardest defects. With a simple prompt to check a statement against the reference paper and repair it, an off-the-shelf coding agent repairs the same 3 of the 5 defects in each of eight runs, and never repairs June p1 or June p23. With our lookup procedure, the same agent repairs those two in 12 of 16 tries (Table B). Critics that only return a verdict do no better. An ensemble of nine single LLM calls finds at most 3 of the 5 defects while flagging 15 of the 82 other, ostensibly clean, references (Table A).

AuthorsOpenReview ↗

Response pt 2

The single call of a frontier model baseline is actually not bad (as described on Lines 421-424). Presumably it is much cheaper than what is proposed here. Cost implications of using the complex pipeline is also something that is missing here.

We now report cost and tokens. One call of GPT-5.6 Sol with the paper costs \0.05andusesabout19kinputand0.4koutputtokens.Thepipelinecostsamedianof0.05 and uses about 19k input and 0.4k output tokens. The pipeline costs a median of \\0.82 per statement (\$93 for the 89 references) and uses about 1.5M input tokens (mostly cached agent context) and 0.2M output tokens.

A single call is a useful first check, but it does not find the hardest defects. We have now run nine models with one call each. They find between 0 and 3 of the 5 defects, and none finds June p1 or June p23 (Table A). Spending more on single calls does not change this, as the ensemble of all nine costs \$0.47 per statement and still finds at most 3 of the 5. Also worth noting is, as run, a single call also only returns a verdict, and does not write a repaired statement.

If cost is the concern, one way to lower the cost of our pipeline is to run only a single attacker, skipping the kernel check and the judge stage. On the five defective references this repairs 4.9 ± 0.1 at \0.33perstatement,against4.8±0.2at0.33 per statement, against 4.8 ± 0.2 at \\3.83 for the full pipeline (Table B). We would also note that the cost of a misformalization grows downstream, as any compute spent proving the wrong statement is wasted. Having more options on the Pareto frontier is genuinely useful for mathematicians/practitioners.

"Adversarial mutation" has the same motivation as something like "diverse solution generation" -- it was done for autoformalization in this work- https://arxiv.org/pdf/2512.00997 [...]

Thank you for the reference, we have added it to our working draft. In IndiMathBench, twelve models each formalize an olympiad problem, GPT-5 ranks the candidates, and two human annotators choose, edit or merge them. The diversity there comes from independent drafts, and humans make the final judgement.

Our attackers do not write independent drafts. Each proposes a single change to a given statement and, as you note, has to link that change to the source text. We think this matters at the research level for two reasons. First, independent drafts are hard to compare. For 88 claims drafted by two different models, the kernel proved only 34 of the pairs equivalent, and a judge called 45 more the same statement. Second, diverse drafts can share the same misreading. None of the nine models flags ContDiff ℝ ⊤ in June p1 with a single call (Table A), and in three of the 32 arXiv papers of Table C, Codex CLI with GPT-6 Astra itself wrote ContDiff ℝ ⊤ where the paper means smooth.

The first concern on leakage is the major one: the most compelling evidence is that the pipeline detected 5 issues in published formalizations, but some knowledge of the defects seems to have gone into the prompts and pipeline design.

We apologize again for the confusion around this, and your concern is reasonable. To summarize our two answers above: In the judge prompt, there are two examples that were related to errors we had found in ArXivLean, but they were not originally part of our pipeline, and added after the fact as a list of examples in attempts to formalize separate papers. We can confirm however that a) removing the examples from the judge prompt does not affect the judging stage’s performance even on ArXivLean, b) removing the judge stage entirely, and simply using the attacker finds all the errors reliably, and c) our pipeline can reliably find misformalizations/produce better formalizations than other methods on examples outside the ArXivLean (cf. comparisons to Theo, and the ArXivMath results in Table C).

Reviewer_yTCSOpenReview ↗

acknowledging author's response

Thank you for your detailed response. One more clarification question: you state in your response - "If cost is the concern, one way to lower the cost of our pipeline is to run only a single attacker, skipping the kernel check and the judge stage. On the five defective references this repairs 4.9 ± 0.1 at $0.33 per statement, against 4.8 ± 0.2 at $3.83 for the full pipeline (Table B)." -- So, does this mean the leaner pipeline is not only more cost efficient, but also performs better on average than the full pipeline?

Thank you once more for your engagement and consideration.

So, does this mean the leaner pipeline is not only more cost efficient, but also performs better on average than the full pipeline?

It is cheaper, but we would not say it performs better. Over eight runs on the five defects, the single attacker repairs 39 of 40 and the full pipeline 38 of 40. This is a difference of one repair, which is within the variation between runs.

What Table B shows is that the repairs come from the attacker's lookup procedure, and that the later stages do not add repairs on these five references. What the later stages can add is a degree of confidence about the change, namely that no change is made without a Lean proof or a quoted sentence of the paper behind it. This isn't strictly necessary, and practitioners could simply opt to run our attack strategy to get a faithful formalization.

For reference, we also ran both of these settings on 20 other references, drawn at random. The single attacker changed 3 of the 20 in each of its two runs, and the full pipeline changed 1 and 1. However, none of these changes is a "false positive" as the statements were proved equivalent to the reference statement.

So the single attacker is definitely a reasonable option for lower cost, and the full pipeline just adds confidence/grounding in a Lean proof or quote from the paper. The full pipeline is also the setting we have tested most, with no harmful change in three runs over all 89 references (Table D).

Are there any other questions we can clarify? We do truly appreciate your continued engagement with our work.

Reviewer_2rVJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper's argument is that faithful statement formalization is what bottlenecks AI driven formal math, and it presents a pipeline for writing Lean 4 statements from research paper claims. A drafting agent with a compiler and Mathlib search writes the statement. A cheap prover then tries to prove it trivially, or tries to prove its negation. Two attacker agents propose one-change variants, each with a quote from the paper. The kernel discards the equivalent ones, and an LLM judge picks the variant the paper actually asserts. Run on 89 ArXivLean reference statements, the pipeline finds 5 that do not assert the paper's theorem and 2 that do not compile, repairs 53 of the 54 planted defects, and finds 2 defects in Theo's 8 released statements.

Critical strengths

Three of the five ArXivLean defects come with a kernel certificate (a degenerate proof for March p4, implications for p25 and June p1). The repairs get shown as diffs, and the authors read every changed reference by hand. Benchmark maintainers should act on this.

There is also a practical lesson here about auditing agents. What Table 2 shows is that attackers with tools who are just told to look for differences will miss the defects that depend on what a Mathlib notion actually means (analytic vs smooth, Sylow over naturals). It also shows that if you force the agent to print the definition of every notion and the range of every hypothesis before it concludes, those defects get recovered. That is useful for anyone building this kind of checker.

Critical weaknesses

Not clear what the novelty is. Faithfulness checking for autoformalization has been around for two years at least by now, and this kind of workflow is pretty much how people build these systems today. You draft with a compiler in the loop, try to break the statement, let a judge decide, repair, repeat. Proving both implications in Lean to decide whether two statements are the same thing is BEq (Liu et al., ICLR 2025, not cited). Proving the negation, or a trivial proof of the draft, is the negation filter from DeepSeek-Prover and FormalMATH. A draft, critique and repair loop with Lean plus a faithfulness judge on research level statements already exists in Theo (cited and compared). So I'm not sure what this adds on top, other than putting the pieces together with a judge that has to quote the paper. And the claim that there is "relatively little work" here is not accurate.

Would a single agent do the same? The paper never compares against one coding agent that gets the compiler, Mathlib search, and a /goal command like "formalize this claim faithfully, check every notion and every hypothesis range, keep going until you are confident it is faithful and compiles". The only simple baseline is a tool-less single call with a condensed prompt. What the big formalization projects of the last year suggest is that less orchestration works better, and that good tools and a clear protocol beat a carefully tuned multi-agent pipeline.

Effort is not matched. Cost gets reported only in dollars for the pipeline (about $3 for a defective one, $93 for the 89 references). There is no token counts, and the single call baselines cost cents and never get a comparable budget.

The sample size of the experiments is small. Everything "in the wild" rests on 5 defects in 89 ArXivLean references plus 2 in Theo's 8 statements, all from a single run. Only the 5 defective references were rerun. No confidence intervals, and no estimate of what got missed in the 84 references left alone. The planted defects are easier than the real ones, as Appendix A says. The Theo and Trellis comparison is on 15 statements.

Questions for authors

Faithfulness checks matter, no doubt about that. But it would help to clarify why someone would actually use this pipeline over just calling a coding agent, or just calling a stronger model. With the models available today, including the cheap open-source ones like DeepSeek and Qwen, the case for carefully tuned pipelines has gotten weaker, and a /goal command along the lines of "formalize this claim faithfully, check every notion and hypothesis, keep going until you are confident" already does most of the work. If the authors want to propose a new pipeline then the burden is on them to show it beats what people use now, on the same statements and with the same budget. So how many of the 5 real and 54 planted defects does such an agent actually find?

4 comments
AuthorsOpenReview ↗

Response pt 1

Thank you for your insightful comments and your time.

Not clear what the novelty is. Faithfulness checking for autoformalization has been around for two years at least by now, and this kind of workflow is pretty much how people build these systems today. [...]

Thank you for the references. We have added BEq and the two negation-filter papers (DeepSeek-Prover, FormalMATH) to our working draft and edited the "relatively little work" sentence, which we meant as a comparison to the amount of work done on the formal proving side. We will upload the revision once the remaining experiments are finished. The draft also states which elements we re-use from existing work, namely: drafting with the compiler in the loop, critic agents, the negation filter, proved equivalence and an LLM judge.

A premise of our paper is that these checks, as they are used today, are insufficient to catch pernicious misformalizations in research math. For example, 5 of the ArXivLean references and 2 of the statements released by Theo, a frontier autoformalization pipeline, contain such errors. The ArXivLean references had passed the benchmark's AI and human review, and Theo's errors had passed its own faithfulness judge and its expert validation. We find more evidence of this by running the published judge prompts of ReForm, FormalMATH, LeanScorer and Beyond Compilation on the five ArXivLean defects, and none finds more than 2 (Table A). A disprover doesn't catch any of the five errors, and three of them actually follow from the paper's own theorem, so they cannot be refuted. BEq scores a candidate against a reference that is taken as correct, whereas in our setting the reference is the statement under test.

Our pipeline adds two key insights. The agent has to look up the definition of every Mathlib notion and the range of every hypothesis and quantifier before it suggests a formalization, and a change to the statement is accepted only with a Lean proof or a sentence quoted from the paper. This procedure is what finds the hardest defects. The same coding agent repairs June p1 and June p23 in 0 of 16 tries without this procedure and in 12 of 16 with it (Table B).

Would a single agent do the same? The paper never compares against one coding agent that gets the compiler, Mathlib search, and a /goal command [...]

To address your comment, we ran this agent on the five defective references, with eight runs per setting (Table B). The agent is Codex CLI with GPT-5.6 Sol, the model of our first attacker, and it has a shell, access to the Lean compiler and the Mathlib source.

With a simple prompt to check a statement against the reference paper and repair it, only 3 of the 5 defects are repaired, the same 3 in every run. But with our lookup procedure, which spells out the checks you name (the definition of every notion and the range of every hypothesis), it repairs 4.5 ± 0.3, against 4.8 ± 0.2 for the full pipeline. One of our attackers deciding alone repairs 4.9 ± 0.1. Indeed, this is one way to lower the cost of our proposed pipeline (by running only a single attacker, and eschewing the kernel check and the judge stage).

The full pipeline does add a level of confidence to the formalization, and changes are required to come with a kernel certified proof or an explicit passage in the paper to back its change. In three full runs it made no harmful change to the other references (Table D).

Effort is not matched. [...]

We now report tokens alongside dollars. Per statement, the pipeline uses about 1.5M input tokens (mostly cached agent context) and 0.2M output tokens, at a median of \0.82overthe89references.OnecallofGPT−5.6Solwiththepaperusesabout19kand0.4k,at0.82 over the 89 references. One call of GPT-5.6 Sol with the paper uses about 19k and 0.4k, at \\0.05. On the five defective references, the coding agent uses about 185k input and 3k output tokens (\0.21)whenaskedviaasimpleprompt,andabout1.0Mand9k(0.21) when asked via a simple prompt, and about 1.0M and 9k (\\0.76) with our procedure.

We have also now tried to match our pipeline's budget in two ways. The first is comparing to an ensemble of nine single LLM calls that costs roughly \0.47perstatement,andfindsatmost3ofthe5defectswhileflagging15ofthe82other,ostensiblyclean,references(TableA).Wealsoreportperformanceofeightrunsofanoff−the−shelfcodingagentcosting0.47 per statement, and finds at most 3 of the 5 defects while flagging 15 of the 82 other, ostensibly clean, references (Table A). We also report performance of eight runs of an off-the-shelf coding agent costing \\1.71 per statement. The agent repairs the same 3 of the 5 in every run (Table B).

AuthorsOpenReview ↗

Response pt 2

The sample size of the experiments is small. [...]

The real defects we find are few partially because reviewed formal research-level statements are scarce. ArXivLean, Theo and Trellis are some of the only sources for such statements where it makes sense to compare performance. If you have any suggestions on research level statements that are likely to contain misformalizations, we would be happy to look into these.

However, to partially assuage concerns with the single run of our pipeline, we have now run the final pipeline three times on all 89 references (Table D). It repaired 13 of the 15 defective cases. The two misses were mechanical failures (for example an API call time limit). In these runs, our pipeline made a total of seven changes to the remaining 81 references, and the grader judged all seven equivalent to the original statement. We also ran the pipeline on the September edition of ArXivLean (53 new statements), where it changed 5, all rewordings, and found no defect.

Tables A and B now give the mean and standard error over runs. None of the other 81 references was changed for the worse in any of the three runs, which gives a 95% upper bound of 3.6% on the rate of harmful changes per reference. For what was missed in the references left unchanged, we are running a human audit, described in our reply to Reviewer 8PwX.

On the planted defects, a single LLM call flags 53 of the 54, so this set does not separate methods, but rather serves as a "first test" for any proposed method as this is the bare minimum for research-level statement formalizers.

[...] why someone would actually use this pipeline over just calling a coding agent, or just calling a stronger model. [...] So how many of the 5 real and 54 planted defects does such an agent actually find?

Of the 5 real defects, the agent repairs 3.0 ± 0.0 over eight runs, and 4.5 ± 0.3 with our procedure as its instructions (Table B). We did not run it on the 54 planted defects, since a single LLM call already flags 53 of them.

On open models, one call with the paper finds 1 of the 5 with DeepSeek V4 Pro, 3 with DeepSeek V4.1 Flash and 0 with Qwen3.8 27B (Table A). Our pipeline can use these models too. Its cheapest setup drafts and attacks with DeepSeek V4.1 Flash, with GPT-5.6 Sol as the judge, and this is the setup that Table C compares against Codex CLI with GPT-6 Astra (19 of our 32 statements are accepted, against 13 of Codex's). Overall, we think it’s very valuable for the community to have several options along the Pareto frontier. On one end, a single call of a frontier LLM may be able to catch easy errors, and give a reasonable first draft. But for more reliable formalizations, our pipeline offers a reasonable tradeoff. Spending a little extra in formalizing statements pays off downstream as wasted proving compute can be avoided.

We are writing to politely follow up from our response to your review. We appreciate your time and effort, and are keen to engage further. Do you have any remaining questions? If your questions have been resolved, we would very much appreciate if you would consider raising your initial score.

Reviewer_2rVJOpenReview ↗

Follow-Up After New Results

Thanks for the detailed response and the extra experiments. Especially the coding agent baseline in Table B, the token accounting, and the repeated runs. Several of my concerns has been addressed and the clarifications help. I'll take the new results into account in my final assessment.

Reviewer_8PwXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Gödel-Scribe, an agentic pipeline for improving the semantic faithfulness of mathematical statement autoformalization. The framework first generates a Lean statement, then employs adversarial agents to identify potential semantic discrepancies and propose localized modifications. A Lean-based verification stage checks the logical relationships between the original and modified statements, while an LLM judge determines which version better matches the source text. Experiments on synthetic defects and research-level mathematical statements demonstrate that the pipeline can identify and repair several errors in existing formalizations, including statements previously reviewed by human experts.

Critical strengths

1.Important and practically relevant problem. The paper addresses a critical challenge in autoformalization: syntactically valid Lean statements may not faithfully preserve the intended mathematical meaning. This problem becomes particularly important as automated theorem proving systems are increasingly applied to research-level mathematics. 2.Valuable empirical findings. The identification of five semantic defects in ArXivLean reference statements and two additional defects in Theo's released formalizations is a meaningful contribution. These case studies demonstrate that semantic errors can survive both automated checks and expert review. The combination of adversarial inspection, Mathlib exploration, and formal verification is also a practically useful engineering approach.

Critical weaknesses

W1. Incomplete discussion of closely related work and unclear incremental contribution. Although the paper discusses FormalEvolve and roundtrip verification, it omits several directly relevant approaches. Li et al. [1] combine theorem-prover-based logical equivalence checks with semantic consistency for candidate selection, which is directly relevant to this work: both methods use machine-checked proofs of logical equivalence between formal statements as evidence for assessing semantic faithfulness. LoC-Decomp [2] introduces fine-grained semantic checking and iterative correction driven by semantic and compiler feedback, while ReForm [3] integrates semantic self-validation and iterative self-correction into autoformalization. Gödel-Scribe incorporates some distinctive design choices. However, the paper does not adequately explain which limitations of existing semantic checking and correction frameworks these mechanisms overcome or what additional capabilities they enable. The authors should systematically position their approach against these closely related works, clearly distinguish established ideas from their proposed contributions, and substantiate the incremental benefits of the latter. Without this comparison, the methodological contribution remains difficult to assess independently of the valuable empirical findings.

W2. The experimental comparisons and ablations do not adequately establish the benefits of the proposed mechanisms. The comparisons in Table 3 are not sufficiently controlled to support the claim that Gödel-Scribe outperforms existing semantic checking approaches. The complete pipeline employs multiple agents, compiler access, Mathlib search, formal proof attempts, and repeated LLM judgments. In contrast, the frontier-model baselines receive only a single inference call. Consequently, the observed performance differences may reflect additional computational resources, tool availability, or problem-domain differences rather than the proposed methodological innovations. Furthermore, the ablations in Table 2 do not independently isolate the effects of the major components. In particular, the full-pipeline setting simultaneously introduces explicit inspection of mathematical definitions, hypothesis-range analysis, and additional convention-checking mechanisms. The study also lacks dedicated ablations of adversarial mutation, kernel-based settling, and the multi-agent design. I recommend evaluating stronger baselines under comparable computational and tool-use budgets, especially a tool-equipped semantic checking and iterative correction pipeline without adversarial mutation. More fine-grained ablations are also needed to establish which proposed components are responsible for the improvements.

W3. Insufficient independent validation of semantic faithfulness. Although the authors manually verify the identified ArXivLean defects, the evaluation of semantic faithfulness still relies substantially on LLM judgments. Notably, Appendix I reports a case where the independent grader confirmed a proposed correction that the authors subsequently determined was not a genuine defect. Moreover, the statements left unchanged are not systematically audited, making it difficult to assess false negatives and overall semantic correctness. I recommend a randomly sampled expert audit covering both modified and unchanged statements to evaluate or calibrate the reliability of the LLM-based assessment. This would remain feasible even with limited human resources.

References: [1] Li, Zenan, et al. "Autoformalize mathematical statements by symbolic equivalence and semantic consistency." Advances in Neural Information Processing Systems 37 (2024): 53598-53625. [2] Shi, Jiangze, et al. "LoC-Decomp: LLM Autoformalization via Logical Concept Decomposition and Iterative Feedback Correction." International Conference on Learning Representations. Vol. 2026. 2026. [3] Chen, Guoxin, et al. "Reform: Reflective autoformalization with prospective bounded sequence optimization." International Conference on Learning Representations. Vol. 2026. 2026.

Questions for authors

1.Can the authors independently evaluate the semantic correctness of the final statements, including those left unchanged? The current experiments mainly evaluate detected and proposed corrections, making it difficult to assess false negatives or the overall faithfulness of the pipeline's outputs. If a comprehensive human evaluation is too costly, the authors could conduct a randomly sampled expert audit to provide a more reliable assessment.

4 comments
AuthorsOpenReview ↗

Response pt 1

Thank you very much for your thoughtful comments and references.

[W1] Incomplete discussion of closely related work and unclear incremental contribution. Although the paper discusses FormalEvolve and roundtrip verification, it omits several directly relevant approaches [...]

Thank you for the references. We have added them to the related work in our working draft, and will upload the revision once the remaining experiments are finished. The three works you passed along all decide faithfulness by having a model adjudicate the statement, either directly or through a back-translation. Li et al. also prove equivalence in Lean, but between candidate formalizations, and faithfulness to the problem is still scored by the embedding similarity of a back-translation. A premise of our paper is that reading a statement, or back-translating it, is not enough to catch subtle misformalizations. The five defective ArXivLean references passed the benchmark's AI and human review, presumably by this sort of reading. We find more evidence of this by running the published judge prompts, ReForm's among them, on the ArXivLean references, and none finds more than 2 of the 5 (Table A). An example of this sort of subtle misformalization is the ContDiff ℝ ⊤ hypothesis of June p1, which "translates" as smooth but actually means analytic in Mathlib.

Our pipeline differs in two key ways. The agent has to look up the definition of every Mathlib notion and the range of every hypothesis and quantifier before it concludes, and a change to the statement is accepted only with a Lean proof or a sentence quoted from the paper. The lookup is what finds the hardest defects. With the model and tools fixed, a coding agent repairs June p1 and June p23 in 0 of 16 tries without it and in 12 of 16 with it (Table B). LoC-Decomp also checks a statement piece by piece, but each piece is again judged by a back-translation.

It's also worth noting that the domains differ. The three works are evaluated on competition and undergraduate benchmarks (MATH, miniF2F, ProofNet, PutnamBench, AIME). These are settings wherein direct judging and back-translation might work perfectly well. We audit research-level statements that others wrote and reviewed, and in this harder setting we find errors in 5 of the 89 ArXivLean references and in 2 of the 8 statements released by a competing frontier pipeline.

Following your comment, though, the draft also distinguishes these explicitly, and states which elements we take from existing work: drafting with the compiler in the loop, critic agents, the negation filter, proved equivalence and an LLM judge.

[W2] The experimental comparisons and ablations do not adequately establish the benefits of the proposed mechanisms. [...]

To address your comment, we added a baseline with the same model and tools, and one at comparable cost.

A coding agent with the model and tools of our first attacker (Codex CLI with GPT-5.6 Sol, a shell, the Lean compiler and the Mathlib source) repairs the same 3 of the 5 defects in each of eight runs and never repairs June p1 or June p23 (Table B). Its eight runs together cost $ 1.71 per statement, and it appears running the agent more often does not find the remaining flaws. Given our lookup procedure as its instructions, the same agent repairs 4.5 ± 0.3 at $0.76 per statement, against 4.8 ± 0.2 for the full pipeline. The tools are the same in both rows, and the eight plain runs cost more than a run with the procedure, so the gain comes from the procedure.

If we instead increase compute with an ensemble of nine single LLM calls ($0.47 per statement, against a median of $0.82 for our pipeline), the ensemble finds at most 3 of the 5 defects while flagging 15 of the 82 other references (Table A).

AuthorsOpenReview ↗

Response pt 2

Furthermore, the ablations in Table 2 do not independently isolate the effects of the major components. [...]

The table below separates the lookup procedure from the convention check.

Lookup procedureConvention checkDefects repaired (of 5)
nono2
noyes3 (March p4, March p25, June p20)
yesno4 (all but March p25)
yesyes5

Rows 1, 2 and 4 are full runs on the 89 references, and row 3 is constructed from the final run by removing the repair that the convention check adopted. The convention check accounts for March p25, and the lookup procedure for June p1 and June p23. Within the procedure, the definition lookup finds June p1 (17 of 18 runs with it, 0 of 6 without) and the hypothesis-range step finds June p23 (16 of 18 against 0 of 6).

For the three components you list, we report the following. The plain Codex agent of Table B is the closest we have to the pipeline without adversarial mutation. It has the same tools but is not asked to write alternative statements, and it has no kernel step and no second agent. It repairs 3.0 ± 0.0 of the five. For the multi-agent design, it turns out that one attacker deciding alone repairs 4.9 ± 0.1 of the five (Table B), so on these defects the second attacker and the judge add no repairs. A single attacker is therefore a cheaper configuration on these five ($ 0.33 against $ 3.83 per statement), and we report it in the revision. For kernel settling, in two full runs the kernel proved 3 of 29 and 6 of 33 proposed changes equivalent to the original statement and discarded them. This step cannot change the number of defects repaired, since a proposal that is provably equivalent to the original is not a repair. Its role is to remove such rewordings before the judge sees them.

I recommend evaluating stronger baselines under comparable computational and tool-use budgets, especially a tool-equipped semantic checking and iterative correction pipeline without adversarial mutation. [...]

The plain Codex row of Table B is the baseline you describe here. An agent with the same tools as our pipeline checks the statement and rewrites it until it is satisfied, with no explicit adversarial mutation. The second Codex row adds our pipeline’s instructions.

[W3] Insufficient independent validation of semantic faithfulness. [...]

The five defects do not rest on an LLM judgment. We confirmed all five by hand, and three of them (March p4, March p25 and June p1) carry a kernel certificate that depends on no model. For the modified statements, as the paper states, we read every change that the grader confirmed on a reference statement. The Theo instance of Appendix I is the one change where we disagreed with the grader, which is why we report it.

For the statements left unchanged, we are running the audit you recommend. Two readers, one of them an author, are reading 16 statements that our pipeline left unchanged, without being told which are which. Ten are chosen at random. The other six are the ones that a single call of GPT-5.6 Sol or Claude Opus 5 reported as defective, since a defect missed by our pipeline is most likely to be among them. We will post the result here as soon as the reading is done.

We have also run the full pipeline three times on the 89 references (Table D). It repaired 13 of the 15 defective cases. It changed 4, 1 and 2 of the other 81 references, and the grader judged all seven equivalent to the original statement.

[Q1] Can the authors independently evaluate the semantic correctness of the final statements, including those left unchanged? [...]

This is the audit described under W3.

AuthorsOpenReview ↗

Random audit of problems

We have gotten back results from a blind review of 10 randomly selected statements our pipeline classified as faithful, as well as 6 statements that either Claude Opus 5, or GPT 5.6 Sol called unfaithful that our pipeline called faithful.

The reader was given the claim, the paper and the Lean statement, and was not told how a statement was selected. We also checked each statement ourselves against the paper's theorem and the Mathlib definitions.

ArXivLean ProblemFrom ours (random) or Opus/SolReader ratingNote
June p43randomfaithful
June p30randomfaithful
June p3randomnot faithfulReader: "No integral appears in the formalization, but one is needed for the t-design definition." The file writes each sphere integral as its value. The values equal the integrals (checked exactly for dimensions 1 to 7).
June p29randomfaithful
June p48randomfaithful
March p32randomfaithful
June p15randomfaithful
June p41randomfaithful
March p6randomfaithful
June p45randomfaithful
March p15OpusfaithfulObjection: entries of modulus 1, where the paper's complex Hadamard matrices have entries in {1, -1, i, -i}. The file follows the benchmark's claim sentence, but the claim sentence departs from the paper, and the statement is trivially true (Fourier matrix).
June p21SolfaithfulObjection: uniform tightness, where the paper proves asymptotic tightness. The file follows the claim sentence, which says "tight".
June p47SolfaithfulObjection: natural instead of binary logarithm. Both entropies in the inequality use the same logarithm.
March p33SolfaithfulObjection: the graph omits the loopless field. The field has a default proof and the file compiles.
March p41SolfaithfulObjection: IsHamiltonian does not require a path. Mathlib defines it as visiting every vertex exactly once.
March p24Solnot faithfulObjection: natural logarithm instead of base 2, which the Θ(1) exponent absorbs. Reader: "The statement considers hypergraphs, but the proof only considers vertex sets." The file quantifies over k-uniform hypergraphs and gives the induced subgraph by its vertex set, as the paper does.

The reader judged 14 of the 16 faithful (9 of 10 of our statements, and 5 of 6 that were called unfaithful by one of two other frontier models). After internal review we maintain that the single statement the blind review disagreed with our pipeline on is still correct, and the unfaithful statement flagged by the reviewer is actually faithful. Our justifications are:

  • June p3 (Our pipeline flagged as faithful): The statement does not write the integral, but it requires each monomial of degree at most 5 to average to an explicit number, and these numbers are exactly the integrals of the monomials over the sphere, so it matches the paper’s definition of a 5-design.
  • March p24 (Sol or Opus flagged as unfaithful): The file does quantify over k-uniform hypergraphs (H is a set of k-element subsets of the N vertices), and it gives the induced subgraph by its vertex set, which is how the paper itself defines a t-independent set.

Based on these selected problems, our pipeline exhibits a much lower false positive rate (flagging statements as unfaithful when they are actually faithful) compared to frontier calls.

Are there any remaining questions we can address? We genuinely value your feedback, and if your questions are addressed, we would appreciate if you would reconsider your initial score

AuthorsOpenReview ↗

Following up

We're following up once more as the end of the review period is quickly approaching. Are there any unresolved questions or concerns that we can engage with? Please let us know if there are remaining obstacles that stand in the way of a higher score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank the reviewers for their time and feedback. Here we aggregate results that several reviewers inquired about including: a tool-using agent as a baseline, comparisons at equal cost, and more than one run. Our related replies to each review refer to the tables here.

We would also like to make two general points. First, unfortunately, there is no established benchmark for checking faithfulness of Lean statements at the research level. Prior work is mostly evaluated on competition style problems, and ArXivLean, whose 89 statements were written from research papers and reviewed by experts, is the best option we know of. Its statements are all presented as correct, so the real defects in our study are genuinely problems at the frontier. Second, the cost of a misformalized statement grows downstream, since a prover that is given a wrong statement spends its whole budget on a theorem that the paper does not state. Our method may incur slightly higher upfront costs compared to, for example, a single call of a frontier model, but having several options on the Pareto frontier for this task is crucial.

Table A. ArXivLean baselines

Here we report expanded results with several single model calls (later we compare to a frontier coding agent). We test on the five defective references our method found, as well as the 82 other ArXivLean references that compile. We know of no defects in the 82 remaining references, so we believe the number of flagged statements in this category serves as a rough proxy for false positives. Every model was tested with access to both the claim and the paper, and current API costs per statement are reported.

MethodDefects found (of 5)Other references flagged (of 82)Cost
Single LLM call, single call prompt from our paper
GPT-5.6 Sol25$0.05
GPT-5.6 Terra24$0.05
GPT-5.6 Luna22$0.01
Claude Opus 521$0.16
Claude Sonnet 502$0.06
Gemini 3.8 Flash11$0.02
DeepSeek V4 Pro (a)12$0.09
DeepSeek V4.1 Flash (a)32$0.01
Qwen3.8 27B02$0.01
Single LLM call, generic prompt
GPT-5.6 Sol30$0.05
Claude Opus 501$0.16
LLM judges from prior work, run on GPT-5.6 Sol
ReForm24$0.06
FormalMATH00$0.05
LeanScorer29$0.11
Beyond Compilation20$0.05
Trained checker
CriticLean-32B113GPU
Ensemble of the nine single calls
Any vote315$0.47
Majority vote20$0.47
Gödel-Scribe (3 runs) (b)4.3 ± 0.32.3 ± 0.9$0.82

Table B. Coding agent baseline on defective references

Here we run Codex CLI with GPT-5.6 Sol, the same model of our first attacker, with eight runs per method. The agent sees both the statement and the full paper, and has shell access and Lean compiler + Mathlib search (the same tools as our proposed method). Interestingly, we find that our method can be made cheaper without compromising capability by limiting the number of attackers, and eliminating the final judging round.

MethodDefects repaired (of 5)June p1June p23Cost
Codex3.0 ± 0.00/80/8$0.21
Codex + our procedure4.5 ± 0.36/86/8$0.76
Single attacker (ours)4.9 ± 0.18/87/8$0.33
Gödel-Scribe4.8 ± 0.28/87/8$3.83

Reported mean ± standard error over the eight runs.

Table C. Formalization from scratch on 32 arXiv papers

To expand the set of research-level questions we attempt to formalize beyond the 89 ArXivLean problems (as well as the other research papers on which we compare to other methods), we further test on a set of 32 arXiv papers from the ArXivMath benchmark. Here, we compare to Codex CLI (which was run at the time with GPT-6 Astra at its highest reasoning effort) given each paper and a general formalization prompt. We compare our pipeline on the same main theorems (With DeepSeek V4.1 Flash drafters and attackers, and GPT-5.6 Sol judges). In the short time window, we were not able to find third party Lean graders, so instead, we opted for LLM grading, in our case a committee of Claude Opus 5 and GPT-5.5 ran three times each. This is certainly not unassailable evidence for our method’s performance, but it serves as another datapoint that our method can expand the Pareto frontier of statement autoformalization.

MethodAccepted by both graders (of 32)Cost
Gödel-Scribe, DeepSeek V4.1 Flash19$102
Codex CLI, GPT-6 Astra13$50

Cost and tokens

The final run on the 89 references cost $93, a median of $0.82 per statement, with about 1.5M input tokens (mostly cached agent context) and 0.2M output tokens per statement. One call of GPT-5.6 Sol with the paper uses about 19k input and 0.4k output tokens and costs $0.05.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.