acceptodds
Under review as a conference paper at ICLR 2027

Self-Evolving Decomposition for Multi-Agent Proof Autoformalization

12 views

Abstract

Large language models are beginning to produce proofs at the frontier of mathematics, whose correctness can no longer rest on human review alone. Formal verification in Lean has thus become the standard for certifying such results, and autoformalization is the bridge from informal proofs to verifiable ones. Existing efforts mostly formalize only the theorem statement. However, verifying the reasoning requires formalizing the entire proof, which is crucial for frontier mathematics yet remains underexplored. Current approaches either train models at high cost or retry blindly at inference time. We introduce ToMap, a pipeline of a Decomposer, a Formalizer, and a Prover that evolves at test time under the guidance of rubrics and Lean verification. Bottleneck analysis identifies the Decomposer as the critical stage, since its proof units govern whether downstream agents succeed. ToMap therefore keeps the Formalizer and the Prover fixed and spends its test time computing on the decomposition alone. At its core, ToMap maintains a pool of candidate decompositions, scores each one with rubrics, and evolves those on the Pareto frontier. A frontier candidate whose rubric scores all exceed a threshold is then verified in Lean. If it fails, its diagnostics guide the next round. The decomposition thus self-evolves at test time. On ProofFlowBench and miniF2F, ToMap improves verified and faithful formalization by an average of 20.3% in absolute terms over the strongest training-free baseline, with less running time, and most of the improvement arrives within the first few iterations.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Self-evolution, Large Language Model, Autoformalization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VkPhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ToMAP to make the decomposition self-evolving while keeping the formalizer and prover fixed. It has a global review to route back both rubrics and compiler feedback to the decomposition stage. On proofflow benchmark and miniF2F benchmark, it improves verified and faithful formalization by 17.3% and 23.3%.

Critical strengths

1.Experiments show the changing proof decomposition is better than changing formal statements or proof scripts.

2.Experiments on both Lean verification and semantic faithfulness improve 17.3% and 23.3% on two benchmarks.

Critical weaknesses

1.Changing only the decomposer performs slightly better changing the Formalizer, 9.2% vs 8.7%. It lacks ablation studies of Pareto selection, rubric scoring and comparison with repeated sampling under the same computing budget.

2.Semantic faithfulness is judged by GPT-5.5, without human checking. It is unfair to compare different models under different computing budgets.

Questions for authors

1.Can you compare the best-of-n and iterative refinement under the same computing budget? Can you add key ablation studies? Is 0.5% improvement significant?

2.Can human experts validate semantic faithfulness? Please report the agreement and error rates.

The authors added comparisons of best-of-10 and iterative refinement. It is helpful. However, It also lacks the agreement between the experts. No statistical test is reported to show the 0.5% improvement is significant. The new experiment changes the feedback condition (lean vs lean+semantic), it may gain from the semantic feedback.

5 comments

We thank Reviewer VkPh for the careful review and for suggesting additional controls on how the test-time budget is spent.

W1. Evidence for prioritizing decomposition in Figure 2

We sincerely appreciate you raising this point and understand your concern about the small SC∧SF gap between D-Intervene and F-Intervene in Figure 2. In this exploratory experiment, we use GPT-5.5 in the Codex agent harness with Lean feedback alone. For this rebuttal, we add semantic feedback so the correction model can compare its candidate with the source statement and proof, checking whether it preserves the assumptions, intermediate claims, and reasoning dependencies. To see whether the same pattern holds with a second model, we repeat the comparison with GPT-5.6-Sol under both feedback conditions. With semantic feedback, revising the decomposition shows a larger SC∧SF advantage over revising the formal statements on both models, supporting our focus on decomposition.

Both models use high reasoning effort for generation and correction. Within each model setting, all three interventions start from the same initial outputs on all 184 ProofFlowBench problems. The table below reports cumulative SC∧SF after five corrections.

Table: Cumulative SC∧SF after five corrections on ProofFlowBench

ModelFeedbackD-Intervene (%)F-Intervene (%)P-Intervene (%)
GPT-5.5Lean only9.28.77.6
GPT-5.5Lean + semantic17.49.88.2
GPT-5.6-SolLean only17.912.514.1
GPT-5.6-SolLean + semantic25.513.614.1

With semantic feedback, D-Intervene achieves higher SC∧SF than F-Intervene on both GPT-5.5 (17.4% vs. 9.8%) and GPT-5.6-Sol (25.5% vs. 13.6%). Across the two models, D-Intervene outperforms F-Intervene by 9.8 percentage points on average, which supports prioritizing decomposition when the goal is to produce Lean proofs that faithfully follow the source argument.

We also examine which stages contribute to SC∧SF failures before any correction in the Figure 2 setting. This analysis covers the 171 ProofFlowBench problems and 217 miniF2F problems that fail SC∧SF on the first attempt. Drawing on ProofFlow’s step-level error analysis [1], we use GPT-6 Astra to inspect each stage’s inputs and outputs and assess whether an error at that stage contributes to the final SC∧SF failure. A problem can appear in more than one row when errors at multiple stages contribute.

Table: Stage contributions to first-attempt SC∧SF failures

StageWhat GPT-6 Astra checksProofFlowBench (171 failures)miniF2F (217 failures)
DecomposerDoes the decomposition omit or alter an assumption, intermediate claim, or reasoning dependency in a way that contributes to the final failure?110/171 (64.3%)99/217 (45.6%)
FormalizerDoes a malformed Lean declaration or a change in the meaning of an obligation contribute to the final failure?83/171 (48.5%)90/217 (41.5%)
ProverDoes the Prover contribute to the final failure by leaving a correctly specified obligation unproved, producing a proof that fails Lean verification, or departing from the source reasoning?59/171 (34.5%)67/217 (30.9%)

Decomposer errors are the most frequent on both benchmarks, contributing to 64.3% of first-attempt SC∧SF failures on ProofFlowBench and 45.6% on miniF2F. Together with the intervention results, this supports prioritizing decomposition when the goal is to produce Lean proofs that faithfully follow the source argument. We have added this analysis of first-attempt failures to the revised manuscript.

[1] Rafael Cabral et al. ProofFlow: A Dependency Graph Approach to Faithful Proof Autoformalization. ICLR 2026.

W1 & Q1. Ablations, alternative baselines, and inference cost

We appreciate your suggestion to control for model choice and inference budget when comparing ToMap with the baselines. In the updated manuscript, we revise the baselines to align the models at each stage and also add comparisons. All six methods in the tables below use Qwen3-30B-A3B-Instruct-2507 as the Decomposer, Goedel-Formalizer-V2-32B as the Formalizer, and Goedel-Prover-V2-32B as the Prover.

The sampling methods use ten independent attempts per problem, while iterative refinement and ToMap run for up to ten rounds. Even with these limits, actual token use varies with proof length, retries, and scoring calls. We therefore report the output tokens each method consumes in thousands per problem (OT, k/problem).

  • Repeated sampling and Best-of-10 use the same ten independently generated proofs for each problem. Each attempt starts from the original statement and informal proof. Once all ten candidates are available, repeated sampling reports their mean performance. Best-of-10 selects the Lean-valid candidate with the highest internal semantic score, or the highest-scoring candidate overall if none passes Lean. Its OT includes the cost of selection.

  • Iterative refinement operates on the full pipeline for up to ten rounds. After each attempt, the pipeline takes the current output and its feedback as input to the next round, which can revise the decomposition, formal statements, or proof scripts. We compare Lean feedback alone with Lean plus semantic feedback. Both variants start from the same initial outputs and use the same round and downstream retry limits.

  • ToMap without rubric scoring or Pareto selection keeps a single verification gate. The Decomposer revises one decomposition for up to ten rounds using Lean diagnostics and its own semantic feedback on whether the decomposition preserves the reasoning in the source proof.

The tables below cover all 184 ProofFlowBench problems and all 244 miniF2F problems. OT includes generation, revisions, downstream calls and retries, and scoring or selection where applicable. We exclude the final SF evaluation from OT for every method.

Table: Performance and output tokens per problem on ProofFlowBench

MethodSC (%)SF (%)SC∧SF (%)OT (k/problem)
Repeated sampling (mean of 10)51.819.613.6198.5
Best-of-10 (same candidates)57.622.816.8199.65
Iterative refinement (Lean only)62.523.419.6120.64
Iterative refinement (Lean + semantic)51.626.121.7181.53
ToMap without rubric scoring or Pareto selection28.827.719.6113.0
ToMap-Qwen30B66.332.127.798.8

Table: Performance and output tokens per problem on miniF2F

MethodSC (%)SF (%)SC∧SF (%)OT (k/problem)
Repeated sampling (mean of 10)69.831.125.4169.2
Best-of-10 (same candidates)7535.729.9171.23
Iterative refinement (Lean only)77.528.326.2128.16
Iterative refinement (Lean + semantic)69.333.632.4138.17
ToMap without rubric scoring or Pareto selection54.537.728.379.6
ToMap-Qwen30B86.9414183.3

Selecting the best of ten independent candidates improves SC∧SF over repeated sampling on both benchmarks. ToMap-Qwen30B improves further over Best-of-10, from 16.8% to 27.7% on ProofFlowBench and from 29.9% to 41.0% on miniF2F, while using about half as many output tokens per problem. ToMap also exceeds full-pipeline iterative refinement with Lean and semantic feedback by 6.0 and 8.6 percentage points, respectively, while using fewer tokens on both benchmarks.

In the ablation, the Decomposer relies on its own semantic feedback to revise a single candidate. Without Pareto selection, greedy revision may encourage reward hacking. Also, when the naive semantic feedback gives little guidance on what to change, the same issue can persist across several rounds. These additional revisions consume tokens and may outweigh the savings from removing rubric scoring. Together, the comparisons support the effectiveness of ToMap’s self-evolving decomposition pipeline. With the same models, guiding revisions with semantic rubrics and selecting candidates from the Pareto frontier yields more faithful, Lean-verified proofs than either independent sampling or full-pipeline iterative refinement.

W2 & Q2. SF evaluator accuracy

Thank you for suggesting that we validate our SF judgments with another model and with human experts. Our use of an LLM to evaluate SF follows prior work on autoformalization. CRAMF [2] and ATLAS [3] use LLM judges to check whether formal statements preserve the meaning of the source statements. ProofFlow [1] reports strong agreement with expert labels on 200 individual proof steps, and ReForm [4] reports 85.8% accuracy for Gemini-2.5-Pro on 859 expert-labeled statement pairs. These results support using LLMs as semantic judges for formal statements and proof steps. For this rebuttal, we examine how well this approach works for complete proofs.

Our original evaluation uses GPT-5.5 with xhigh reasoning effort, with GPT-6 Astra reviewing difficult or ambiguous cases. We also use the latest model, GPT-6.1-Sol, to rescore the same outputs with the same reasoning effort and Astra review procedure. This covers all six baselines and both ToMap methods on all 184 ProofFlowBench problems and 244 miniF2F problems. Both evaluations take place after candidate selection. For each output, we combine the new SF label with the original Lean verdict to compute SC∧SF.

Table: GPT-6.1-Sol reevaluation on both benchmarks

MethodProofFlowBench SF (%)ProofFlowBench SC∧SF (%)miniF2F SF (%)miniF2F SC∧SF (%)
ProofBridge21.720.727.525.8
Codex End2End25.017.932.026.6
M2F37.010.345.97.0
Codex D-Correction22.311.430.319.3
Monotonic19.616.831.131.1
ProofFlow40.821.250.034.0
ToMap-Qwen30B31.023.440.236.9
ToMap-Gemini52.742.462.758.2

The GPT-6.1-Sol evaluation supports the same conclusions as the original GPT-5.5 evaluation. On both benchmarks, ToMap-Gemini ranks first on SF and SC∧SF, and ToMap-Qwen30B outperforms all six baselines on SC∧SF.

To check how often each evaluator agrees with experts on individual proofs, we also conduct a human evaluation. Given the time available during the rebuttal period, we limit this evaluation to 40 complete ToMap-Gemini proofs, 20 from each benchmark. Two PhD researchers specializing in mathematical formalization assess the proofs. Both have prior experience annotating full-proof autoformalization outputs for foundation model training. For each item, they compare the generated Lean proof with the source statement and informal proof. They assign a positive SF label when the Lean proof preserves the source assumptions, essential intermediate claims, and reasoning dependencies.

The table below reports the error rate for each evaluator. We count an error whenever its final SF label differs from the expert label. We sincerely hope that these additional results address your concern about the reliability of our SF evaluation for complete proofs.

Table: SF evaluator error rates against expert judgments

Evaluation setGPT-5.5 + GPT-6 AstraGPT-6.1-Sol + GPT-6 Astra
ProofFlowBench20.0% (4/20)10.0% (2/20)
miniF2F10.0% (2/20)5.0% (1/20)
Overall15.0% (6/40)7.5% (3/40)

References

[1] Rafael Cabral et al. ProofFlow: A Dependency Graph Approach to Faithful Proof Autoformalization. ICLR 2026.

[2] Wangyue Lu et al. Automated Formalization via Conceptual Retrieval-Augmented LLMs. ICLR 2026.

[3] Xiaoyang Liu et al. ATLAS: Autoformalizing Theorems through Lifting, Augmentation, and Synthesis of Data. NeurIPS 2025.

[4] Guoxin Chen et al. ReForm: Reflective Autoformalization with Prospective Bounded Sequence Optimization. ICLR 2026.

Reviewer_V5PwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ToMap, a multi-agent framework for translating informal mathematical proofs into Lean. It focuses on improving proof decomposition while keeping the Formalizer and Prover fixed. The method uses rubric-based evaluation, Pareto selection, and Lean feedback to iteratively refine proof decompositions at test time. Experiments on ProofFlowBench and miniF2F show improvements in both verification success and semantic faithfulness compared with existing methods.

Critical strengths
  1. The paper addresses an important distinction between proving a theorem and faithfully formalizing its original argument. Revising upstream decompositions is also a reasonable strategy for correcting missing assumptions and dependencies.
  2. The experimental results are encouraging, particularly the improvements in jointly achieving syntactic correctness and semantic faithfulness. The authors also conduct stage-wise intervention experiments to explain why the Decomposer deserves more attention. This analysis is useful for understanding the different roles of agents in the formalization pipeline.
Critical weaknesses
  1. The paper provides evidence that revising the decomposition can recover failed proofs, but does not quantify what proportion of all initially failed problems originate from decomposition errors rather than formalization or proof-search failures. An audit across both benchmarks would strengthen the motivation for prioritizing upstream optimization.
  2. The evidence from the weak-link analysis is not entirely convincing when considering semantic faithfulness. In Figure 2, D-Intervene clearly outperforms the other interventions in syntactic correctness, but its advantage over F-Intervene in faithful verification is relatively small (9.2% versus 8.7%). Given that faithful verification is the main objective of this paper, I think this difference deserves further discussion.
  3. The evaluation of semantic faithfulness relies heavily on LLM-based judgments. Although the authors provide detailed evaluation instructions, it remains unclear how reliable these judgments are across different models or evaluators.
Questions for authors
  1. Please report the proportions of initial failures caused by decomposition, formalization, and proof search on both benchmarks, with sample sizes and attribution criteria.
  2. Could the authors provide some evidence of the reliability of the semantic faithfulness evaluation, such as agreement with human judgments on a subset of examples?
5 comments

We thank Reviewer V5Pw for the careful review and constructive feedback.

W1 & Q1. Attribution of initial failures on both benchmarks

Thank you for suggesting an audit of the initial failures on both benchmarks. To address this directly, we examine the first attempt, before any correction in the Figure 2 setting. 171 of 184 ProofFlowBench problems and 217 of 244 miniF2F problems fail SC∧SF on the first attempt. These initially failed problems form the denominator for each benchmark.

For each failed problem, GPT-6 Astra inspects the inputs and outputs of the Decomposer, Formalizer, and Prover. Informed by ProofFlow’s step-level error analysis [1], we use the criteria below to assess whether each stage introduces an error that contributes to the final SC∧SF failure.

Table: Criteria for attributing first-attempt SC∧SF failures

StageGPT-6 Astra’s attribution criterion
DecomposerDid the decomposition omit or change a required assumption, intermediate claim, or reasoning dependency in a way that contributed to the final SC∧SF failure?
FormalizerDid the Formalizer contribute to the final SC∧SF failure by producing a malformed Lean declaration or changing the quantifiers, domains, assumptions, or conclusion of an obligation it received?
ProverDid the Prover contribute to the final SC∧SF failure by leaving a correctly specified obligation unproved, producing a proof that failed Lean verification, or departing from the reasoning in the source proof?

A problem can receive more than one stage label when multiple stages introduce contributing errors. For example, suppose the Decomposer omits a required assumption from one claim, and the Formalizer translates "at least one" as "exactly one" in another claim. If both errors contribute to the final failure, both stages receive a label. The table below reports the resulting proportions.

Table: Stage contributions to first-attempt SC∧SF failures

StageProofFlowBench (171 failures)miniF2F (217 failures)
Decomposer110/171 (64.3%)99/217 (45.6%)
Formalizer83/171 (48.5%)90/217 (41.5%)
Prover59/171 (34.5%)67/217 (30.9%)

Decomposer errors are the most frequent on both benchmarks. The difference is larger on ProofFlowBench. On miniF2F, Formalizer errors are also common, contributing to 41.5% of initial failures compared with 45.6% for the Decomposer. We have added this analysis of first-attempt failures to the revised manuscript.

W2. The SC∧SF difference in Figure 2

We agree that the small SC∧SF gap between D-Intervene and F-Intervene in Figure 2 needs more discussion. In this exploratory experiment, we use GPT-5.5 in the Codex agent harness with Lean feedback alone. For this rebuttal, we add semantic feedback so the correction model can also check its candidate against the source statement and proof. To see whether the same pattern holds with a second model, we repeat the comparison with GPT-5.6-Sol under both feedback conditions. In our follow-up experiment with semantic feedback, this gap grows to 9.8 percentage points on average across GPT-5.5 and GPT-5.6-Sol, in favor of D-Intervene. Both models use high reasoning effort for generation and correction.

Within each model setting, D-Intervene, F-Intervene, and P-Intervene start from the same outputs on all 184 ProofFlowBench problems. To provide semantic feedback, the correction model compares its candidate with the source statement and proof, checking assumptions, intermediate claims, and their dependencies. The table below reports cumulative SC∧SF after five corrections.

Table: SC∧SF after five corrections on ProofFlowBench

ModelFeedbackD-Intervene (%)F-Intervene (%)P-Intervene (%)
GPT-5.5Lean only9.28.77.6
GPT-5.5Lean + semantic17.49.88.2
GPT-5.6-SolLean only17.912.514.1
GPT-5.6-SolLean + semantic25.513.614.1

Adding semantic feedback widens the D-Intervene advantage over F-Intervene from 0.5 to 7.6 percentage points with GPT-5.5 and from 5.4 to 11.9 percentage points with GPT-5.6-Sol, using the percentages shown in the table. With semantic feedback, D-Intervene outperforms F-Intervene by 9.8 percentage points on average across the two models. The first-attempt error attribution in W1 & Q1 provides complementary evidence for prioritizing decomposition: among proofs that fail SC∧SF before any correction, the Decomposer is the most frequent contributor on both ProofFlowBench (64.3%) and miniF2F (45.6%). We have added this analysis of first-attempt failures to the revised manuscript.

W3 & Q2. Reliability of semantic faithfulness evaluation

We fully understand your concern. Our use of an LLM to evaluate SF follows prior work on autoformalization. CRAMF [2] and ATLAS [3] rely on LLM judgments for the semantic checks in their automated evaluation of formal statements. ProofFlow [1] reports strong agreement between LLM and expert judgments on 200 individual proof steps, and ReForm [4] reports 85.8% accuracy on 859 expert-labeled statement pairs. These findings support the use of frontier LLMs as semantic judges for formal statements and individual proof steps.

In the intervention experiments, GPT-5.5 and GPT-5.6-Sol both use high reasoning effort for generation and correction. After candidate selection, we evaluate SF using GPT-5.5 with xhigh reasoning effort, with GPT-6 Astra reviewing difficult or ambiguous cases before we finalize the labels.

For this rebuttal, we also use the latest model, GPT-6.1-Sol with xhigh reasoning effort, to rescore the same outputs under the same evaluation procedure, and GPT-6 Astra reviews difficult or ambiguous cases in both evaluations. We rescore the same outputs from all six baselines and both ToMap methods on all 184 ProofFlowBench and 244 miniF2F problems. For each output, SC∧SF combines the new SF label with its original Lean verdict.

Table: GPT-6.1-Sol reevaluation on both benchmarks

MethodProofFlowBench SF (%)ProofFlowBench SC∧SF (%)miniF2F SF (%)miniF2F SC∧SF (%)
ProofBridge21.720.727.525.8
Codex End2End25.017.932.026.6
M2F37.010.345.97.0
Codex D-Correction22.311.430.319.3
Monotonic19.616.831.131.1
ProofFlow40.821.250.034.0
ToMap-Qwen30B31.023.440.236.9
ToMap-Gemini52.742.462.758.2

The GPT-6.1-Sol evaluation supports the same conclusions as the original GPT-5.5 evaluation. On both benchmarks, ToMap-Gemini ranks first on SF and SC∧SF, and ToMap-Qwen30B outperforms all six baselines on SC∧SF.

To further assess whether the evaluators correctly judge individual proofs, we compare their final SF labels with expert judgments. Our comparison assesses complete proofs, including whether the generated proof preserves the dependencies between steps.

Two PhD researchers specializing in mathematical formalization evaluate 40 complete ToMap-Gemini proofs, 20 from each benchmark. Both have prior experience annotating full-proof autoformalization outputs for foundation model training. For each item, they compare the generated Lean proof with the source statement and informal proof. A positive SF label requires preservation of the source assumptions, essential intermediate claims, and reasoning dependencies.

The table below reports accuracy against these expert labels. On the same 40 proofs, GPT-5.5 with GPT-6 Astra review correctly labels 34/40 (85.0%), and GPT-6.1-Sol with GPT-6 Astra review correctly labels 37/40 (92.5%).

Table: SF evaluator accuracy on 40 expert-labeled ToMap-Gemini proofs

Evaluation setGPT-5.5 (xhigh) + GPT-6 AstraGPT-6.1-Sol (xhigh) + GPT-6 Astra
ProofFlowBench, 20 outputs80.0% (16/20)90.0% (18/20)
miniF2F test, 20 outputs90.0% (18/20)95.0% (19/20)
Overall, 40 outputs85.0% (34/40)92.5% (37/40)

References

[1] Rafael Cabral et al. ProofFlow: A Dependency Graph Approach to Faithful Proof Autoformalization. ICLR 2026.

[2] Wangyue Lu et al. Automated Formalization via Conceptual Retrieval-Augmented LLMs. ICLR 2026.

[3] Xiaoyang Liu et al. ATLAS: Autoformalizing Theorems through Lifting, Augmentation, and Synthesis of Data. NeurIPS 2025.

[4] Guoxin Chen et al. ReForm: Reflective Autoformalization with Prospective Bounded Sequence Optimization. ICLR 2026.

Dear Reviewer V5Pw,

Thank you again for your time and effort! With the discussion deadline approaching, could you let us know whether these results address your main concerns? We would welcome the opportunity to clarify anything that remains open.

Best regards,

The Authors

Reviewer_KxkDOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents ToMap, a training-free framework for full-proof autoformalization in Lean. It combines a Decomposer, a Formalizer, and a Prover, and iteratively revises the proof decomposition while keeping the downstream models fixed. Candidate decompositions are assessed using semantic rubrics, maintained through Pareto selection, and submitted to the downstream pipeline after passing a quality gate. Lean diagnostics guide subsequent revisions. The paper motivates this design through stage-wise intervention experiments and reports improvements in verified and faithful formalization on ProofFlowBench and miniF2F.

Critical strengths

Evaluation distinguishes formal validity from proof faithfulness.

  • The paper explicitly evaluates whether an output both passes Lean and preserves the supplied informal argument. This distinction is consequential for full-proof autoformalization, where proving a related or simplified proposition does not establish successful translation. Evaluation on two existing benchmarks, together with runtime and token reporting, provides useful evidence of the system’s effectiveness and computational requirements.
Critical weaknesses

The bottleneck diagnosis is stronger for compilation than for faithful formalization.

  • In Figure 2, Decomposer intervention clearly outperforms Formalizer intervention on syntactic correctness. However, their final SC∧SF scores are approximately 9.2% and 8.7%, a difference of roughly one problem out of 184. This provides limited support for identifying the Decomposer as the highest-leverage stage for the paper’s actual objective. Figure 4 also reports only syntactic correctness, leaving unclear whether the gains from additional evolution rounds similarly improve faithful formalization.
Questions for authors

What improvement does the proposed evolution mechanism provide over direct Lean-feedback-guided decomposition revision under the same models and iteration budget? In particular, how does ToMap compare with the D-Intervene procedure from Section 4 under matched settings?

Can the authors provide SC∧SF results for the iteration budgets in Figure 4? These would clarify whether additional evolution rounds improve faithful formalization alongside compilation success, particularly given the small separation between Decomposer and Formalizer intervention on this metric in Figure 2.

6 comments

We thank Reviewer KxkD for the careful review and constructive feedback.

W1. Evidence for prioritizing decomposition in Figure 2

We understand your concern that Figure 2 shows a clearer advantage for D-Intervene on SC than on SC∧SF. Figure 2 is an exploratory experiment that uses GPT-5.5 in the Codex agent harness with Lean feedback alone. In this setting, D-Intervene and F-Intervene achieve 9.2% and 8.7% SC∧SF, respectively, a difference of one problem out of 184.

To examine whether revising the decomposition also helps preserve the source reasoning, we add semantic feedback and evaluate both feedback conditions with GPT-5.6-Sol as well. Adding semantic feedback widens D-Intervene’s lead over F-Intervene on SC∧SF with both GPT-5.5 and GPT-5.6-Sol. Both models use high reasoning effort for generation and correction. For semantic feedback, the correction model compares its candidate with the source statement and proof, checking assumptions, intermediate claims, and their dependencies. Within each model setting, all three interventions start from the same initial outputs.

With semantic feedback, D-Intervene yields proofs that satisfy SC∧SF on 14 more problems than F-Intervene with GPT-5.5 and 22 more with GPT-5.6-Sol. Across the two models, this amounts to an average of 18 additional problems out of 184, or 9.8 percentage points. Together, these results support prioritizing decomposition when the goal is to produce Lean proofs that faithfully follow the source argument.

Table: SC∧SF after five corrections on ProofFlowBench

ModelFeedbackD-Intervene (%)F-Intervene (%)P-Intervene (%)
GPT-5.5Lean only9.28.77.6
GPT-5.5Lean + semantic17.49.88.2
GPT-5.6-SolLean only17.912.514.1
GPT-5.6-SolLean + semantic25.513.614.1

We also examine which stages contribute to failure on the first attempt, before any correction in the Figure 2 setting. This analysis covers the 171 of 184 ProofFlowBench problems and 217 of 244 miniF2F problems that fail SC∧SF. Drawing on ProofFlow’s step-level error analysis [1], we use GPT-6 Astra to inspect each stage’s inputs and outputs against the criteria below. A problem can appear in more than one row when multiple stages contribute to its failure.

Table: Criteria for attributing first-attempt SC∧SF failures

StageGPT-6 Astra’s attribution criterion
DecomposerDid the decomposition omit or change a required assumption, intermediate claim, or reasoning dependency in a way that contributed to the final SC∧SF failure?
FormalizerDid the Formalizer contribute to the final SC∧SF failure by producing a malformed Lean declaration or changing the quantifiers, domains, assumptions, or conclusion of an obligation it received?
ProverDid the Prover contribute to the final SC∧SF failure by leaving a correctly specified obligation unproved, producing a proof that failed Lean verification, or departing from the reasoning in the source proof?

Table: Stage contributions to first-attempt SC∧SF failures

StageProofFlowBench (171 failures)miniF2F (217 failures)
Decomposer110/171 (64.3%)99/217 (45.6%)
Formalizer83/171 (48.5%)90/217 (41.5%)
Prover59/171 (34.5%)67/217 (30.9%)

Decomposer errors are the most frequent on both benchmarks, contributing to 64.3% of first-attempt SC∧SF failures on ProofFlowBench and 45.6% on miniF2F. This supports prioritizing decomposition under the same SC∧SF criterion we use in Figure 2. We have added this analysis of first-attempt failures to the revised manuscript.

[1] Rafael Cabral et al. ProofFlow: A Dependency Graph Approach to Faithful Proof Autoformalization. ICLR 2026.

Q1. ToMap’s evolution mechanism versus direct revision with Lean feedback

We compare ToMap’s evolution mechanism with a baseline that revises the decomposition directly using Lean feedback. This baseline follows D-Intervene from Figure 2, where we use a common GPT-5.5 setup in the Codex harness to study which stage benefits most from correction. For the comparison below, we keep the D-Intervene procedure in Codex and match all three models and the round limit to ToMap-Qwen30B. We denote this configuration as D-Intervene (ToMap-Qwen30B).

Both methods use Qwen3-30B-A3B-Instruct-2507 as the Decomposer, Goedel-Formalizer-V2-32B as the Formalizer, and Goedel-Prover-V2-32B as the Prover, with at most ten rounds. At each round, D-Intervene revises a single decomposition using only the Lean diagnostics from the previous attempt. ToMap maintains a pool of candidate decompositions, scores them with semantic rubrics, and selects candidates from the Pareto frontier for revision using rubric and Lean feedback.

Table: Direct Lean-guided revision and ToMap evolution with matched models and a ten-round limit

BenchmarkMethodSC (%)SF (%)SC∧SF (%)
ProofFlowBenchD-Intervene (ToMap-Qwen30B)68.524.517.4
ProofFlowBenchToMap-Qwen30B66.332.127.7
miniF2FD-Intervene (ToMap-Qwen30B)79.932.828.3
miniF2FToMap-Qwen30B86.941.041.0

Under these matched settings, ToMap improves SC∧SF over direct Lean-guided revision by 10.3 percentage points on ProofFlowBench and 12.7 points on miniF2F. On ProofFlowBench, D-Intervene achieves comparable SC, but its lower SF limits the proportion of proofs that both pass Lean verification and preserve the source reasoning.

W1 & Q2. Semantic faithfulness across evolution rounds in Figure 4

We understand your concern that Figure 4 reports how often proofs pass Lean verification, while leaving open whether more of those proofs also preserve the source reasoning as the round limit increases. To address this, we report SC∧SF alongside SC for both ToMap variants at the same round limits as in Figure 4.

Table: SC and SC∧SF across evolution rounds on ProofFlowBench (184 problems)

MethodMax. roundsSC (%)SC∧SF (%)
ToMap-Qwen30B139.715.2
ToMap-Qwen30B352.720.7
ToMap-Qwen30B559.224.5
ToMap-Qwen30B1066.327.7
ToMap-Gemini135.316.3
ToMap-Gemini350.527.7
ToMap-Gemini556.532.6
ToMap-Gemini1063.638.0

Table: SC and SC∧SF across evolution rounds on miniF2F (244 problems)

MethodMax. roundsSC (%)SC∧SF (%)
ToMap-Qwen30B157.825.0
ToMap-Qwen30B375.434.4
ToMap-Qwen30B581.138.5
ToMap-Qwen30B1086.941.0
ToMap-Gemini161.945.1
ToMap-Gemini375.453.3
ToMap-Gemini579.156.1
ToMap-Gemini1082.858.6

SC∧SF increases at each successive round limit for both models on both benchmarks. From 1 to 10 rounds, the gains range from 12.5 to 21.7 percentage points. In all four settings, more problems yield proofs that pass Lean verification and preserve the source reasoning, although the overall gain in SC∧SF is smaller than the gain in SC. We have added the SC∧SF results for all four round limits to the revised manuscript.

Dear Reviewer KxkD,

Thank you again for your time and effort in reviewing our paper. As the discussion draws to a close, we would be happy to address any remaining questions. If the additional results in our rebuttal resolve your main concerns, we would be sincerely grateful if you would consider raising your overall score.

Best regards,

The Authors

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

Thank you for your time and thoughtful feedback. As LLMs produce proofs at the frontier of mathematics, verifying their reasoning at the level of a complete proof becomes increasingly important. ToMap addresses this need through full-proof autoformalization, translating an informal argument into a Lean proof that preserves its intermediate claims and the dependencies between them.

Reviewers KxkD and V5Pw highlight the importance of checking both whether the formal proof passes Lean verification and whether it faithfully follows the supplied argument. Reviewers V5Pw and VkPh also recognize the value of our stage-wise intervention experiments in explaining why ToMap focuses on the Decomposer. As the ICLR discussion period draws to a close, we highlight two rebuttal results that address the reviewers’ concerns.

1. Evidence for prioritizing decomposition

Reviewers KxkD, V5Pw, and VkPh ask whether the small SC∧SF gap in the original Figure 2 supports prioritizing the Decomposer. We extend the GPT-5.5 experiment by adding semantic feedback and repeat both feedback conditions with GPT-5.6-Sol. With semantic feedback, D-Intervene exceeds F-Intervene by 9.8 percentage points on average across the two models. The table reports cumulative SC∧SF after five corrections on 184 ProofFlowBench problems.

ModelFeedbackD-Intervene (%)F-Intervene (%)P-Intervene (%)
GPT-5.5Lean only9.28.77.6
GPT-5.5Lean + semantic17.49.88.2
GPT-5.6-SolLean only17.912.514.1
GPT-5.6-SolLean + semantic25.513.614.1

Reviewer V5Pw also asks where failures arise before any correction. GPT-6 Astra examines each first-attempt SC∧SF failure and labels every stage that contributes to an error. Decomposer errors are the most frequent on both benchmarks. The revised manuscript includes the results and attribution criteria.

StageProofFlowBench (171 initial failures)miniF2F (217 initial failures)
Decomposer110/171 (64.3%)99/217 (45.6%)
Formalizer83/171 (48.5%)90/217 (41.5%)
Prover59/171 (34.5%)67/217 (30.9%)

2. The benefit of decomposition evolution under matched settings

Reviewer KxkD asks how ToMap compares with direct Lean-guided revision, and Reviewer VkPh asks for comparisons controlling for model choice and inference budget. All methods below use the same Decomposer, Formalizer, and Prover models. D-Intervene runs in the Codex agent harness and revises one decomposition using Lean feedback alone. With the same ten-round limit, ToMap exceeds D-Intervene in SC∧SF by 10.3 percentage points on ProofFlowBench and 12.7 points on miniF2F.

MethodProofFlowBench SC (%)ProofFlowBench SF (%)ProofFlowBench SC∧SF (%)miniF2F SC (%)miniF2F SF (%)miniF2F SC∧SF (%)
D-Intervene (ToMap-Qwen30B model config)68.524.517.479.932.828.3
Repeated sampling (mean of 10)51.819.613.669.831.125.4
Best-of-1057.622.816.875.035.729.9
Iterative refinement (Lean only)62.523.419.677.528.326.2
Iterative refinement (Lean + semantic)51.626.121.769.333.632.4
ToMap without rubric scoring or Pareto selection28.827.719.654.537.728.3
ToMap-Qwen30B66.332.127.786.941.041.0

The sampling methods share ten independent candidates, and all iterative methods use at most ten rounds. The revised manuscript reports actual output tokens per problem. In response to Reviewer KxkD’s question about Figure 4, we also report SC∧SF at 1, 3, 5, and 10 rounds. SC∧SF increases at every reported round limit for both Decomposer models on both benchmarks.

We sincerely hope these results address the reviewers’ concerns. We would be grateful for any further feedback and would be happy to address remaining questions before the discussion closes.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.