- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ToMAP to make the decomposition self-evolving while keeping the formalizer and prover fixed. It has a global review to route back both rubrics and compiler feedback to the decomposition stage. On proofflow benchmark and miniF2F benchmark, it improves verified and faithful formalization by 17.3% and 23.3%.
Critical strengths
1.Experiments show the changing proof decomposition is better than changing formal statements or proof scripts.
2.Experiments on both Lean verification and semantic faithfulness improve 17.3% and 23.3% on two benchmarks.
Critical weaknesses
1.Changing only the decomposer performs slightly better changing the Formalizer, 9.2% vs 8.7%. It lacks ablation studies of Pareto selection, rubric scoring and comparison with repeated sampling under the same computing budget.
2.Semantic faithfulness is judged by GPT-5.5, without human checking. It is unfair to compare different models under different computing budgets.
Questions for authors
1.Can you compare the best-of-n and iterative refinement under the same computing budget? Can you add key ablation studies? Is 0.5% improvement significant?
2.Can human experts validate semantic faithfulness? Please report the agreement and error rates.
The authors added comparisons of best-of-10 and iterative refinement. It is helpful. However, It also lacks the agreement between the experts. No statistical test is reported to show the 0.5% improvement is significant. The new experiment changes the feedback condition (lean vs lean+semantic), it may gain from the semantic feedback.
5 comments
W1 & Q1. Ablations, alternative baselines, and inference cost
We appreciate your suggestion to control for model choice and inference budget when comparing ToMap with the baselines. In the updated manuscript, we revise the baselines to align the models at each stage and also add comparisons. All six methods in the tables below use Qwen3-30B-A3B-Instruct-2507 as the Decomposer, Goedel-Formalizer-V2-32B as the Formalizer, and Goedel-Prover-V2-32B as the Prover.
The sampling methods use ten independent attempts per problem, while iterative refinement and ToMap run for up to ten rounds. Even with these limits, actual token use varies with proof length, retries, and scoring calls. We therefore report the output tokens each method consumes in thousands per problem (OT, k/problem).
-
Repeated sampling and Best-of-10 use the same ten independently generated proofs for each problem. Each attempt starts from the original statement and informal proof. Once all ten candidates are available, repeated sampling reports their mean performance. Best-of-10 selects the Lean-valid candidate with the highest internal semantic score, or the highest-scoring candidate overall if none passes Lean. Its OT includes the cost of selection.
-
Iterative refinement operates on the full pipeline for up to ten rounds. After each attempt, the pipeline takes the current output and its feedback as input to the next round, which can revise the decomposition, formal statements, or proof scripts. We compare Lean feedback alone with Lean plus semantic feedback. Both variants start from the same initial outputs and use the same round and downstream retry limits.
-
ToMap without rubric scoring or Pareto selection keeps a single verification gate. The Decomposer revises one decomposition for up to ten rounds using Lean diagnostics and its own semantic feedback on whether the decomposition preserves the reasoning in the source proof.
The tables below cover all 184 ProofFlowBench problems and all 244 miniF2F problems. OT includes generation, revisions, downstream calls and retries, and scoring or selection where applicable. We exclude the final SF evaluation from OT for every method.
Table: Performance and output tokens per problem on ProofFlowBench
| Method | SC (%) | SF (%) | SC∧SF (%) | OT (k/problem) |
|---|---|---|---|---|
| Repeated sampling (mean of 10) | 51.8 | 19.6 | 13.6 | 198.5 |
| Best-of-10 (same candidates) | 57.6 | 22.8 | 16.8 | 199.65 |
| Iterative refinement (Lean only) | 62.5 | 23.4 | 19.6 | 120.64 |
| Iterative refinement (Lean + semantic) | 51.6 | 26.1 | 21.7 | 181.53 |
| ToMap without rubric scoring or Pareto selection | 28.8 | 27.7 | 19.6 | 113.0 |
| ToMap-Qwen30B | 66.3 | 32.1 | 27.7 | 98.8 |
Table: Performance and output tokens per problem on miniF2F
| Method | SC (%) | SF (%) | SC∧SF (%) | OT (k/problem) |
|---|---|---|---|---|
| Repeated sampling (mean of 10) | 69.8 | 31.1 | 25.4 | 169.2 |
| Best-of-10 (same candidates) | 75 | 35.7 | 29.9 | 171.23 |
| Iterative refinement (Lean only) | 77.5 | 28.3 | 26.2 | 128.16 |
| Iterative refinement (Lean + semantic) | 69.3 | 33.6 | 32.4 | 138.17 |
| ToMap without rubric scoring or Pareto selection | 54.5 | 37.7 | 28.3 | 79.6 |
| ToMap-Qwen30B | 86.9 | 41 | 41 | 83.3 |
Selecting the best of ten independent candidates improves SC∧SF over repeated sampling on both benchmarks. ToMap-Qwen30B improves further over Best-of-10, from 16.8% to 27.7% on ProofFlowBench and from 29.9% to 41.0% on miniF2F, while using about half as many output tokens per problem. ToMap also exceeds full-pipeline iterative refinement with Lean and semantic feedback by 6.0 and 8.6 percentage points, respectively, while using fewer tokens on both benchmarks.
In the ablation, the Decomposer relies on its own semantic feedback to revise a single candidate. Without Pareto selection, greedy revision may encourage reward hacking. Also, when the naive semantic feedback gives little guidance on what to change, the same issue can persist across several rounds. These additional revisions consume tokens and may outweigh the savings from removing rubric scoring. Together, the comparisons support the effectiveness of ToMap’s self-evolving decomposition pipeline. With the same models, guiding revisions with semantic rubrics and selecting candidates from the Pareto frontier yields more faithful, Lean-verified proofs than either independent sampling or full-pipeline iterative refinement.
W2 & Q2. SF evaluator accuracy
Thank you for suggesting that we validate our SF judgments with another model and with human experts. Our use of an LLM to evaluate SF follows prior work on autoformalization. CRAMF [2] and ATLAS [3] use LLM judges to check whether formal statements preserve the meaning of the source statements. ProofFlow [1] reports strong agreement with expert labels on 200 individual proof steps, and ReForm [4] reports 85.8% accuracy for Gemini-2.5-Pro on 859 expert-labeled statement pairs. These results support using LLMs as semantic judges for formal statements and proof steps. For this rebuttal, we examine how well this approach works for complete proofs.
Our original evaluation uses GPT-5.5 with xhigh reasoning effort, with GPT-6 Astra reviewing difficult or ambiguous cases. We also use the latest model, GPT-6.1-Sol, to rescore the same outputs with the same reasoning effort and Astra review procedure. This covers all six baselines and both ToMap methods on all 184 ProofFlowBench problems and 244 miniF2F problems. Both evaluations take place after candidate selection. For each output, we combine the new SF label with the original Lean verdict to compute SC∧SF.
Table: GPT-6.1-Sol reevaluation on both benchmarks
| Method | ProofFlowBench SF (%) | ProofFlowBench SC∧SF (%) | miniF2F SF (%) | miniF2F SC∧SF (%) |
|---|---|---|---|---|
| ProofBridge | 21.7 | 20.7 | 27.5 | 25.8 |
| Codex End2End | 25.0 | 17.9 | 32.0 | 26.6 |
| M2F | 37.0 | 10.3 | 45.9 | 7.0 |
| Codex D-Correction | 22.3 | 11.4 | 30.3 | 19.3 |
| Monotonic | 19.6 | 16.8 | 31.1 | 31.1 |
| ProofFlow | 40.8 | 21.2 | 50.0 | 34.0 |
| ToMap-Qwen30B | 31.0 | 23.4 | 40.2 | 36.9 |
| ToMap-Gemini | 52.7 | 42.4 | 62.7 | 58.2 |
The GPT-6.1-Sol evaluation supports the same conclusions as the original GPT-5.5 evaluation. On both benchmarks, ToMap-Gemini ranks first on SF and SC∧SF, and ToMap-Qwen30B outperforms all six baselines on SC∧SF.
To check how often each evaluator agrees with experts on individual proofs, we also conduct a human evaluation. Given the time available during the rebuttal period, we limit this evaluation to 40 complete ToMap-Gemini proofs, 20 from each benchmark. Two PhD researchers specializing in mathematical formalization assess the proofs. Both have prior experience annotating full-proof autoformalization outputs for foundation model training. For each item, they compare the generated Lean proof with the source statement and informal proof. They assign a positive SF label when the Lean proof preserves the source assumptions, essential intermediate claims, and reasoning dependencies.
The table below reports the error rate for each evaluator. We count an error whenever its final SF label differs from the expert label. We sincerely hope that these additional results address your concern about the reliability of our SF evaluation for complete proofs.
Table: SF evaluator error rates against expert judgments
| Evaluation set | GPT-5.5 + GPT-6 Astra | GPT-6.1-Sol + GPT-6 Astra |
|---|---|---|
| ProofFlowBench | 20.0% (4/20) | 10.0% (2/20) |
| miniF2F | 10.0% (2/20) | 5.0% (1/20) |
| Overall | 15.0% (6/40) | 7.5% (3/40) |
References
[1] Rafael Cabral et al. ProofFlow: A Dependency Graph Approach to Faithful Proof Autoformalization. ICLR 2026.
[2] Wangyue Lu et al. Automated Formalization via Conceptual Retrieval-Augmented LLMs. ICLR 2026.
[3] Xiaoyang Liu et al. ATLAS: Autoformalizing Theorems through Lifting, Augmentation, and Synthesis of Data. NeurIPS 2025.
[4] Guoxin Chen et al. ReForm: Reflective Autoformalization with Prospective Bounded Sequence Optimization. ICLR 2026.
We thank Reviewer VkPh for the careful review and for suggesting additional controls on how the test-time budget is spent.
W1. Evidence for prioritizing decomposition in Figure 2
We sincerely appreciate you raising this point and understand your concern about the small SC∧SF gap between D-Intervene and F-Intervene in Figure 2. In this exploratory experiment, we use GPT-5.5 in the Codex agent harness with Lean feedback alone. For this rebuttal, we add semantic feedback so the correction model can compare its candidate with the source statement and proof, checking whether it preserves the assumptions, intermediate claims, and reasoning dependencies. To see whether the same pattern holds with a second model, we repeat the comparison with GPT-5.6-Sol under both feedback conditions. With semantic feedback, revising the decomposition shows a larger SC∧SF advantage over revising the formal statements on both models, supporting our focus on decomposition.
Both models use high reasoning effort for generation and correction. Within each model setting, all three interventions start from the same initial outputs on all 184 ProofFlowBench problems. The table below reports cumulative SC∧SF after five corrections.
Table: Cumulative SC∧SF after five corrections on ProofFlowBench
With semantic feedback, D-Intervene achieves higher SC∧SF than F-Intervene on both GPT-5.5 (17.4% vs. 9.8%) and GPT-5.6-Sol (25.5% vs. 13.6%). Across the two models, D-Intervene outperforms F-Intervene by 9.8 percentage points on average, which supports prioritizing decomposition when the goal is to produce Lean proofs that faithfully follow the source argument.
We also examine which stages contribute to SC∧SF failures before any correction in the Figure 2 setting. This analysis covers the 171 ProofFlowBench problems and 217 miniF2F problems that fail SC∧SF on the first attempt. Drawing on ProofFlow’s step-level error analysis [1], we use GPT-6 Astra to inspect each stage’s inputs and outputs and assess whether an error at that stage contributes to the final SC∧SF failure. A problem can appear in more than one row when errors at multiple stages contribute.
Table: Stage contributions to first-attempt SC∧SF failures
Decomposer errors are the most frequent on both benchmarks, contributing to 64.3% of first-attempt SC∧SF failures on ProofFlowBench and 45.6% on miniF2F. Together with the intervention results, this supports prioritizing decomposition when the goal is to produce Lean proofs that faithfully follow the source argument. We have added this analysis of first-attempt failures to the revised manuscript.
[1] Rafael Cabral et al. ProofFlow: A Dependency Graph Approach to Faithful Proof Autoformalization. ICLR 2026.