acceptodds
Under review as a conference paper at ICLR 2027

From Natural-Language Propositions to Lean Proofs: Proof-Search Feedback for Source-Grounded Re-Formalization

Abstract

End-to-end formalization and proving transforms a natural-language mathematical proposition into a Lean theorem together with a kernel-checked proof. In the proposition-only setting, no proof is supplied, so the system must discover the proof itself. Once a formalization is accepted, however, many pipelines keep it fixed during proof search, leaving proof failures unable to expose semantic defects in the statement. We call the resulting rate of self-accepted but unproved formalizations the operational semantic–proof closure gap. We study whether concrete proof-failure information produced by a model can help that same model discover such defects. We instantiate a verify-before-revise loop at the proposition-only Lean formalization–proof boundary: when proof search produces a suspected-false signal, a separate semantic audit compares the root formalization with the original proposition, and a revision is allowed only when the audit identifies a concrete, proposition-grounded discrepancy. Across two models and two datasets, the closed loop reduces the closure gap and improves externally validated end-to-end success, including BEq-validated success.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Autoformalization, Automated theorem proving, Large language model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_kG1SOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the gap between Lean formalization and successful proofs. When a proof fails, the method checks the Lean statement again with the original proposition. It doesn't directly change the theorem based on proof errors. Experiments on 3 models and 4 datasets show better end2end results than the baseline.

Critical strengths

1.When a proof fails, the method checks the Lean statement again with the original proposition. Semantic checking is used to avoid wrong changes.

2.Experiments on 3 models and 4 datasets show better end2end results than the baseline, and using the same prover models and token budgets.

Critical weaknesses

1.Signal-only already achieves most of the improvement. Full gives only 0-2.8 pp. The paper lacks statistical testing and ablation studies for key components.

2.Semantic checking depends on LLMs. Only 5 of 137 cases were correct at first. So, 0 harmful changes in these 5 cases is not enough to prove the safety. The evaluation only covers two math datasets.

Questions for authors

1.Can the authors test more formalizations that are correct at first and report the harmful rate? Can the authors also report the agreement between annotators?

2.Is the improvement between Full and Signal-only significant? Can the authors test 3 pre-pruning checks separately to show each contribution?

Based on the new experiments, Full outperforms Signal-only among 8 experimental settings. It is helpful. However, I still concern the number of initial faithful cases is relatively small, even though the new version increased from 5 to 21. The authors still don't report the agreement between annotators.

3 comments
AuthorsOpenReview ↗

Response to Reviewer kG1S

Thank you for your thoughtful suggestions.

1. The incremental value of Full over Signal-only and its statistical significance (Weakness 1; Question 2).

Signal-only already captures much of the benefit of allowing semantic revision. Comparing Full with Signal-only asks whether concrete diagnoses provide further help beyond a generic proof-failure signal. The table below reports this difference across all eight settings; complete results and the statistical procedure appear in the common response, “Extending Evaluation to Undergraduate-Level Datasets,” Tables C1–C3.

Full–Signal-only paired effects (percentage points [pointwise 95% CI]). We use paired percentile-bootstrap intervals without multiple-comparison adjustment and consider a difference significant when its interval excludes zero.

ModelDatasetASCC+Proof effect [95% CI]BEq+Proof effect [95% CI]
DeepSeek V4 FlashminiF2F0.41 [-0.61, 1.43]0.41 [-0.41, 1.23]
DeepSeek V4 FlashFormalMath5000.78 [0.00, 1.81]0.26 [-0.78, 1.55]
MiMo v2.5miniF2F2.05 [0.20, 3.89]0.82 [-0.61, 2.46]
MiMo v2.5FormalMath5002.58 [0.52, 4.65]2.58 [0.52, 4.91]
MiMo v2.6 FlashminiF2F3.07 [0.82, 5.53]1.64 [-0.20, 3.48]
MiMo v2.6 FlashFormalMath5004.13 [1.29, 6.98]4.39 [1.81, 7.24]
MiMo v2.6 FlashProofNet-Verified3.84 [1.37, 6.30]4.11 [1.64, 6.85]
MiMo v2.6 FlashPutnamBench2.53 [1.19, 4.02]1.93 [0.74, 3.27]

Full achieves higher success rates in all eight settings under both validators, with significant gains in 6/8 settings under ASCC+Proof and 4/8 under BEq+Proof. Concrete diagnoses therefore provide an additional benefit in several settings, although the remaining intervals include zero, particularly for DeepSeek V4 Flash. Notably, the gains are significant on both ProofNet-Verified and PutnamBench under both metrics, suggesting that diagnostic information may be especially useful for more challenging problems.

2. Separate contributions of the three pre-pruning checks (Weakness 1; Question 2).

We have clarified that pre-pruning is an auxiliary proof-search module. Full, Signal-only, and Fixed all use the same pruning configuration, allowing us to examine our core question: whether proof-stage feedback improves semantic revision and end-to-end outcomes. We no longer present pre-pruning as a standalone contribution and have moved its detailed analysis and grouped ablation to Appendix D, “Auxiliary Analysis: Grouped Pre-Pruning Ablation.” Separate ablations of the three checks remain for future work.

3. Harmful revisions, reliance on LLM checking, and annotator agreement (Weakness 2; Question 1).

We agree that the original 0/5 observation cannot establish safety. The additional 100-record gate comparison includes more initially faithful statements: Full introduces errors in 3/21, compared with 7/21 for Gate-off. It also repairs more initially incorrect statements (53/79 versus 21/79). These results show a better balance between repair and harm with the semantic gate, while confirming that harmful revisions still occur. Sampling details and results appear in the common response, “Repairs and harmful revisions among root-reaching inputs,” Table C6.

To assess faithful task completion beyond the root-triggered subset, we also manually evaluated 150 randomly sampled FormalMath500 model inputs. Human-faithful+Proof success is 68.67% for Fixed, 76.00% for Signal-only, and 78.67% for Full, an observed Full–Fixed gain of 10.00 pp. The common response, “End-to-end human evaluation,” Table C4, provides the full results.

This sample also lets us compare the automatic validators with human judgments. Across its 184 distinct formalizations, agreement is 75.54% for ASCC and 83.70% for BEq. These findings support the usefulness of the validators while showing that neither fully captures human judgments. Confusion matrices, precision, and recall appear in the common response, “Human labels agreement with ASCC-3 and BEq,” Table C5 and its accompanying text.

Time and staffing constraints prevented us from obtaining two independent annotations for every sample, so we do not report inter-annotator agreement. We explicitly acknowledge this limitation in the revised manuscript’s Limitations section.

4. Evaluation limited to two datasets (Weakness 2).

To broaden the evaluation, we added ProofNet-Verified and PutnamBench, two undergraduate-level datasets, using MiMo v2.6 Flash. Both preserve the ordering Full > Signal-only > Fixed under both validators. BEq+Proof success for Full / Signal-only / Fixed is 41.37% / 37.26% / 35.89% on ProofNet-Verified and 15.18% / 13.24% / 12.20% on PutnamBench. Complete results and confidence intervals appear in the common response, “Extending Evaluation to Undergraduate-Level Datasets,” Tables C1–C3.

Thank you again for your thoughtful review; if time permits, we would appreciate hearing whether our response has addressed your concerns or whether any further clarification would be helpful.

Reviewer kG1S, thanks for your review. Could you take a look at the authors’ response and comment on which of your concerns have or have not been resolved?

Reviewer_FYDiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper studies the proposition-only end-to-end autoformalization problem. They provide a three-stage framework in a verify-before-revise manner, which first autoformalizes with Lean and semantic validation, then recursively proves with a knowledge-accumulating pre-pruning module and failure diagnosis, and finally runs the audit-gated reformalization. Experiments with DeepSeek-V4-Flash and Mimo-v2.5 on MiniF2F and FormalMath500 indicate that the proposed method can reduce the closure gap and improve the BEq-validated end-to-end success rate over a fixed formalization baseline.

Critical strengths
  1. This paper provides a practical framework that may be useful for resolving formalization in agent-oriented math research.
  2. The work conducts relatively comprehensive experiments on the MiniF2F and FormalMath500 benchmarks, using multiple metrics and controls to study different aspects of the method.
Critical weaknesses
  1. Saturated benchmarks: Although the improvements are relatively significant, the authformalization on MiniF2F and FormalMath500 is close to saturation. The fixed baseline already reaches close to 80% accuracy in BEq-validated end-to-end success. Furthermore, these problems are likely included in the training corpora of all current models. Evaluating on more challenging benchmarks like ProofNet or PutnamBench would better demonstrate the paper’s improvements.
  2. No end-to-end human evaluation: The human audit covers only root-triggered revisions on 137 inputs, and the end-to-end evaluation relies entirely on ASCC-3 and BEq. These validators agree on only ~75% of the final authformalization. Since autoformalization currently lacks a widely accepted substitute for human judgment, an end-to-end human evaluation is recommended to make the experiments comprehensive.
  3. The current paper-writing style reads more like an experimental report than an academic paper, especially in the experimental section, where there is no clear separation between the main results and auxiliary studies, which limits the paper's readability.
Questions for authors
  1. How does the framework perform on more challenging benchmarks such as ProofNet or PutnamBench? Do the gains over Fixed persist?
  2. Can the authors provide an end-to-end human evaluation on a sample of final outputs from Fixed and Full, and report its agreement with ASCC-3 and BEq?
  3. Can the authors revise the writing for the experiment section?

I am happy to increase my score if the above questions are resolved

3 comments
AuthorsOpenReview ↗

Response to Reviewer FYDi

Thank you for your thoughtful suggestions.

1. Benchmark difficulty, possible saturation, and generalization (Weakness 1; Question 1).

To test whether the gains persist on more challenging problems, we added ProofNet-Verified and PutnamBench and evaluated Full, Signal-only, and Fixed with MiMo v2.6 Flash on these datasets as well as miniF2F and FormalMath500.

Across the four datasets, Full > Signal-only > Fixed under both automatic evaluation schemes. On ProofNet-Verified, BEq+Proof success is 41.37% / 37.26% / 35.89% for Full / Signal-only / Fixed; on PutnamBench, it is 15.18% / 13.24% / 12.20%. Thus, the gains over Fixed extend to the two more challenging datasets. The common response, “Extending Evaluation to Undergraduate-Level Datasets,” reports all eight settings in Table C1 and the paired comparisons in Tables C2 and C3.

The concern about training exposure is also relevant to interpreting these results. All three policies use the same model, problems, initial formalization, and execution up to the intervention, so any prior exposure is shared. Our central question is whether proof-search feedback helps a model discover semantic inconsistencies that it misses in its own formalization. Prior exposure may affect absolute performance, but the comparison holds that exposure fixed and provides evidence for this question within the evaluated model–benchmark settings.

2. End-to-end human evaluation and agreement with ASCC and BEq (Weakness 2; Question 2).

Following your recommendation, we manually evaluated 150 randomly sampled FormalMath500 model inputs, yielding 184 distinct formalizations across Fixed, Signal-only, and Full. Human-faithful+Proof success is 68.67% for Fixed, 76.00% for Signal-only, and 78.67% for Full, an observed Full–Fixed gain of 10.00 pp. Sampling details and full results appear in the common response, “End-to-end human evaluation,” Table C4.

We also compared ASCC and BEq with the human labels after deduplicating the formalizations. BEq achieves higher agreement, precision, and recall than ASCC; the confusion matrices and metric values appear in the common response, “Human labels agreement with ASCC-3 and BEq,” Table C5 and its accompanying text.

3. Organization and readability of the experimental section (Weakness 3; Question 3).

We have reorganized the experimental section around the main research questions, keeping the key results in the main text and moving supporting analyses to the appendix. The revised discussion connects each question to its comparison and findings, making the progression of the experiments easier to follow.

Thanks a lot for your response. My major concerns has been resolved and I have increased my rating from 2 to 3.

Thank you for your thoughtful review and constructive feedback, and for the time and care you devote to serving the academic community.

Reviewer_1MehOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to obtain both a faithful Lean statement and a kernel-checked proof from a natural-language proposition without a supplied proof. It generates and checks an initial formalization, performs recursive proof search, and uses suspected-false diagnoses reaching the root to trigger a source-grounded semantic audit; discrepancies that survive rechecking can authorize revision before further proving. The authors compare Fixed, Signal-only, and Full across two models and two datasets, using fixed offline semantic validators and a paired human audit. They report higher validated end-to-end success than when the initial statement remains fixed.

Critical strengths
  1. The method addresses a practical need in end-to-end formalization: an initially accepted statement can still be wrong, and proof exploration can provide useful reasons to revisit it. The system permits revision of the root theorem while requiring a source-grounded semantic audit, discrepancy rechecking, and renewed validation. Combining revisable targets with constraints from the original proposition is a useful design for handling formalization errors that proof search alone cannot resolve.

  2. The paper directly audits whether revisions restore the intended meaning or weaken the target, with Gate-off providing a comparison without semantic gating. Among 137 root-triggered inputs, Full repairs 96/132 initially inconsistent statements, compared with 81/132 for Gate-off; method identities, automatic scores, and proof outcomes are hidden from the annotator (Section 5.7 and Appendix A.8). This provides valuable evidence about semantic gating and evaluates revision beyond proof completion alone.

Critical weaknesses
  1. The comparisons do not sufficiently establish the additional value of interleaving revision with proving relative to stronger initial review. Initialization uses ASCC-1 with the same base model as generation and proving, potentially sharing semantic errors; other fixed models are used only offline. Fixed retains initialization repair and local proof repair but freezes the subsequent root statement, so Full–Fixed primarily measures the benefit of enabling audit and revision after the current initialization. Mathesis changes candidate generation and selection as well as scoring, so it does not isolate initial-review strength. A comparison that independently reviews and iteratively repairs the same initial candidates before freezing them is missing, as is evidence of the remaining loop benefit when initial statements are independently judged faithful. This limits the assessment of when revision during proving is necessary or useful. Such comparisons need to include initial review, repair, and proving in total costs.

  2. The relationship between final source fidelity and proof success needs more direct analysis. Appendix A.6, Table 10 reports 111 modified-and-proved Full endpoints, whereas Appendix A.8, Table 11 gives only 101 final human-faithful endpoints among the same 137 records: 96 repaired and five retained faithful. Under the manuscript's same-endpoint definitions, at least 10 modified-and-proved endpoints must therefore remain human-labeled inconsistent, but the precise intersection and failure types are not reported. The authors provide ASCC+Proof and BEq+Proof and acknowledge the successful-deviation blind spot, yet how those automatic evaluations correspond to these human outcomes remains unclear. A selected, single-annotator audit with only five initially faithful records also limits estimates of residual errors and harmful revisions. This affects how much of the success-rate improvement can be interpreted as faithfully completing the source task.

Questions for authors
  1. Have you evaluated stronger independent reviewers, multi-model review, or upfront refutation search on the same initial candidates, followed by source-grounded iterative repair and then a frozen statement for the same prover? A comparison under total budgets including initial review and repair, together with Full's remaining gain on statements independently judged faithful before proof search, would help establish the additional value of revision during proving.

  2. Can you provide the final human-fidelity-by-proof-success cross-tabulation for the 137 records, confirm that Tables 10 and 11 refer to the same endpoints, and explain the modified-and-proved endpoints that remain human-labeled inconsistent? How do they depart from the source, do ASCC and BEq identify them, and have the labels been checked by another annotator? Additional human sampling of successful proof endpoints across the complete datasets would help assess the remaining blind spot outside the revision-triggered population.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 1Meh

Thank you for your careful assessment.

1. Stronger initial review, shared-model errors, and the value of revision during proving (Weakness 1; Question 1).

Following your suggestion, we added More, a baseline that performs additional source-grounded semantic review and iterative repair before freezing the statement for proof search. More and Full start from the same initial candidate and use the same prover; Full allows semantic audit and revision to continue during proving, guided by proof-stage feedback.

We use the same MiMo model throughout to examine whether proof-search feedback helps a model identify semantic errors that it misses during direct review. Keeping the model fixed lets us assess the contribution of this feedback while controlling for differences in reviewer capability.

Table R1.1. Upfront review versus proof-triggered revision with MiMo v2.6 Flash. Rates are percentages over all inputs. Evaluation criteria are described in the common response, “Stricter BEq protocol” and “Extending Evaluation to Undergraduate-Level Datasets.”

DatasetNMore ASCC+PSignal ASCC+PFull ASCC+PMore BEq+PSignal BEq+PFull BEq+P
miniF2F48871.7274.5977.6665.9868.8570.49
FormalMath50038768.7371.3275.4565.6368.4872.87
ProofNet-Verified36540.8242.4746.3037.2637.2641.37
PutnamBench67213.8415.7718.3012.0513.2415.18

Full outperforms More on all four datasets under both ASCC+Proof and BEq+Proof. These results show that allocating more of the same budget to upfront review does not fully recover the gains from allowing proof feedback to guide subsequent revision.

2. Human fidelity versus proof success, and the relationship between the original tables (Weakness 2; Question 2).

The two original tables you mentioned refer to the same 137 Full endpoints. Table R1.2 brings the proof outcomes and human judgments together, confirming that 13 of the 111 proved outputs remain unfaithful to the source task.

Table R1.2. Human fidelity and proof success for the 137 Full endpoints. All 111 proved outputs were modified.

Proof outcomeHuman-faithfulHuman-inconsistentTotal
Proved9813111
Unproved32326
Total10136137

In all 13 cases, the initial formalization was already unfaithful, and revision failed to restore fidelity. The final statements can nevertheless be proved because they contain contradictory assumptions (4 cases), assume or restate the target (6), or express a weaker conclusion (3). Detailed explanations appear in Appendix A.9, “Why Semantically Unfaithful Outputs Can Still Be Proved.”

The automatic validators also miss some of these errors: ASCC accepts 9/13 (69.23%) and BEq accepts 2/13 (15.38%), although no case is accepted by both.

3. Harmful revisions and end-to-end human audit (Weakness 2; Question 2).

To examine whether the gains extend beyond the original root-triggered sample, we randomly sampled 150 model inputs from FormalMath500 for human evaluation. Human-faithful+Proof success reaches 78.67% for Full, compared with 76.00% for Signal-only and 68.67% for Fixed, an observed Full–Fixed gain of 10.00 pp. Sampling details and results are provided in the common response, “End-to-end human evaluation,” Table C4. We also assess how well ASCC and BEq agree with human judgments on the 184 distinct formalizations from this sample; the confusion matrices, agreement, precision, and recall appear in “Human labels agreement with ASCC-3 and BEq,” Table C5 and its accompanying text.

A separate audit of 100 root-reaching records examines both repairs and harmful revisions. Full repairs more initially incorrect statements than Gate-off (53/79 versus 21/79) and damages fewer initially faithful statements (3/21 versus 7/21). These results show that the semantic gate improves the balance between repair and harm, although it does not eliminate harmful revisions. Details appear in the common response, “Repairs and harmful revisions among root-reaching inputs,” Table C6.

Time and staffing constraints prevented us from obtaining two independent annotations for every sample. We explicitly acknowledge this limitation in the revised manuscript’s Limitations section.

Thank you for the additional experiments and the detailed response. I am raising my rating from 2 to 3, Weak acceptance. The new human evaluation was the main reason for this change. The cross-tabulation confirms that 13 of the 111 modified and proved statements still departed from the source, and the case analysis explains how. On the same 150 inputs sampled from the full FormalMath500 runs, the number of results judged faithful by human reviewers and verified by Lean rose from 103 with Fixed to 114 with Signal-only and 118 with Full. This supports the value of revisiting statements during proof search. The specific diagnosis provided a further net gain of four successful results in this human evaluation sample.

The More comparison also addresses part of my concern about upfront review. I accept that Full outperformed the tested upfront review strategy with the same model, initial statement, prover, and post-initialization output token cap. As a suggestion for future work, please clarify whether the two strategies use the same audit prompts, criteria for allowing a revision, and revalidation procedure, and report their actual resource use by stage. This would help establish how much benefit comes from interleaving proof search with semantic revision when the review procedures are comparable.

Thank you for your positive assessment and thoughtful suggestions, which have helped us substantially improve the paper. We offer one further clarification simply to make our discussion more complete: More and Full use the same semantic-review procedure, including the audit prompts, criteria for permitting revisions, and revalidation steps. The only difference in this procedure is that More receives no proof-stage feedback. In future work, we plan to investigate how token allocation across review, repair, and proof search influences end-to-end gains. Thanks again for the time and care you devote to serving the academic community.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Common response to all reviewers(1/3)

We thank all reviewers for their thoughtful comments on benchmark breadth, the contribution of proof-failure diagnoses, and the reliability of semantic evaluation. Below, we describe the stricter BEq protocol, four additional model–dataset settings, paired analyses across all eight settings, and new human evaluations. Together, these results provide further evidence that source-grounded revision improves validated end-to-end outcomes, while clarifying where concrete diagnoses add value and where semantic errors remain.

Stricter BEq protocol

We adopted a stricter BEq protocol to improve the reliability of semantic evaluation. Manual inspection of the original protocol revealed that equivalence proofs could establish the two propositions independently, without using either to derive the other. The revised protocol checks for this behavior before accepting a proof.

Under the revised protocol, every completed bidirectional equivalence proof must pass three consecutive checks for this behavior. If any check fails, another proof must be attempted. We also replaced the evaluation model with GLM-5.3-Flash.

To assess the change, we manually inspected proofs from both protocols on the same randomly sampled 80 instances, with 20 from each of the four original model–dataset settings. Accepted proofs exhibiting this behavior decreased from 18/80 (22.50%) to 5/80 (6.25%), a reduction of 16.25 percentage points. Of the 69 instances accepted by the revised protocol, 5 (7.25%) still exhibited the problem. The revised protocol also failed to accept one previously accepted proof that manual review judged free of this behavior (1/80). It therefore substantially reduces these false acceptances, although it does not eliminate them.

We further assessed the revised BEq protocol against human labels in the end-to-end audit described below. On 184 distinct formalizations, BEq achieves precision of 91.41% and recall of 86.03%; see “Human labels agreement with ASCC-3 and BEq,” Table C5 and its accompanying text.

All BEq results reported below and in the main experiments use this stricter protocol. Although the absolute success rates change, the ordering Full > Signal-only > Fixed is preserved for BEq+Proof across all eight settings, so the main comparative conclusion remains consistent. The separate Mathesis comparison in Appendix A.7 uses DeepSeek V4 Flash without the proof-dependence checks; those scores are used only to compare Full and Mathesis under that shared protocol.

AuthorsOpenReview ↗

Common response to all reviewers(2/3)

Extending Evaluation to Undergraduate-Level Datasets

To test whether the gains extend to more challenging problems, we evaluated Full, Signal-only, and Fixed with MiMo v2.6 Flash on miniF2F, FormalMath500, ProofNet-Verified, and PutnamBench, with 488, 387, 365, and 672 inputs, respectively. Apart from the models and datasets, all settings remain as described in the main text.

Table C1. Validated end-to-end success (%). Every success requires both a proof and acceptance by the specified semantic validator.

ModelDatasetFixed ASCC+PSignal ASCC+PFull ASCC+PFixed BEq+PSignal BEq+PFull BEq+P
DeepSeek V4 FlashminiF2F82.5885.4585.8672.1373.7774.18
DeepSeek V4 FlashFormalMath50083.2086.0586.8277.2679.8480.10
MiMo v2.5miniF2F73.3677.6679.7165.3769.2670.08
MiMo v2.5FormalMath50070.5474.9477.5267.7072.0974.68
MiMo v2.6 FlashminiF2F68.8574.5977.6662.7068.8570.49
MiMo v2.6 FlashFormalMath50063.0571.3275.4561.5068.4872.87
MiMo v2.6 FlashProofNet-Verified40.5542.4746.3035.8937.2641.37
MiMo v2.6 FlashPutnamBench14.5815.7718.3012.2013.2415.18

The ordering Full > Signal-only > Fixed holds under both evaluation schemes in the new experiments. Relative to Fixed, Full raises BEq+Proof success from 35.89% to 41.37% on ProofNet-Verified and from 12.20% to 15.18% on PutnamBench. The benefit thus extends to both evaluated undergraduate-level datasets.

Statistical analysis. To assess the differences among Full, Signal-only, and Fixed, we use paired comparisons. A difference is considered significant when its two-sided, pointwise 95% paired percentile-bootstrap confidence interval excludes zero, without multiple-comparison adjustment. Computational details appear in Appendix A.4 (Budget Accounting, Shared Prefixes, and Statistics) of the revised paper.

Table C2. Paired comparisons under ASCC+Proof. Each cell is effect in pp [95% CI]; boldface indicates statistical significance.

ModelDatasetFull − FixedSignal − FixedFull − Signal
DeepSeek V4 FlashminiF2F3.28 [1.84, 4.92]2.87 [1.43, 4.51]0.41 [-0.61, 1.43]
DeepSeek V4 FlashFormalMath5003.62 [1.81, 5.43]2.84 [1.29, 4.65]0.78 [0.00, 1.81]
MiMo v2.5miniF2F6.35 [4.30, 8.61]4.30 [2.66, 6.15]2.05 [0.20, 3.89]
MiMo v2.5FormalMath5006.98 [4.65, 9.56]4.39 [2.58, 6.46]2.58 [0.52, 4.65]
MiMo v2.6 FlashminiF2F8.81 [6.35, 11.27]5.74 [3.69, 7.99]3.07 [0.82, 5.53]
MiMo v2.6 FlashFormalMath50012.40 [9.04, 15.76]8.27 [5.68, 11.11]4.13 [1.29, 6.98]
MiMo v2.6 FlashProofNet-Verified5.75 [3.29, 8.49]1.92 [0.55, 3.56]3.84 [1.37, 6.30]
MiMo v2.6 FlashPutnamBench3.72 [2.23, 5.36]1.19 [0.30, 2.23]2.53 [1.19, 4.02]

Table C3. Paired comparisons under BEq+Proof. Cell format and the interval procedure are the same as in Table C2.

ModelDatasetFull − FixedSignal − FixedFull − Signal
DeepSeek V4 FlashminiF2F2.05 [0.82, 3.48]1.64 [0.61, 2.87]0.41 [-0.41, 1.23]
DeepSeek V4 FlashFormalMath5002.84 [1.29, 4.65]2.58 [1.03, 4.39]0.26 [-0.78, 1.55]
MiMo v2.5miniF2F4.71 [2.87, 6.56]3.89 [2.25, 5.74]0.82 [-0.61, 2.46]
MiMo v2.5FormalMath5006.98 [4.39, 9.56]4.39 [2.58, 6.46]2.58 [0.52, 4.91]
MiMo v2.6 FlashminiF2F7.79 [5.53, 10.25]6.15 [4.10, 8.40]1.64 [-0.20, 3.48]
MiMo v2.6 FlashFormalMath50011.37 [8.27, 14.47]6.98 [4.65, 9.56]4.39 [1.81, 7.24]
MiMo v2.6 FlashProofNet-Verified5.48 [3.01, 7.95]1.37 [0.00, 3.01]4.11 [1.64, 6.85]
MiMo v2.6 FlashPutnamBench2.98 [1.49, 4.46]1.04 [0.15, 2.08]1.93 [0.74, 3.27]

The paired analyses show significant Full–Fixed gains in all eight settings under both ASCC+Proof and BEq+Proof. The additional gain from Full over Signal-only is significant in six of eight settings under ASCC+Proof and four of eight under BEq+Proof, including both undergraduate-level datasets, ProofNet-Verified and PutnamBench, under both metrics.

AuthorsOpenReview ↗

Common response to all reviewers(3/3)

End-to-end human evaluation

To evaluate performance directly against human judgments, we randomly sampled 50 records from the FormalMath500 results of each of three models: DeepSeek V4 Flash, MiMo v2.5, and MiMo v2.6 Flash. Sampling covered each model’s results, including cases outside the root-reaching subset, unchanged outputs, and unsuccessful proof searches. These 150 model inputs yield 184 distinct formalizations across Fixed, Signal-only, and Full. We matched the annotated formalizations to their experimental endpoints to obtain the corresponding proof outcomes.

Human annotators assess whether each formalization faithfully expresses the source task. Combining these labels with the proof outcomes gives Human-faithful+Proof, which requires both a faithful final statement and a kernel-checked proof of that statement.

Table C4. Paired human evaluation on FormalMath500. Entries are counts. The pooled row weights the three model samples equally (50 records each).

ModelNFixed faithfulSignal faithfulFull faithfulFixed faithful+ProofSignal faithful+ProofFull faithful+Proof
DeepSeek V4 Flash50414242404141
MiMo v2.550374244354042
MiMo v2.6 Flash50343840283335
Pooled150112122126103114118

Across the sampled inputs, human-fidelity rates are 74.67%, 81.33%, and 84.00% for Fixed, Signal-only, and Full, respectively. Requiring a proof as well as fidelity gives end-to-end success rates of 68.67%, 76.00%, and 78.67%, preserving the same ordering.

Human labels agreement with ASCC-3 and BEq

The same human labels also allow us to assess the automatic validators. We pool outputs from the three policies and count each unique problem–formalization pair once, yielding 184 distinct formalizations for comparison with ASCC and BEq.

Table C5. Automatic validation against human labels on the 184 distinct formalizations. Human labels serve as the reference; “positive” denotes faithful.

ValidatorTPFPFNTN
ASCC116252023
BEq117111937

Against these human labels, agreement, precision, and recall are 75.54%, 82.27%, and 85.29% for ASCC and 83.70%, 91.41%, and 86.03% for BEq, respectively.

Time and staffing constraints prevented us from obtaining two independent annotations for every sample, so we do not report inter-annotator agreement. This limitation is explicitly stated in the revised manuscript’s Limitations section.

Repairs and harmful revisions among root-reaching inputs

To examine how revisions change source fidelity, we conducted a separate audit of root-reaching records. The new MiMo v2.6 Flash experiments yielded 552 records that backtracked to the root. We randomly sampled 100 of these, with 25 each from miniF2F, FormalMath500, ProofNet-Verified, and PutnamBench, and compared the Initial, Full, and Gate-off formalizations. We count a repair when an initially incorrect statement becomes faithful, and a harmful transition when an initially faithful statement becomes incorrect.

Table C6. Semantic transitions in the 100-record gate comparison.

DatasetNInitially faithfulFull repairsGate-off repairsFull harmful transitionsGate-off harmful transitions
miniF2F25217/238/230/20/2
FormalMath50025115/245/240/10/1
ProofNet-Verified25414/216/213/43/4
PutnamBench25147/112/110/144/14
Pooled1002153/7921/793/217/21

Full produces 71/100 faithful final statements, compared with 35/100 for Gate-off. This difference reflects both more repairs and fewer harmful revisions: Full repairs 53/79 (67.09%) initially incorrect statements, versus 21/79 (26.58%) for Gate-off, and damages 3/21 (14.29%) initially faithful statements, versus 7/21 (33.33%). All three harmful Full revisions occur on ProofNet-Verified.

Pooling these 100 records with the original 137-record audit described in Appendix A.8, “Human Audit Protocol,” gives 237 audited records: 211 initially incorrect and 26 initially faithful. Across the two audits, Full repairs 149/211 (70.62%), compared with 102/211 (48.34%) for Gate-off, and causes 3/26 (11.54%) harmful transitions, compared with 9/26 (34.62%). The resulting final fidelity rates are 172/237 (72.57%) for Full and 119/237 (50.21%) for Gate-off.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Invitation for Further Discussion

We sincerely thank all the reviewers for your constructive feedback, which has been invaluable in helping us improve the quality of our manuscript. We would greatly appreciate it if the reviewers could let us know whether our rebuttal has adequately addressed your concerns. We would also be happy to provide further clarification and look forward to continuing the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.