- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the gap between Lean formalization and successful proofs. When a proof fails, the method checks the Lean statement again with the original proposition. It doesn't directly change the theorem based on proof errors. Experiments on 3 models and 4 datasets show better end2end results than the baseline.
Critical strengths
1.When a proof fails, the method checks the Lean statement again with the original proposition. Semantic checking is used to avoid wrong changes.
2.Experiments on 3 models and 4 datasets show better end2end results than the baseline, and using the same prover models and token budgets.
Critical weaknesses
1.Signal-only already achieves most of the improvement. Full gives only 0-2.8 pp. The paper lacks statistical testing and ablation studies for key components.
2.Semantic checking depends on LLMs. Only 5 of 137 cases were correct at first. So, 0 harmful changes in these 5 cases is not enough to prove the safety. The evaluation only covers two math datasets.
Questions for authors
1.Can the authors test more formalizations that are correct at first and report the harmful rate? Can the authors also report the agreement between annotators?
2.Is the improvement between Full and Signal-only significant? Can the authors test 3 pre-pruning checks separately to show each contribution?
Based on the new experiments, Full outperforms Signal-only among 8 experimental settings. It is helpful. However, I still concern the number of initial faithful cases is relatively small, even though the new version increased from 5 to 21. The authors still don't report the agreement between annotators.
3 comments
Thank you again for your thoughtful review; if time permits, we would appreciate hearing whether our response has addressed your concerns or whether any further clarification would be helpful.
Reviewer kG1S, thanks for your review. Could you take a look at the authors’ response and comment on which of your concerns have or have not been resolved?
Response to Reviewer kG1S
Thank you for your thoughtful suggestions.
1. The incremental value of Full over Signal-only and its statistical significance (Weakness 1; Question 2).
Signal-only already captures much of the benefit of allowing semantic revision. Comparing Full with Signal-only asks whether concrete diagnoses provide further help beyond a generic proof-failure signal. The table below reports this difference across all eight settings; complete results and the statistical procedure appear in the common response, “Extending Evaluation to Undergraduate-Level Datasets,” Tables C1–C3.
Full–Signal-only paired effects (percentage points [pointwise 95% CI]). We use paired percentile-bootstrap intervals without multiple-comparison adjustment and consider a difference significant when its interval excludes zero.
Full achieves higher success rates in all eight settings under both validators, with significant gains in 6/8 settings under ASCC+Proof and 4/8 under BEq+Proof. Concrete diagnoses therefore provide an additional benefit in several settings, although the remaining intervals include zero, particularly for DeepSeek V4 Flash. Notably, the gains are significant on both ProofNet-Verified and PutnamBench under both metrics, suggesting that diagnostic information may be especially useful for more challenging problems.
2. Separate contributions of the three pre-pruning checks (Weakness 1; Question 2).
We have clarified that pre-pruning is an auxiliary proof-search module. Full, Signal-only, and Fixed all use the same pruning configuration, allowing us to examine our core question: whether proof-stage feedback improves semantic revision and end-to-end outcomes. We no longer present pre-pruning as a standalone contribution and have moved its detailed analysis and grouped ablation to Appendix D, “Auxiliary Analysis: Grouped Pre-Pruning Ablation.” Separate ablations of the three checks remain for future work.
3. Harmful revisions, reliance on LLM checking, and annotator agreement (Weakness 2; Question 1).
We agree that the original 0/5 observation cannot establish safety. The additional 100-record gate comparison includes more initially faithful statements: Full introduces errors in 3/21, compared with 7/21 for Gate-off. It also repairs more initially incorrect statements (53/79 versus 21/79). These results show a better balance between repair and harm with the semantic gate, while confirming that harmful revisions still occur. Sampling details and results appear in the common response, “Repairs and harmful revisions among root-reaching inputs,” Table C6.
To assess faithful task completion beyond the root-triggered subset, we also manually evaluated 150 randomly sampled FormalMath500 model inputs. Human-faithful+Proof success is 68.67% for Fixed, 76.00% for Signal-only, and 78.67% for Full, an observed Full–Fixed gain of 10.00 pp. The common response, “End-to-end human evaluation,” Table C4, provides the full results.
This sample also lets us compare the automatic validators with human judgments. Across its 184 distinct formalizations, agreement is 75.54% for ASCC and 83.70% for BEq. These findings support the usefulness of the validators while showing that neither fully captures human judgments. Confusion matrices, precision, and recall appear in the common response, “Human labels agreement with ASCC-3 and BEq,” Table C5 and its accompanying text.
Time and staffing constraints prevented us from obtaining two independent annotations for every sample, so we do not report inter-annotator agreement. We explicitly acknowledge this limitation in the revised manuscript’s Limitations section.
4. Evaluation limited to two datasets (Weakness 2).
To broaden the evaluation, we added ProofNet-Verified and PutnamBench, two undergraduate-level datasets, using MiMo v2.6 Flash. Both preserve the ordering Full > Signal-only > Fixed under both validators. BEq+Proof success for Full / Signal-only / Fixed is 41.37% / 37.26% / 35.89% on ProofNet-Verified and 15.18% / 13.24% / 12.20% on PutnamBench. Complete results and confidence intervals appear in the common response, “Extending Evaluation to Undergraduate-Level Datasets,” Tables C1–C3.