- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
LeanPolish is a symbolic framework for generating verified local edits to Lean 4 proof and using those edits to investigate what LLMs learn from search-generated supervision.
It optimizes existing & compiling Lean proofs using four main transformations: tactic replacement, local fact generalization, unused-fact removal, and unreachable-code cleanup. Each accepted modification is Lean-checked, and the system records the original proof fragment, its replacement, the associated proof state, and attempted alternatives.
Contributions:
- A verified proof-editing pipeline and supervision dataset
- A analysis of selection artifacts in search-generated supervision
- A investigation of whether learned editing improves complete proofs
Critical strengths
The paper demonstrates that the validity of individual training examples does not imply a supervised learning task measures the intended capability. The first-success experiment is convincing and insightful. A trained ranker achieves 100p accuracy, but the same result is obtained by selecting the last candidate in a fixed search order, without using the goal or mathematical context. They construct complete candidate pools and repeat the analysis under a protocol that removes the deterministic ordering shortcut. The deletion control further strengthens the contribution here. Overall, I see this as the No.1 strength: the compelling diagnosis of selection bias in verified supervision.
The second strength is on the infra side: the combination of a practically useful data resource and a careful evaluation methodology. The dataset and experimental protocol create a useful foundation for future work on proof-edit supervision. The paper's willingness to report negative or qualified findings increases the credibility of its central claims and makes the resource more scientifically valuable.
Critical weaknesses
The paper convincingly demonstrates that fine-tuning improves performance on teacher-selected local editing tasks. However, these improvements do not consistently translate into better whole-proof optimization. Table 4 seems to show that, when the sampling and verification loops are matched, a frozen DeepSeek editor performs comparably to or slightly better than the trained DeepSeek editor. Although alternating symbolic and neural optimization achieves reduction on PutnamBench compared with symbolic fixed-point, the paper does not talk about how much of this advantage is specifically attributable to training? A corresponding frozen-editor alternating baseline is not here.
The whole-proof rewriting result is positive but modest in absolute terms and is evaluated on relatively small test sets. The trained rewriter still substantially underperforms the symbolic fixed-point optimizer on miniF2F. The comparisons with broader proof-optimization systems are also limited. Although the authors appropriately avoid treating published results from different proof samples as directly comparable, this leaves the practical competitiveness of this work insufficiently established.
The complete-menu experiment is conceptually important but seems that its evaluation has a gap between predictive accuracy and practical utility. The authors first collected pools for over 6k held-out states. Approximately 2/3 of the total are however excluded because they lack an acceptable target. So the result does not establish that the model can reliably distinguish states where a useful edit exists from those where it does not. Also, first-success verified search achieves higher best-candidate accuracy on the same evaluated pools, exceeding the trained ranker. These results do not invalidate the demonstration that goal-dependent signal exists, but they substantially weaken the case that the learned ranker provides a practical improvement over simply executing the symbolic search.
Questions for authors
Clarifications / additional analysis that resolve some of the main concerns above would be helpful. For example, some questions worth answering might be: Can the complete-menu ranker be evaluated on all candidate pools, including states without an acceptable edit? How robust are the improvements attributed to whole-proof fine-tuning, and can they be separated from additional search?
2 comments
Thank you for the detailed rebuttal and additional experiments. I appreciate that the authors have directly addressed my main concerns with new controlled comparisons.
The budgeted-search evaluation across all 6641 held-out states is particularly convincing. Recovering 81% of exhaustive savings with one Lean check per state, compared with 64% for the best frozen baseline, establishes a concrete practical benefit of the learned ranker beyond predictive accuracy. The matched frozen-editor alternating baseline also resolves my attribution concern, although it clarifies that the additional compression comes from generate-and-verify rather than training.
The whole-proof results are more nuanced. The improvement under small verification budgets provides meaningful evidence of training utility. However, the weaker results at larger budgets, lack of reliable gains on PutnamBench, and degradation for Qwen suggest that the benefits remain model- and setting-dependent. I appreciate that the revision explicitly acknowledges these limitations. The matched AXLE comparison also substantially strengthens the practical evaluation.
Overall, the new experiments address my principal objections and make the paper's contributions more convincing. I continue to view the selection-bias analysis and verified supervision resource as its strongest contributions, now complemented by clearer evidence of practical utility. I am raising my recommendation from Weak Acceptance to Acceptance, together with corresponding subscores. Well done!
Thank you for the careful reading. Each of your points is answered by a new experiment; the shared results are in the general response.
Q1. Can the ranker be evaluated on all candidate pools, including states without an acceptable edit? Yes. In a search with a fixed budget of Lean checks over all 6,641 held-out states (4,253 without an acceptable edit), ranker ordering recovers 81% of the exhaustive token savings with one check per state, against 64% for the best frozen model and 8% for symbolic menu order (table in the reply to qSEi). The ranker also indicates whether a state has any acceptable edit: its maximum score separates the two kinds of states with AUROC 0.83 (frozen DeepSeek 0.79), and skipping the lowest-scored half of the states keeps 71% of the exhaustive savings with half the checks. The ranker weights and the search script are now released (anonymous dataset and supplement).
First-success search is more accurate (76.9% vs 70.1%). Yes, but it pays in Lean checks. With at most 8 checks per state, ranker order recovers 99.96% of the exhaustive savings using 34,855 checks; menu order recovers 75.6% using 42,336, and needs 63,458 checks to reach 100%. The ranker's value is in needing fewer checks, not in replacing verification.
Attribution of the alternating gain to training. We added the missing frozen-editor alternation (dgCC, Q3; same schedule, sampling and verification, three sampling seeds each). Alternation still gains 2.6 to 3.5 pp over the symbolic fixed point, and the trained editor reaches the same reduction within noise (−0.1 and −0.6 pp, intervals include zero) at half the GPU time or less. Table 4 and §5 now attribute this gain to generate-and-verify.
Q2. How robust is the whole-proof gain, and can it be separated from additional search? Trained and frozen rewriters use the same search (16 samples per file, the same decoding seeds, the same verifier). The frozen model gets the prompt that works best for it (it does worse with the training prompt), so the comparison is conservative and the difference comes from training. The gain is largest where search is smallest (new Table 14, a secondary analysis of the same runs): with one sample and one Lean check per proof on all 351 miniF2F proofs, 8.63% vs 3.37% (+5.26 pp [+3.52, +7.13]); its interval stays above zero up to 8 samples, and 8 frozen plus 8 trained samples beat 16 of either.
Over five training seeds with three decoding seeds each (general response, point 1), the 16-sample gain has the same direction for every seed on miniF2F (+5.2 pp on the original 99 files, +1.5 pp on the other 252). It does not extend to PutnamBench, whose proofs are 8 to 19 times longer than the training proofs, or to a second base model (Qwen2.5-Coder), which already cuts aggressively and becomes more conservative. We agree the trained rewriter stays below the symbolic fixed point; the revision states the result as transfer of the teacher's edits, bounded by the teacher.
Competitiveness with broader optimizers. On identical inputs and toolchain we now compare with a fuller simplifier, AXLE, and with prompted and fine-tuned LLM rewriters (16 verified samples each). AXLE removes more on PutnamBench, LeanPolish removes more on miniF2F, and running both is best on both (35.9% and 28.5%; reply to qSEi). The LLM rewriters stay below LeanPolish's fixed point (11.7% and 16.9% vs 29.8% on miniF2F-99) and gain from it: a frozen rewriter reaches 27.6% after one LeanPolish pass, against 11.7% alone (Table 4).
LeanPolish is meant to work with trained whole-proof optimizers such as ProofOptimizer, not to replace them: it runs on CPU, verifies every edit, shortens their inputs, and releases the edits and candidate pools they can train on. We make no claim of out-compressing them. Their published reductions are on their own proof samples and, as you note, not directly comparable, and we found no public ProofOptimizer model.