acceptodds
Under review as a conference paper at ICLR 2027

LeanPolish: Verified Supervision for Lean Proof Compression

20 views

Abstract

Verified proof edits offer a natural source of supervision for improving language-model-generated Lean proofs. Yet verification establishes that an edit is correct, not that its training signal is free of search artifacts. We introduce LeanPolish, a symbolic Lean 4 pipeline that releases 33,402 accepted local edits and 65,596 same-state failed attempts, and use it to study what models learn from this supervision. First-success search admits a goal-independent rule with perfect ranking accuracy; teacher-selected evaluation sites also reward trivial deletions. Continuing menu evaluation beyond the first success removes the ordering shortcut: a trained ranker selects the best candidate on 70.1% of evaluated held-out states, versus 36.9% for the strongest frozen baseline. For compression, iterating the symbolic pass raises miniF2F savings from 19.7% to 27.5%, exceeding the neural hybrids we test there. Verified neural editing helps on other proof sources, but matched frozen-model controls show that its gains need not come from training. The supervision does improve whole-proof rewriting: fine-tuning raises verified token reduction from 2.8% to 5.5% on 19 PutnamBench proofs. Together, the released edits, complete candidate pools, and controlled evaluations separate learning to imitate a search policy from improving on that search. They provide a reproducible basis for studying proof improvement while keeping correctness, compression, and edit policy distinct.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Lean 4, formal theorem proving, proof compression, neurosymbolic, verified supervision, proof editing, selection effects, program synthesis

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 4.00 over 3 reviews, above 99% of the others.
Reviewer_M71cOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

LeanPolish is a symbolic framework for generating verified local edits to Lean 4 proof and using those edits to investigate what LLMs learn from search-generated supervision.

It optimizes existing & compiling Lean proofs using four main transformations: tactic replacement, local fact generalization, unused-fact removal, and unreachable-code cleanup. Each accepted modification is Lean-checked, and the system records the original proof fragment, its replacement, the associated proof state, and attempted alternatives.

Contributions:

  • A verified proof-editing pipeline and supervision dataset
  • A analysis of selection artifacts in search-generated supervision
  • A investigation of whether learned editing improves complete proofs
Critical strengths

The paper demonstrates that the validity of individual training examples does not imply a supervised learning task measures the intended capability. The first-success experiment is convincing and insightful. A trained ranker achieves 100p accuracy, but the same result is obtained by selecting the last candidate in a fixed search order, without using the goal or mathematical context. They construct complete candidate pools and repeat the analysis under a protocol that removes the deterministic ordering shortcut. The deletion control further strengthens the contribution here. Overall, I see this as the No.1 strength: the compelling diagnosis of selection bias in verified supervision.

The second strength is on the infra side: the combination of a practically useful data resource and a careful evaluation methodology. The dataset and experimental protocol create a useful foundation for future work on proof-edit supervision. The paper's willingness to report negative or qualified findings increases the credibility of its central claims and makes the resource more scientifically valuable.

Critical weaknesses

The paper convincingly demonstrates that fine-tuning improves performance on teacher-selected local editing tasks. However, these improvements do not consistently translate into better whole-proof optimization. Table 4 seems to show that, when the sampling and verification loops are matched, a frozen DeepSeek editor performs comparably to or slightly better than the trained DeepSeek editor. Although alternating symbolic and neural optimization achieves reduction on PutnamBench compared with symbolic fixed-point, the paper does not talk about how much of this advantage is specifically attributable to training? A corresponding frozen-editor alternating baseline is not here.

The whole-proof rewriting result is positive but modest in absolute terms and is evaluated on relatively small test sets. The trained rewriter still substantially underperforms the symbolic fixed-point optimizer on miniF2F. The comparisons with broader proof-optimization systems are also limited. Although the authors appropriately avoid treating published results from different proof samples as directly comparable, this leaves the practical competitiveness of this work insufficiently established.

The complete-menu experiment is conceptually important but seems that its evaluation has a gap between predictive accuracy and practical utility. The authors first collected pools for over 6k held-out states. Approximately 2/3 of the total are however excluded because they lack an acceptable target. So the result does not establish that the model can reliably distinguish states where a useful edit exists from those where it does not. Also, first-success verified search achieves higher best-candidate accuracy on the same evaluated pools, exceeding the trained ranker. These results do not invalidate the demonstration that goal-dependent signal exists, but they substantially weaken the case that the learned ranker provides a practical improvement over simply executing the symbolic search.

Questions for authors

Clarifications / additional analysis that resolve some of the main concerns above would be helpful. For example, some questions worth answering might be: Can the complete-menu ranker be evaluated on all candidate pools, including states without an acceptable edit? How robust are the improvements attributed to whole-proof fine-tuning, and can they be separated from additional search?

2 comments

Thank you for the careful reading. Each of your points is answered by a new experiment; the shared results are in the general response.

Q1. Can the ranker be evaluated on all candidate pools, including states without an acceptable edit? Yes. In a search with a fixed budget of Lean checks over all 6,641 held-out states (4,253 without an acceptable edit), ranker ordering recovers 81% of the exhaustive token savings with one check per state, against 64% for the best frozen model and 8% for symbolic menu order (table in the reply to qSEi). The ranker also indicates whether a state has any acceptable edit: its maximum score separates the two kinds of states with AUROC 0.83 (frozen DeepSeek 0.79), and skipping the lowest-scored half of the states keeps 71% of the exhaustive savings with half the checks. The ranker weights and the search script are now released (anonymous dataset and supplement).

First-success search is more accurate (76.9% vs 70.1%). Yes, but it pays in Lean checks. With at most 8 checks per state, ranker order recovers 99.96% of the exhaustive savings using 34,855 checks; menu order recovers 75.6% using 42,336, and needs 63,458 checks to reach 100%. The ranker's value is in needing fewer checks, not in replacing verification.

Attribution of the alternating gain to training. We added the missing frozen-editor alternation (dgCC, Q3; same schedule, sampling and verification, three sampling seeds each). Alternation still gains 2.6 to 3.5 pp over the symbolic fixed point, and the trained editor reaches the same reduction within noise (−0.1 and −0.6 pp, intervals include zero) at half the GPU time or less. Table 4 and §5 now attribute this gain to generate-and-verify.

Q2. How robust is the whole-proof gain, and can it be separated from additional search? Trained and frozen rewriters use the same search (16 samples per file, the same decoding seeds, the same verifier). The frozen model gets the prompt that works best for it (it does worse with the training prompt), so the comparison is conservative and the difference comes from training. The gain is largest where search is smallest (new Table 14, a secondary analysis of the same runs): with one sample and one Lean check per proof on all 351 miniF2F proofs, 8.63% vs 3.37% (+5.26 pp [+3.52, +7.13]); its interval stays above zero up to 8 samples, and 8 frozen plus 8 trained samples beat 16 of either.

Over five training seeds with three decoding seeds each (general response, point 1), the 16-sample gain has the same direction for every seed on miniF2F (+5.2 pp on the original 99 files, +1.5 pp on the other 252). It does not extend to PutnamBench, whose proofs are 8 to 19 times longer than the training proofs, or to a second base model (Qwen2.5-Coder), which already cuts aggressively and becomes more conservative. We agree the trained rewriter stays below the symbolic fixed point; the revision states the result as transfer of the teacher's edits, bounded by the teacher.

Competitiveness with broader optimizers. On identical inputs and toolchain we now compare with a fuller simplifier, AXLE, and with prompted and fine-tuned LLM rewriters (16 verified samples each). AXLE removes more on PutnamBench, LeanPolish removes more on miniF2F, and running both is best on both (35.9% and 28.5%; reply to qSEi). The LLM rewriters stay below LeanPolish's fixed point (11.7% and 16.9% vs 29.8% on miniF2F-99) and gain from it: a frozen rewriter reaches 27.6% after one LeanPolish pass, against 11.7% alone (Table 4).

LeanPolish is meant to work with trained whole-proof optimizers such as ProofOptimizer, not to replace them: it runs on CPU, verifies every edit, shortens their inputs, and releases the edits and candidate pools they can train on. We make no claim of out-compressing them. Their published reductions are on their own proof samples and, as you note, not directly comparable, and we found no public ProofOptimizer model.

Thank you for the detailed rebuttal and additional experiments. I appreciate that the authors have directly addressed my main concerns with new controlled comparisons.

The budgeted-search evaluation across all 6641 held-out states is particularly convincing. Recovering 81% of exhaustive savings with one Lean check per state, compared with 64% for the best frozen baseline, establishes a concrete practical benefit of the learned ranker beyond predictive accuracy. The matched frozen-editor alternating baseline also resolves my attribution concern, although it clarifies that the additional compression comes from generate-and-verify rather than training.

The whole-proof results are more nuanced. The improvement under small verification budgets provides meaningful evidence of training utility. However, the weaker results at larger budgets, lack of reliable gains on PutnamBench, and degradation for Qwen suggest that the benefits remain model- and setting-dependent. I appreciate that the revision explicitly acknowledges these limitations. The matched AXLE comparison also substantially strengthens the practical evaluation.

Overall, the new experiments address my principal objections and make the paper's contributions more convincing. I continue to view the selection-bias analysis and verified supervision resource as its strongest contributions, now complemented by clearer evidence of practical utility. I am raising my recommendation from Weak Acceptance to Acceptance, together with corresponding subscores. Well done!

Reviewer_qSEiOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3(+1)
Presentation
3
Soundness
3
Read review
Paper summary

LeanPolish shortens Lean proofs with a rule-based tool and checks that the edits preserve correctness. It releases successful edits, failed attempts, and complete sets of candidate edits. The experiments expose shortcuts in editing and ranking tests. Fine-tuning helps whole-proof rewriting, but repeated rule-based optimization and models without fine-tuning often match or exceed the trained editors.

Critical strengths
  1. The dataset links each successful edit to the original text, proof state, and failed alternatives. Complete candidate sets distinguish valid edits, failures, timeouts, and edits rejected by the tool’s rules. Together with the verification scripts and model outputs, this makes the release useful for studying edit generation and selection.
  2. The controls expose concrete shortcuts. A rule that ignores the mathematical goal matches perfect ranking scores by exploiting search order. Complete candidate sets remove this shortcut, and the goal ablation shows useful information remains. Separating deletion sites from tactic replacements also reveals how a prompt cue can inflate editing scores.
Critical weaknesses
  1. The compression method is incremental. Trying shorter proof commands, removing unused facts, and deleting unnecessary steps overlap with existing tools. Combining similar local facts is a more distinct part of the method, but it accounts for only 0.8% of accepted edits and has no demonstrated overall compression benefit. The main external comparison on identical inputs is with Lean’s unused-tactic linter, which covers only part of this functionality. A comparison with a fuller simplifier such as AXLE or ProofOptimizer on the same proofs and Lean version, even on a shared subset, would better show what LeanPolish adds.
  2. The benefit of training for complete-proof compression is still modest and needs more testing. The authors appropriately show that models without fine-tuning can match trained local editors, and that repeated rule-based cleanup can outperform the tested neural pipelines on miniF2F. Whole-proof fine-tuning improves compression on 19 PutnamBench and 99 miniF2F proofs, but stays below the repeated rule-based baseline and adds no consistent benefit when used after LeanPolish. These whole-proof experiments use one trained checkpoint per setup. The three-seed tests cover only local editing at sites selected by the tool. Repeating the whole-proof training comparison across seeds and adding more PutnamBench proofs would show how stable the positive result is.
Questions for authors
  1. Can the learned ranker be tested inside a search procedure with a fixed budget of Lean checks? The current ranking test excludes 4,253 states with no acceptable edit. It also ranks candidates by character length, while the main compression results count tokens. Reporting verified token savings, failed proposals, and the number of Lean checks across all held-out states would show whether the ranking improvement is useful in practice.
  2. Can Table 4 report runtime or verification-call counts for repeated rule-based cleanup and the pipelines using models with and without fine-tuning? The release run and clean rerun give quite different PutnamBench results because of resource limits. Comparing methods under a stated computation budget would make their compression gains easier to interpret.
2 comments

Thank you for the careful reading.

W1. What LeanPolish adds over a fuller simplifier, on the same proofs and Lean version. We ran AXLE's simplify_theorems (Lean 4.21.0) on the Table 4 inputs. The Goedel-Prover-V2 files disable Lean's linters in their header, and AXLE then makes no edit, so we removed those lines for AXLE and restored the header before counting.

Whole-corpus reduction (%)miniF2F-99PutnamBench-19
LeanPolish, fixed point29.816.1
AXLE27.125.6
LeanPolish, then AXLE35.928.5
AXLE, then LeanPolish34.028.1

All 395 changed final files pass a fresh-process audit (compilation, unchanged statements, no new sorry or axioms).

  • Each tool wins on different inputs. AXLE removes more on PutnamBench, almost entirely unused haves and inert <;> try arms; LeanPolish removes more on miniF2F.
  • They are complementary. LeanPolish adds 6.9 pp (miniF2F) and 2.6 pp (PutnamBench) after AXLE, and the combination is our best result on both.
  • What LeanPolish adds is verified tactic replacement under a specificity policy, which a deletion tool does not attempt, and the recorded supervision: every edit with its proof state, failed alternatives and complete candidate pool.
  • AxiomProver gives no comparison. AXLE returns all 12 files unchanged and reports a failed simplification on 7 of them (App. F.4).

We agree on fact generalization (0.8% of edits, no aggregate benefit); the paper makes no compression claim for it. Of the two systems you suggested, AXLE is publicly runnable and we found no public ProofOptimizer model.

W2. Whole-proof training across seeds, and more PutnamBench proofs. Please see general response, point 1: five training seeds, each with three decoding seeds, and 15 new PutnamBench problems (every further problem with a compiling proof). The gain holds on miniF2F for every seed, most clearly at small budgets (one sample per proof: 8.6% vs 3.4% on all 351 proofs; Table 14), and not on PutnamBench, whose proofs are 8 to 19 times longer than the training proofs, so the revision no longer claims a gain there.

Q1. The ranker inside a search with a fixed budget of Lean checks, on all held-out states, in tokens. On all 6,641 held-out states, including the 4,253 without an acceptable edit, each policy orders the candidates using only information available before verification, spends at most b Lean checks per state, and keeps the first acceptable edit. Share of the exhaustive token savings (38,144 tokens):

Ordering policyb = 1b = 4Checks to try every candidate
Symbolic menu order7.7%46.9%63,458
Per-tactic prior (training pools)34.8%66.9%60,132
Frozen DeepSeek-7B log-prob63.5%89.3%56,632
Trained ranker81.1%95.4%54,886

Failed proposals: with one check per state the ranker finds an acceptable edit in 27.4% of states (menu order 4.2%; 36.0% have one), so 4,772 of its 6,594 checks fail, against 6,318 for menu order. Characters vs tokens: menu tactics are single tokens, so choosing the character-shortest acceptable candidate loses none of the 38,144 tokens.

Q2. Runtime and verification counts for Table 4; release run vs clean rerun. Per run on PutnamBench-19 (neural pipelines: mean per seed; Table 18; Table 19 covers the other two input sets). REPL checks test a proposal on its proof state; file compilations are fresh whole-file checks:

MethodREPL checksFile compilationsGPU-s (H100)Reduction (%)
LeanPolish, one pass019011.9
LeanPolish, fixed point055016.1
Hybrid, frozen / trained editor1,048 / 83936 / 43128 / 769.3 / 9.0
Alternation, frozen / trained editor3,000 / 1,980160 / 156438 / 20218.7 / 18.6
Whole-proof rewrite, frozen / trained047 / 134430 / 4503.6 / 5.2
AXLE018025.6

LeanPolish also runs in-process tactic checks that it does not all log; a replay gives about 4,200 for one pass on these files. Hybrid and alternation start from the release LeanPolish output (6.1%), whose cost is not included. The symbolic fixed point and AXLE need no GPU; the neural pipelines add GPU time and thousands of REPL checks for +2.6 pp. Two comparisons are made at a stated budget: the ranker (Lean checks per state, Q1) and the rewriter (samples per proof, Table 14). Across pipelines we report costs and do not equalize them.

Release vs rerun (6.1% vs 11.9%): five files carry 99% of the gap. Re-running the same binary under load (10.4%, vs 11.2% idle) reproduces the release values on three of them exactly or closely. The other two (76% of the gap) have no release output. They are the two slowest files of the cohort, and the slowest runs at the release's 30-minute limit under load, so we attribute this to time limits on the shared release server (App. I).

Reviewer_qSEiOpenReview ↗

Response

Dear Authors,

Thank you for your thoughtful rebuttal and for the additional experiments. Your responses have addressed all of my concerns except the one regarding computational costs. I am therefore happy to increase both informativeness and general rating.

Best of luck,

Reviewer

Reviewer_dgCCOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LeanPolish produces verified local proof edits through symbolic tactic replacement, fact generalization, and removal of unused material. The release contains 33,402 accepted edits, 65,596 failed attempts, and complete candidate pools for about 60,000 states. The paper studies whether these records teach useful editing or merely reproduce the search procedure. It identifies shortcuts from first-success logging and teacher-selected evaluation sites. Complete candidate pools retain learnable information: ranking accuracy reaches 70.1%, versus 51.0% without the goal and 36.9% for the strongest frozen baseline. Whole-proof experiments give a mixed result. Symbolic iteration outperforms the tested neural hybrids on miniF2F. Fine-tuning improves standalone rewriting, but provides no consistent advantage after symbolic optimization.

Critical strengths

The controls materially change the interpretation of the results. First-success ranking reaches 100% using menu order alone. A deletion rule reproduces 558 reference edits without a model. Fixed-point and frozen-model controls further separate training gains from additional search. These findings provide useful guidance for evaluating supervision produced by search (§§4–5).

The resource supports inspection and controlled experiments. Local records connect edits, states, alternatives, and verification outcomes. Complete pools distinguish failures, timeouts, policy rejections, and length exclusions. The goal ablation provides evidence of useful information beyond the original proof span. Together, these features support research beyond a single compression benchmark (§3, §4.2).

Critical weaknesses
  1. The practical value of learned editing remains limited.

    Frozen local editors match or exceed trained editors under the same hybrid loop. Symbolic iteration also outperforms the tested neural hybrids on miniF2F. The alternating pipeline improves compression on PutnamBench and AxiomProver, but lacks a matched alternating frozen-editor control. Consequently, these additional savings do not establish a benefit from training.

    The paper acknowledges these limitations. However, they constrain the practical significance of the learned components. The experiments establish useful supervision and conditional imitation more clearly than improvements beyond strong symbolic and frozen-model baselines (§5, Table 4).

  2. The positive whole-proof results need clearer uncertainty reporting and consistent baseline selection.

    Whole-proof experiments use one trained checkpoint per configuration, with 99 miniF2F files and only 19 PutnamBench proofs. The three-seed checks cover conditional local editing, so they do not establish the stability of the whole-proof gains.

    An additional analysis of the released outputs reproduces the reported trained-model reductions. Against the stronger released frozen control, paired file bootstrapping gives:

    DatasetTraining gainApproximate 95% confidence interval
    miniF2F, 99 files+3.84 percentage points−3.18 to +11.09
    PutnamBench, 19 files+2.48 percentage points+0.03 to +5.32

    These intervals come from 100,000 paired file resamples using the paper’s whole-corpus token-reduction metric. They measure variation across files, not training seeds. The miniF2F interval includes zero, while the PutnamBench interval barely excludes zero. These results support a possible benefit, but leave substantial uncertainty about its magnitude and stability.

    Baseline selection also needs clarification. Table 4 reports 2.76% reduction for the frozen PutnamBench rewriter, whereas the released whole-proof results report 3.05%. Against that stronger control, the trained model’s 5.53% reduction gives a gain of 2.48 percentage points. The authors should reconcile these runs and report uncertainty for the comparisons supporting the claimed training benefit (§5.2, Appendix E).

Questions for authors
  1. Clarification: Which frozen run supports the 2.76% PutnamBench result, and why does the released whole-proof experiment report 3.05%?
  2. Validation request: How stable are the whole-proof gains under paired file resampling and additional training seeds?
  3. Validation request: Does the alternating trained editor outperform an alternating frozen editor with matched iteration counts, sampling, and specificity constraints?
2 comments

Thank you for the detailed analysis; it directly shaped the new experiments.

Q1. Which frozen run gives 2.76%, and why does the whole-proof experiment report 3.05%? Both are the same configuration (frozen DeepSeek-Prover-V2-7B, same prompt, token budget, sampling settings and seed): 2.76% is the run in the submitted Table 4, 3.05% the run of the whole-proof experiment. Their raw samples are identical on short generations and diverge on long ones (GPU nondeterminism), which changes the outcome on 3 of 19 files. The trained 5.53% comes from the whole-proof run, so 3.05% is its matched comparator (+2.48 pp, as you computed). Because single runs vary in this way, the revision replaces both by means over three decoding seeds for every model (frozen 3.6%, trained 5.2%) and reports the two frozen single runs in a footnote of Table 4 and in App. F.2. The new runs used other GPUs, and single-run values moved in both directions (frozen, seed 42: 12.5% to 11.7% on miniF2F, 3.05% to 4.2% on PutnamBench), so the mean gains below differ from the single-run gains you computed.

Q2. Stability under paired file resampling and additional training seeds. Your intervals reproduce with our scripts (miniF2F +3.84 pp [−3.21, +11.06]; PutnamBench +2.48 pp [+0.04, +5.30]). We then trained four more rewriters (seeds 1 to 4, identical configuration) and decoded every model three times (table in the general response):

  • Training seeds: on miniF2F-99 every seed beats the frozen model (15.0% to 17.6% vs 11.7%), and again on the 252 other miniF2F problems (+1.5 pp).
  • Decoding seeds: SD 0.3 to 0.5 pp for the frozen model and 0.1 to 1.4 pp per checkpoint.
  • Files: on miniF2F-99 the trained rewriter is better on 76 files and worse on 15 (sign test p < 1e−10); the token-weighted interval still includes zero because three long files dominate the sum.
  • Sample budget (new Table 14): on all 351 miniF2F proofs the gain is +5.3 pp [+3.5, +7.1] with one sample per proof, its interval stays above zero up to 8 samples, and it narrows to +2.4 pp [−1.1, +6.0] at 16.
  • PutnamBench: +1.6 pp on the original 19 and −0.5 pp on 15 new problems, so the revision no longer claims a gain there. The two sets do not differ significantly (2.2 pp [−1.3, +6.5]); the 19 are the set used in every table of the submission, and the 15 are all remaining problems with a compiling proof.
  • A second base model (Qwen2.5-Coder), which already cuts aggressively, is made more conservative by the same fine-tuning (general response).

The revision states the result accordingly: training on LeanPolish pairs transfers the teacher's edits, which makes a prover's rewrites reliable at small budgets, and does not go beyond the teacher.

Q3. Does the alternating trained editor outperform an alternating frozen editor? No: it reaches the same reduction at lower GPU cost. We ran the paper's alternating schedule with a frozen and with a trained editor, with identical iteration counts, sampling (greedy plus 4 samples per site), verification and specificity setting, three sampling seeds each (one trained editor checkpoint):

Whole-corpus reduction (%)PutnamBench-19AxiomProver-12
Symbolic fixed point16.071.76
Alternation, frozen editor (3 seeds)18.715.24
Alternation, trained editor (3 seeds)18.634.62
Frozen alternation minus fixed point+2.63 [+0.76, +4.37]+3.48 [+2.27, +4.75]
Trained minus frozen−0.08 [−1.62, +1.53]−0.62 [−2.00, +0.65]

The alternating gain is real and comes from generate-and-verify: the two editors reach the same reduction within noise. The trained editor needs no few-shot prompt, so it uses half the GPU time or less; on AxiomProver it is slightly lower in every seed. We used one trained editor here because trained editors are stable across training seeds on the local task (three seeds within 1.4 points, §4). Our single trained run in the submission (20.25% / 5.05%) started from a hybrid output produced on other hardware and lies above the three-seed mean; Table 4 now reports the means for both editors.

On the practical value of learned components. We agree that trained local editors do not beat frozen ones in our pipelines, and the revision says so. Training helps where it saves verification. The ranker trained on complete pools, in a budgeted verified search over all held-out states, recovers 81% of the exhaustive savings with one Lean check per state, against 64% for the best frozen model and 8% for menu order (details in the reply to qSEi); its weights are now in the anonymous dataset. The fine-tuned rewriter, with one sample per proof, shortens 58% of the 351 miniF2F proofs against 12% for the frozen model.

Reviewer_dgCCOpenReview ↗

Follow-up on the author response

Thank you for the detailed response and additional experiments. The reported additions substantially address my concerns.

The baseline clarification resolves the comparison issue: 3.05% is the matched frozen comparator for the original 5.53% trained result. Replacing single-run comparisons with repeated decoding, and reporting both original runs, improves the presentation.

The additional training seeds and paired uncertainty estimates address my stability concern. The results support a narrower and more useful conclusion. Training helps DeepSeek on miniF2F at small verification budgets, while the aggregate token-saving advantage remains uncertain at 16 samples. The low-budget analysis should remain identified as secondary. The revised treatment of PutnamBench and the negative Qwen result appropriately clarify the limits.

The matched alternating frozen-editor control also resolves my attribution concern. It supports attributing the additional compression to generate-and-verify, without establishing a compression advantage from training.

The budgeted ranker evaluation provides useful new evidence of practical value, including on states without acceptable edits. The AXLE comparison further establishes complementarity with a stronger simplifier.

These results strengthen the resource and evaluation contributions that I valued in the original submission. I am increasing my rating to clear acceptance and updating my soundness and informativeness assessments.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response

We thank the reviewers, who raise three shared questions: (1) is the whole-proof training gain stable, (2) does training explain the gain of the alternating pipeline, and (3) are the learned ranker and LeanPolish useful in practice against stronger baselines. New experiments answer each; seeds, cohorts and budgets of the main comparisons were fixed in a written protocol beforehand, and every such configuration is reported. Changes in the revised PDF are in blue; new experiments are in Appendix F. Scripts, outputs and the ranker weights are in the updated supplement and the anonymous dataset.

1. Whole-proof training gain. We trained five DeepSeek-Prover-V2-7B rewriters with different seeds and decoded each, and the frozen model, with three seeds (16 samples per file). The frozen model uses the prompt that works best for it. We also trained Qwen2.5-Coder-7B with the same data and recipe (two seeds). Token reduction (%):

Model, cohortFrozenTrained (mean ± SD)Gain, pp [95% CI]Seeds above frozen
DeepSeek, miniF2F original 9911.716.9 ± 1.1+5.2 [−0.7, +11.2]5/5
DeepSeek, miniF2F other 252 (one decoding)13.715.2 ± 0.8+1.5 [−2.7, +5.8]5/5
DeepSeek, PutnamBench original 193.65.2 ± 0.9+1.6 [−0.7, +5.1]5/5
DeepSeek, PutnamBench 15 new problems4.03.5 ± 0.8−0.5 [−3.3, +1.9]1/5
Qwen, miniF2F original 9916.99.2 ± 0.7−7.7 [−12.9, −3.4]0/2

On miniF2F every trained DeepSeek seed beats the frozen model (better on 76 vs 15 and 177 vs 53 files). With 16 samples its size is uncertain (all 351 files: +2.4 pp [−1.1, +6.0]). For Qwen, training lowers the reduction. This is not a prompt effect: frozen Qwen with the training prompt reaches 14.0%.

Where the trained rewriter is useful: small verification budgets (new Table 14, a secondary analysis of the same runs). On all 351 miniF2F proofs, with one sample per proof, it saves 8.63% vs 3.37% (+5.26 pp [+3.52, +7.13]) and shortens 58% of the proofs vs 12%; the interval stays above zero up to 8 samples. The two are complementary: 8 frozen plus 8 trained samples give 20.7%, above 16 of either (+5.1 pp [+3.2, +7.2] over trained).

The samples suggest why: fine-tuning makes a model edit like LeanPolish, with frequent, small, verified cuts. This helps DeepSeek-Prover, which returns the proof unchanged in 77% of prompted samples. It makes Qwen, which already makes large cuts, more reliable but more conservative (median cut 31% to 8%). PutnamBench proofs are 8 to 19 times longer than the training proofs, so the learned cuts are small there.

This is the paper's distinction between imitating a search and improving on it: training on the teacher's final outputs teaches its edits, not how to go beyond them. On PutnamBench the gain is not reliable on either set: +1.6 pp on the 19 proofs used in every table of the submission, and −0.5 pp on the 15 new problems, which are all remaining problems with a compiling proof and about twice as long. The two sets do not differ significantly (2.2 pp [−1.3, +6.5]). The revision therefore drops the single-run figure (2.8% to 5.5%) and states the whole-proof result as transfer of the teacher's edits, useful when few samples can be verified.

2. Training in the alternating pipeline (dgCC, M71c). With a matched frozen editor (same schedule, sampling and verification; three sampling seeds each), alternation still beats the symbolic fixed point (+2.6 pp on PutnamBench, +3.5 pp on AxiomProver, intervals above zero), so the gain is from generate-and-verify. The trained editor reaches the same reduction within noise (−0.1 and −0.6 pp, intervals include zero) at half the GPU time or less: it needs no few-shot prompt.

3. The learned ranker in practice (qSEi, M71c). In a search with a fixed budget of Lean checks over all 6,641 held-out states, including the 4,253 with no acceptable edit, ranker ordering recovers 81% of the exhaustive token savings with one check per state, against 64% for the best frozen model, 35% for a per-tactic prior and 8% for menu order. This is the clearest benefit of training on LeanPolish data.

4. External comparison (qSEi, M71c). On identical inputs the tools are complementary: AXLE removes more dead code on PutnamBench (25.6% vs 16.1%), LeanPolish compresses more on miniF2F (29.8% vs 27.1%), and LeanPolish followed by AXLE gives our best result on both (35.9% and 28.5%). On AxiomProver, AXLE returns every file unchanged and reports a failed simplification on 7 of 12, so we draw no comparison there. All 395 changed files pass a fresh-process audit.

In short. The contributions the reviewers valued most, the released supervision with complete pools and the diagnosis of selection effects, are unchanged. The new experiments show where training pays off (the ranker and the rewriter both cut the verification a search needs) and that LeanPolish adds to a stronger deletion tool.

AuthorsOpenReview ↗

Follow-up response

We thank all three reviewers for the follow-up comments and for raising their scores. Two points remain that we address below.

Reviewer qSEi: computational cost. The revised PDF now compares the two alternating editors at a stated budget (new Table 20, App. F.3; a secondary analysis of the saved runs). Both pipelines save their output after every round (one editor pass, then one LeanPolish pass), so we can report what each reaches for a given amount of compute. On PutnamBench-19, mean of three seeds:

EditorRounds doneREPL checksGPU-secondsReduction (%)
Frozen11,04812815.5
Frozen22,01927717.7
Frozen33,00043818.7
Trained18397615.9
Trained21,41813717.8
Trained31,98020218.6

A REPL check is one test in Lean of one replacement proposed by the editor. Both pipelines start from the release LeanPolish output, where the proofs are already 6.1% shorter. LeanPolish repeated on its own reaches 16.1%, so the alternation passes that level in round 2, not round 1.

Reading the table at a fixed budget:

  • About 130 GPU-seconds: the trained editor has completed two rounds (17.8%), the frozen editor one (15.5%). Difference +2.3 pp [+0.2, +4.3]; the trained editor used more REPL checks.
  • About 2,000 REPL checks: the trained editor has completed all three rounds (18.6%), the frozen editor two (17.7%). Difference +0.9 pp [−0.8, +2.5], within noise.

So the trained editor reaches a given reduction with less GPU time, and the final reductions do not differ. On AxiomProver-12 the two editors are within noise at every round (Table 20).

Other pipelines: Table 18 gives the same counts for every pipeline, including whole-proof rewriting. LeanPolish and AXLE need no GPU (16.1% and 25.6%; 28.5% combined), and 438 GPU-seconds of alternation add 2.6 pp to LeanPolish alone.

Units: Budgets are stated in Lean checks and GPU-seconds, which do not depend on machine load. We do not compare CPU wall-clock times, because the CPU stages ran under varying load (App. F.3).

Reviewers dgCC and M71c. The sample-budget analysis will stay identified as a secondary analysis in the final version, and the limits of the whole-proof result (larger budgets, PutnamBench, Qwen) will stay stated.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.