- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes RAF, a training free framework for whole proof generation and repair in Lean 4. Given a theorem task, Direct first generates a complete proof; optional Mathlib declaration retrieval supplies formal context; if the proof is incomplete, Apollo applies a broader repair procedure and Lightweight Sorrifier then replaces only Lean diagnosed failing source spans and fills the resulting holes with local tactics. Every candidate is checked by a shared Lean verifier, and only a candidate that is accepted with no unresolved sorry is counted as “complete". The main experiment compares Direct, Direct+Apollo, and cumulative RAF on all 244 miniF2F test problems for five LLM backends. Relative to Apollo, Lightweight adds 2, 0, 5, 11, and 25 COMPLETE proofs (from 0 to 10.25 percentage points). Retrieval and DSP initial generation are reported separately in the appendix and should not be mixed with the main comparison.
Critical strengths
- The shared Lean verification boundary and explicit COMPLETE definition are important strengths: accepted sources containing sorry are not reported as completed proofs, reducing the risk that a repair method gains credit by deleting proof content.
- The main experiment fixes the full set of 244 problems and reports the complete outcome distribution for five backends. The model dependent marginal coverage of Lightweight, especially for o4 mini and o3 mini, gives some evidence for the local claim that heterogeneous failure modes may need different repair boundaries.
Critical weaknesses
- The paper does not establish sufficient novelty. RAF organizes whole proof generation, formal library retrieval, structural repair, local hole filling, and shared verification, but it lacks adequate related work positioning, strong independent baselines, and component level ablations showing that this combination itself is a new methodological contribution.
- The performance gains are small and unstable, and lack budget matched attribution. Relative to Apollo, Lightweight adds 2, 0, 5, 11, and 25 COMPLETE proofs (from 0 to 10.25 percentage points) across backends. The paper does not report Direct+Lightweight, alternative orders, an independent lightweight repair baseline, or a matched total model/Lean call budget, so the gains may reflect extra attempts rather than complementary coverage.
- The correctness and contribution of retrieval are unclear. The appendix only describes extracting syntax cues, BM25 ranking Mathlib declaration headers, and returning the top 10 names and type signatures. It reports no retrieval precision/recall, false positive/false negative analysis, index or Lean/Mathlib versions, exclusion rule for target theorems/declarations, or per problem evidence that retrieved hints are relevant. Retrieval even decreases completion for one backend, so the hints cannot be treated as reliable external knowledge without further validation.
- Reproducibility and cost reporting are too incomplete for the broad framework positioning. Exact Lean 4/Mathlib versions, Lightweight's M, the full tactic list and order, per problem time limits, model sampling settings, stage level call counts and latency, and replayable proof artifacts are not fully provided.
Questions for authors
- Please provide adequate related work positioning and strong independent baselines that do not rely on a single cumulative pipeline, and state precisely what new scientific contribution RAF adds beyond existing generation, retrieval, structural repair, and local repair components.
- Can the authors provide Direct+Lightweight, reverse order, independent lightweight, and matched total model/Lean call budget controls, with theorem level success set overlaps, stage call counts, and wall clock cost? This is needed to distinguish complementary coverage from extra budget.
- Please justify the Apollo followed by Lightweight order, the restart from the original Direct candidate, and the fixed tactic portfolio, with component ablations, representative failure cases, and duplicate/stopping rules.
- What exactly does “at least 5%” in the abstract mean: percentage points, relative percentage, or another comparison? How is it compatible with zero gain for GPT 5.6 Luna and 0.82 percentage points for Goedel V2 8B?
- RAF is described as a general compositional framework, but the main experiment evaluates only the sequence Direct, then Apollo, then Lightweight. Which claims are specific to that instantiation, and what evidence supports transfer to other generators, repairers, or execution orders?