- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper introduces self-advertisement, an approach to selecting mathematical methods for LLM-based formal theorem proving. The motivation is that conventional lexical and embedding-based retrieval methods identify mathematically related content but do not necessarily determine whether a method can be applied to the current proof goal.
The authors introduce Method Contracts, which describe a mathematical method through its applicability conditions, intended action, expected proof obligations, Mathlib references and Lean-checked execution scaffolds. Given a formal proof goal, an LLM examines the entire contract library and generates a problem-specific applicability proposal for each method. As the next step (which is optional), the remaining proposals are compared to improve top-ranked selections.
The authors construct 82 Method Contracts from Putnam 2000–14 solutions and evaluate method selection on 120 Putnam 2015–25 problems and 60 IMO ProofBench problems. The proposed method improves shortlist coverage.
The paper also evaluates the selected contracts in a Lean theorem-proving pipeline. Adding the top-ranked contract improves proof success on IMO ProofBench. The paper additionally provides (theoretical) observations concerning the limitations of similarity-only representations, ranking errors and conditional soundness of verified execution scaffolds.
Critical strengths
- A practically relevant problem.
- Substantial improvements in retrieval coverage on Putnam problems, and there is some evidence that the selected methods can help downstream tasks.
- Verified mathematical artifacts and clearly stated limitations.
Critical weaknesses
-
Limited methodological novelty
The central idea of selecting mathematical methods according to their preconditions, intended effects and applicability to a proof goal has been extensively explored in traditional automated theorem proving and proof planning.
The proposed self-advertisement mechanism appears to implement the general idea through LLM prompting: the model examines descriptions of candidate methods, generates structured applicability proposals, and ranks them according to its own applicability estimates. The use of an LLM and the Method Contract representation essentially provides an implementation of very traditional ideas.
Moreover, the theoretical results largely formalize elementary observations about information loss, candidate-pool restrictions, ranking accuracy and the soundness of applying verified proof terms. They do not establish novel properties of the proposed self-advertisement algorithm itself.
-
Unfair comparison Self-advertisement evaluates all 82 Method Contracts, whereas the embedding-based LLM rerankers are restricted to the top 10 candidates returned by the initial retrieval stage. As a result, the proposed method has access to substantially more candidate methods. Its superior hit@5 and recall@5 may therefore be explained, at least partly, by its ability to examine the entire library rather than by the effectiveness of the self-advertisement mechanism itself.
Indeed, the paper explicitly emphasizes that many correct methods fall outside the embedding-based candidate pools. This demonstrates a limitation of the chosen two-stage baselines but does not establish that self-advertisement is superior to an LLM reranker that also examines all 82 methods.
-
Experiment design does not isolate the contribution of method selection.
The downstream proving experiment compares an Ax-Prover-style baseline with the same prover augmented by the top-ranked Method Contract. However, the contract contains substantial additional information (e.g., mathematical descriptions, Mathlib references, verified scaffolds, worked examples and expected proof obligations). It is possible that the observed improvement arises from supplying additional mathematical knowledge or formal proof guidance, rather than from the particular method-selection algorithm.
(The paper explicitly acknowledges this limitation, but it appears to be a critical weakness because the main contribution is presented as a new method-selection approach.)
-
Insufficient ablation
The proposed ranking combines several decisions, for instance, generating applicability proposals, assigning scores, classifying methods into main or supporting roles, applying a commitment gate and optionally running a pairwise comparison procedure. However, the paper does not provide ablation experiments examining these components. In particular, it is unclear whether the improvement comes from examining all candidates, requiring structured applicability explanations, the commitment gate or the model's own applicability scores.
-
The theoretical analysis has little to do with the effectiveness of self-advertisement.
The theoretical propositions are reasonable but are not about the proposed algorithm. For instance, Prop. 1 follows directly from the standard typing and application rules of Lean. Prop. 2 states that retaining more information cannot reduce the optimal achievable prediction performance. Corollary 1 follows immediately from restricting reranking to a fixed candidate pool. Prop. 3 provides a conventional top-k ranking-regret bound under a uniform accuracy assumption on the relevant scores. The paper does not demonstrate that self-advertisement satisfies this assumption, nor does it estimate the relevant approximation error empirically. Prop. 4 applies the standard submodularity of coverage objectives, but the implementation does not use the associated greedy selection algorithm.
It appears that these propositions do not explain why the specific self-advertisement procedure should outperform alternatives or provide theoretical guarantees about its behaviour.
Questions for authors
Q1. What is the precise methodological contribution of self-advertisement beyond conventional applicability-aware method selection and proof planning?
Q2. Can the authors provide comparisons against full-library LLM ranking or applicability assessment, where every method is visible to all competing selectors? How much of the improvement remains when candidate coverage is controlled?
Q3. Can the authors provide ablation experiments isolating the effects of major components?
Q4. Can the authors identify any theoretical result specific to the proposed self-advertisement mechanism?