acceptodds
Under review as a conference paper at ICLR 2027

Let the Library Speak: Self-Advertised Method Selection for Formal Proving

Abstract

LLM-based formal provers can retrieve relevant lemmas and prior proofs, but relevance alone does not say whether a mathematical method can be used on the current theorem. A method has prerequisites, a target, an intended action, and obligations that its use leaves to prove. Methods that look equally related to a theorem may therefore differ substantially in whether they offer a plausible next step. We formulate this as an applicability-aware method-selection problem and introduce self-advertisement: before candidates are ranked, a model generates a problem-specific proposal for each one, stating what part of the goal it targets, what action it would take, and what conditions that action requires. We organize 82 reusable methods from Putnam 2000–2014 as Method Contracts, which pair applicability descriptions with Mathlib anchors, a checked example or scaffold, and expected proof obligations. A single batched call elicits proposals across the library; vague or unsupported proposals are demoted, yielding a ranked shortlist accompanied by inspectable claims about each candidate's use. We analyze when similarity-based representations cannot distinguish methods with different applicability, how errors in applicability estimates affect shortlist quality, and what a checked scaffold guarantees under its stated assumptions. Against lexical, embedding, and embedding-plus-LLM reranking baselines, self-advertisement achieves 95.0% hit@5 on Putnam 2015–2025, compared with 84.2% for the strongest reranker. On IMO ProofBench, it achieves 91.7% compared with 88.3%. These results indicate improved coverage of annotated methods in the retrieved shortlists, particularly on Putnam.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
formal theorem proving, large language models, applicability-guided retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_TbU1OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces self-advertisement, an approach to selecting mathematical methods for LLM-based formal theorem proving. The motivation is that conventional lexical and embedding-based retrieval methods identify mathematically related content but do not necessarily determine whether a method can be applied to the current proof goal.

The authors introduce Method Contracts, which describe a mathematical method through its applicability conditions, intended action, expected proof obligations, Mathlib references and Lean-checked execution scaffolds. Given a formal proof goal, an LLM examines the entire contract library and generates a problem-specific applicability proposal for each method. As the next step (which is optional), the remaining proposals are compared to improve top-ranked selections.

The authors construct 82 Method Contracts from Putnam 2000–14 solutions and evaluate method selection on 120 Putnam 2015–25 problems and 60 IMO ProofBench problems. The proposed method improves shortlist coverage.

The paper also evaluates the selected contracts in a Lean theorem-proving pipeline. Adding the top-ranked contract improves proof success on IMO ProofBench. The paper additionally provides (theoretical) observations concerning the limitations of similarity-only representations, ranking errors and conditional soundness of verified execution scaffolds.

Critical strengths
  • A practically relevant problem.
  • Substantial improvements in retrieval coverage on Putnam problems, and there is some evidence that the selected methods can help downstream tasks.
  • Verified mathematical artifacts and clearly stated limitations.
Critical weaknesses
  • Limited methodological novelty

    The central idea of selecting mathematical methods according to their preconditions, intended effects and applicability to a proof goal has been extensively explored in traditional automated theorem proving and proof planning.

    The proposed self-advertisement mechanism appears to implement the general idea through LLM prompting: the model examines descriptions of candidate methods, generates structured applicability proposals, and ranks them according to its own applicability estimates. The use of an LLM and the Method Contract representation essentially provides an implementation of very traditional ideas.

    Moreover, the theoretical results largely formalize elementary observations about information loss, candidate-pool restrictions, ranking accuracy and the soundness of applying verified proof terms. They do not establish novel properties of the proposed self-advertisement algorithm itself.

  • Unfair comparison Self-advertisement evaluates all 82 Method Contracts, whereas the embedding-based LLM rerankers are restricted to the top 10 candidates returned by the initial retrieval stage. As a result, the proposed method has access to substantially more candidate methods. Its superior hit@5 and recall@5 may therefore be explained, at least partly, by its ability to examine the entire library rather than by the effectiveness of the self-advertisement mechanism itself.

    Indeed, the paper explicitly emphasizes that many correct methods fall outside the embedding-based candidate pools. This demonstrates a limitation of the chosen two-stage baselines but does not establish that self-advertisement is superior to an LLM reranker that also examines all 82 methods.

  • Experiment design does not isolate the contribution of method selection.

    The downstream proving experiment compares an Ax-Prover-style baseline with the same prover augmented by the top-ranked Method Contract. However, the contract contains substantial additional information (e.g., mathematical descriptions, Mathlib references, verified scaffolds, worked examples and expected proof obligations). It is possible that the observed improvement arises from supplying additional mathematical knowledge or formal proof guidance, rather than from the particular method-selection algorithm.

    (The paper explicitly acknowledges this limitation, but it appears to be a critical weakness because the main contribution is presented as a new method-selection approach.)

  • Insufficient ablation

    The proposed ranking combines several decisions, for instance, generating applicability proposals, assigning scores, classifying methods into main or supporting roles, applying a commitment gate and optionally running a pairwise comparison procedure. However, the paper does not provide ablation experiments examining these components. In particular, it is unclear whether the improvement comes from examining all candidates, requiring structured applicability explanations, the commitment gate or the model's own applicability scores.

  • The theoretical analysis has little to do with the effectiveness of self-advertisement.

    The theoretical propositions are reasonable but are not about the proposed algorithm. For instance, Prop. 1 follows directly from the standard typing and application rules of Lean. Prop. 2 states that retaining more information cannot reduce the optimal achievable prediction performance. Corollary 1 follows immediately from restricting reranking to a fixed candidate pool. Prop. 3 provides a conventional top-k ranking-regret bound under a uniform accuracy assumption on the relevant scores. The paper does not demonstrate that self-advertisement satisfies this assumption, nor does it estimate the relevant approximation error empirically. Prop. 4 applies the standard submodularity of coverage objectives, but the implementation does not use the associated greedy selection algorithm.

    It appears that these propositions do not explain why the specific self-advertisement procedure should outperform alternatives or provide theoretical guarantees about its behaviour.

Questions for authors

Q1. What is the precise methodological contribution of self-advertisement beyond conventional applicability-aware method selection and proof planning?

Q2. Can the authors provide comparisons against full-library LLM ranking or applicability assessment, where every method is visible to all competing selectors? How much of the improvement remains when candidate coverage is controlled?

Q3. Can the authors provide ablation experiments isolating the effects of major components?

Q4. Can the authors identify any theoretical result specific to the proposed self-advertisement mechanism?

Reviewer_QkuBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies method selection for LLM-based formal proving. It organizes 82 mathematical methods into contracts containing applicability descriptions and checked Lean execution material. A model generates a goal-specific proposal for each contract, and the proposals are screened and ranked. The reported results show improved reference-method coverage, particularly on Putnam, and preliminary proving gains when the selected contract is supplied to a prover.

Critical strengths

The motivation is sound: retrieving a related method does not explain how to instantiate it or satisfy its prerequisites. Connecting method descriptions to reusable Lean scaffolds is also a useful engineering contribution. The Putnam shortlist gains are substantial, and I appreciate the authors’ explicit distinction between shortlist coverage and completed proofs.

Critical weaknesses
  1. The main comparison does not isolate the proposed mechanism. The rerankers only see the embedding retriever’s top 10 candidates, whereas self-advertisement considers all 82 contracts . Consequently, the gains may largely reflect access to a larger candidate pool rather than the benefit of generating and screening proposals. The candidate-pool ceiling explains this limitation of the baseline, but does not establish an advantage specific to self-advertisement. A direct LLM ranking of all 82 contracts, using the same descriptions, backbone, and a comparable token budget, is the essential missing control.
  2. The evaluation does not establish that the generated proposals are actually applicable. The labels identify methods appearing in reference solutions, while the gate checks proposal specificity and the presence of supporting citations, not whether the proposed binding is correct or its prerequisites can be established. A selector can therefore receive credit for naming the right method while proposing an incorrect use. Since problem-specific applicability is the central motivation, reference-method hit rates alone leave an important part of the contribution untested.
  3. The proving experiment does not close this gap. Table 2 compares a prover without contracts against one receiving the selected contract as a whole; it does not compare selectors using identical execution material. The improvement could therefore come from the scaffolds and Mathlib guidance rather than better selection. Moreover, the prover receives only the top-ranked contract, while the strongest retrieval result concerns top-five coverage. With Luna, self-advertisement’s Putnam hit@1 is below the strongest reranker, including after runoff (as what we‘ve seen in Table 1). This makes a selector-controlled proving comparison especially important.
Questions for authors
  1. Does self-advertisement outperform direct LLM ranking when both methods consider all 82 contracts under comparable budgets?
  2. How often are the proposed bindings, actions, and prerequisite claims correct, independently of whether the contract matches a reference label?
  3. Under the same prover and execution material, do contracts selected by self-advertisement yield more completed proofs than those selected by the strongest reranker?
Reviewer_f7dvOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes self-advertisement, an applicability-aware method selection approach for formal theorem proving. It constructs a library of 82 Method Contracts containing applicability conditions, Mathlib references, and Lean-checked scaffolds, and uses LLM-generated proposals to rank their suitability for a given theorem. Experiments on Putnam and IMO ProofBench show improved Hit@5 over similarity-based retrieval and LLM reranking baselines. Adding the selected contract to a prover also improves proof success rates, although the contributions of selection and contract content are not isolated.

Critical strengths

The method improves shortlist coverage across different models and benchmarks, particularly on Putnam problems. The downstream experiments also show that providing method contracts can help existing provers solve more problems.

Critical weaknesses
  1. Unfair candidate-pool comparison. Self-advertisement considers all 82 contracts, while the LLM reranking baselines only examine the top 10 retrieved candidates. Without a full-library LLM reranking baseline under a comparable budget, it is unclear whether the improvement comes from applicability reasoning or simply broader candidate coverage.

  2. Insufficient evidence for downstream effectiveness. The main improvement is in Hit@5, while the downstream prover only uses the top-ranked contract, where self-advertisement often underperforms reranking. Moreover, the proving experiment does not separate the benefits of method selection from the additional information provided by the contract.

  3. Limited component ablations. The paper does not isolate the effects of structured applicability proposals and the commitment gate. In particular, the gate checks the specificity of proposals rather than whether their prerequisites are actually satisfied.

  4. Limited theoretical contribution. Most propositions establish general or conditional properties of retrieval, ranking, and proof reuse. They do not provide specific guarantees for the proposed applicability estimates or explain the observed empirical improvements.

  5. Limited scalability evaluation. The method is evaluated with only 82 contracts, allowing the LLM to inspect the entire library in one call. Its effectiveness and computational efficiency for substantially larger method libraries remain unclear.

Questions for authors
  1. Can the authors compare self-advertisement against an LLM reranker that considers all 82 contracts under a comparable token budget? How much improvement remains after controlling for candidate-pool size?

  2. Can the authors provide ablations for structured proposals and the commitment gate, and evaluate downstream proving with contracts selected by different retrieval methods but supplied with identical execution information?

  3. How often do the selected methods have prerequisites that are actually satisfiable in the target context? Can the authors provide an empirical assessment of proposal correctness beyond agreement with reference-method annotations?

Reviewer_pN27OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the selection of reusable mathematical methods for formal theorem proving. Rather than ranking methods by their similarity to a given theorem, the authors propose self-advertisement, where each candidate method generates a problem-specific proposal describing its target, intended action, and required conditions.

The approach is built on a library of 82 Method Contracts extracted from Putnam solutions, each containing applicability descriptions and Lean-checked execution guidance. The proposals are screened and ranked, with an optional runoff stage to improve the top-ranked choice.

Experiments on Putnam 2015–2025 and IMO ProofBench show improved coverage of annotated reference methods compared with lexical, embedding, and LLM-reranking baselines. Adding the selected contract to an existing Lean proving loop also improves proof success rates, although this experiment does not separate the effects of method selection and contract content.

Critical strengths
  1. A well-motivated retrieval problem with convincing improvements in shortlist coverage: The distinction between method relevance and applicability is well motivated, and Table 1 demonstrates substantial improvements in shortlist coverage over the evaluated baselines.
  2. A practical representation of reusable mathematical methods: The Method Contracts connect applicability conditions and intended actions with formal execution guidance, including Lean-checked scaffolds and worked examples. The downstream proving results are also encouraging.
Critical weaknesses
  1. The main comparison does not isolate the benefit of self-advertisement from the benefit of considering the entire library: In Section 5.1, self-advertisement evaluates all 82 contracts, whereas the LLM reranking baselines consider only the top 10 candidates retrieved by embedding similarity. However, these results do not establish how much of the improvement comes from generating structured applicability proposals, as opposed to simply allowing an LLM to evaluate all 82 method descriptions. A direct full-library LLM ranking baseline, using the same descriptions and backbone but without self-advertisement, would help resolve this distinction. I consider this more important than adding further benchmarks.
  2. The downstream improvement is not clearly attributable to self-advertisement: Table 2 compares a prover without contract guidance to one given the contract selected by self-advertisement. This shows that adding a contract helps, but does not establish whether self-advertisement selects better methods than existing rerankers. With GPT-5.6 Luna, self-advertisement does not outperform the strongest reranker in hit@1 on either benchmark, even with the optional runoff. Since the downstream prover receives only the top-ranked contract, it remains unclear whether the observed proving gains are attributable to better method selection. Since the prover only receives the top-ranked contract, this difference is relevant. Although the authors acknowledge this limitation, the absence of a controlled comparison leaves the contribution of the selection mechanism to downstream proof success unresolved.
Questions for authors
  1. How would a direct LLM ranking or applicability-scoring baseline perform if it were given the same complete library of 82 contracts? This would help determine whether the structured self-advertisement step contributes beyond removing the similarity-based candidate restriction.
  2. Could the authors clarify how unsupported prerequisites are detected by the commitment gate? The stated gate does not appear to explicitly check whether the required conditions hold in the current goal. For example, is the missing nonnegativity assumption in Figure 1 detected through the model's own nomination decision or through an independent check?
  3. For the downstream experiment, how would proof success change if the same Method Contracts were supplied using the strongest reranker's selections? Was the optional runoff used to select the contract in Table 2?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.