acceptodds
Withdrawn from ICLR 2027

Library Card: Automated Repository Knowledge Interface for Lean Theorem Proving

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Automated theorem proving (ATP) in large Lean repositories requires both retrieving relevant premises and providing repository-specific information that supports their effective use. Premise retrieval is central to repository-aware theorem proving, but premise availability and proof-time context are often evaluated jointly. To address this, we introduce Library Card, an automated repository knowledge interface for Lean theorem proving. Library Card automatically constructs and organizes repository-specific knowledge, then uses it to independently control the premises available for selection and the context provided around selected premises. This separation enables a controlled study of whether proving performance is limited by finding additional premises or by using available repository knowledge more effectively. On a 50-theorem benchmark from Optlib, the baseline formal-plus-semantic retriever recovers all 37 observable earlier-same-file source-proof premise occurrences at , while Card-based expansion rarely changes the selected premise set. With selected premises and their order fixed, the observed solve rate is 54% with rich context versus 44% with compact context, although the paired difference is statistically inconclusive. Additional controls show no aggregate advantage for the tested structured or compiled representations. These results suggest that when the evaluated retrieval budget already covers the observable premises in scope, progress may depend less on expanding candidate availability and more on how repository-specific knowledge is provided to the prover. Our source code and data are available at https://anonymous.4open.science/r/Library-Card.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Automated Theorem Proving, Formal Reasoning, Lean, Premise Retrieval, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_8JtKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a knowledge repository system named Library Card for repository-aware Lean theorem proving. It separates the premises available for selection and the context about the selected premises, so that the most effective factor for proving performance can be checked. The experiment on a 50-theorem benchmark from Optlib is conducted to show the effectiveness of the proposed method. The work is significant for improving automated theorem proving techniques.

Critical strengths

None

Critical weaknesses

The contrast experiments to support benefits of the proposed method are not conducted. The controls that separate the premises and the context show no advantage over flat or raw context. In addition, the optimization strategy on retrieval efficiency is not studied.

Questions for authors

None

1 comment

We thank the reviewer for the constructive assessment and for highlighting the need to better isolate Library Card's contributions. To address these concerns, we conducted two additional controlled evaluations that provide more direct evidence of the benefits of repository-grounded knowledge and historical usage examples.

Additional Experimental Evidence.

First, we evaluated Compact retrieval context against a theorem-only No-Card baseline on the same 50 Optlib targets, with three independent runs per target and arm. Compact achieved 69/147 (46.94%) Lean-verified successes versus 51/147 (34.69%) for No-Card. Across 144 complete pairs, the equal-target-weighted improvement was +12.33 percentage points (target-cluster bootstrap 95% CI [+2.00,+23.00] pp). This provides repeated evidence that explicitly supplying retrieved repository knowledge improves observed downstream proof performance.

Second, to investigate the contribution of historical usage examples, we completed a new 450-trajectory, three-arm ablation (50 targets × 3 arms × 3 runs), comparing:

  • FORMAL_ONLY: identical frozen selected formal premises.
  • VERIFIED_USAGE: the same premises plus historically grounded, Lean-verified usage examples.
  • RAW_USAGE: the same premises plus raw Lean snippets from matching historical usage occurrences.

Unlike our earlier compiled/raw comparison, this experiment controls the underlying source occurrences, allowing us to distinguish the value of additional usage information from that of its representation.

The original execution produced 423 scientific outcomes and 27 infrastructure-unresolved cases. A separately documented supplemental recovery obtained scientific outcomes for 26 additional cases while preserving the original records.

In the supplemental analysis, VERIFIED_USAGE achieved 85/150 (56.67%) successes versus 78/150 (52.00%) for FORMAL_ONLY, an improvement of +4.67 percentage points. More importantly, among the 37 example-eligible targets defined before execution, VERIFIED_USAGE improved over FORMAL_ONLY by +9.91 percentage points, with a descriptive target-cluster bootstrap 95% interval of [+1.80,+18.92] pp.

These results provide encouraging evidence for our central motivation: concrete historical usage patterns can help provers apply retrieved declarations beyond simply supplying their formal statements.

Interpretation and Future Improvements.

The reviewer also noted the limited advantages observed in our earlier structured/flat and compiled/raw comparisons. We believe the new experiments help clarify this observation.

The structured/flat comparison examined different presentations of similar information, and its comparable results suggest that the prover can utilize such knowledge without requiring a particular layout. Similarly, in the new source-matched ablation, VERIFIED_USAGE and RAW_USAGE both achieved 67/111 (60.36%) successes on the 37 eligible targets.

Since both conditions provide the same underlying historical usage occurrences, these findings are consistent with the interpretation that the additional theorem-application knowledge, rather than its particular presentation format, is the primary source of the observed benefit. We will make this distinction clearer in the revised manuscript.

Regarding retrieval efficiency, Library Card primarily aims to improve how retrieved knowledge is utilized in downstream proving, rather than introduce a faster search algorithm. The new ablation demonstrates the practical value of usage information under identical Top-10 premise selection and prover settings, without increasing the number of retrieved formal premises. We also appreciate the reviewer's suggestion concerning retrieval efficiency and plan to investigate more efficient indexing, retrieval, and context selection strategies in future work.

Summary.

Together, these two evaluations strengthen the empirical foundation of Library Card. The repeated No-Card comparison demonstrates the incremental benefit of retrieved repository knowledge, while the new source-matched ablation provides evidence that historically grounded, Lean-verified usage examples improve proof success beyond formal premises alone, particularly when eligible examples are available.

These findings reinforce Library Card's central contribution: automatically constructing reusable repository knowledge and providing automated provers with concrete, verifiable theorem-application experience. We will incorporate these experimental findings and the refined mechanism analysis into the revised manuscript.

Reviewer_9quxOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Library Card, a system that automatically organizes knowledge from a Lean repository into structured cards and uses it both to add candidate premises before selection and to present the selected premises with additional descriptions, dependencies, and usage examples.

Critical strengths

The paper asks an clear question: whether proving success in a Lean repository depends more on which premises are retrieved or on how those premises are presented. And experiments are conducted separately.

Critical weaknesses
  1. Limited novelty and limited significance of the studied setting. Most of the main components of Library Card have close precedents in related works. For example, Natural-language descriptions and dependency information for declarations are provided by LeanExplore. Structured domain knowledge exposed to provers appears in LAMP. Moreover, many recent prover systems let the prover query the library itself during proof search. In this setting, the question of which fixed 10 premises to provide, and how to present them, matters much less.

  2. Insufficient experimental evidence. (a) All proving experiments use one repository, 50 theorems and a single run per theorem. No comparison is statistically significant. The main result (rich 27/50 vs. compact 22/50, p = 0.30) is within the variation the authors observe between nearly identical inputs. (b) Candidates are limited to earlier declarations in the same file (median about 16), so K = 50 usually returns all of them, so the full recovery of premises at K = 50 says little about retrieval quality. (c) Flattened text performs the same as structured cards, and raw source context is no worse than curated context. Hence it seems any benefit cannot be attributed to Library Card.

Questions for authors
  1. Could the authors clarify which aspects of Library Card they see as most novel relative to related systems such as LeanExplore and LAMP?
  2. Given the single run per theorem and condition, it would be helpful to see results from multiple trajectories with per-theorem solve rates.
  3. Since Optlib is publicly available, could the authors comment on possible training-data contamination?
  4. Four runs are reported as execution-unresolved (one in A10 and three in A11). Could the authors briefly explain what caused these runs to be unresolved and whether they were rerun?
1 comment

We thank the reviewer for the constructive comments. We conducted two matched evaluations of retrieved knowledge and verified usage examples.

Q1 — Novelty relative to LeanExplore, LAMP, and tool-using provers.

Library Card automatically constructs and delivers repository-grounded knowledge through declaration, concept, theorem-family, and strategy records, linking retrieved premises to historical, Lean-verified usage examples. The contribution is not individual descriptions, dependencies, or a new ranking algorithm.

Beyond declaration discovery emphasized by LeanExplore, we address how provers apply retrieved lemmas, including instantiation and composition. Unlike LAMP's domain-specific ontology-guided approach, we emphasize automatic population of repository knowledge and its integration into proving. Cross-repository generalization remains untested. Fixed Top-10 is an experimental control; usage examples could also complement dynamic retrieval.

Q2 — Evidence and ablations.

(1) Compact vs. No-Card: On 50 Optlib targets with three runs per arm (300 trajectories), Compact achieved 69/147 (46.94%) successes versus 51/147 (34.69%) for theorem-only No-Card. Across 144 complete pairs, the equal-target-weighted difference was +12.33 pp (target-cluster bootstrap 95% CI [+2,+23] pp). Conditional sign-swap p=0.0317, but significance depends on six unresolved outcomes. This supports incremental value over theorem-only input, not superiority over conventional retrieval.

(2) Usage ablation: We ran 450 slots (50 targets × 3 arms × 3 runs), comparing identical frozen premises under FORMAL_ONLY, VERIFIED_USAGE, and RAW_USAGE. The latter two use matching historical source occurrences, separating usage information from its presentation.

The original execution produced 423 scientific outcomes and 27 infrastructure-unresolved cases. Separately documented supplemental recovery obtained 26 additional outcomes without overwriting the original record.

In supplemental results, FORMAL_ONLY achieved 78/150 (52.00%), VERIFIED_USAGE 85/150 (56.67%), and RAW_USAGE 86–87/150 (57.33%–58.00%), with one outcome missing.

Among 37 example-eligible targets defined before execution, VERIFIED_USAGE exceeded FORMAL_ONLY by +9.91 pp (descriptive target-cluster bootstrap 95% interval [+1.80,+18.92] pp); the all-50 difference was +4.67 pp. This provides encouraging evidence for concrete lemma-usage information. VERIFIED_USAGE and source-matched RAW_USAGE both achieved 67/111 (60.36%) on eligible targets. Hence, the benefit may stem from usage knowledge itself, not its compiled format. We will distinguish these claims. Post hoc recovery does not restore original confirmatory significance.

Regarding saturation, 40/50 targets have at most 50 eligible earlier-same-file candidates. All 37 observable reference-premise occurrences entered first-stage retrieval, but only 30 survived Top-10. We will clarify this selection loss and limitations of earlier structured/flat and compiled/raw comparisons.

Q3 — Potential training-data contamination.

We acknowledge that the external prover API's model may have encountered public Optlib during pretraining; we cannot verify its training corpus or exclude context-triggered recall. However, the Compact–No-Card comparison directly tests whether explicitly supplied Library Card knowledge helps despite possible shared exposure. Using the same model, targets, and proof protocol, Compact increased observed Lean-verified success from 34.69% to 46.94% (+12.33 pp paired target-weighted). Thus, the observed benefit does not rely on assuming an uncontaminated model: supplied repository knowledge provides incremental assistance beyond theorem-only prompting under the same model conditions. We will clarify historical-source provenance limitations.

Q4 — Four unresolved historical runs.

These reflect infrastructure, not proof failures: two model-identity mismatches, one missing durable completion, and one repeated connection failure. Independent recovery runs did not replace primary records; unresolved outcomes were excluded from applicable complete-pair analyses.

Summary. New evidence supports automatic repository knowledge construction and concrete usage assistance, without establishing an independent advantage for compiled presentation over source-matched raw examples.

Reviewer_LrwPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Library Card, which organizes declaration, concept, theorem-family, and strategy knowledge for Lean repositories and separates candidate premise discovery from the information presented around selected premises. Formal, semantic, and optional Card-link retrieval feed a shared reranker that selects at most ten premises, followed by Compact or Rich context. The main evaluation uses 50 Optlib theorems, with retrieval restricted to legal earlier declarations in the same file. The authors report that Card expansion rarely changes the selected premises. With premise identities and order fixed, Rich solves 27 tasks versus 22 for Compact, although the paired difference is statistically inconclusive. Additional controls show no aggregate advantage for the tested structured or compiled representations.

Critical strengths

Separating candidate discovery from the use of selected premises is a useful experimental design. The Compact/Rich pairs fix premise identities and order, reducing the possibility that an apparent improvement is simply due to one condition receiving more useful lemmas. The paper also includes an information- and budget-matched flat control, a post-hoc raw-relevant control, results in both the Rich-only and Compact-only directions, and statistical uncertainty. These analyses give the work diagnostic value: it goes beyond end-to-end scores to examine discovery, selection, and use, while also reporting conditions that do not establish an advantage.

Critical weaknesses

The evaluation does not establish added value over straightforward source access. Retrieval is restricted to earlier declarations in the same file, and 40/50 targets have at most 50 legal candidates under a per-channel budget of K=50. Baseline discovery already covers 37/37 observable source-proof premise occurrences, while Cards change only 2/50 final Top-10 sets. Candidate discovery is therefore largely saturated.

Source prefixes are a relevant baseline: across the 50 targets, the median prefix contains 329.5 lines and 4,909 tokens. For the highlighted e₂_subseq_converge_zero case, an adjacent preceding proof matches the original target proof after variable substitutions. Comparisons should include complete prefixes, a file-tool agent, reranking all legal declarations, and established retrievers on the same Optlib scope. The existing raw-relevant control fixes selected premises and leaves these alternatives unevaluated.

Cost effectiveness also remains unclear. Rich increases mean context from 1,592 to 15,431 tokens for 27 versus 22 successes, with a paired 95% interval of [-4,24] percentage points. Without independent identical-input repeats and complete execution costs, the evidence does not establish a reliable benefit that justifies the additional expense.

The provenance-isolation guarantee for Rich lacks independently verifiable implementation and examples. A legal declaration identity does not ensure legal evidence for its annotations. If an earlier lemma's summary sees the target or later proofs during whole-library preprocessing, it may absorb solution hints without retaining the target's name. Deleting a target reference from used_by at evaluation time does not guarantee removal of those hints. Lean verification that a usage example is valid also does not establish that it was available before the target.

The paper states that target-time filtering is applied, but the actual masker, field-level transitive provenance, verified-use materialization, and complete Rich inputs are unavailable, as acknowledged in Appendix J. It is therefore unclear whether filtering precedes generation or operates on already-generated summaries, and how information derived from unavailable evidence is handled. The paper's projection example has a source preceding the target and is not itself evidence of leakage. The concern is an inability to audit a guarantee directly affecting the validity of the Compact/Rich comparison, rather than an established leakage finding.

Questions for authors
  1. Which declarations and proofs are accessible when generating semantic records and usage evidence? If summaries are generated over the full library, how is transitive provenance tracked, and how is target- or later-proof information already absorbed into natural language removed at evaluation time? Please show the generation inputs, field provenance, and Rich contents before and after filtering for one actual evaluation target. This would directly affect the assessment of provenance isolation.

  2. Can the prover read the target prefix, imported project files, and Mathlib sources, or use file search? Please provide the complete input templates, tool permissions, and the exact raw material included in the raw-relevant control. In particular, could the prover see the preceding e₁ proof for e₂_subseq_converge_zero? These details would establish whether the existing controls already cover preceding source context as a natural alternative.

  3. Please specify the representations, tokenization, and indices used for first-stage retrieval in the frozen experiments, and provide per-target legal universes, channel candidate pools, and final Top-10 lists. For targets whose legal universe is fully covered at K=50, how do Cards change the candidates or final selection? Could you compare against reranking all legal declarations, providing the complete legal prefix, and established retrievers on the same Optlib scope, reporting proof success and total cost to distinguish the contributions of candidate discovery, reranking, and added context?

  4. Are independent runs with identical complete model inputs available, together with per-target online calls, total tokens, and elapsed time including failures and retries, and separate offline construction costs? If so, please provide these records and an analysis of paired variability to assess the reliability and cost effectiveness of 27/50 versus 22/50. If unavailable, repeated runs and resource-matched comparisons would provide additional evidence.

Comments on the paper

2 comments

Dear Reviewer LrwP,

The authors have responded to your comments. Could you please review their responses and update your reviews?

Thanks for the reminder. I’ve read the authors’ response and posted a follow-up. The new Prefix experiment and provenance checks are helpful, but my main concerns about the missing Rich–Prefix comparison and the provenance of the Rich context remain. I’m keeping my original rating for now.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.