acceptodds
Under review as a conference paper at ICLR 2027

PocketLean: A Lightweight Mathlib Retriever for Proof Agents

Abstract

We present PocketLean, a lightweight Mathlib retriever for proof agents that learns from their own retrieval calls. Agents query Mathlib, a large library that changes across versions, repeatedly as they write, check, and revise, so retrieval must be cheap to run and to re-index. Pairing each recorded call with the lemmas its Lean-checked final proof uses yields query-level weak supervision. After multi-source query training, REvo (Replay-to-Rollout Evolution) applies this supervision over generations, first on recorded trajectories and then on rollouts collected with the updated retriever, so that the supervision follows the queries each update elicits. We also introduce ProofCall, a benchmark of 1,999 recorded agent calls in five sets, each labeled with the proof-used lemmas judged relevant to it, complementing the static queries of the public Lean Finder benchmark. Agents not told how to phrase their queries mostly issue approximate formal statements rather than natural-language questions, the form on which PocketLean is strongest. The 0.6B PocketLean reaches 78.9% mean Recall@10 on the Lean Finder benchmark against 77.9% for the 7B Lean Finder model, and the highest hit@10 on all five ProofCall sets, 4.3 to 7.6 points above the strongest external system. In end-to-end proving on subsets we curate from miniF2F and ProofNet, it more than doubles the success rate without retrieval, scores highest among the evaluated retrievers, and cuts the prover's tokens per model call by 36%. It needs only a single encoder and a per-version Mathlib index, and on a CPU it encodes 15 times as many queries per second as Lean Finder; we will release the weights, code, and data.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
formal theorem proving, Lean 4, Mathlib, LLM agents, self-improving retrieval

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ePmzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PocketLean, a lightweight Mathlib retriever for theorem proving agents. The approach involves first performing a multi-source contrastive training on query–declaration pairs. It then uses R2 Evo, which improves the retriever using supervision extracted from Lean-verified proofs. This process involves replay-based training on historical agent trajectories as well as rollout-based training on newly-collected trajectories using updated retrievers.

The authors also introduce ProofCall–a benchmark of 1,999 recorded retrieval calls from proof agents. PocketLean attains 78.9% Recall@10 on the Lean Finder benchmark and 74.0% hit@10 on ProofCall, outperforming larger retrieval baselines. In end-to-end theorem proving, it attains 8.8% average proof success versus 3.4% without retrieval and 7.7% with LeanSearch v2.

Critical strengths
  1. Practical and efficient Mathlib retrieval: PocketLean achieves strong retrieval performance compared with substantially larger models while only requiring a 0.6B parameter encoder. The reported CPU query-encoding throughput is about 14.6x that of Lean Finder. This is a significant practical contribution, especially for proof agents which can involve frequent querying of large, version dependent theorem libraries.

  2. Evaluation on realistic agent queries and analysis of proof-derived supervision: ProofCall tackles an important shortcoming of static retrieval benchmarks by evaluating on queries generated during attempts to perform theorem proving. The attribution experiments also provide valuable insight that associating all queries with all lemmas used in a proof generates noisy supervision, whereas call level attribution generates more query relevant training examples.

Critical weaknesses
  1. The methodological contribution of R2 Evo is not well isolated: Training retrievers from agent trajectories is similar in spirit to prior approaches such as LRAT, and the rollout loop resembles expert iteration, as the authors note. Proof-verified supervision and query-level attribution make PocketLean different, but the experiments don't clearly isolate the benefits of collecting fresh rollouts with updated retrievers. B1 and B2 use different data sources, attribution approaches, and training procedures, so the gains from the B2 updates do not demonstrate that it is the rollout feedback loop - rather than additional or better training data - that is responsible for them. While retrieval performance is measured for G0-G5, the end-to-end proving experiments only use the final checkpoint. As written, the paper does not establish that R2 Evo improves downstream proof success beyond the initial contrastive training step.

  2. Limited evidence for downstream superiority over competing retrievers: The end-to-end experiments only use 69 theorem proving tasks. PocketLean improves average proof success from 7.7% with LeanSearch v2 to 8.8% (1.1 percentage points). While the experiment includes multiple provers and multiple attempts, the paper does not report confidence intervals or perform paired significance tests for these results. Thus, we do not know how generalizable this relatively small improvement is. The efficiency gains are useful in and of themselves, but more statistical power is needed to claim downstream accuracy advantages.

Questions for authors
  1. Contribution of individual training stages: The authors should report end-to-end theorem proving performance for G0, G3 and G5 using the same evaluation protocol. This would show whether improvements in retrieval quality lead to downstream proof success.

  2. Effect of fresh rollouts: Can the authors compare B2 to a training condition that uses the same amount of extra historical data, controlling for attribution strategy and training budget? This would allow isolation of the contribution of updating the data distribution by way of agent interaction.

  3. Statistical reliability. Could the authors provide paired confidence intervals or statistical significance analysis for the end-to-end comparisons, particularly PocketLean vs. LeanSearch v2 and Lean Finder?

2 comments
AuthorsOpenReview ↗

Response to Reviewer ePmz

We thank the reviewer for the clear summary and for highlighting the practical value of efficient retrieval and our attribution analysis.

On isolating the effect of fresh rollouts (W1, Q2). Controls (b) and (c) of the General Response separate fresh rollouts from additional data. In control (b), starting from G3, we repeated the exact G4 and G5 recipes but replaced only the rollout-derived (B2) pairs with historical trajectories upsampled to the same pair counts (31,218 and 30,287 call pairs, and 40,000 proof-state pairs per round), keeping attribution, retained data, optimization budget, seeds, and averaging weights identical. G5 leads this historical analogue by +1.1 [+0.2, +2.0] in ProofCall hit@10 and +1.5 [+0.2, +2.8] in hit@1. Control (c) isolates the retriever deployed during collection: we re-collected the second rollout round with G3 deployed instead of G4 and kept everything else identical. G5 leads the resulting model, G5c, by +1.2 [+0.4, +2.1] in hit@1 and +0.3 [−0.5, +1.0] in hit@10, and the last round's hit@1 gain over G4 appears only when the updated retriever is deployed. A single non-iterative run on all lineage data with matched training examples is 2.5 [+0.9, +4.0] points behind G5 in hit@10 and 5.2 points behind in hit@1 (control (a)). G5 is ahead in all three controls, and its hit@1 gain is significant in each (+5.2, +1.5, +1.2).

Regarding LRAT and expert iteration, the revision will state explicitly that our distinguishing contributions are proof-verified, call-level supervision and its application across retriever generations. Controls (b) and (c) address the reviewer's remaining question, whether collecting fresh rollouts with the updated retriever adds value beyond this supervision.

On end-to-end results for G0 and G3 (W1, Q1). End-to-end proof success increases monotonically from G0 to G3 to G5, matching the retrieval gains across generations. We evaluated G0, G3, and G5 under one protocol, with the 69 problems and five provers of Table 4: eight attempts per prover-problem pair, one submission without compiler feedback, and at most four searches per attempt. Because the unversioned deepseek-v4-flash endpoint from Table 4 was retired, all three stages use the dated snapshot deepseek-v4-flash-0731. Success rates (%):

Stageqwen3.8-flashqwen3.8-maxglm-5.2deepseek-v4-flash-0731gemini-3.7-flashAll
G06.59.84.78.712.38.4
G38.510.55.89.112.19.2
G58.310.76.210.015.910.2

Average proof success rises from 8.4% (G0) to 9.2% (G3) to 10.2% (G5), and G5 outperforms G0 on every prover.

On statistical reliability of the end-to-end comparison (W2, Q3). Under paired problem-level testing, PocketLean significantly beats no retrieval and matches or slightly exceeds much larger retrievers. Pairing systems across the 69 problems in the Table 4 runs (40 attempts per problem, bootstrap over problems), the paired bootstrap 95% CI of the mean difference is +5.4 [+2.2, +9.2] over no retrieval, +1.1 [−0.4, +2.7] against LeanSearch v2 (better on 17 problems, worse on 7), and +1.3 [−0.1, +2.9] against Lean Finder. Under the Wilcoxon signed-rank test, p = 0.03 against LeanSearch v2 and p = 0.08 against Lean Finder. The direction is the same for every prover: per-prover differences range from 0.0 to +2.0 points against LeanSearch v2 and from +0.4 to +1.8 points against Lean Finder. The revision will report these paired statistics and state that PocketLean matches or slightly exceeds the end-to-end proof success of much larger retrievers at a fraction of their cost.

Dear Reviewer ePmz, as the discussion period closes shortly, we wanted to check whether our response has addressed your questions, especially the matched controls showing that fresh rollouts with the updated retriever deployed outperform both equal-volume historical data and rollouts collected with the older retriever. Thank you again for your review.

Reviewer_iRC4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a lightweight Mathlib retriever trained using proof-agent trajectories. Its main contribution is a two-stage training procedure that first learns from historical retrieval calls and then collects new proof trajectories using updated retrievers. The authors also introduce ProofCall, a benchmark of 1999 agent retrieval calls labeled using dependencies from Lean-verified proofs and LLM-based relevance judgments. PocketLean achieves competitive retrieval performance against larger systems, while offering better inference efficiency. End-to-end proving experiments also show modest improvements over existing retrieval systems.

Critical strengths
  • paper is clearly written and easy to read

  • this is a practical and well-motivated problem as Mathlib retrieval is important for theorem proving

  • it shows strong retrieval efficiency. The 0.6B model achieves better retrieval performance than several substantially larger baselines, with higher CPU query-encoding throughput

Critical weaknesses
  • ProofCall only evaluates retrieval calls from successful proof trajectories, excluding failures where better retrieval may be most needed. Additionally, relevance labels are derived from historical proofs, potentially penalizing alternative valid lemmas.

  • MathlibQR [1] is an expert-curated benchmark for Lean retrieval and widely used in Lean search engines. Evaluating PocketLean on MathlibQR would provide important evidence on the of its effectiveness in real world performance.

  • The ablations do not isolate the benefit of iterative self-improvement from simply using more training data or additional fine-tuning. A data- and compute-matched comparison against non-iterative training is missing.

[1] Gao et al., LeanSearch v2: Global Premise Retrieval for Lean 4 Theorem Proving, 2026

Questions for authors
  1. How does the exclusion of failed proof trajectories affect the representativeness of ProofCall? Could the authors also evaluate retrieval quality on unsuccessful proof attempts?

  2. Could the authors evaluate PocketLean on MathlibQR benchmark to better assess its capability?

  3. How does the framework compare against a non-iterative training baseline using the same amount of training data, training steps, and compute budget? How much of the improvement can be attributed specifically to iterative rollout collection rather than additional training?

I think this is an interesting paper and would be willing to raise score if my concerns are addressed.

2 comments
AuthorsOpenReview ↗

Response to Reviewer iRC4

We thank the reviewer for the constructive feedback and willingness to reconsider the score.

On iterative vs. non-iterative training with matched data and compute (W3, Q3). Control (a) of the General Response is the requested baseline: a single run from Init on the deduplicated union of all G0 to G5 pairs, with hard negatives re-mined by Init, for the same number of training examples as the full lineage. G5 leads this baseline on ProofCall hit@10 (74.0 vs. 71.5; +2.5 [+0.9, +4.0]), on hit@1 (46.9 vs. 41.7; +5.2 [+3.2, +7.1]), and on Lean Finder Recall@10 (78.9 vs. 77.4). Two further controls separate rollout collection from additional fine-tuning: replacing the B2 rollouts with an equal volume of historical data leaves G5 ahead by +1.1 [+0.2, +2.0] in hit@10 and +1.5 [+0.2, +2.8] in hit@1 (control (b)), and collecting the last round with the older retriever deployed leaves G5 ahead by +1.2 [+0.4, +2.1] in hit@1 (G5 − G5c; control (c)). G5 is ahead in all three controls, and its hit@1 gain is significant in each (+5.2, +1.5, +1.2).

On calls from failed trajectories (W1, Q1). PocketLean leads on all 2,489 dependency-labeled calls (Table 14), and under a new proof-free label set it also has the highest hit@10 on calls from failed attempts. Across all 2,489 calls (without the GPT-5.5 filter), PocketLean reaches 66.3 hit@10 versus 57.3 for Lean Finder and 54.3 for LeanSearch v2, with all retrievers evaluated without reranking.

To evaluate failed attempts, which have no verified proof, we used proof-free pooled labels: for every recorded call, we pooled the top-10 results of all compared systems and had GPT-5.5 score each lemma with the ProofCall 0–3 rubric, blind to system, rank, and final proof. On the 396 calls whose final proof failed to compile, hit@10 is 97.2 for PocketLean, 92.4 for LeanSearch v2, and 96.0 for Lean Finder, and hit@1 is 74.7, 74.5, and 79.0. Under the same labels, PocketLean reaches 96.0 hit@10 on the 2,489 calls from successful attempts, so its top-10 retrieval does not depend on the trajectory succeeding.

On alternative valid lemmas (W1). Under pooled relevance labels that credit any valid lemma, PocketLean's hit@10 is level with both baselines, so the Table 1 margins reflect its strength at ranking the lemmas that verified proofs actually use, the lemmas the agent goes on to use. Because the pooled evaluation judges every system's top-10 candidates, valid alternative lemmas count as hits even if unused in the recorded proof. Across all 2,489 calls, pooled hit@10 is 96.0 for PocketLean, 95.7 for LeanSearch v2, and 96.8 for Lean Finder (paired 95% CIs: +0.3 [−0.7, +1.3] and −0.8 [−1.7, +0.0]), while hit@1 is 75.8, 73.4, and 79.5. The revision will report both label sets.

On MathlibQR (W2, Q2). On MathlibQR, the 0.6B PocketLean achieves the highest nDCG@1 and nDCG@10 among the evaluated systems and ties the 8B LeanSearch v2 on Recall@10. Using MathlibQR's released queries and metric code, we ran a controlled comparison over the 310,579 declarations of the LeanSearch v2 corpus (Mathlib v4.28.0-rc1), re-running all baselines per our appendix adaptations and using LeanSearch v2's released document embeddings (Lean Finder documents follow its released corpus format). Below are the results on the 810-query fair subset used in the main table of the LeanSearch v2 paper (each query has one target declaration, so nDCG@1 equals Recall@1):

SystemnDCG@1nDCG@10Recall@10
PocketLean (0.6B)0.4230.5390.668
LeanSearch v2 (8B)0.3350.4960.668
Lean Finder (6.49B)0.3360.4610.593
LeanSearch v1 (7B)0.2700.3970.532
LeanExplore0.1900.3140.464
Qwen3-Embedding-0.6B (untrained)0.2440.3680.506

Target-declaration-clustered 95% CIs show significant nDCG@10 gains over LeanSearch v2 (+0.044 [+0.014, +0.073]), Lean Finder (+0.078 [+0.048, +0.110]), and the untrained backbone (+0.171 [+0.141, +0.202]). For Recall@10, PocketLean ties LeanSearch v2 (0.000 [−0.031, +0.031]) and outperforms Lean Finder (+0.075 [+0.038, +0.114]). We used the LeanSearch v2 corpus because our theorem-only Mathlib 4.17 index covers only 66 of MathlibQR's 200 target declarations; using our Mathlib 4.32 index yields similar PocketLean scores (0.547 nDCG@10, 0.688 Recall@10). The revision will include these results.

Dear Reviewer iRC4, as the discussion period closes shortly, we wanted to check whether our response, especially the MathlibQR evaluation and the matched non-iterative baseline you requested, has addressed your concerns. Thank you for your openness to reconsidering your score.

Reviewer_zZCEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PocketLean is a lightweight Mathlib retriever designed for Lean proof agents.

Contributions:

  • They train a 0.6B embedding model using diverse query sources, followed by Replay-to-Rollout Evolution. This method extracts weak supervision from Lean-verified proofs, first using historical agent trajectories and then collecting new rollouts with updated retrievers.
  • Also a new benchmark containing 1999 recorded retrieval calls across 5 settings, with query-relevant labels derived from proof dependencies and refined using LLM judgments.
  • The empirical performance is impressive, and it also provides substantially higher CPU query-encoding throughput.
Critical strengths

ProofCall evaluates retrieval on actual agent queries rather than only synthetic or static queries. Connecting recorded retrieval calls to Lean-verified proof dependencies provides a scalable supervision mechanism. The comparison of attribution strategies also gives convincing evidence that accurate call-level attribution matters for learning. This is, in my opinion, a meaningful contribution beyond introducing another specialized embedding model.

PocketLean consistently outperforms evaluated alternatives across all five ProofCall sets while using substantially fewer parameters. The improvements over its pretrained backbone, along with the replay/rollout ablations, support the effectiveness of specialized training. The CPU throughput and downstream proving results further strengthen its practical value.

Critical weaknesses

ProofCall evaluates 1999 calls selected from 3083 recorded calls, requiring verified final proofs and at least one automatically judged query-relevant lemma. This excludes unsuccessful or insufficiently labeled retrieval interactions, potentially favoring queries associated with successful proof construction. Also, relevance is determined by GPT 5.5 without reported independent human validation. Multiple calls also originate from the same trajectories or problems, introducing statistical dependence.

The ablations do not completely separate the benefit of collecting rollouts with updated retrievers from additional training data, proof-state supervision, and parameter averaging? A matched-budget comparison against continued offline training would better establish the importance of the proposed on-policy mechanism. Also, the end-to-end advantage over the strongest competing retriever is only 1.1p on selected problems, without confidence intervals or any other thing that helps determine how reliably the retrieval improvements translate into superior proving performance.

Questions for authors

Clarifications / additional analysis that resolve some of the main concerns above would be helpful. For example, some questions worth answering might be: Is there some human judgment/confirmation for the GPT 5.5 judge? How does the performance comparison change when considering retrieval calls from unsuccessful trajectories? Can you compare rollout-based evolution against continued offline training with matched training-pair counts, proof-state supervision, optimization budgets, and parameter averaging? Are the differences between PocketLean and LeanSearch v2 in end-to-end proving statistically significant under paired, problem-level analysis?

3 comments
AuthorsOpenReview ↗

Response to Reviewer zZCE (1/2)

We thank the reviewer for recognizing ProofCall's real agent queries, our proof-verified call-level supervision, and the attribution study.

On statistical dependence between calls (W1). PocketLean's ProofCall gains are statistically significant when calls are resampled by problem or by trajectory. Within-trajectory dependence occurs in three of the five sets: Library-Theorem, Free-Form, and Form-Guided retain full trajectories, whereas the two Olympiad sets sample one random call per trajectory (Appendix B.2) and contain nearly one call per problem (203 calls from 202 problems; 208 calls from 207 problems). We recomputed every ProofCall comparison with a paired cluster bootstrap (10,000 resamples): within each set we resample problems (or trajectories) with replacement, and all systems share the same draws. Problem-clustered 95% percentile CIs for hit@10 (all retrievers evaluated without reranking), PocketLean minus baseline:

BaselineOlympiad 4.32Olympiad 4.17Library-TheoremFree-FormForm-GuidedFive-set mean
LeanSearch v2+9.4 [+5.4, +13.4]+5.8 [+1.9, +10.1]+13.2 [+7.1, +19.6]+6.9 [+2.0, +11.2]+10.0 [+4.4, +15.8]+9.0 [+6.8, +11.4]
Lean Finder+4.4 [+0.5, +8.8]+6.2 [+1.9, +10.6]+7.6 [+2.5, +13.1]+6.6 [+1.9, +11.4]+9.3 [+4.3, +15.2]+6.9 [+4.7, +9.0]

No per-set interval includes zero, and trajectory-clustered intervals are nearly identical (for example, five-set mean vs. LeanSearch v2: +9.0 [+6.9, +11.1]). For hit@1, PocketLean's five-set mean gain is +11.0 [+8.4, +13.6] over LeanSearch v2 and +6.1 [+3.5, +8.6] over Lean Finder. The revision will report these intervals.

On the GPT-5.5 judge (W1, Q1). We checked the GPT-5.5 judge in two ways: by re-judging with Gemini 4 Argon and Claude Opus 5.5, which come from other model families, and by having two human annotators score a sample of candidates. Under either alternative judge, PocketLean still ranks first and Lean Finder second, and GPT-5.5 agrees with the annotators about as often as they agree with each other. First, we re-judged all 17,087 candidate pairs of the 2,489 dependency-labeled calls with both models. Each judge used the exact same prompt and 0–3 rubric (Appendix B.3), blind to retriever names and ranks.

Judge pairQuadratic-weighted κ (4-level) [95% CI]Agreement on relevant (score ≥ 2)
GPT-5.5 vs. Gemini 4 Argon0.671 [0.657, 0.685]82.0%
GPT-5.5 vs. Claude Opus 5.50.694 [0.681, 0.706]84.2%
Gemini 4 Argon vs. Claude Opus 5.50.785 [0.773, 0.797]94.9%

System rankings are consistent across the three judges. Rescoring ProofCall five-set mean hit@10 under GPT-5.5 / Gemini 4 Argon / Claude Opus 5.5 yields 74.0 / 69.1 / 70.0 for PocketLean, 67.2 / 64.5 / 65.1 for Lean Finder, and 65.0 / 62.9 / 62.9 for LeanSearch v2 (and 70.9, 65.9, and 63.6 under majority-of-three labels). Under each judge, PocketLean ranks first and Lean Finder ranks second. Across all eight evaluated systems, Kendall's τ with the GPT-5.5 ranking is 0.929 for Gemini 4 Argon and 0.857 for Claude Opus 5.5.

Second, the two annotators, both familiar with Lean and Mathlib, independently scored a stratified sample of 100 candidate pairs using the identical inputs and rubric, blind to all judge scores. Stratified by GPT-5.5 score and three-judge agreement, the sample uses inverse-probability weights to reflect the full 17,087-pair population. On binary relevance (score ≥ 2), GPT-5.5 agrees with one annotator on 79.7% of pairs (κ 0.50 [0.44, 0.57]) and with the other on 93.3% (κ 0.85 [0.77, 0.92]), while the two annotators agree with each other on 84.8% (κ 0.59 [0.50, 0.68]). Its agreement with humans thus lies within the range of agreement between humans.

AuthorsOpenReview ↗

Response to Reviewer zZCE (2/2)

On calls from unsuccessful or insufficiently labeled interactions (W1, Q2). PocketLean leads on all 2,489 dependency-labeled calls (Table 14), and under a new proof-free label set it also has the highest hit@10 on calls from failed attempts. These 2,489 calls include the 490 calls removed by the GPT-5.5 filter, and their labels involve no LLM judge: PocketLean reaches 66.3 hit@10, ahead of Lean Finder (57.3) and LeanSearch v2 (54.3).

Because failed attempts lack verified proofs for dependency extraction, we also built a proof-free label set covering all recorded calls, including the 396 calls whose final proofs failed to compile. For each call, we pooled the top-10 results across all evaluated systems and had GPT-5.5 score every candidate with the ProofCall 0–3 rubric (Appendix B.3; score ≥ 2 is relevant), blind to retriever, rank, and final proof. On those 396 unsuccessful-attempt calls, hit@10 is 97.2% for PocketLean, 92.4% for LeanSearch v2, and 96.0% for Lean Finder, and hit@1 is 74.7, 74.5, and 79.0. Under the same labels, PocketLean reaches 96.0 hit@10 on the 2,489 calls from successful attempts, so its top-10 retrieval does not depend on the trajectory succeeding.

On matched comparison against continued offline training (W2, Q3). Control (b) of the General Response is this comparison. Replacing the fresh rollout (B2) pairs with an equal volume of historical data, with matched proof-state counts, optimization budget, and parameter averaging, leaves G5 ahead by +1.1 [+0.2, +2.0] in hit@10 and +1.5 [+0.2, +2.8] in hit@1. G5 also leads a single non-iterative run with matched training examples by +2.5 [+0.9, +4.0] in hit@10 and +5.2 in hit@1 (control (a)), and the model trained on rollouts collected with the older retriever by +1.2 [+0.4, +2.1] in hit@1 (control (c)). G5 is ahead in all three controls, and its hit@1 gain is significant in each (+5.2, +1.5, +1.2).

On "selected problems" in end-to-end proving (W2). Problem selection is independent of any retriever. miniF2F-hard comprises all 37 miniF2F problems where Goedel-Prover-V2-32B failed across all eight attempts. The ProofNet subset is a uniform random sample of 32 problems after removing entries with faulty formal statements (Appendix D). Both selection rules will be stated in the abstract and Section 3.1.

On the paired end-to-end analysis (Q4). Under problem-level paired testing, PocketLean significantly outperforms no retrieval and matches or slightly exceeds 6.5B–8B retrievers. For each of the 69 problems, we compute the solve rate across 40 attempts (5 provers × 8 samples) and run a 10,000-resample paired bootstrap over problems:

PocketLean minusMean difference, points [95% CI]Problems better / worse / tiedWilcoxon signed-rank p
No retrieval+5.4 [+2.2, +9.2]18 / 7 / 440.003
LeanSearch v2+1.1 [−0.4, +2.7]17 / 7 / 450.03
Lean Finder+1.3 [−0.1, +2.9]20 / 9 / 400.08

The gain over no retrieval is significant, and the signed-rank test is significant against LeanSearch v2 as well (p = 0.03). PocketLean is not worse than either baseline with any of the five provers. Using a 0.6B encoder, PocketLean matches or slightly exceeds the end-to-end proof success of 6.5B–8B retrievers at roughly 14.6 times the CPU query-encoding throughput of Lean Finder. The revision will report these paired statistics and frame the end-to-end claim in these terms.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address the points raised in my review.

The new analyses are helpful. In particular, the problem-clustered bootstrap results and cross-judge / human validation strengthen confidence in the ProofCall evaluation. The matched-budget controls are especially helpful: they provide good evidence that fresh rollouts and deploying updated retrievers contribute beyond simply increasing the amount of training data.

My main remaining concerns are about the end-to-end results. While PocketLean clearly improves over no retrieval, its advantage over LeanSearch v2 remains modest (+1.1 pts). I would therefore encourage the authors to frame the end-to-end contribution primarily as achieving comparable or slightly better proving performance with substantially lower retrieval cost (which seems true), rather than establishing a clear improvement in proof success over the strongest retrievers. The proof-free evaluation on failed trajectories is also good, although its reliance on LLM-generated relevance labels remains a limitation.

Overall, I find the rebuttal helpful and believe the additional evidence meaningfully improves the paper. I am updating my rating from 2 (Weak Rejection) to 3 (Weak Acceptance), conditional on incorporating the promised analyses and appropriately qualifying the end-to-end claims (the rebuttal was submitted near the end of the discussion period, so there might not be sufficient time for authors to qualify the claims now; however, I think these edits should not be major, and I thus assume the authors would do them eventually).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their careful reading. PocketLean trains a Mathlib retriever on proof agents' own retrieval calls, labeling each call with the lemmas used in its Lean-checked final proof. R2Evo repeats this across generations, first replaying recorded trajectories and then collecting new rollouts with the updated retriever deployed (B2). All three reviewers asked whether the gains come from this iterative collection or simply from more data and training. We answer this shared question below with three matched controls. Reviewer-specific points, including statistical analyses and the MathlibQR evaluation, are answered in the individual replies, and all new numbers will be added to the revision.

Is the gain from iterative rollout collection, or just from more data and training? (zZCE, iRC4, ePmz)

Iterative rollout collection improves retrieval beyond adding more data or training steps. We ran three complementary controls to separate the effects of training schedule, data source, and the retriever used during rollout collection.

(a) Non-iterative training with the same data and the same number of training examples (iRC4). Iterative training outperforms a single run on the combined dataset with a matched number of training examples. Starting from the same Init checkpoint as G0, we train once on the deduplicated union of all G0 to G5 training pairs (774,826 pairs). The Init encoder re-mines hard negatives so the baseline does not inherit negatives from later generations. Total training examples match the full G0 to G5 lineage (1,540,800 vs. 1,540,832, i.e., 5,350 optimizer steps at a global batch of 288). As a single run, it uses no weight averaging (Eq. 3); all other settings match the lineage runs.

ProofCall hit@10 (five-set mean)ProofCall hit@1Lean Finder bench. R@10 (mean)
Non-iterative, matched data and examples71.541.777.4
G5 (PocketLean)74.046.978.9
G5 − non-iterative (problem-clustered 95% CI)+2.5 [+0.9, +4.0]+5.2 [+3.2, +7.1]+1.5

(b) Equal volume of historical data in place of fresh rollouts (ePmz, zZCE). Fresh agent rollouts yield stronger retrievers than an equal volume of historical trajectories. Starting from G3, we repeat the exact G4 and G5 recipes, replacing only the rollout-derived (B2) pairs with pairs from 4,883 unseen historical Library-Theorem trajectories (filtered to exclude any problem matching ProofCall or our end-to-end tasks). We upsample these historical pairs to match the B2 counts (31,218 and 30,287 call pairs, and 40,000 proof-state pairs per round). Retained data, attribution rule, pair builder, negative miner, optimizer, seeds, steps, and averaging weights remain unchanged. Two details differ: the 2,985 state-intent pairs have no historical counterpart, so one of the two training runs averaged into the G5 analogue takes 480 steps instead of 490.

ProofCall hit@10 (mean)ProofCall hit@1Lean Finder bench. R@10 (mean)
Historical-data analogue of G572.945.478.3
G5 (PocketLean)74.046.978.9
G5 − historical (problem-clustered 95% CI)+1.1 [+0.2, +2.0]+1.5 [+0.2, +2.8]+0.6

Control (c) next isolates the effect of updating the retriever deployed during rollout collection.

(c) Same rollout tasks, older retriever deployed (ePmz, zZCE). Deploying the updated retriever during rollout collection significantly improves hit@1. We re-collected the second rollout round with G3 serving the proof agents instead of G4, holding problems, policy models, prompts, and verification fixed (20,395 tasks, 19,702 with a verified proof). Using the same attribution rule and pair builder, we rebuilt the B2 pairs, sampled them to the original counts, and trained and averaged identically to G5. The resulting model, G5c, differs from G5 only in which retriever served the agents during the final data collection round.

ProofCall hit@10 (mean)ProofCall hit@1Lean Finder bench. R@10 (mean)
G5c (second round collected with G3 deployed)73.845.678.8
G5 (PocketLean)74.046.978.9
G5 − G5c (problem-clustered 95% CI)+0.3 [−0.5, +1.0]+1.2 [+0.4, +2.1]+0.1

Relative to G4, the final round improves hit@1 by +1.4 [+0.4, +2.5] when G4 is deployed during collection, versus +0.2 [−1.0, +1.3] when G3 is deployed.

Summary. G5 is ahead in all three controls, and its hit@1 gain is statistically significant in each (+5.2, +1.5, +1.2). Iterative training outperforms a single example-matched run on the same data, fresh rollouts beat an equal volume of historical trajectories, and deploying the updated retriever during collection raises top-1 accuracy. The revision will report all three controls.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.