- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces PocketLean, a lightweight Mathlib retriever for theorem proving agents. The approach involves first performing a multi-source contrastive training on query–declaration pairs. It then uses R2 Evo, which improves the retriever using supervision extracted from Lean-verified proofs. This process involves replay-based training on historical agent trajectories as well as rollout-based training on newly-collected trajectories using updated retrievers.
The authors also introduce ProofCall–a benchmark of 1,999 recorded retrieval calls from proof agents. PocketLean attains 78.9% Recall@10 on the Lean Finder benchmark and 74.0% hit@10 on ProofCall, outperforming larger retrieval baselines. In end-to-end theorem proving, it attains 8.8% average proof success versus 3.4% without retrieval and 7.7% with LeanSearch v2.
Critical strengths
-
Practical and efficient Mathlib retrieval: PocketLean achieves strong retrieval performance compared with substantially larger models while only requiring a 0.6B parameter encoder. The reported CPU query-encoding throughput is about 14.6x that of Lean Finder. This is a significant practical contribution, especially for proof agents which can involve frequent querying of large, version dependent theorem libraries.
-
Evaluation on realistic agent queries and analysis of proof-derived supervision: ProofCall tackles an important shortcoming of static retrieval benchmarks by evaluating on queries generated during attempts to perform theorem proving. The attribution experiments also provide valuable insight that associating all queries with all lemmas used in a proof generates noisy supervision, whereas call level attribution generates more query relevant training examples.
Critical weaknesses
-
The methodological contribution of R2 Evo is not well isolated: Training retrievers from agent trajectories is similar in spirit to prior approaches such as LRAT, and the rollout loop resembles expert iteration, as the authors note. Proof-verified supervision and query-level attribution make PocketLean different, but the experiments don't clearly isolate the benefits of collecting fresh rollouts with updated retrievers. B1 and B2 use different data sources, attribution approaches, and training procedures, so the gains from the B2 updates do not demonstrate that it is the rollout feedback loop - rather than additional or better training data - that is responsible for them. While retrieval performance is measured for G0-G5, the end-to-end proving experiments only use the final checkpoint. As written, the paper does not establish that R2 Evo improves downstream proof success beyond the initial contrastive training step.
-
Limited evidence for downstream superiority over competing retrievers: The end-to-end experiments only use 69 theorem proving tasks. PocketLean improves average proof success from 7.7% with LeanSearch v2 to 8.8% (1.1 percentage points). While the experiment includes multiple provers and multiple attempts, the paper does not report confidence intervals or perform paired significance tests for these results. Thus, we do not know how generalizable this relatively small improvement is. The efficiency gains are useful in and of themselves, but more statistical power is needed to claim downstream accuracy advantages.
Questions for authors
-
Contribution of individual training stages: The authors should report end-to-end theorem proving performance for G0, G3 and G5 using the same evaluation protocol. This would show whether improvements in retrieval quality lead to downstream proof success.
-
Effect of fresh rollouts: Can the authors compare B2 to a training condition that uses the same amount of extra historical data, controlling for attribution strategy and training budget? This would allow isolation of the contribution of updating the data distribution by way of agent interaction.
-
Statistical reliability. Could the authors provide paired confidence intervals or statistical significance analysis for the end-to-end comparisons, particularly PocketLean vs. LeanSearch v2 and Lean Finder?
2 comments
Dear Reviewer ePmz, as the discussion period closes shortly, we wanted to check whether our response has addressed your questions, especially the matched controls showing that fresh rollouts with the updated retriever deployed outperform both equal-volume historical data and rollouts collected with the older retriever. Thank you again for your review.
Response to Reviewer ePmz
We thank the reviewer for the clear summary and for highlighting the practical value of efficient retrieval and our attribution analysis.
On isolating the effect of fresh rollouts (W1, Q2). Controls (b) and (c) of the General Response separate fresh rollouts from additional data. In control (b), starting from G3, we repeated the exact G4 and G5 recipes but replaced only the rollout-derived (B2) pairs with historical trajectories upsampled to the same pair counts (31,218 and 30,287 call pairs, and 40,000 proof-state pairs per round), keeping attribution, retained data, optimization budget, seeds, and averaging weights identical. G5 leads this historical analogue by +1.1 [+0.2, +2.0] in ProofCall hit@10 and +1.5 [+0.2, +2.8] in hit@1. Control (c) isolates the retriever deployed during collection: we re-collected the second rollout round with G3 deployed instead of G4 and kept everything else identical. G5 leads the resulting model, G5c, by +1.2 [+0.4, +2.1] in hit@1 and +0.3 [−0.5, +1.0] in hit@10, and the last round's hit@1 gain over G4 appears only when the updated retriever is deployed. A single non-iterative run on all lineage data with matched training examples is 2.5 [+0.9, +4.0] points behind G5 in hit@10 and 5.2 points behind in hit@1 (control (a)). G5 is ahead in all three controls, and its hit@1 gain is significant in each (+5.2, +1.5, +1.2).
Regarding LRAT and expert iteration, the revision will state explicitly that our distinguishing contributions are proof-verified, call-level supervision and its application across retriever generations. Controls (b) and (c) address the reviewer's remaining question, whether collecting fresh rollouts with the updated retriever adds value beyond this supervision.
On end-to-end results for G0 and G3 (W1, Q1). End-to-end proof success increases monotonically from G0 to G3 to G5, matching the retrieval gains across generations. We evaluated G0, G3, and G5 under one protocol, with the 69 problems and five provers of Table 4: eight attempts per prover-problem pair, one submission without compiler feedback, and at most four searches per attempt. Because the unversioned deepseek-v4-flash endpoint from Table 4 was retired, all three stages use the dated snapshot deepseek-v4-flash-0731. Success rates (%):
Average proof success rises from 8.4% (G0) to 9.2% (G3) to 10.2% (G5), and G5 outperforms G0 on every prover.
On statistical reliability of the end-to-end comparison (W2, Q3). Under paired problem-level testing, PocketLean significantly beats no retrieval and matches or slightly exceeds much larger retrievers. Pairing systems across the 69 problems in the Table 4 runs (40 attempts per problem, bootstrap over problems), the paired bootstrap 95% CI of the mean difference is +5.4 [+2.2, +9.2] over no retrieval, +1.1 [−0.4, +2.7] against LeanSearch v2 (better on 17 problems, worse on 7), and +1.3 [−0.1, +2.9] against Lean Finder. Under the Wilcoxon signed-rank test, p = 0.03 against LeanSearch v2 and p = 0.08 against Lean Finder. The direction is the same for every prover: per-prover differences range from 0.0 to +2.0 points against LeanSearch v2 and from +0.4 to +1.8 points against Lean Finder. The revision will report these paired statistics and state that PocketLean matches or slightly exceeds the end-to-end proof success of much larger retrievers at a fraction of their cost.