- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents Lean Finder V2, a semantic search engine for the Lean mathematical library Mathlib that targets the short and ambiguous queries issued by humans and coding agents. At the model level, a retriever is first fine-tuned on more than one million context-rich query–declaration pairs; a query generator is then trained with reinforcement learning using a reward that multiplies an alignment gate (the query is faithful to the target declaration) by a difficulty term (the current retriever struggles to retrieve it), and the generated short queries are used to retrain the retriever, yielding more than 600K pairs over two rounds. At the input level, GEPA is used to optimize a query-rewriting prompt on the same synthetic data. On the 810 short human-written queries of MathlibQR, Lean Finder V2 reaches a Recall@10 of 0.893 versus 0.780 for the previous best system, LeanSearch V2; it is judged most helpful by two LLM judges on 200 unlabeled real queries; and with Claude Code it reaches a 39.3% proof success rate on FATE-H versus 36.0% for the next best tool.
Critical strengths
-
It addresses a real data gap and clearly improves over the main existing systems. The paper shows that real queries are much shorter than typical training data and fills this gap with scalable adversarial generation. Compared with LeanSearch V2, Lean Finder, LeanExplore, TheoremGraph and Claude Code on MathlibQR, it raises Recall@10 from the previous best of 0.780 to 0.893, while maintaining performance on context-rich queries. This has direct practical value for the Lean community and for agents.
-
The ablations attribute the gains to the generator design. A frozen generator, an SFT-only generator and a non-updated RL generator all underperform the round-by-round RL generator under the same source declarations, initialization and retriever-update budget, supporting that generating queries against the current retriever is what brings the gains.
Critical weaknesses
The methodological novelty is limited. Training retrievers on synthetic queries and generating hard examples adversarially both have precedent in information retrieval; IRGAN and AR2, cited by the paper, generate hard negative documents, whereas this work generates hard queries and adds an alignment gate and a difficulty window. Overall, it is an incremental improvement over the earlier Lean Finder.
This affects how I position the contribution: its value lies mainly in effectively applying these ideas to Mathlib retrieval with clear gains, rather than in the method itself.
Questions for authors
-
The "ambiguity gap" in Figure 1 compares two different benchmarks (short queries from MathlibQR and long queries from the Lean Finder test set) with different target declarations. Could you construct both short and long queries for the same set of target declarations, to directly measure the ambiguity gap and how much Lean Finder V2 reduces it?
-
The gap of 39.3% versus 36.0% on FATE-H amounts to about three of 100 problems, and the standard deviation over three runs does not reflect the uncertainty from problem sampling. Could you provide a per-problem paired comparison (e.g., which problems are solved only with a particular tool), or report results on more problems?
-
The alignment audit in Appendix I uses a sample constructed with 40 misaligned and 60 aligned pairs. Could you randomly sample the final training data at its natural proportions and report the fraction of queries that are faithful to their target declarations?
4 comments
Looking forward to more discussions
Dear Reviewer aY2t,
Thank you again for your positive review and constructive feedback.
As the author-reviewer discussion period is coming to an end, we will appreciate if you could check our response to your review comments soon. This way, if you have further questions and comments, we can still reply before the author-reviewer discussion period ends. If our response resolves your concerns, we kindly ask you to consider raising the rating of our work. Thank you very much for your time and efforts.
Best regards,
Authors of Submission 23348
Thank you for the additional experiments. The matched-query comparison and paired FATE-H analysis address my main questions. The methodological novelty remains modest, but the paper demonstrates a practical improvement for Mathlib search. I maintain my rating of 3.
Thank you for your response
Dear Reviewer aY2t,
Thank you for taking the time to review our responses and additional experiments. We are glad that they have addressed your main questions. We sincerely appreciate your recognition of the practical value of our work, your constructive feedback, and your positive rating of 3.
Thank you again for your time and support.
Best regards,
Authors of Submission 23348
We thank Reviewer aY2t for the constructive review, and for recognizing that our work addresses a real data gap, achieves clear improvements over existing systems, and provides practical value to the Lean community and coding agents.
We thank the reviewer for this suggestion. For each MathlibQR target declaration, we get the context-rich query (from Lean Finder context-rich test set) for the same target declaration, and show the performance gap in recall@10 like our Figure 1.
Lean Finder V2 has the smallest ambiguity gap (3.6 percentage points, vs. 8.0–22.2 for the baselines) and the best performance on both query types, consistent with Figure 1. We will update Figure 1 with this matched comparison in camera ready.
We thank the reviewer for the suggestion. Across the 100 problems, Lean Finder V2 achieves a higher success fraction across the three runs on 10 problems and a lower fraction on 2 problems compared with the strongest baseline LeanSearch v2. The mean improvement is 3.3 percentage points, and a two-sided exact permutation test gives p = 0.033, indicating statistical significance. Our theorem-proving experiments have already incurred approximately 1500 dollars in token costs, and budget constraints prevent evaluation on additional problems.
We thank the reviewer for the suggestion. We randomly sample 300 query–declaration pairs from the generated training data used to train the retriever. Using two independent frontier LLM judges (GPT-5.6-Sol and Claude Opus 5) and consistent evaluation protocol with the paper, we obtain alignment rates of 98.3% and 98.7%, respectively. These results provide additional evidence of the alignment quality of our training data.
The core novelty of our work lies in identifying ambiguous queries, commonly issued by humans and coding agents in real-world interactions, as a central bottleneck of Lean/Mathlib search, and addressing this problem. This bottleneck has not been discussed or explored in prior Lean search engine papers, including the earlier Lean Finder, which also degrades substantially on ambiguous queries (Fig. 1b). Lean Finder V2 therefore targets a failure mode that prior search engines leave unresolved through our complementary model level adversarial training and optimized input-level query rewriting. As reviewers aY2t and 1fws acknowledge, our paper identifies and addresses a real gap and problem in existing Lean search engines.
Prior work like IRGAN and AR2 co-train two ranking models by sampling hard negative documents for a fixed set of labeled queries, yet in this setting labeled ambiguous queries data doesn’t exist. We therefore train the query generator itself using RL with our retriever-guided rewards, producing large-scale, effective training data. Furthermore, we reuse the same synthetic data to optimize a query-rewriting prompt, enabling complementary disambiguation before retrieval.
As all reviewers recognize, our proposed methods effectively address this ambiguous query problem, supported by extensive evaluations across diverse benchmarks and tasks, and our state-of-the-art performance on MathlibQR. Reviewers aY2T and 1fws further highlight the practical value and contributions of our work to the Lean community. We will further expand our related work section to discuss this.