acceptodds
Under review as a conference paper at ICLR 2027

Lean Finder V2: Semantic Search for Mathlib That Disambiguates User Queries

Abstract

We introduce Lean Finder V2, a semantic search framework for Mathlib that disambiguates user queries through complementary model-level adversarial training and input-level query rewriting. Human mathematicians and coding agents rarely search Mathlib by providing detailed context. In practice, they query short properties, notation, aliases, or hypotheses, leaving much of the intended mathematics implicit. In contrast, existing Mathlib search engines unrealistically assume context-rich queries. The scarcity of real-world short queries paired with ground-truth Mathlib statements makes this ambiguity even more intractable. We argue that this is essentially a data problem. Through adversarial training, we scale up the generation of labeled short and ambiguous queries, and learn to disambiguate user queries at both model-level fine-tuning and input-level query rewriting. On the human-curated MathlibQR benchmark, our Lean Finder V2 sets a new state of the art, with at least 11% relative improvement over the previous best Mathlib semantic search engines. On top of our state-of-the-art retriever, optimized query rewriting further improves MathlibQR retrieval. On unlabeled queries from human and coding-agent interactions, Lean Finder V2 is ranked as the strongest system. Integrated with Claude Code, Lean Finder V2 improves proof success on downstream theorem-proving task. We will release our code, model, and data upon acceptance.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Lean, mathlib, code search, theorem retrieval

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_aY2tOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents Lean Finder V2, a semantic search engine for the Lean mathematical library Mathlib that targets the short and ambiguous queries issued by humans and coding agents. At the model level, a retriever is first fine-tuned on more than one million context-rich query–declaration pairs; a query generator is then trained with reinforcement learning using a reward that multiplies an alignment gate (the query is faithful to the target declaration) by a difficulty term (the current retriever struggles to retrieve it), and the generated short queries are used to retrain the retriever, yielding more than 600K pairs over two rounds. At the input level, GEPA is used to optimize a query-rewriting prompt on the same synthetic data. On the 810 short human-written queries of MathlibQR, Lean Finder V2 reaches a Recall@10 of 0.893 versus 0.780 for the previous best system, LeanSearch V2; it is judged most helpful by two LLM judges on 200 unlabeled real queries; and with Claude Code it reaches a 39.3% proof success rate on FATE-H versus 36.0% for the next best tool.

Critical strengths
  1. It addresses a real data gap and clearly improves over the main existing systems. The paper shows that real queries are much shorter than typical training data and fills this gap with scalable adversarial generation. Compared with LeanSearch V2, Lean Finder, LeanExplore, TheoremGraph and Claude Code on MathlibQR, it raises Recall@10 from the previous best of 0.780 to 0.893, while maintaining performance on context-rich queries. This has direct practical value for the Lean community and for agents.

  2. The ablations attribute the gains to the generator design. A frozen generator, an SFT-only generator and a non-updated RL generator all underperform the round-by-round RL generator under the same source declarations, initialization and retriever-update budget, supporting that generating queries against the current retriever is what brings the gains.

Critical weaknesses

The methodological novelty is limited. Training retrievers on synthetic queries and generating hard examples adversarially both have precedent in information retrieval; IRGAN and AR2, cited by the paper, generate hard negative documents, whereas this work generates hard queries and adds an alignment gate and a difficulty window. Overall, it is an incremental improvement over the earlier Lean Finder.

This affects how I position the contribution: its value lies mainly in effectively applying these ideas to Mathlib retrieval with clear gains, rather than in the method itself.

Questions for authors
  1. The "ambiguity gap" in Figure 1 compares two different benchmarks (short queries from MathlibQR and long queries from the Lean Finder test set) with different target declarations. Could you construct both short and long queries for the same set of target declarations, to directly measure the ambiguity gap and how much Lean Finder V2 reduces it?

  2. The gap of 39.3% versus 36.0% on FATE-H amounts to about three of 100 problems, and the standard deviation over three runs does not reflect the uncertainty from problem sampling. Could you provide a per-problem paired comparison (e.g., which problems are solved only with a particular tool), or report results on more problems?

  3. The alignment audit in Appendix I uses a sample constructed with 40 misaligned and 60 aligned pairs. Could you randomly sample the final training data at its natural proportions and report the fraction of queries that are faithful to their target declarations?

4 comments

We thank Reviewer aY2t for the constructive review, and for recognizing that our work addresses a real data gap, achieves clear improvements over existing systems, and provides practical value to the Lean community and coding agents.

Q1. Measuring the ambiguity gap with matched target declarations

We thank the reviewer for this suggestion. For each MathlibQR target declaration, we get the context-rich query (from Lean Finder context-rich test set) for the same target declaration, and show the performance gap in recall@10 like our Figure 1.

Recall@10 (%)LeanExploreLean FinderLeanSearch v2 (w/ reranker)Lean Finder V2
Ambiguous56.969.878.089.3
Context-rich67.792.086.092.9
Gap (closer to 0 is better)−10.8−22.2−8.0−3.6

Lean Finder V2 has the smallest ambiguity gap (3.6 percentage points, vs. 8.0–22.2 for the baselines) and the best performance on both query types, consistent with Figure 1. We will update Figure 1 with this matched comparison in camera ready.

Q2. Problem-wise paired comparison with LeanSearch v2 on FATE-H

We thank the reviewer for the suggestion. Across the 100 problems, Lean Finder V2 achieves a higher success fraction across the three runs on 10 problems and a lower fraction on 2 problems compared with the strongest baseline LeanSearch v2. The mean improvement is 3.3 percentage points, and a two-sided exact permutation test gives p = 0.033, indicating statistical significance. Our theorem-proving experiments have already incurred approximately 1500 dollars in token costs, and budget constraints prevent evaluation on additional problems.

Q3. Alignment audit on randomly sampled training data

We thank the reviewer for the suggestion. We randomly sample 300 query–declaration pairs from the generated training data used to train the retriever. Using two independent frontier LLM judges (GPT-5.6-Sol and Claude Opus 5) and consistent evaluation protocol with the paper, we obtain alignment rates of 98.3% and 98.7%, respectively. These results provide additional evidence of the alignment quality of our training data.

Q4. Methodological Novelty and Contributions

The core novelty of our work lies in identifying ambiguous queries, commonly issued by humans and coding agents in real-world interactions, as a central bottleneck of Lean/Mathlib search, and addressing this problem. This bottleneck has not been discussed or explored in prior Lean search engine papers, including the earlier Lean Finder, which also degrades substantially on ambiguous queries (Fig. 1b). Lean Finder V2 therefore targets a failure mode that prior search engines leave unresolved through our complementary model level adversarial training and optimized input-level query rewriting. As reviewers aY2t and 1fws acknowledge, our paper identifies and addresses a real gap and problem in existing Lean search engines.

Prior work like IRGAN and AR2 co-train two ranking models by sampling hard negative documents for a fixed set of labeled queries, yet in this setting labeled ambiguous queries data doesn’t exist. We therefore train the query generator itself using RL with our retriever-guided rewards, producing large-scale, effective training data. Furthermore, we reuse the same synthetic data to optimize a query-rewriting prompt, enabling complementary disambiguation before retrieval.

As all reviewers recognize, our proposed methods effectively address this ambiguous query problem, supported by extensive evaluations across diverse benchmarks and tasks, and our state-of-the-art performance on MathlibQR. Reviewers aY2T and 1fws further highlight the practical value and contributions of our work to the Lean community. We will further expand our related work section to discuss this.

AuthorsOpenReview ↗

Looking forward to more discussions

Dear Reviewer aY2t,

Thank you again for your positive review and constructive feedback.

As the author-reviewer discussion period is coming to an end, we will appreciate if you could check our response to your review comments soon. This way, if you have further questions and comments, we can still reply before the author-reviewer discussion period ends. If our response resolves your concerns, we kindly ask you to consider raising the rating of our work. Thank you very much for your time and efforts.

Best regards,

Authors of Submission 23348

Thank you for the additional experiments. The matched-query comparison and paired FATE-H analysis address my main questions. The methodological novelty remains modest, but the paper demonstrates a practical improvement for Mathlib search. I maintain my rating of 3.

AuthorsOpenReview ↗

Thank you for your response

Dear Reviewer aY2t,

Thank you for taking the time to review our responses and additional experiments. We are glad that they have addressed your main questions. We sincerely appreciate your recognition of the practical value of our work, your constructive feedback, and your positive rating of 3.

Thank you again for your time and support.

Best regards,


Authors of Submission 23348

Reviewer_1fwsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets semantic search over Mathlib when queries are short and underspecified (median ~15–16 tokens for human and coding-agent queries, versus ~82 tokens for the statement informalizations on which existing Lean search engines are trained). The authors frame this as a data problem and propose two uses of one synthetic dataset. (i) Model level: a Qwen3-4B query generator is trained with GRPO plus an SFT anchor on deployment queries; its reward is an LLM-judged alignment gate multiplied by a rank-based difficulty term computed from the current retriever (Qwen3-Embedding-8B, masked InfoNCE). The generator and retriever are updated alternately for two rounds, yielding a corpus of 600K+ ambiguous query–declaration pairs. (ii) Input level: GEPA optimizes a query-rewriting prompt on the same synthetic pairs using a rank-based score. Validation: on the 810-query MathlibQR fair set the final retriever reaches Recall@10 0.893 versus 0.780 for LeanSearch V2 with reranker, while staying on par with the best systems on a 2,526-query context-rich set. Two frontier LLM judges prefer it on 200 unlabeled deployment queries. GEPA rewriting raises nDCG@1 from 0.558 to 0.588. Plugged into Claude Code (Sonnet 5), it lifts FATE-H pass rate to 39.3% versus 36.0% for LeanSearch V2 and 34.0% without semantic search.

Critical strengths
  1. The paper identifies a real problem and isolates the source of its gain with careful ablations. The query-length analysis (Fig. 1a) and the even-function example make the ambiguity gap concrete. The ablations in Appendix B carry much of the paper's weight: with matched generator, sampling and retriever budgets, difficulty-only rewards hurt top-rank metrics, alignment-only rewards give small gains, and only the gated product helps (Table 6). Together these attribute the gain to retriever-adaptive generation rather than to more data, which is the paper's central claim.

  2. The evidence spans several levels rather than a single benchmark. The authors report labeled retrieval on both short and context-rich queries, preference on held-out deployment queries, query rewriting with three repetitions, and end-to-end proving. Context-rich performance does not degrade (Table 2), which rules out the obvious failure mode of overfitting to short queries. This breadth makes the practical claim credible for the Lean community.

Critical weaknesses
  1. The downstream and preference evidence is thinner than the headline wording suggests. The proving result rests on one benchmark of 100 problems, three runs and a $1 per-problem cap. The 3.3-point gap over LeanSearch V2 (39.3±1.2 vs 36.0±1.0) amounts to about three problems, and no significance test is reported. The preference study uses LLM judges only, and the validation in Table 10 checks a different task: picking the ground truth among five candidates with the answer inserted, not ranking five result lists. It is also not stated which search engines the 200 deployment queries (and the SFT anchor queries) were logged from. If they come largely from Lean Finder's own service, the query distribution may favor this system. These issues do not undermine the retrieval results, but they weaken contribution claims 1 and 4.

  2. Methodological novelty is moderate relative to the dense retrieval literature. Training a query generator against a retriever and feeding its hard, faithful queries back as supervision is close in spirit to prior synthetic-query and adversarial retriever training for dense retrieval (e.g., InPars, Promptagator, AR2). The related-work paragraph claims RL has "primarily optimized queries against fixed retrievers", but does not engage with this line. The contribution is mainly a careful adaptation to Mathlib plus the reward design; that is valuable, but the positioning should be accurate.

Questions for authors

The following clarifications could change my assessment.

  1. Where do the 200 deployment evaluation queries and the SFT expert queries come from (which search engines and platforms, collected when)? How much of each comes from Lean Finder itself?

  2. Can you report a paired significance test on FATE-H (per-problem outcomes across the three runs), and results with a larger per-problem budget? Is the advantage over "no semantic retrieval" stable when the cost cap is raised?

  3. The abstract claims "at least 11% relative improvement" but the introduction says 14%. On nDCG@1 the gain over Claude Code is about 6% (0.558 vs 0.527). Which metric and baseline does each number refer to?

  4. Retriever results appear to come from single training runs. How large is the variance across seeds for Round 0→1→2?

  5. Beyond excluding ground-truth declarations, did you check near-duplicate overlap between the generated queries and MathlibQR queries?

4 comments

We thank Reviewer 1fws for the constructive and careful review, and for recognizing that our work identifies a real problem in Lean search engines and that our comprehensive evaluations across diverse types of benchmarks demonstrate practical value for the Lean community.

Q1. Source of deployment queries and SFT queries

Deployment queries and the SFT queries were collected from queries to Lean Finder and LeanSearch. Importantly, these are naturally occurring user queries, rather than queries constructed or selected to favor any particular retrieval system. Similar to the human-curated MathlibQR benchmark, they reflect how users might search for or describe mathematical Lean statements in practice.

Q2. Paired significance test with closest baseline LeanSearch v2 for downstream theorem proving tasks, and discussion on budget.

We thank the reviewer for this suggestion. Using a two-sided exact permutation test that pairs the two systems' per-problem success fractions (each averaged over three runs), Lean Finder V2 shows an improvement of 3.3 percentage points over the strongest baseline LeanSearch v2, with p = 0.033, which is statistically significant.

Our theorem-proving experiments have already incurred approximately 1500 dollars in token costs, and our budget constraints prevent further evaluation at higher per-problem budget caps than 1 dollar. We use the same FATE-H benchmark as LeanSearch V2 to provide a focused downstream assessment of semantic retrieval’s effectiveness in theorem proving. The shared 1 dollar per-problem cap evaluates whether retrieval improves proof success under a matched inference budget, thereby also assessing the potential efficiency benefits that effective semantic retrieval can bring. This budget is comparable in scale to LeanSearch v2’s reported approximately 1.10 dollar per-query inference cost.

Q3. Confusion on numbers listed in abstract and introduction

We thank the reviewer for pointing out this potential confusion. As specified in the introduction, the 14% relative improvement refers to Recall@10 on MathlibQR over the previous best (LeanSearch v2 with reranking). The comparison listed in abstract (11% improvement) is stated to be among semantic search systems. To avoid confusion, we will clarify the abstract in the camera-ready version to simply state 14% relative improvement in Recall@10 over the previous best.

Q4. More retriever training runs across seeds

We train each round's retriever with three random seeds. Below are mean ± std over the 3 seeds on both test sets, next to the strongest baseline on each: LeanSearch V2 (retriever + reranker) on MathlibQR, and Lean Finder on the context-rich test set.

MathlibQR

ModelnDCG@1 (%)nDCG@5 (%)nDCG@10 (%)R@10 (%)R@50 (%)R@100 (%)
LeanSearch V2 (retriever + reranker)47.060.162.378.084.785.8
Ours, Round 046.26 ± 0.1958.76 ± 0.2560.90 ± 0.5075.87 ± 0.8786.08 ± 0.3090.16 ± 0.19
Ours, Round 152.19 ± 0.4766.00 ± 0.5568.37 ± 0.6984.20 ± 0.6292.27 ± 0.7195.20 ± 0.56
Ours, Round 255.51 ± 0.4070.30 ± 0.4472.56 ± 0.6488.99 ± 0.4195.37 ± 0.3797.08 ± 0.26

Context-rich test set

ModelnDCG@1 (%)nDCG@5 (%)nDCG@10 (%)R@10 (%)R@50 (%)R@100 (%)
Lean Finder54.472.074.291.498.098.9
Ours, Round 056.26 ± 0.1871.80 ± 0.2074.19 ± 0.3491.05 ± 0.5896.55 ± 0.1798.22 ± 0.14
Ours, Round 157.04 ± 0.3973.00 ± 0.3974.36 ± 0.4991.54 ± 0.5098.47 ± 0.4499.38 ± 0.54
Ours, Round 257.48 ± 0.3972.89 ± 0.1974.99 ± 0.2091.85 ± 0.1098.27 ± 0.3399.25 ± 0.07

We observe that seed variance is small: std ≤ 0.87 on every metric of both test sets. On MathlibQR, this variance is negligible compared to our gains over the strongest baseline on this benchmark (LeanSearch v2). On the context-rich set, the results are consistent with our claim that our Lean Finder V2 retains strong performance on context-rich queries against the strongest baseline (Lean Finder). We will add both tables to the camera ready.

Q5. Clarification on query overlap

As stated in Section 4.1, we exclude all test set ground-truth declarations and their associated queries from training. We additionally exclude exact matches to test set queries, regardless of their associated target declarations. We do not additionally filter retriever training queries based on similarity to test queries when the training queries target declarations are not in any test set. Lexically similar queries paired with different ground-truth declarations can still be valid training examples, since small changes in wording can require different declarations. Established code-search papers, such as CodeT5+ and CoCoSoDa, likewise do not perform a separate cross-split query-only near-duplicate filter (Wang et al., 2023; Shi et al., 2023).

Wang et al. (2023), CodeT5+: Open Code Large Language Models for Code Understanding and Generation.

Shi et al. (2023), CoCoSoDa: Effective Contrastive Learning for Code Search.

Q6. Reliability of LLM-based preference evaluation and its validation

First, we note that using LLM judges as a proxy for human preference is an established practice in Lean/Mathlib retrieval, adopted by prior work such as LeanSearch v2, LeanExplore, and Lean Finder. Given the substantial cost of expert human evaluations, we follow this established protocol by using two best LLMs (at the time of submission), GPT-5.6-Sol and Opus 5, as judges.

We further strengthen Table 10 (validation of LLM judge) by evaluating both judges on the top 30 native retrieval results from Lean Finder V2 without insertion of ground truth on MathlibQR. Claude Opus 5 and GPT-5.6 Sol achieve 90.7% and 90.1% agreement with human-curated labels, respectively. These results demonstrate that both frontier models can reliably identify the human-preferred mathematical statement among similar candidates that naturally appear in top retrievals. Consistently, both judges independently choose Lean Finder V2’s retrievals as the best most frequently in our preference study (Table 3).

Q7. Methodological Novelty and Contributions

The core novelty of our work lies in identifying ambiguous queries, commonly issued by humans and coding agents in real-world interactions, as a central bottleneck of Lean/Mathlib search, and addressing this problem. This bottleneck has not been discussed or explored in prior Lean search engine papers, and we address it through our complementary model level adversarial training and optimized input-level query rewriting. As reviewers aY2t and 1fws acknowledge, our paper identifies and addresses a real gap and problem in existing Lean search engines.

Prior work such as InPars, Promptagator, and Lean Finder uses fixed LLMs to generate synthetic training queries, while AR2 adversarially mines hard negative documents for existing labeled queries. In contrast, we train the query generator itself using RL with our retriever-guided rewards, producing large-scale and effective training data that improves retrieval on ambiguous real-world queries. Furthermore, we reuse the same synthetic data to optimize a query-rewriting prompt, enabling complementary disambiguation before retrieval.

As all reviewers recognize, our proposed methods effectively address this ambiguous query problem, supported by extensive evaluations across diverse benchmarks and tasks, and our state-of-the-art performance on MathlibQR. Reviewers aY2T and 1fws further highlight the practical value and contributions of our work to the Lean community.

We will expand the related-work discussion to include InPars and Promptagator in camera ready.

Thank you for the detailed response, which resolves most of my concerns. My view on novelty is unchanged, so I keep my rating at 3.

AuthorsOpenReview ↗

Thank you for your response

Dear Reviewer 1fws,

Thank you for taking the time to review our responses. We are glad that our responses have addressed your concerns, and we sincerely appreciate your constructive feedback and your positive rating of 3.

Thank you again for your time and support.

Best regards, Authors of Submission 23348

Reviewer_a7dROpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The authors introduce Lean Finder V2, a semantic search framework for Mathlib. The method reaches SOTA on MathlibQR and wins previous best system by frontier LLM judges. Lean Finder V2 is trained using retriever-guided adversarial training to disambiguate user queries. GEPA-optimized query rewriting further improves MathlibQR retrieval for Lean Finder V2. The authors also provide an extensive experiment on the downstream tasks to prove the advantage of Lean Finder V2.

Critical strengths
  1. The paper is well written and easy to read.
  2. The methodology is clear and strong. The authors propose Lean Finder V2 as a dedicated solution to disambiguate user queries, and adversarial training sounds very reasonable.
  3. The authors prove the advantage of Lean Finder V2 through extensive experiments. The results are clearly very strong.

Overall, I think there is no major flaw for the authors to close their claim in this paper.

Critical weaknesses

The paper has no critical weaknesses to me.

Questions for authors
  1. I believe the authors have the necessary expertise to justify this but I feel it would be helpful to provide some baselines other than adversarial training.
  2. I am also interested about the performance when it is used by other frontier models than Claude, but I guess that really depends on whether authors can find enough credits for it.
1 comment

We thank Reviewer a7dR for the constructive review, and for recognizing the effectiveness of our approach to disambiguating user queries, and the strong results demonstrated through extensive experiments.

Q1. Additional baselines

In the paper, we compare against all six publicly available baselines that we are aware of at the time of submission. We would be happy to include additional baselines if the reviewer has any further suggestions.

Q2. Interest in seeing performance with other frontier models

We thank the reviewer for their interest. We additionally evaluate Lean Finder V2 with the latest frontier model GPT-6.1-Sol (OpenCode harness) using the same setup and benchmark as in the paper. Due to budget constraints, we conduct one run to compare Lean Finder V2 against the baseline without semantic search:

  • Without semantic search: 95/100, $0.19/problem
  • With Lean Finder V2: 97/100, $0.15/problem

While the success rates are nearly saturated with small differences, Lean Finder V2 configuration solves 2 more problems while reducing inference cost. This indicates that Lean Finder V2 remains beneficial even when integrated with the latest frontier model.

Comments on the paper

1 comment

We sincerely thank all reviewers for their positive review and constructive feedback. We are glad that Reviewers 1fws and aY2t recognized that our work identifies a real gap and bottleneck in existing Lean search engines: the ambiguous queries commonly issued by humans and coding agents. They further highlighted the practical value of our work to the Lean community. We are also grateful for the unanimous recognition of our method’s effectiveness and its improvements over prior work, demonstrated through comprehensive experiments on diverse benchmarks.

We have conducted additional experiments and have addressed all questions from reviewers. Below, we address a question on novelty; all other questions are addressed in the individual responses.

Novelty

Our core novelty lies in identifying ambiguous queries as a central bottleneck of Mathlib search, one that prior Lean search engines leave unaddressed, and addressing it through complementary model-level adversarial training and input-level query rewriting. We thank Reviewers 1fws and aY2t for acknowledging this real bottleneck.

Prior information retrieval methods like IRGAN and AR2 mine hard negative documents for a fixed set of labeled queries, but labeled ambiguous queries do not exist in our setting. Other works like InPars, Promptagator, and Lean Finder generate synthetic queries with fixed LLMs. In contrast, we train the query generator itself with RL using retriever-guided rewards, producing large-scale and effective training data for the retriever. We further reuse the synthetic data to optimize a query-rewriting prompt, which disambiguates user queries before retrieval.

As all reviewers recognize, our proposed methods effectively address this ambiguous query problem, supported by extensive evaluations across diverse benchmarks and tasks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.