acceptodds
Under review as a conference paper at ICLR 2027

Superintelligent Retrieval Agent: The Next Frontier of Agentic Retrieval

Abstract

Retrieval-augmented agents are increasingly the interface to large organizational and public knowledge bases, yet most still treat retrieval as a black box: they issue exploratory queries, inspect returned snippets, and iteratively reformulate until useful evidence emerges. This resembles how a newcomer searches an unfamiliar database rather than how an expert navigates it with strong priors about terminology, constraints, and likely evidence, leading to unnecessary retrieval rounds, increased latency, and poor recall. We introduce Superintelligent Retrieval Agent (SIRA), which defines superintelligence in retrieval as the ability to compress multi-round exploratory search into a single corpus-discriminative retrieval action. SIRA does not merely ask what terms are relevant to the query; it asks which terms are likely to separate the desired evidence from corpus-level confusers. On the corpus side, an LLM enriches each document offline with missing search vocabulary; on the query side, it predicts evidence vocabulary omitted by the query; and corpus statistics are used as tool calls to filter proposed terms that are absent, overly common, or unlikely to create retrieval margin. In the main BEIR and QA setting, the final retrieval step is a single weighted BM25 call combining the original query with the validated expansion. Across ten BEIR benchmarks, SIRA achieves the strongest average retrieval performance in our comparison, outperforming dense retrievers, learned sparse retrievers, and LLM-based search-agent baselines while using no relevance labels or retriever fine-tuning. On question answering, SIRA's retrieval-only answer coverage exceeds recent RL-trained agentic QA systems on NQ and HotpotQA. Finally, we introduce BrowseComp-Wikipedia, a hard-search benchmark of 232 BrowseComp-derived queries grounded in a 25,587,229-document English Wikipedia index. Even without index-time LLM document enrichment, using only grounded Wikipedia categories as corpus-visible structure, SIRA outperforms multi-round Perplexity agents at every retrieval budget, reaching 9.70% Recall@1, 15.27% Recall@10, and 36.14% Recall@100. These results show that pre-planned, corpus-grounded lexical retrieval can outperform substantially more expensive multi-round search while remaining interpretable, training-free, and efficient.

Primary area:
foundation or frontier models, including LLMs
Keywords:
BM25, Retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_wyzKOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a new retrieval method combining LLM reasoning and BM25 and get better results on a lot of QA benchmarks.

Critical strengths

none

Critical weaknesses
  1. This paper is totally AI-written. The feature is : very long abstract, related work in the introduction, text covered by line in the figure.
  2. Overclaim of superintelligence but the method is only combination of BM25 an LLM.
Questions for authors

none

Reviewer_QnryOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SIRA uses a frozen LLM to expand document and query vocabulary, filters the proposed terms using corpus statistics, and retrieves documents with weighted BM25. It aims to reduce the need for repeated search. The paper reports strong average results on ten BEIR datasets and introduces BrowseComp-Wikipedia to test retrieval over a large Wikipedia index.

Critical strengths
  1. The idea is simple and practical. Checking whether proposed search terms actually appear in the corpus is a useful way to guide LLM query expansion.
Critical weaknesses
  1. My main concern is the QA comparison in Section 3.3. SIRA is measured by whether a retrieved document contains the answer string, while the other methods are measured by whether they produce the correct answer. These are different tasks. Finding the answer string does not show that a model can identify and use the right evidence, especially on HotpotQA.

  2. The ablation study is missing. It is unclear how much of the gain comes from document expansion, query expansion, or the DF filter. Testing these parts separately would help. It would also be useful to run a query-expansion baseline on the same enriched index, so the benefit of the query method is not mixed with the benefit of changing the corpus representation.

Questions for authors

Please see the weaknesses above. I also have the following concerns and questions.

  1. The BrowseComp protocol needs clarification. Section 3.4 says that retrieved page content cannot guide later queries. However, Appendix C allows fetch_page calls for ranking or verification, and the trace in Appendix D shows these calls between searches. How is the model prevented from using the fetched content when writing its next query? It is also unclear whether Recall@1 and Recall@10 are measured before or after the model has inspected and ranked candidates from the full 100-page budget.

  2. The BrowseComp comparison uses different search backends. SIRA has direct access to a local Wikipedia index and its categories, while the baseline uses Perplexity search restricted to Wikipedia. Differences in coverage and ranking may therefore affect the results. Could the authors run an iterative search agent on the same local index to make the comparison more controlled?

  3. The efficiency claims need measurements. A single final BM25 call does not account for the LLM work done beforehand or the cost of enriching every document. The paper itself says that full document enrichment is too expensive at Wikipedia scale. Could the authors report the actual indexing time and average query latency on the BEIR datasets?

  4. The DF filter removes overly common terms and, on the query side, checks whether they exist in the index. However, these checks do not establish that the terms are relevant to the query or faithful to the document. A rare but incorrect expansion could still pass the filter and receive a high BM25 weight. How often does this happen, and how much does it affect retrieval?

Reviewer_69DUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a so-called "Superintelligent Retrieval Agent (SIRA)", which claims to be corpus-discriminative retrieval. The method enriches potential search keywords using an LLM for the corpus and enriches the query with missing words for the query, while corpus word statistics are used as supplementary information for word choices. Experiments on BEIR and a newly proposed dataset BrowseComp-Wikipedia, show better performance than several baselines.

Critical strengths

None

Critical weaknesses
  1. The headline "Superintelligence" is a significant overclaim. The method itself is essentially a line of query expansion work, similar to many existing works, e.g., Query2Doc or HYDE, and nothing "superintelligent". I do not see novelty in the method itself.

  2. The effectiveness claim is not well supported and not well justified by the experiments. The evaluation is conducted mainly on BEIR, which is a relatively old benchmark, and compared against relatively old baselines on a single LLM backbone only. Although BrowseComp-Wikipedia is also reported, the corpus is largely involved in LLM training, so the knowledge tested is mostly in the LLM's parametric memory.

Questions for authors
  1. Did you compare with modern dense models e.g., Qwen3-Embedding and BGE-M3, and other variants like ColBERT and SPLADE?
  2. Why do HyDE and many baselines generally perform worse than BM25 in the experiments?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.