acceptodds
Under review as a conference paper at ICLR 2027

FORAGE: Related Works Prediction as a Benchmark for Agentic Retrieval

Abstract

Agentic Retrieval Augmented Generation, where language models autonomously plan queries against a retriever, has several important applications, but lacks rigorous evaluation: single-turn benchmarks ignore iteration, multi-turn benchmarks reward fact lookup over document understanding, and web-based benchmarks are prone to contamination. We introduce FORAGE (**F**inding **O**ut **R**elated **A**rticles via **G**enerative **E**xploration), a live agentic retrieval benchmark built on a related works prediction (RWP) task: given a research paper with its related works section, citations, and bibliography redacted, the system must query a scientific corpus to rank candidate papers by likelihood of being cited as related work. We instantiate FORAGE separately on NeurIPS and ICLR 2025 papers and evaluate three open-source LLMs along with a leading close-source model across sparse, dense, and late-interaction retrievers. The best configuration recovers nearly a third of each paper's cited related works, and an oracle probe that removes the retriever from the loop recovers substantially more, placing the bottleneck in query generation and retrieval. FORAGE provides substantial headroom for future work on agentic retrieval, and is designed to be refreshed each conference cycle to remain contamination-resistant as models evolve. We release the dataset [here](https://anonymous-hf.com/a/o5o42hifzs1a/).

Primary area:
datasets and benchmarks
Keywords:
information retrieval, retrieval-augmented generation (RAG), agentic retrieval, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mTLhOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

FORAGE proposes Related Works Prediction as a benchmark for agentic retrieval. An agent receives a scientific paper with the related-work section and references removed, then searches for and ranks papers cited in the original related-work section. The authors evaluate several models and retrievers and include oracle and input-length analyses. The task is practical to construct, but I have concerns about what its labels and metrics actually measure.

Critical strengths

Using full scientific papers as search inputs is an interesting alternative to short standalone queries. The work also assembles a usable evaluation pipeline across recent conference papers and tests several retrieval configurations. The diagnostic experiments are a welcome attempt to move beyond a simple leaderboard.

Critical weaknesses

1.

Matching the original bibliography is not the same as finding relevant literature. Which papers an author cites depends on framing, space, and individual choices. A search system may return equally relevant papers that were never cited and receive no credit. The human study does not settle this: it covers 120 source–candidate pairs across 20 papers, and only 71.1% of the evaluated automatic false positives were judged not citable. That leaves room for valid alternatives being treated as errors. Independent relevance judgments for uncited predictions are needed before citation overlap can serve as the main quality measure.

Recall@k is defined in a misleading way. Section 3.4 divides the number of hits by min(k, |R_i|), not by the number of relevant references |R_i|. When a paper has at least k relevant references, this is effectively Precision@k, not standard Recall@k. For example, recovering 4 of 24 references in the top 10 gives 40% under the paper's formula, but 16.7% standard recall. Since the dataset has about 22–24 related-work citations per paper on average, this matters for how readers interpret the headline results. At a minimum, the metric should be renamed and standard Recall@k reported alongside it.

2.

The experiments do not show that adaptive agentic search is necessary. A strong non-agentic system could generate multiple queries from the input paper, retrieve candidates in parallel, and rerank once. There is no clearly budget-matched comparison with that approach. The paper's observation that Gemma issues most queries at the start yet performs competitively makes this control particularly important. The oracle study is also difficult to use for bottleneck attribution: it covers only 16 papers and changes the candidate pool and ranking procedure. Finally, the retrieval corpus itself is drawn from citations in the evaluation papers, creating a curated setting that differs from open literature search.

The evaluation idea is useful, but both the target labels and the headline metric need stronger justification. Until those are addressed, the results are hard to interpret as evidence of better agentic retrieval. My recommendation is Clear rejection (1/4).

Questions for authors
  1. Could you report standard Recall@k and Precision@k, and revisit the conclusions that rely on the current Recall@k definition?
  2. How do system rankings change when independent experts judge uncited but potentially relevant retrieved papers?
  3. Under the same retrieval budget, does adaptive search beat one-shot multi-query retrieval and reranking? Can the oracle comparison use the same examples and ranking setup?
3 comments
AuthorsOpenReview ↗

Response to Reviewer mTLh [1/3]

We thank the reviewer for their comments.


W1 (part 1) / Q2

W1 (part 1). Matching the original bibliography is not the same as finding relevant literature. Which papers an author cites depends on framing, space, and individual choices. A search system may return equally relevant papers that were never cited and receive no credit. The human study does not settle this: it covers 120 source–candidate pairs across 20 papers, and only 71.1% of the evaluated automatic false positives were judged not citable. That leaves room for valid alternatives being treated as errors. Independent relevance judgments for uncited predictions are needed before citation overlap can serve as the main quality measure.

Q2. How do system rankings change when independent experts judge uncited but potentially relevant retrieved papers?

We agree that citation overlap is not strictly the same as relevance. Authors leave out relevant work for reasons of framing and space, so a system can be penalized for valid alternatives. However, we provide justifications for why it is still a valuable metric for evaluation.

  1. Comparisons use the same labels on both sides. Incomplete labels lower absolute scores for every system alike. They don’t favor any configuration unless some systems find uncited relevant papers far more often than others.
  2. To quantify the impact of incomplete ground truths on evaluation robustness, we perform the Ground Truth Incompleteness Study, which simulates incomplete GT to analyse how benchmark rankings fare.

Ground Truth Incompleteness Study. For each ground-truth list of related works, we simulate a random dropout of p% of the works (mirroring the real world case where definitively related works are missed). We re-score each configuration on the new ground truth and measure correlation of the new rankings, based on recall@10 (using Kendall’s Tau) with the original rankings. Each scoring is repeated across 200 different seeds. This follows standard practice for evaluating retrieval under incomplete relevance judgments (Buckley & Voorhees, 2004). We use the 12 full retrieval configurations.

pKendall τ (mean ± sd)top-3 set unchanged
0%1.000 ± 0.000100%
5%0.955 ± 0.030100%
10%0.931 ± 0.037100%
15%0.910 ± 0.045100%
20%0.896 ± 0.04498%
30%0.866 ± 0.05484%
40%0.819 ± 0.05350%

Rankings stay highly stable even with 30% of positives hidden, showing that evaluation is robust to incomplete ground truth. We acknowledge that this is not fully representative of the real world; dropout is not random and less popular papers are more likely to be missed, but it is a strong indication towards robustness.

We are unable to perform a wide-ranging expert judge review of the predictions at this time.

AuthorsOpenReview ↗

Response to Reviewer mTLh [2/3]

W1 (part 2) / Q1

W1 (part 2). Recall@k is defined in a misleading way. Section 3.4 divides the number of hits by min(k, |R_i|), not by the number of relevant references |R_i|. When a paper has at least k relevant references, this is effectively Precision@k, not standard Recall@k. For example, recovering 4 of 24 references in the top 10 gives 40% under the paper's formula, but 16.7% standard recall. Since the dataset has about 22–24 related-work citations per paper on average, this matters for how readers interpret the headline results. At a minimum, the metric should be renamed and standard Recall@k reported alongside it.

Q1. Could you report standard Recall@k and Precision@k, and revisit the conclusions that rely on the current Recall@k definition?

The reviewer is correct. Dividing by min(k,∣Ri∣)\text{min}(k, |R_i|) gives the capped recall used in BEIR (Thakur et al., 2021). Because most papers have at least 10 related-work citations, it is effectively Precision@10. We rename the metric to CappedRecall@10 and report (True)Recall@10, where Recall@kk = ∣top-k∩Ri∣∣Ri∣\frac{|\text{top-}k ∩ R_i|}{|R_i|}:

ModelBM25GemmaColBERTQwen
GPT-OSS17.1017.4215.8017.57
Nemotron13.6514.3714.7014.91
Gemma 4 31B15.7216.6915.5117.05
ExperimentTrueRecall@10
Oracle Linear Scan - Gemma-4-31B24.23 ± 1.88

Therefore, the abstract’s claim is better put as “the best configuration recovers under a fifth of a paper’s cited related works.” However, as is apparent, the headline claim for the oracle’s performance is preserved even under true recall, as the oracle outperforms all retrieval setups under the new metric as well.


W2 (part 1) / Q3 (part 1)

W2 (part 1). The experiments do not show that adaptive agentic search is necessary. A strong non-agentic system could generate multiple queries from the input paper, retrieve candidates in parallel, and rerank once. There is no clearly budget-matched comparison with that approach. The paper's observation that Gemma issues most queries at the start yet performs competitively makes this control particularly important.

Q3 (part 1). Under the same retrieval budget, does adaptive search beat one-shot multi-query retrieval and reranking?

We perform the experiment recommended by the reviewer, where a system generates multiple queries, reranks and reports its prediction. It is budget-matched with the unconstrained retrieval setup, with a maximum of 8 queries permitted. We perform it on GPT-OSS-120B, as in the unconstrained setup, it tends to prefer making one query per step, as opposed to making multiple at once.

Experiment(True)Recall@10CappedRecall@10
GPT-OSS-120B (single shot)18.2536.88

We would like to clarify that this experiment is still agentic, as the LLM still generates all queries autonomously. Additionally, we do not claim that adaptive search is necessary for this task. Our unconstrained evaluation setup gives the LLM the option to search adaptively, because adaptivity is something an agent could benefit from in principle, and FORAGE is meant to measure whether agents actually do.

We observe that the single-shot experiment outperforms the retrieval setups. Because the budgets are identical, this gap can’t come from extra computation. This finding is reinforced by Gemma, as it is able to make most queries in a single step and still perform competitively.

  • Studies like Trivedi et al., Shao et al., and Khattab et al. have shown that iterative retrieval allows the model to improve retrieval performance on Q/A tasks. By conditioning subsequent queries on past query-result pairs, models are able to elicit better recall and surface more relevant documents.
  • The FORAGE benchmark itself is not ill-conditioned for iterative retrieval: the model is allowed to make up to 8 retrieval calls in any order and grouping it wants, with query outputs available in the very next step. There is no constraint that favours burst strategies, and models should, in theory, be able to improve their retrieval based on past outputs.

These points suggest that current agents do not exploit adaptivity well. In theory, adaptivity should not harm performance as the computation and retrieval budgets are the same. We see this as a finding that FORAGE makes visible, not a failure of the benchmark.

AuthorsOpenReview ↗

Response to Reviewer mTLh [3/3]

W2 (part 2) / Q3 (part 2)

W2 (part 2). The oracle study is also difficult to use for bottleneck attribution: it covers only 16 papers and changes the candidate pool and ranking procedure. Finally, the retrieval corpus itself is drawn from citations in the evaluation papers, creating a curated setting that differs from open literature search.

Q3 (part 2). Can the oracle comparison use the same examples and ranking setup?

We’d like to clarify the oracle setup with respect to the reviewer’s concerns:

  • It uses the same candidates. The oracle draws its candidates from the same FORAGE corpus and is scored against the same targets Rᵢ as the retrieval configurations. For a set of source papers, its candidate pool is the union of their citations, ⋃Aᵢ. Each paper’s own related works therefore compete against the other papers’ citations, which are topically close ML papers and hence hard negatives. This is the same kind of competition the agent faces after retrieval. The oracle is meant to simulate a perfect retriever: every true target is guaranteed to be in the pool, alongside realistic distractors. The two-stage classify-then-rank procedure is needed only because full texts for hundreds of candidates don’t fit in one context window. The LLM, the source-paper input and the output format are the same as in the agentic runs.

  • The 16 papers are a deliberate choice. For the oracle to simulate retrieval faithfully, the number of candidates it must sort through has to match what the agent faces. The pool grows with every source paper added, because each paper contributes its full citation list. At 16 papers it reaches about 600 candidates. That is already well above the most the agent can see in a session: 64 passages, from which agents surface 23–36 distinct papers (Table 9). Adding source papers would quickly inflate the pool toward the full 25,050-paper corpus. The oracle would then measure ranking over the whole corpus instead of ranking after perfect retrieval, and it would face far more hard negatives than any real retriever passes on. With 16 papers, the oracle’s pool is larger than the agent’s, so if anything it is a conservative estimate of what perfect retrieval would yield. Should the reviewer prefer a larger sample, the design scales by adding further disjoint groups of 16 papers, each with its own pool of about 600. That keeps the difficulty fixed while increasing the number of examples.

We report stdev for the oracle across 3 runs:

Experiment(True)Recall@10
Oracle Linear Scan - Gemma-4-31B24.23 ± 1.88

We hope we have interpreted and responded to the reviewer’s concerns appropriately.

Reviewer_UgiFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FORAGE, an agentic retrieval benchmark built on a Related Works Prediction (RWP) task: given a recent conference paper with its related-work section, in-text citations, and bibliography redacted, an LLM agent queries a retriever (≤8 calls) over a corpus of cited arXiv papers and ranks candidate related works. It is instantiated on NeurIPS'25 (1,024 papers) and ICLR'25 (772) and intended to be refreshed each cycle. Three open-weight LLMs and GPT-5.6-Luna are evaluated with four retrievers, plus a no-retrieval floor and an oracle linear-scan ceiling. Best configurations reach ~35 Recall@10 vs. 53.4 for the oracle; the authors attribute the gap to query generation and retrieval rather than ranking.

Critical strengths
  • Clear motivation: literature search over long documents is a natural testbed for deep-research-style agentic retrieval, and the refreshable design addresses contamination.
  • Ground truth comes from peer-reviewed author citations — human-produced, verifiable, and free to obtain at scale.
  • Reasonably broad analysis (multiple retrievers, no-retrieval floor, oracle ceiling, truncation ablations, retrieval diagnostics, second instantiation with variance); dataset and code released.
Critical weaknesses

1. Citation-context leakage undermines the "holistic understanding" premise. Citations outside the related-work section become [citation] but their surrounding sentences are kept (Sec. 3.2), so any related-work paper also cited in the intro/method/experiments is recoverable from local context — exactly what Sec. 3.1 calls "largely predictable from local context." The fraction of R_i cited only in related works is never reported, and the truncation ablation (Table 5) cannot separate global signal from leaked citation sentences.

2. The agentic framing is untested and the evaluation is thin. There is no non-agentic baseline (e.g., a single title+abstract query with the same budget), and the best model, Gemma-4-31B, issues nearly all queries in the first step (Sec. 4.1), so the gain cannot be attributed to agency. Only one frontier model with one retriever is evaluated, the oracle probe uses 16 papers, and variance is reported for a single configuration.

3. "Recall@10" is precision@10, and the headline claim depends on it. With h = min(k, |R_i|) and |R_i| ≈ 24, the Sec. 3.4 metric is |top-10 ∩ R_i| / 10, i.e., precision. The abstract's "recovers nearly a third of cited related works" therefore does not follow: P@10 = 34.75 is ~3.5 of ~24 related works (~14% recall), and session-level retrieval recall (Table 9) is only 22–24%.

4. Missing related work with the same core design. DeepScholar-Bench (Patel et al., 2025, arXiv 2508.20033) is a live benchmark that also uses recent arXiv papers' related-work sections as ground truth and scores retrieval against cited references, which directly overlaps the novelty claim in Sec. 1. CiteME (Press et al., 2024), PaSa (He et al., 2025), and ScholarQABench (Asai et al., 2024) are also absent.

Questions for authors
  1. What fraction of R_i is cited only in the related-work section, and what is performance on that subset?
  2. How does a single-shot retrieval baseline with the same budget compare?
  3. Please confirm "Recall@10" is precision@10 and report true recall.
  4. How does FORAGE differ from DeepScholar-Bench?
2 comments
AuthorsOpenReview ↗

Response to Reviewer UgiF [1/2]

We thank the reviewer for their careful and precise review. The reviewer’s comments have helped us improve the contribution.


Q3/W3:

W3. "Recall@10" is precision@10, and the headline claim depends on it. With h = min(k, |R_i|) and |R_i| ≈ 24, the Sec. 3.4 metric is |top-10 ∩ R_i| / 10, i.e., precision. The abstract's "recovers nearly a third of cited related works" therefore does not follow: P@10 = 34.75 is 3.5 of ~24 related works (14% recall), and session-level retrieval recall (Table 9) is only 22–24%.

Q3. Please confirm "Recall@10" is precision@10 and report true recall.

The reviewer’s criticism is correct. This is the capped recall used in BEIR (Thakur et al., 2021), but because |Rᵢ| ≥ 10 for ~86% of source papers, it equals precision@10 in majority cases.

We rename the metric to CappedRecall@10.

We report values for true Recall@10:

ModelBM25GemmaColBERTQwen
GPT-OSS17.1017.4215.8017.57
Nemotron13.6514.3714.7014.91
Gemma 4 31B15.7216.6915.5117.05
ExperimentTrueRecall@10
Oracle Linear Scan - Gemma-4-31B24.23 \pm 1.88

Where Recall@kk = \frac{\text{top-10} ∩ R_i\}{R_i}

Therefore, the abstract’s claim is better put as “the best configuration recovers under a fifth of a paper’s cited related works.” However, as is apparent, the headline claim for the oracle’s performance is preserved even under true recall, as the oracle outperforms all retrieval setups under the new metric as well.

We would like to clarify that the retrieval recall values reported are indeed true recall measurements.


Q1/W1:

W1. Citation-context leakage undermines the "holistic understanding" premise. Citations outside the related-work section become [citation] but their surrounding sentences are kept (Sec. 3.2), so any related-work paper also cited in the intro/method/experiments is recoverable from local context — exactly what Sec. 3.1 calls "largely predictable from local context." The fraction of R_i cited only in related works is never reported, and the truncation ablation (Table 5) cannot separate global signal from leaked citation sentences.

Q1. What fraction of R_i is cited only in the related-work section, and what is performance on that subset?

We thank the reviewer for raising this point. We found that approximately 60% of RiR_i candidates are only cited in the related works section of the paper.

When restricting the ground truth to candidates only cited in the related works section, we find the (True)Recall@10 to be as follows:

ModelBM25QwenGemmaColBERT
GPT-OSS7.127.806.486.91
Nemotron3.324.133.363.58
Gemma 4 31B2.663.142.802.90

The oracle run, in contrast, achieves a recall@10 of 10.28, outperforming the retrieval setups even under the new subset. The difference in scores between the full ground truth and subset indicates the score gained by reporting related works which are cited elsewhere. Therefore, FORAGE is also vulnerable to the issue of citation leakage and will remove it as a differentiating factor for FORAGE.

The scores indicate that, even for citations which have been leaked, the model is not able to achieve strong performance. Hence, in-line citations are not largely predictable from context.


AuthorsOpenReview ↗

Response to Reviewer UgiF [2/2]

Q4/W4

W4. Missing related work with the same core design. DeepScholar-Bench (Patel et al., 2025, arXiv 2508.20033) is a live benchmark that also uses recent arXiv papers' related-work sections as ground truth and scores retrieval against cited references, which directly overlaps the novelty claim in Sec. 1. CiteME (Press et al., 2024), PaSa (He et al., 2025), and ScholarQABench (Asai et al., 2024) are also absent.

Q4. How does FORAGE differ from DeepScholar-Bench?

Thank you for pointing us to these references. We will add these to Table 1.

FORAGE differs from DeepScholarBench (DSB) in several key ways:

MetricFORAGEDeepScholarBenchTakeaway
InputRedacted full-textTitle + abstractFORAGE stresses holistic document understanding while DSB does not
OutputRanked predictionsRecreated related works sectionFORAGE measures retrieval directly, without the variance of text-based eval.
Corpus size1024 papers each for 2 conferences63 papersFORAGE encompasses a much larger variety of papers.
EvaluationComparison with author citationsLLM-as-a-judgeHeavy reliance on LLMs in DSB evaluation introduced potential biases beyond retrieving relevant works.

We will add DeepScholarBench and the other references to Table 1 and the related works section:

BenchmarkAgenticDeep Doc.DomainRefreshable
CiteMEYesNoScientificNo
PaSaYesNoScientificNo
ScholarQABenchPartialNoScientificNo
DeepScholarBenchPartialNoScientificYes

We thank the reviewer for their effort. We hope we have interpreted and addressed their concerns appropriately.

Reviewer_GGS2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces FORAGE, a continually updated benchmark addressing gaps in evaluating autonomous query generation, multi-step retrieval, and long-document understanding while reducing contamination risk. Its core task is related work prediction: given a redacted paper, an agent retrieves and ranks candidate papers by their likelihood of being cited, using the authors’ actual citations as ground truth. The benchmark is refreshed with each conference cycle. Experiments on NeurIPS and ICLR papers evaluate four LLMs paired with three types of retrievers against a no-retrieval baseline and an oracle linear-scan upper bound, revealing a substantial performance gap between the evaluated systems and the upper bound.

Critical strengths
  1. Practical relevance. The task combines paper understanding, autonomous query generation, and multi-step retrieval, closely reflecting the needs of literature discovery in scientific research.
  2. Refreshable benchmark. Using conference papers and their citations as ground truth allows the benchmark to be updated each conference cycle, helping mitigate contamination associated with static benchmarks.
Critical weaknesses
  1. Unclear distinction from prior work. The approach overlaps with AgentExpt in retrieving scholarly resources from paper content and evaluating predictions against observed citations or resource usage. The paper should clarify its additional contributions in task design and evaluation objectives.
  2. Experimental comparability requires clarification. The paper should explain how retrieval calls, the number of returned documents, and computational budgets are controlled across configurations, to determine whether performance differences reflect strategy quality or additional computation.
  3. Potentially incomplete ground truth. Author citations do not exhaustively cover all relevant work. Relevant but uncited papers may therefore be counted as errors, affecting the reliability of the evaluation.
  4. Limited coverage of agent architectures. Although the paper aims to evaluate agentic retrieval capabilities, the experiments primarily compare LLM–retriever combinations under a shared tool-calling setup, without examining representative agent architectures with different planning and iterative retrieval mechanisms.
Questions for authors
  1. How does FORAGE differ from AgentExpt in task formulation and evaluation objectives?
  2. Do the main conclusions hold under matched retrieval and inference budgets?
  3. How does incomplete citation-based ground truth affect the reported scores and model rankings?
  4. Do the findings generalize to agent architectures with different planning and iterative retrieval mechanisms?
2 comments
AuthorsOpenReview ↗

Response to Reviewer GGS2 [1/2]

We thank the reviewer for their careful reading of our paper and for the constructive feedback. We are glad the reviewer found the task practically relevant to literature discovery and appreciated the refreshable design of the benchmark. Below, we address each weakness and question in turn.


W1/Q1

W1. Unclear distinction from prior work. The approach overlaps with AgentExpt in retrieving scholarly resources from paper content and evaluating predictions against observed citations or resource usage. The paper should clarify its additional contributions in task design and evaluation objectives.

Q1. How does FORAGE differ from AgentExpt in task formulation and evaluation objectives?

Thank you for pointing us to AgentExpt. We now include it in the Table 1 comparison and the related work section. The AgentExpt task and dataset differ from FORAGE in both task design and evaluation objectives:

Task design:

  • Targets: AgentExpt focuses on recommending resources (baselines and datasets) given a paper, while FORAGE focuses on related works. By our classification, only about 2–3% of related works fall into this category (Datasets/Benchmarks, Table 3). Even if the "Competing" category overlaps with baselines, more than 50% of FORAGE targets are outside AgentExpt's scope.
  • Inputs: FORAGE provides the LLM with the redacted full text of the paper, averaging about 20k tokens, while AgentExpt tests on an AI-generated summary of the paper. This distinction matters because it stresses long-document and paper understanding (see the truncated-input ablation), one of the main things that make FORAGE practically relevant.
  • Corpora: In AgentExpt, like the input papers, the candidates are short entity descriptions made up of self-descriptions and LLM summaries. In FORAGE, the system must search through actual full-text research papers, simulating deployment scenarios.

Evaluation objectives:

  • Refreshability: FORAGE is designed to be live and refreshable, so it stays unsaturated for newer models. The AgentExpt dataset is inherently non-refreshable because it relies on a candidate's later usage to describe it, so a new candidate may not have enough usage to construct the dataset.
BenchmarkAgenticDeep Doc.DomainRefreshable
AgentExpt (dataset)YesNoScientificNo

W2/Q2

W2. Experimental comparability requires clarification. The paper should explain how retrieval calls, the number of returned documents, and computational budgets are controlled across configurations, to determine whether performance differences reflect strategy quality or additional computation.

Q2. Do the main conclusions hold under matched retrieval and inference budgets?

We thank the reviewer for pointing this out. To make the controls clearer, we are adding a paragraph that discusses them explicitly:

Every configuration and test runs under the same computational budget. The LLM is allowed at most 8 tool calls per sample (set empirically, based on when the context length runs out). Each call returns 8 chunks, using the same chunking setup (1024 tokens, 128 overlap). The context window (131k) and decoding parameters are also the same. A sample is retried at most once if it fails (context window exceeded, invalid JSON response, etc.).

Eval ParameterValue
Max tool calls per sample8
No. of chunks returned per call8
Max retries1
Temperature1.0
Top P1
Reasoning EffortHigh
Context Length131072 tokens

The variation in the number of distinct papers retrieved comes from differences in query generation and retriever performance. Depending on how specific or general a query is, an LLM may retrieve many distinct papers or many chunks from the same paper. This is an important signal when comparing LLMs' search and retrieval performance.


AuthorsOpenReview ↗

Response to Reviewer GGS2 [2/2]

W3/Q3

W3. Potentially incomplete ground truth. Author citations do not exhaustively cover all relevant work. Relevant but uncited papers may therefore be counted as errors, affecting the reliability of the evaluation.

Q3. How does incomplete citation-based ground truth affect the reported scores and model rankings?

We agree that author citations are not exhaustive, and we discuss this in Sections 3.1 and 5. We have added a new analysis that shows how this affects scores and rankings.

Ground Truth Incompleteness Study

This study simulates the exact situation the reviewer describes, in which authors leave relevant papers uncited. For each ground-truth list of related works, we randomly drop p% of the works, mirroring the real-world case where definitively related works are missed. We re-score each configuration on the new ground truth and use Kendall's τ to measure how well the new recall@10 rankings correlate with the original rankings. Each scoring is repeated across 200 random seeds. This follows standard practice for evaluating retrieval under incomplete relevance judgments (Buckley & Voorhees, 2004). We use the 12 full retrieval configurations.

pKendall τ (mean ± sd)Top-3 set unchanged
0%1.000 ± 0.000100%
5%0.955 ± 0.030100%
10%0.931 ± 0.037100%
15%0.910 ± 0.045100%
20%0.896 ± 0.04498%
30%0.866 ± 0.05484%
40%0.819 ± 0.05350%

Rankings remain highly stable even with 30% of positives hidden, which shows that the evaluation is robust to incomplete ground truth. We acknowledge that this is not fully representative of the real world: dropout is not random, and less popular papers are more likely to be missed. Still, it is a strong indication of robustness.


W4/Q4

W4. Limited coverage of agent architectures. Although the paper aims to evaluate agentic retrieval capabilities, the experiments primarily compare LLM–retriever combinations under a shared tool-calling setup, without examining representative agent architectures with different planning and iterative retrieval mechanisms.

Q4. Do the findings generalize to agent architectures with different planning and iterative retrieval mechanisms?

Our main setup uses unconstrained native tool calling. Within the budget, the LLM may plan, batch or interleave its queries however it likes, and it decides when to stop. We chose this so that the benchmark measures each model's own retrieval strategy rather than one we impose. Different strategies do emerge: Gemma-4-31B issues most of its queries upfront, while GPT-OSS and Nemotron interleave about one query per step (Sec. 4.1).

To separate the effect of the architecture from that of the model, we also evaluate LLMs under constrained agent architectures. Since Gemma naturally prefers to issue most of its queries in one step and GPT-OSS queries one at a time, we impose the opposite architecture on each model: ReAct (a strict thought → tool call → observe-and-analyse loop) on Gemma, and plan-then-execute on GPT-OSS. Results are summarized below.

Model – ArchitectureRecall@10
Gemma-4-31B – ReAct31.59
GPT-OSS-120B – Plan-then-Execute36.88

Ethics Review

Flag For Ethics Review: Yes

We would be grateful if the reviewer could clarify the reason for the ethics review flag, so that we can address any concerns directly.


We thank the reviewer again for their thoughtful feedback, which has helped us strengthen the paper. In summary, we have (1) clarified FORAGE's distinction from AgentExpt and added it to our comparisons, (2) documented the matched computational budget used across all configurations, (3) added a ground-truth incompleteness study showing that rankings are robust to missing citations, and (4) added experiments with constrained agent architectures.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.