- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
FORAGE proposes Related Works Prediction as a benchmark for agentic retrieval. An agent receives a scientific paper with the related-work section and references removed, then searches for and ranks papers cited in the original related-work section. The authors evaluate several models and retrievers and include oracle and input-length analyses. The task is practical to construct, but I have concerns about what its labels and metrics actually measure.
Critical strengths
Using full scientific papers as search inputs is an interesting alternative to short standalone queries. The work also assembles a usable evaluation pipeline across recent conference papers and tests several retrieval configurations. The diagnostic experiments are a welcome attempt to move beyond a simple leaderboard.
Critical weaknesses
1.
Matching the original bibliography is not the same as finding relevant literature. Which papers an author cites depends on framing, space, and individual choices. A search system may return equally relevant papers that were never cited and receive no credit. The human study does not settle this: it covers 120 source–candidate pairs across 20 papers, and only 71.1% of the evaluated automatic false positives were judged not citable. That leaves room for valid alternatives being treated as errors. Independent relevance judgments for uncited predictions are needed before citation overlap can serve as the main quality measure.
Recall@k is defined in a misleading way. Section 3.4 divides the number of hits by min(k, |R_i|), not by the number of relevant references |R_i|. When a paper has at least k relevant references, this is effectively Precision@k, not standard Recall@k. For example, recovering 4 of 24 references in the top 10 gives 40% under the paper's formula, but 16.7% standard recall. Since the dataset has about 22–24 related-work citations per paper on average, this matters for how readers interpret the headline results. At a minimum, the metric should be renamed and standard Recall@k reported alongside it.
2.
The experiments do not show that adaptive agentic search is necessary. A strong non-agentic system could generate multiple queries from the input paper, retrieve candidates in parallel, and rerank once. There is no clearly budget-matched comparison with that approach. The paper's observation that Gemma issues most queries at the start yet performs competitively makes this control particularly important. The oracle study is also difficult to use for bottleneck attribution: it covers only 16 papers and changes the candidate pool and ranking procedure. Finally, the retrieval corpus itself is drawn from citations in the evaluation papers, creating a curated setting that differs from open literature search.
The evaluation idea is useful, but both the target labels and the headline metric need stronger justification. Until those are addressed, the results are hard to interpret as evidence of better agentic retrieval. My recommendation is Clear rejection (1/4).
Questions for authors
- Could you report standard Recall@k and Precision@k, and revisit the conclusions that rely on the current Recall@k definition?
- How do system rankings change when independent experts judge uncited but potentially relevant retrieved papers?
- Under the same retrieval budget, does adaptive search beat one-shot multi-query retrieval and reranking? Can the oracle comparison use the same examples and ranking setup?
3 comments
Response to Reviewer mTLh [2/3]
W1 (part 2) / Q1
W1 (part 2). Recall@k is defined in a misleading way. Section 3.4 divides the number of hits by min(k, |R_i|), not by the number of relevant references |R_i|. When a paper has at least k relevant references, this is effectively Precision@k, not standard Recall@k. For example, recovering 4 of 24 references in the top 10 gives 40% under the paper's formula, but 16.7% standard recall. Since the dataset has about 22–24 related-work citations per paper on average, this matters for how readers interpret the headline results. At a minimum, the metric should be renamed and standard Recall@k reported alongside it.
Q1. Could you report standard Recall@k and Precision@k, and revisit the conclusions that rely on the current Recall@k definition?
The reviewer is correct. Dividing by gives the capped recall used in BEIR (Thakur et al., 2021). Because most papers have at least 10 related-work citations, it is effectively Precision@10. We rename the metric to CappedRecall@10 and report (True)Recall@10, where Recall@ = :
| Model | BM25 | Gemma | ColBERT | Qwen |
|---|---|---|---|---|
| GPT-OSS | 17.10 | 17.42 | 15.80 | 17.57 |
| Nemotron | 13.65 | 14.37 | 14.70 | 14.91 |
| Gemma 4 31B | 15.72 | 16.69 | 15.51 | 17.05 |
| Experiment | TrueRecall@10 |
|---|---|
| Oracle Linear Scan - Gemma-4-31B | 24.23 ± 1.88 |
Therefore, the abstract’s claim is better put as “the best configuration recovers under a fifth of a paper’s cited related works.” However, as is apparent, the headline claim for the oracle’s performance is preserved even under true recall, as the oracle outperforms all retrieval setups under the new metric as well.
W2 (part 1) / Q3 (part 1)
W2 (part 1). The experiments do not show that adaptive agentic search is necessary. A strong non-agentic system could generate multiple queries from the input paper, retrieve candidates in parallel, and rerank once. There is no clearly budget-matched comparison with that approach. The paper's observation that Gemma issues most queries at the start yet performs competitively makes this control particularly important.
Q3 (part 1). Under the same retrieval budget, does adaptive search beat one-shot multi-query retrieval and reranking?
We perform the experiment recommended by the reviewer, where a system generates multiple queries, reranks and reports its prediction. It is budget-matched with the unconstrained retrieval setup, with a maximum of 8 queries permitted. We perform it on GPT-OSS-120B, as in the unconstrained setup, it tends to prefer making one query per step, as opposed to making multiple at once.
| Experiment | (True)Recall@10 | CappedRecall@10 |
|---|---|---|
| GPT-OSS-120B (single shot) | 18.25 | 36.88 |
We would like to clarify that this experiment is still agentic, as the LLM still generates all queries autonomously. Additionally, we do not claim that adaptive search is necessary for this task. Our unconstrained evaluation setup gives the LLM the option to search adaptively, because adaptivity is something an agent could benefit from in principle, and FORAGE is meant to measure whether agents actually do.
We observe that the single-shot experiment outperforms the retrieval setups. Because the budgets are identical, this gap can’t come from extra computation. This finding is reinforced by Gemma, as it is able to make most queries in a single step and still perform competitively.
- Studies like Trivedi et al., Shao et al., and Khattab et al. have shown that iterative retrieval allows the model to improve retrieval performance on Q/A tasks. By conditioning subsequent queries on past query-result pairs, models are able to elicit better recall and surface more relevant documents.
- The FORAGE benchmark itself is not ill-conditioned for iterative retrieval: the model is allowed to make up to 8 retrieval calls in any order and grouping it wants, with query outputs available in the very next step. There is no constraint that favours burst strategies, and models should, in theory, be able to improve their retrieval based on past outputs.
These points suggest that current agents do not exploit adaptivity well. In theory, adaptivity should not harm performance as the computation and retrieval budgets are the same. We see this as a finding that FORAGE makes visible, not a failure of the benchmark.
Response to Reviewer mTLh [3/3]
W2 (part 2) / Q3 (part 2)
W2 (part 2). The oracle study is also difficult to use for bottleneck attribution: it covers only 16 papers and changes the candidate pool and ranking procedure. Finally, the retrieval corpus itself is drawn from citations in the evaluation papers, creating a curated setting that differs from open literature search.
Q3 (part 2). Can the oracle comparison use the same examples and ranking setup?
We’d like to clarify the oracle setup with respect to the reviewer’s concerns:
-
It uses the same candidates. The oracle draws its candidates from the same FORAGE corpus and is scored against the same targets Rᵢ as the retrieval configurations. For a set of source papers, its candidate pool is the union of their citations, ⋃Aᵢ. Each paper’s own related works therefore compete against the other papers’ citations, which are topically close ML papers and hence hard negatives. This is the same kind of competition the agent faces after retrieval. The oracle is meant to simulate a perfect retriever: every true target is guaranteed to be in the pool, alongside realistic distractors. The two-stage classify-then-rank procedure is needed only because full texts for hundreds of candidates don’t fit in one context window. The LLM, the source-paper input and the output format are the same as in the agentic runs.
-
The 16 papers are a deliberate choice. For the oracle to simulate retrieval faithfully, the number of candidates it must sort through has to match what the agent faces. The pool grows with every source paper added, because each paper contributes its full citation list. At 16 papers it reaches about 600 candidates. That is already well above the most the agent can see in a session: 64 passages, from which agents surface 23–36 distinct papers (Table 9). Adding source papers would quickly inflate the pool toward the full 25,050-paper corpus. The oracle would then measure ranking over the whole corpus instead of ranking after perfect retrieval, and it would face far more hard negatives than any real retriever passes on. With 16 papers, the oracle’s pool is larger than the agent’s, so if anything it is a conservative estimate of what perfect retrieval would yield. Should the reviewer prefer a larger sample, the design scales by adding further disjoint groups of 16 papers, each with its own pool of about 600. That keeps the difficulty fixed while increasing the number of examples.
We report stdev for the oracle across 3 runs:
| Experiment | (True)Recall@10 |
|---|---|
| Oracle Linear Scan - Gemma-4-31B | 24.23 ± 1.88 |
We hope we have interpreted and responded to the reviewer’s concerns appropriately.
Response to Reviewer mTLh [1/3]
We thank the reviewer for their comments.
W1 (part 1) / Q2
We agree that citation overlap is not strictly the same as relevance. Authors leave out relevant work for reasons of framing and space, so a system can be penalized for valid alternatives. However, we provide justifications for why it is still a valuable metric for evaluation.
Ground Truth Incompleteness Study. For each ground-truth list of related works, we simulate a random dropout of p% of the works (mirroring the real world case where definitively related works are missed). We re-score each configuration on the new ground truth and measure correlation of the new rankings, based on recall@10 (using Kendall’s Tau) with the original rankings. Each scoring is repeated across 200 different seeds. This follows standard practice for evaluating retrieval under incomplete relevance judgments (Buckley & Voorhees, 2004). We use the 12 full retrieval configurations.
Rankings stay highly stable even with 30% of positives hidden, showing that evaluation is robust to incomplete ground truth. We acknowledge that this is not fully representative of the real world; dropout is not random and less popular papers are more likely to be missed, but it is a strong indication towards robustness.
We are unable to perform a wide-ranging expert judge review of the predictions at this time.