- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ScholarQuest is a benchmark for agentic academic paper search in computer science.
Topic seeds come from ACM CCS nodes, and an LLM writes queries for four research intents: method-oriented, setting-anchored, comparison-based and scope-controlled. Answer sets are built automatically with LLM query rewrites over Google, arXiv and Semantic Scholar, citation expansion and multi-stage LLM relevance judging.
The result is 1,111 queries with about 49K gold papers, plus a 100-query subset curated by PhD annotators. The authors also release ScholarBase, a frozen million-scale arXiv index with sparse, dense and hybrid search plus citation APIs, and a shared Search/Expand harness.
They evaluate full search systems (PaSa, SPAR, PaperScout and commercial engines) and compare seven LLM planners inside the shared harness.
Agentic systems beat single-shot retrieval at R@100, but the best system reaches only 0.312 R@100, and model strengths vary by intent. The frozen environment and harness are the most useful part. The main open question is whether automatically built gold sets are complete enough for a recall benchmark, and the paper isn't positioned against the closest recent benchmarks.
Critical strengths
-
A reusable, frozen literature environment. ScholarBase plus the shared Search/Expand harness lets three open agentic systems and seven LLM planners run on the same backend, and the release is promised. This matters because earlier paper-search agents were each evaluated on different live APIs, so results weren't comparable, and this part stays useful even if the labels are revised later.
-
Larger scale than earlier paper-search benchmarks, with an intent split that shows where agents struggle. Scope and exclusion queries are hard for every system (R@100 at most 0.193), and agent results are averaged over three runs. This gives agent builders a concrete target beyond one aggregate recall number.
Critical weaknesses
-
For a recall benchmark, the gold sets aren't shown to be complete, and they come from one search-and-expand pipeline that looks a lot like the agents being tested. The audit only checks precision. Some cases suggest heavy incompleteness: 'Which papers report that transfer learning outperforms random initialization' has 15 gold papers over all of arXiv, and on the CIFAR-10 image search query all three agents get 0/30, which the paper blames on agent drift without checking whether the retrieved papers were relevant. This matters because if many relevant papers are missing, low recall partly measures disagreement with the construction pipeline, and pipeline-like systems get an edge. The headline 0.312 R@100, the scope finding and the failure analysis all depend on this, so pooled re-judging of system outputs (as in TREC) is needed.
-
Novelty is modest, and the closest benchmarks aren't discussed. Table 1 compares only with PaSa's datasets and SPARBench, while LitSearch (EMNLP 2024), DORIS-MAE (NeurIPS 2023), PaperFindingBench in AstaBench (2025) and AutoResearchBench (April 2026, also a controlled arXiv environment with set retrieval and LLM-judged expansion) are not cited. Table 1 also lists '1,000+ CS topics' as the topic scope, while App. A.3 says the release covers 373 ACM nodes. For a benchmark paper, how it differs from existing benchmarks is the core question, and right now the new parts are mainly the intent templates and the scale.
Questions for authors
- For a random sample of queries, can humans judge the papers that systems return but that aren't in the gold set, e.g. from GPT-5.6-Sol's pools or the zero-recall CIFAR-10 case? What fraction are strict matches, and do the system rankings and the 0.312 R@100 reading still hold after pooled re-judging? This would most change my score.
2 comments
Response to Weakness 1 and Question 1
Thank you for raising this concern about gold-answer completeness. To further assess answer-set coverage, we drew an intent-balanced sample of 20 queries from the 100 queries in ScholarQuest-100, with five queries from each of the four research intents. The four answer sets were obtained as follows:
- Automatic Construction: We directly reused the answer set produced during the original ScholarQuest construction process, which combines query rewriting, multi-source retrieval, citation expansion, and LLM-based relevance adjudication.
- Pooled Search: We pooled all papers returned by nine system-level retrieval methods, merged and deduplicated them, applied Selector-7B as an initial filter, and included the retained candidates in the shared candidate pool for final adjudication by GPT-5.6-Sol (ultra) under the strict-match criterion.
- Independent Human Search: Researchers independently searched the literature according to the query requirements and assessed the relevance of each paper, producing a human-retrieved answer set independent of the automatic construction process.
- Human-in-the-Loop: We directly reused the previously completed final Human-in-the-Loop answer set. Starting from Automatic Construction, researchers iteratively reviewed the existing answers and searched for missing papers until the process converged.
For each query, we pooled and deduplicated the four answer sets. We retained verified judgments for Pooled Search and Independent Human Search, and re-adjudicated Automatic Construction and Human-in-the-Loop candidates using GPT-5.6-Sol (ultra) under the same strict-match criterion. Accepted strict matches formed the shared reference set, giving Pooled Search and Independent Human Search precision 1.0000 by construction. Precision and recall were computed per query and macro-averaged across 20 queries:
| Answer-acquisition method | Precision | Recall |
|---|---|---|
| Automatic Construction | 0.899 | 0.786 |
| Pooled Search | 1.000 | 0.878 |
| Independent Human Search | 1.000 | 0.792 |
| Human-in-the-Loop (Ours) | 0.979 | 0.958 |
1. The coverage gap in automatic construction. Automatic Construction achieves a macro-averaged recall of 0.786 against the shared reference set, leaving an average coverage gap of 21.4%, while approaching the recall of Independent Human Search (0.792). Increasing the retrieval and citation-expansion budgets may further improve its coverage.
2. Does performance change under the expanded shared reference set? Using the same 20 queries, we recomputed query-level Recall@25 and Recall@100 against both the original Gold and the expanded pooled Gold assembled from the four routes, and then macro-averaged the results across the 20 queries (Original → Pooled):
| System | Recall@25 | Recall@100 |
|---|---|---|
| PaperScout | 0.228 → 0.237 | 0.309 → 0.348 |
| SPAR | 0.199 → 0.207 | 0.287 → 0.326 |
| PaSa | 0.216 → 0.224 | 0.299 → 0.339 |
| Hybrid | 0.080 → 0.085 | 0.215 → 0.229 |
| Dense | 0.105 → 0.112 | 0.221 → 0.232 |
| DeepXiv | 0.061 → 0.079 | 0.108 → 0.160 |
| Google Search | 0.092 → 0.099 | 0.131 → 0.145 |
| Semantic Scholar | 0.043 → 0.051 | 0.066 → 0.097 |
| Google Scholar | 0.069 → 0.082 | 0.098 → 0.114 |
All nine systems improve under the expanded reference set. Recall@25 increases by 0.5–1.8 percentage points with unchanged rankings; Recall@100 increases by 1.1–5.2 percentage points, with DeepXiv overtaking Google Search as the only rank change. PaperScout remains first, and the top-three order (PaperScout > PaSa > SPAR) is unchanged, leaving the main system-level comparison intact on this sample.
3. CIFAR-10 case review. For the original query, “Which papers evaluate image search techniques on the CIFAR-10 dataset?”, we reviewed all 70 PaperScout outputs and the top 200 from each of SPAR and PaSa (470 records; 308 unique papers). AI-assisted review of titles and abstracts yielded 0 STRICT, 1 PARTIAL, 190 MISMATCH, and 117 INSUFFICIENT, with no additional strict match confirmed. NASNet [1], ranked first by both SPAR and PaSa, performs classification architecture search on CIFAR-10. PaperScout's top-ranked retrieval paper [2] reports CUB-200-2011 and CIFAR-100 in its abstract, without confirming CIFAR-10 evaluation. P-DARTS [3] reports CIFAR-10 architecture experiments and ReID transfer evaluation, without establishing image retrieval on CIFAR-10. These examples support the original analysis of deviations from the query's topic and setting constraints.
References:
[1] Zoph, Barret, et al. "Learning transferable architectures for scalable image recognition." 2018 IEEE/CVF conference on computer vision and pattern recognition. IEEE, 2018.
[2] Venkataramanan, et al. "Integrating visual and semantic similarity using hierarchies for image retrieval." International Conference on Computer Vision Systems. Cham: Springer Nature Switzerland, 2023.
Response to Weakness 2
Thank you for pointing out these omissions in our discussion of related work. We will include all four benchmarks at the Related Work and clarify their differences from ScholarQuest. LitSearch [1] provides 597 realistic scholarly search queries, but its corpus contains approximately 64,000 papers, with only 1.07 annotated relevant papers per query on average, limiting its ability to assess multi-turn exploration and set retrieval in large-scale literature environments. DORIS-MAE [2] and AutoResearchBench [3] also combine retrieval expansion with relevance judgments to obtain reference annotations, but construct queries from preselected papers without explicit control over query generation and distribution across predefined research intents. This limits analysis of intent-dependent agent performance. The semantic subtask of PaperFindingBench in AstaBench [4] uses an LLM to judge the relevance of papers retrieved by the evaluated systems, adding per-run evaluation costs and potentially introducing variability due to the judging model.
ScholarQuest addresses these limitations through a query-first design: queries are generated from taxonomy topics under four predefined research intents, followed by multi-source retrieval and citation expansion to construct reference sets of 5–200 papers per query. Fixed reference annotations enable direct metric computation and reduce the cost of repeated LLM-based judging. Together with ScholarBase and the shared evaluation harness, this design supports analysis of recall, search efficiency, and exploration behavior across research intents. We will also correct the topic-coverage statement: “1,000+ CS topics” refers to the construction pool, whereas the released benchmark contains 1,111 queries spanning 373 distinct ACM taxonomy nodes. The main text and Table 1 will consistently report coverage of the released benchmark.
References:
[1] Ajith, Anirudh, et al. "Litsearch: A retrieval benchmark for scientific literature search." Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.
[2] Wang, Jianyou Andre, et al. "Scientific document retrieval using multi-level aspect-based queries." Advances in Neural Information Processing Systems 36 (2023): 38404-38419.
[3] Xiong, Lei, et al. "AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery." arXiv preprint arXiv:2604.252
[4] Bragg, Jonathan, et al. "Astabench: Rigorous benchmarking of ai agents with a scientific research suite." International conference on learning representations. Vol. 2026. 2026.