acceptodds
Under review as a conference paper at ICLR 2027

ScholarQuest: A Taxonomy-Guided Benchmark for Agentic Academic Paper Search in Open Literature Environments

Abstract

Academic paper search is a core step in scientific research, and LLM-based search agents are emerging as a promising paradigm for iterative, intent-driven literature exploration. However, existing benchmarks are insufficient for systematically evaluating agentic academic search under realistic open literature environments. We propose ScholarQuest, a taxonomy-guided benchmark for agentic academic paper search. ScholarQuest is constructed over 1,000 computer science topics and organized around four research intents, including method-oriented, setting-anchored, comparison-based, and scope-controlled queries. It further provides scalable answer construction and a shared retrieval backend ScholarBase for reproducible evaluation. We evaluate at both the system and model levels, with nine representative literature search systems and six leading foundation models, respectively. The latter are evaluated using a shared multi-turn search harness. Benchmarking results show that agentic methods outperform single-shot retrieval baselines, yet the best-performing agent among these systems achieves only 0.312 Recall@100 and 0.352 Recall@All, indicating substantial room for improvement. In addition, analyses of search efficiency, intent-level robustness, and failure cases further highlight the benchmark’s ability to provide multi-dimensional evaluation signals for academic paper search agents. Our code and data are publicly available。

Primary area:
datasets and benchmarks
Keywords:
academic paper search, LLM agents, information retrieval, benchmark, evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_XaTMOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ScholarQuest is a benchmark for agentic academic paper search in computer science.

Topic seeds come from ACM CCS nodes, and an LLM writes queries for four research intents: method-oriented, setting-anchored, comparison-based and scope-controlled. Answer sets are built automatically with LLM query rewrites over Google, arXiv and Semantic Scholar, citation expansion and multi-stage LLM relevance judging.

The result is 1,111 queries with about 49K gold papers, plus a 100-query subset curated by PhD annotators. The authors also release ScholarBase, a frozen million-scale arXiv index with sparse, dense and hybrid search plus citation APIs, and a shared Search/Expand harness.

They evaluate full search systems (PaSa, SPAR, PaperScout and commercial engines) and compare seven LLM planners inside the shared harness.

Agentic systems beat single-shot retrieval at R@100, but the best system reaches only 0.312 R@100, and model strengths vary by intent. The frozen environment and harness are the most useful part. The main open question is whether automatically built gold sets are complete enough for a recall benchmark, and the paper isn't positioned against the closest recent benchmarks.

Critical strengths
  1. A reusable, frozen literature environment. ScholarBase plus the shared Search/Expand harness lets three open agentic systems and seven LLM planners run on the same backend, and the release is promised. This matters because earlier paper-search agents were each evaluated on different live APIs, so results weren't comparable, and this part stays useful even if the labels are revised later.

  2. Larger scale than earlier paper-search benchmarks, with an intent split that shows where agents struggle. Scope and exclusion queries are hard for every system (R@100 at most 0.193), and agent results are averaged over three runs. This gives agent builders a concrete target beyond one aggregate recall number.

Critical weaknesses
  1. For a recall benchmark, the gold sets aren't shown to be complete, and they come from one search-and-expand pipeline that looks a lot like the agents being tested. The audit only checks precision. Some cases suggest heavy incompleteness: 'Which papers report that transfer learning outperforms random initialization' has 15 gold papers over all of arXiv, and on the CIFAR-10 image search query all three agents get 0/30, which the paper blames on agent drift without checking whether the retrieved papers were relevant. This matters because if many relevant papers are missing, low recall partly measures disagreement with the construction pipeline, and pipeline-like systems get an edge. The headline 0.312 R@100, the scope finding and the failure analysis all depend on this, so pooled re-judging of system outputs (as in TREC) is needed.

  2. Novelty is modest, and the closest benchmarks aren't discussed. Table 1 compares only with PaSa's datasets and SPARBench, while LitSearch (EMNLP 2024), DORIS-MAE (NeurIPS 2023), PaperFindingBench in AstaBench (2025) and AutoResearchBench (April 2026, also a controlled arXiv environment with set retrieval and LLM-judged expansion) are not cited. Table 1 also lists '1,000+ CS topics' as the topic scope, while App. A.3 says the release covers 373 ACM nodes. For a benchmark paper, how it differs from existing benchmarks is the core question, and right now the new parts are mainly the intent templates and the scale.

Questions for authors
  1. For a random sample of queries, can humans judge the papers that systems return but that aren't in the gold set, e.g. from GPT-5.6-Sol's pools or the zero-recall CIFAR-10 case? What fraction are strict matches, and do the system rankings and the 0.312 R@100 reading still hold after pooled re-judging? This would most change my score.
2 comments

Response to Weakness 2

Thank you for pointing out these omissions in our discussion of related work. We will include all four benchmarks at the Related Work and clarify their differences from ScholarQuest. LitSearch [1] provides 597 realistic scholarly search queries, but its corpus contains approximately 64,000 papers, with only 1.07 annotated relevant papers per query on average, limiting its ability to assess multi-turn exploration and set retrieval in large-scale literature environments. DORIS-MAE [2] and AutoResearchBench [3] also combine retrieval expansion with relevance judgments to obtain reference annotations, but construct queries from preselected papers without explicit control over query generation and distribution across predefined research intents. This limits analysis of intent-dependent agent performance. The semantic subtask of PaperFindingBench in AstaBench [4] uses an LLM to judge the relevance of papers retrieved by the evaluated systems, adding per-run evaluation costs and potentially introducing variability due to the judging model.

ScholarQuest addresses these limitations through a query-first design: queries are generated from taxonomy topics under four predefined research intents, followed by multi-source retrieval and citation expansion to construct reference sets of 5–200 papers per query. Fixed reference annotations enable direct metric computation and reduce the cost of repeated LLM-based judging. Together with ScholarBase and the shared evaluation harness, this design supports analysis of recall, search efficiency, and exploration behavior across research intents. We will also correct the topic-coverage statement: “1,000+ CS topics” refers to the construction pool, whereas the released benchmark contains 1,111 queries spanning 373 distinct ACM taxonomy nodes. The main text and Table 1 will consistently report coverage of the released benchmark.

References:

[1] Ajith, Anirudh, et al. "Litsearch: A retrieval benchmark for scientific literature search." Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.

[2] Wang, Jianyou Andre, et al. "Scientific document retrieval using multi-level aspect-based queries." Advances in Neural Information Processing Systems 36 (2023): 38404-38419.

[3] Xiong, Lei, et al. "AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery." arXiv preprint arXiv:2604.252

[4] Bragg, Jonathan, et al. "Astabench: Rigorous benchmarking of ai agents with a scientific research suite." International conference on learning representations. Vol. 2026. 2026.

Response to Weakness 1 and Question 1

Thank you for raising this concern about gold-answer completeness. To further assess answer-set coverage, we drew an intent-balanced sample of 20 queries from the 100 queries in ScholarQuest-100, with five queries from each of the four research intents. The four answer sets were obtained as follows:

  • Automatic Construction: We directly reused the answer set produced during the original ScholarQuest construction process, which combines query rewriting, multi-source retrieval, citation expansion, and LLM-based relevance adjudication.
  • Pooled Search: We pooled all papers returned by nine system-level retrieval methods, merged and deduplicated them, applied Selector-7B as an initial filter, and included the retained candidates in the shared candidate pool for final adjudication by GPT-5.6-Sol (ultra) under the strict-match criterion.
  • Independent Human Search: Researchers independently searched the literature according to the query requirements and assessed the relevance of each paper, producing a human-retrieved answer set independent of the automatic construction process.
  • Human-in-the-Loop: We directly reused the previously completed final Human-in-the-Loop answer set. Starting from Automatic Construction, researchers iteratively reviewed the existing answers and searched for missing papers until the process converged.

For each query, we pooled and deduplicated the four answer sets. We retained verified judgments for Pooled Search and Independent Human Search, and re-adjudicated Automatic Construction and Human-in-the-Loop candidates using GPT-5.6-Sol (ultra) under the same strict-match criterion. Accepted strict matches formed the shared reference set, giving Pooled Search and Independent Human Search precision 1.0000 by construction. Precision and recall were computed per query and macro-averaged across 20 queries:

Answer-acquisition methodPrecisionRecall
Automatic Construction0.8990.786
Pooled Search1.0000.878
Independent Human Search1.0000.792
Human-in-the-Loop (Ours)0.9790.958

1. The coverage gap in automatic construction. Automatic Construction achieves a macro-averaged recall of 0.786 against the shared reference set, leaving an average coverage gap of 21.4%, while approaching the recall of Independent Human Search (0.792). Increasing the retrieval and citation-expansion budgets may further improve its coverage.

2. Does performance change under the expanded shared reference set? Using the same 20 queries, we recomputed query-level Recall@25 and Recall@100 against both the original Gold and the expanded pooled Gold assembled from the four routes, and then macro-averaged the results across the 20 queries (Original → Pooled):

SystemRecall@25Recall@100
PaperScout0.228 → 0.2370.309 → 0.348
SPAR0.199 → 0.2070.287 → 0.326
PaSa0.216 → 0.2240.299 → 0.339
Hybrid0.080 → 0.0850.215 → 0.229
Dense0.105 → 0.1120.221 → 0.232
DeepXiv0.061 → 0.0790.108 → 0.160
Google Search0.092 → 0.0990.131 → 0.145
Semantic Scholar0.043 → 0.0510.066 → 0.097
Google Scholar0.069 → 0.0820.098 → 0.114

All nine systems improve under the expanded reference set. Recall@25 increases by 0.5–1.8 percentage points with unchanged rankings; Recall@100 increases by 1.1–5.2 percentage points, with DeepXiv overtaking Google Search as the only rank change. PaperScout remains first, and the top-three order (PaperScout > PaSa > SPAR) is unchanged, leaving the main system-level comparison intact on this sample.

3. CIFAR-10 case review. For the original query, “Which papers evaluate image search techniques on the CIFAR-10 dataset?”, we reviewed all 70 PaperScout outputs and the top 200 from each of SPAR and PaSa (470 records; 308 unique papers). AI-assisted review of titles and abstracts yielded 0 STRICT, 1 PARTIAL, 190 MISMATCH, and 117 INSUFFICIENT, with no additional strict match confirmed. NASNet [1], ranked first by both SPAR and PaSa, performs classification architecture search on CIFAR-10. PaperScout's top-ranked retrieval paper [2] reports CUB-200-2011 and CIFAR-100 in its abstract, without confirming CIFAR-10 evaluation. P-DARTS [3] reports CIFAR-10 architecture experiments and ReID transfer evaluation, without establishing image retrieval on CIFAR-10. These examples support the original analysis of deviations from the query's topic and setting constraints.

References:

[1] Zoph, Barret, et al. "Learning transferable architectures for scalable image recognition." 2018 IEEE/CVF conference on computer vision and pattern recognition. IEEE, 2018.

[2] Venkataramanan, et al. "Integrating visual and semantic similarity using hierarchies for image retrieval." International Conference on Computer Vision Systems. Cham: Springer Nature Switzerland, 2023.

Reviewer_A7scOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

ScholarQuest is a benchmark for agentic academic paper search. It includes 1,111 queries drawn from four research-intent categories and evaluates search agents in a shared literature-search environment, ScholarBase. I like the move away from single-query retrieval toward more realistic search workflows. However, I am not yet convinced that the gold sets are complete enough, or that the experiments isolate the benefit of agentic search.

Critical strengths

ScholarBase gives the community a useful common environment for multi-step search. The query taxonomy is a sensible way to create varied retrieval tasks, and the paper goes beyond a single recall score by reporting additional retrieval metrics, run variability, and failure analyses. These are meaningful strengths.

Critical weaknesses

1.

The main uncertainty is gold-answer completeness. The reported human check finds 129 strict matches among 150 sampled candidates labeled as strict matches by the pipeline (86%). That raises a precision question, but the larger issue is coverage: checking papers already found by the pipeline says little about relevant papers it never discovered. A system that finds an overlooked but valid paper may receive no credit. The additional curation for ScholarQuest-100 helps, but its stopping criterion does not establish that the answer sets are close to complete. I would put much more weight on a pooled judgment study that includes papers retrieved by the evaluated systems but absent from the current gold sets. For comparison-oriented queries, some relevance decisions may also require the full paper rather than the title and abstract alone.

The released queries may not represent the intended search space. Appendix A.3 reports that the final dataset covers 373 distinct ACM taxonomy nodes, despite the emphasis on a generation pool of more than 1,000 topics. It also shows a reduction from 39,888 generated queries to 1,111 released queries, partly because queries must have 5–200 automatically identified strict matches. That filter could systematically remove queries that are harder to annotate. The share of comparison-based queries falls from 30.9% to 15.3%, which makes this concern more concrete. The paper should characterize what was lost in filtering, rather than using the generation pool as a proxy for final coverage.

2.

The comparisons do not isolate agentic planning. PaperScout reaches Recall@100 of 0.312, versus 0.212 for hybrid retrieval, but the agent can search repeatedly, rewrite queries, and expand citations while the basic retriever starts from a single query. This establishes the value of a larger search workflow, not necessarily adaptive planning. A strong multi-query retriever with fixed citation expansion and the same search budget would be a more informative baseline. The foundation-model comparisons have a related issue: configurations use different numbers of calls and candidates, and results exclude different failed-query subsets. For example, the GPT-5.6-Sol configuration uses 64.3 tool calls on average versus 32.4 for Gemini 3.1 Pro, while their Recall@50 differs by only 0.038 (Appendix B.3).

The benchmark infrastructure is promising, but the quality of the reference sets and the fairness of the main comparisons need firmer evidence. These issues affect the headline conclusions, so my recommendation is Weak rejection (2/4).

Questions for authors
  1. If you pool the newly retrieved papers from all systems and judge them independently, how many additional valid answers emerge, and do system rankings change?
  2. What does the final query distribution look like relative to the generated pool, especially for comparison-based queries and the 373 covered taxonomy nodes?
  3. How much of PaperScout's advantage remains against budget-matched multi-query retrieval with fixed citation expansion? Do the model rankings hold when failures count as zero?
4 comments

R1: Response to Gold-Answer Completeness and Pooled Judgment (from Weakness 1 & Question 1)

We sincerely thank the reviewer for highlighting this important concern. To address omissions that cannot be measured by our original audit of 450 query–paper pairs, we drew an intent-balanced sample of 20 queries from the 100 queries in ScholarQuest-100, with five queries from each of the four research intents. The four answer sets were obtained as follows:

  • Automatic Construction: We directly reused the answer set produced during the original ScholarQuest construction process, which combines query rewriting, multi-source retrieval, citation expansion, and LLM-based relevance adjudication.
  • Pooled Search: We pooled all papers returned by nine system-level retrieval methods, merged and deduplicated them, applied Selector-7B as an initial filter, and included the retained candidates in the shared candidate pool for final adjudication by GPT-5.6-Sol (ultra) under the strict-match criterion.
  • Independent Human Search: Researchers independently searched the literature according to the query requirements and assessed the relevance of each paper, producing a human-retrieved answer set independent of the automatic construction process.
  • Human-in-the-Loop: We directly reused the previously completed final Human-in-the-Loop answer set. Starting from Automatic Construction, researchers iteratively reviewed the existing answers and searched for missing papers until the process converged.

For each query, we merged and deduplicated candidates from all four routes to form a shared candidate pool. The shared reference set retained the existing verified judgments associated with each answer set: Pooled Search reused the strict-match decisions produced by GPT-5.6-Sol (ultra), whereas Independent Human Search reused the paper-level judgments made by the researchers. Their accepted answers were therefore included directly in the shared reference set, making their precision 1.0000 by construction. Candidates from Automatic Construction and Human-in-the-Loop were re-adjudicated by GPT-5.6-Sol (ultra) under the same strict-match criterion. All papers confirmed as strict matches through these respective procedures jointly formed the shared reference set. We then computed precision and recall for each method on each query and macro-averaged the results across the 20 queries:

Answer-acquisition methodMacro PrecisionMacro Recall
Automatic Construction0.89850.7858
Pooled Search1.00000.8789
Independent Human Search1.00000.7921
Human-in-the-Loop (Ours)0.97950.9576

1. How much relevant literature did the automatic construction pipeline miss? Under its fixed construction budget, Automatic Construction achieves a macro-averaged recall of 0.7858 against the shared reference, leaving an average coverage gap of 21.42%. This recall is close to that of Independent Human Search (0.7921), suggesting that the pipeline can reduce manual search effort as a scalable first-stage discovery method, although final human verification remains necessary. Increasing the number of query rewrites, retrieval depth, and citation-expansion rounds may further improve coverage under the same strict-match criterion.

2. How much additional coverage did Pooled Search provide? Pooled Search achieves a macro-averaged recall of 0.8789. Compared with Automatic Construction, it increases average per-query reference-set coverage by 9.31 percentage points, showing that pooling outputs from multiple systems covers a larger share of the relevant papers in the shared reference set.

3. Do system-level performance and rankings change under the expanded reference set? Using the same 20 queries, we recomputed query-level Recall@25 and Recall@100 against both the original Gold and the expanded Gold from the four routes, and then macro-averaged the results over the 20 queries (Original → Pooled):

SystemMacro Recall@25Macro Recall@100
PaperScout0.228 → 0.2370.309 → 0.348
SPAR0.199 → 0.2070.287 → 0.326
PaSa0.216 → 0.2240.299 → 0.339
Hybrid0.080 → 0.0850.215 → 0.229
Dense0.105 → 0.1120.221 → 0.232
DeepXiv0.061 → 0.0790.108 → 0.160
Google Search0.092 → 0.0990.131 → 0.145
Semantic Scholar0.043 → 0.0510.066 → 0.097
Google Scholar0.069 → 0.0820.098 → 0.114

All nine systems improve under the expanded reference set. Recall@25 increases by 0.5–1.8 percentage points with unchanged rankings; Recall@100 increases by 1.1–5.2 percentage points, with DeepXiv overtaking Google Search as the only rank change. PaperScout remains first, and the top-three order (PaperScout > PaSa > SPAR) is unchanged, leaving the main system-level conclusion intact.

R2: Response to the Full-text Evidence Concern (from Weakness 1)

Thank you for raising the issue of full-text evidence. Most existing academic paper retrieval systems, including all baselines in our study, assess relevance primarily from titles and abstracts for efficiency and scalability, without accessing full-text sections such as the introduction, methods, or experiments. We therefore deliberately adopted the same evidence scope when constructing ScholarQuest. For comparison-based queries, a paper is labeled a strict match only when the required comparative relation is explicitly supported by its title or abstract; evidence verifiable only from the full text falls outside the scope of the current evaluation. This design does not discount the value of full text, but avoids information asymmetry between annotators and evaluated systems, thereby ensuring evaluation fairness.

 

R3: Response to Final Query Distribution and Taxonomy Coverage (from Weakness 2 & Question 2)

Thank you for asking us to clarify the distribution of the final query set. Query generation produced 39,888 queries, of which 13,097 remained after exact deduplication. To control answer-set construction costs, we stratified by the four research intents and sampled 625 queries per intent (2,500 in total). After answer retrieval, we retained queries with 5–200 strict-match papers, yielding 1,111 queries (44.4%).

The final set contains 302 method-oriented (27.2%), 322 setting-anchored (29.0%), 317 scope-controlled (28.5%), and 170 comparison-based queries (15.3%). Relative to the balanced pool of 625 queries per intent, the retention rates for the first three intents are 48.3%, 51.5%, and 50.7%, respectively, compared with only 27.2% for comparison-based queries. This lower retention is consistent with their smaller answer sets: across the 625 comparison-based queries, the mean number of strict matches is 5.23 and the median is 1; 455 queries (72.8%) have fewer than five matches, while none exceeds 200. Thus, attrition for comparison-based queries is driven primarily by the lower answer-count threshold.

Across the 373 covered ACM taxonomy nodes, the final queries are not concentrated in a small number of popular topics. The median is 3 queries per node (IQR: 2–4); 271/373 nodes (72.7%) contain at most 3 queries, and the 10 most represented nodes account for only 8.1% of all queries. We will revise the coverage description to distinguish clearly between the construction-stage topic pool and the 373 taxonomy nodes represented in the final release.

R4: Budget-Matched Baseline (from Question 3)

Thank you for highlighting this issue. PaSa, our system-level baseline, directly instantiates the suggested fixed pipeline: a preset number of Search calls followed by preset Expand calls on retrieved papers in descending order of similarity. Only the Search query strings are rewritten by the LLM based on accumulated context, and the tool schedule remains fixed. PaSa and PaperScout share the same ScholarBase and tool interface.

We further conduct a budget-matched comparison in Figure 7(a), evaluating PaSa and PaperScout at the same number of reconstructed tool steps. Under this matched tool budget, PaSa leads initially, but PaperScout overtakes it after approximately 15 steps. At 45 steps, PaperScout achieves Recall@All/Recall@50 of 0.354/0.270, versus 0.278/0.234 for PaSa—absolute gains of approximately 0.075/0.037. This advantage does not come from greater resource use: Figure 6 shows 45.0 actual calls and 408 candidates for PaperScout, versus 60.1 calls and 744 candidates for PaSa. Thus, PaperScout retains its advantage over fixed multi-query retrieval with citation expansion despite using fewer calls and examining fewer candidates.

 

R5: Response to Model Rankings after Completing Failed Runs (from Question 3)

Due to API provider issues, a small number of model-level results were missing. After completing these runs under the original configurations, the results are as follows (Before →\rightarrow After; absolute change in parentheses):

Model  Completed Queries  Recall@25  Recall@100  Recall@All
GPT-5.6-Sol94 →\rightarrow 1000.225 →\rightarrow 0.2390.421 →\rightarrow 0.4450.609 →\rightarrow 0.635
Gemini 3.1 Pro94 →\rightarrow 1000.205 →\rightarrow 0.2110.338 →\rightarrow 0.3510.395 →\rightarrow 0.406
Kimi-K2.5100 →\rightarrow 1000.182 →\rightarrow 0.1820.313 →\rightarrow 0.3130.391 →\rightarrow 0.391
DeepSeek-v4-Pro93 →\rightarrow 1000.169 →\rightarrow 0.1810.269 →\rightarrow 0.3000.332 →\rightarrow 0.372
Qwen3.8-Max100 →\rightarrow 1000.161 →\rightarrow 0.1610.286 →\rightarrow 0.2860.361 →\rightarrow 0.361
PaperScout-4B100 →\rightarrow 1000.151 →\rightarrow 0.1510.253 →\rightarrow 0.2530.291 →\rightarrow 0.291
GLM-5.3100 →\rightarrow 1000.160 →\rightarrow 0.1600.255 →\rightarrow 0.2550.278 →\rightarrow 0.278

Completing the missing runs increases the absolute scores of GPT-5.6-Sol, Gemini 3.1 Pro, and DeepSeek-v4-Pro, while the scores of the already complete configurations remain unchanged. The broad model-level conclusion is stable: GPT-5.6-Sol remains first on all three metrics. Under the primary Recall@All metric, the completed ranking is GPT-5.6-Sol >> Gemini 3.1 Pro >> Kimi-K2.5 >> DeepSeek-v4-Pro >> Qwen3.8-Max >> PaperScout-4B >> GLM-5.3. The exact middle ranks are not completely unchanged: DeepSeek moves ahead of Qwen after the missing runs are recovered, with similar cutoff-specific swaps at Recall@25 and Recall@100.

Thanks for your effort. The matched-step PaSa comparison and the completed model runs address my main concerns about comparability. The pooled audit also makes the ranking on the 20 audited queries more credible. Coverage of the full release remains uncertain because the sample comes from ScholarQuest-100 and the pooled reference inherits some of the contributing judgments. I would retain my rating of 2.

Reviewer_mkDQOpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
3(+1)
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ScholarQuest, a large-scale benchmark for agentic academic paper retrieval. The authors construct 1,111 queries covering four types of research intent, and through an innovative iterative manual screening and refinement process, build a high-quality subset of 100 queries, ScholarQuest-100. The authors also construct ScholarBase, a million-scale closed literature retrieval environment, and design a unified multi-round evaluation framework for comparing different methods at both the system level and the model level. In addition, the authors conduct experiments on traditional retrieval methods, academic search agents, and foundation models.

Critical strengths
  1. Contributed a high-quality benchmark for agent-oriented academic retrieval to the community.
  2. The benchmark's multi-round iterative annotation is highly novel and worth learning from.
  3. The evaluations at both the system level and the model level provide a comprehensive examination of existing methods, revealing their key limitations while offering valuable insights for future research.
Critical weaknesses

1.Large-scale query filtering and the imbalanced distribution of research intents may limit the diversity and representativeness of the benchmark. 2.Relevance judgments mainly rely on paper titles, abstracts, and metadata. 3.ScholarBase is a closed paper collection and cannot fully simulate realistic paper queries in an open web environment.

Questions for authors

For comparative queries and complex constraint queries, if the necessary evidence is not in the paper title or abstract, how can the reliability of relevance judgments be ensured? Can verification based on the full text of the paper change these judgment results?

1 comment

Thank you for the response, I will increase my confidence and Informativeness score.

Comments on the paper

1 comment
Area_Chair_qWJkOpenReview ↗

Discussion Window Closing in 2 days. Please Respond to the Author's Rebuttal!

Dear Reviewers,

The authors have thoughtfully responded to your reviews and the discussion window is coming to a close on November 18th. Please take a minute to engage with their response and discuss whether it changes your opinions.

Thank you for your time and effort!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.