- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes INSPIRE, a benchmark designed to evaluate scientific literature search agents in an open-ended setting where the final solution and relevance criteria are hidden from the agent. The benchmark formulates literature search as a three-stage process consisting of resource exposure, selection, and ranking, and introduces stage-wise metrics to diagnose different sources of retrieval failure.
The authors construct the benchmark by collecting future computer science target papers and using their pre-cutoff cited antecedents as hidden relevance labels. Experiments on 476 evaluation targets compare several LLM-based search agents under a shared protocol, and a hindsight-SFT study demonstrates that benchmark-derived trajectories can improve search performance. Overall, the paper provides a systematic evaluation framework for scientific search agents, while the validity of using realized citation lineages as a proxy for scientific relevance remains an important consideration.
Critical strengths
- The paper proposes a carefully designed benchmark setting for an important but underexplored problem. The proposed INSPIRE benchmark provides a structured evaluation environment for scientific literature search agents and highlights the difficulty of searching for relevant prior work when the final solution and relevance criteria are hidden. This is a meaningful direction for evaluating AI systems for scientific discovery.
- The paper provides a valuable diagnostic perspective on search failures. The exposure-selection-ranking decomposition offers a more interpretable analysis than a single retrieval metric. This framework helps distinguish whether failures originate from missing relevant resources, inappropriate selection, or ineffective ranking.
Critical weaknesses
- The benchmark may not faithfully measure scientific literature search ability. The central evaluation assumption of INSPIRE is that the realized citation lineage of a future target paper provides an appropriate ground truth for evaluating scientific search agents. However, citation behavior does not necessarily capture all scientifically useful prior work. Important uncited papers, alternative solution paths, or interdisciplinary connections may provide valuable insights but receive no benchmark credit. Therefore, the proposed benchmark may measure the ability to recover future citation antecedents rather than the broader capability of scientific discovery or literature exploration. Although the authors acknowledge this limitation in Sec. 3.1 and Sec. 6, its impact on the validity of the benchmark conclusions remains substantial.
- The construction of hidden relevance labels relies heavily on automated judging and is insufficiently validated. The benchmark depends on LLM-based grading of cited references into relevance tiers and roles (e.g., T1/T2/T3 and conceptual/infrastructural categories). While the paper describes the grading pipeline and sensitivity analyses, there is limited evidence demonstrating that these automatically generated labels consistently agree with expert assessments across the benchmark. Since these labels directly determine the evaluation scores, potential annotation bias or judge-model inconsistency could substantially affect the reported comparisons between agents.
Questions for authors
- How do the authors justify that realized citation lineages are an adequate proxy for scientific literature search quality? The benchmark uses graded antecedents from future target papers as hidden relevance labels. However, useful prior work may not always be cited by the target paper, and alternative research directions may also provide valuable information. Could the authors provide additional validation (e.g., expert evaluation, comparison with independent relevance judgments, or analysis of uncited but potentially relevant works) to demonstrate that INSPIRE measures broader scientific search ability rather than citation-lineage recovery?
- How reliable are the LLM-generated relevance labels used for benchmark evaluation? The benchmark relies on judge models to assign citation roles and relevance tiers (T1/T2/T3). Could the authors provide more details or evidence regarding agreement between these automated labels and independent human expert judgments? Since these labels directly determine nDCG and stage-wise metrics, understanding their reliability is important for interpreting the benchmark results.
- To what extent do the reported results generalize beyond computer science literature? The current benchmark focuses on computer-science targets. Could the authors clarify whether they expect the proposed evaluation framework to transfer to other scientific domains with different citation practices and knowledge structures? Have any cross-domain validation experiments been considered or conducted?
We thank the reviewer for the thoughtful review. We conducted additional human relevance evaluations, annotation studies, and cross-domain experiments to address the three questions.
Q1 / W1: Do citation-derived antecedents reflect scientific literature-search relevance?
We conducted a human relevance assessment of 350 candidates across 50 research problems, including 100 cited antecedents and 250 uncited agent recommendations. Two researchers assessed candidates using four relevance levels: directly actionable (3), substantively relevant (2), background (1), and irrelevant (0).
Human judgments closely reflect the intended tier semantics. All sampled T3 references and 67.7% of T2 references were rated substantively relevant or actionable, while 93.8% of T1 references were classified as background, consistent with their intended role.
Among 250 uncited recommendations, only 3 (1.2%) received substantive-relevance scores, with none classified as directly actionable. These findings provide empirical support for the scientific relevance captured by citation-derived labels, with few high-relevance omissions in the audited recommendation pool.
We further recomputed nDCG@10 using human relevance judgments on the 50-problem subset:
All six model rankings remain unchanged under human reassessment, demonstrating the stability of INSPIRE's system comparisons across automated and human relevance judgments.
These results establish an empirical connection between citation-derived antecedents and independently assessed scientific relevance. We will incorporate the human evaluation and rescoring analysis into the revised paper.
Q2 / W2: How reliable are the LLM-generated relevance tiers?
INSPIRE uses three independent LLM judges to assign relevance tiers and citation roles, with consensus annotations recorded during construction. We further evaluated annotation reliability using two human researchers on 200 citation–target pairs.
The results demonstrate substantial agreement between automated relevance grading and human assessments. Human–LLM weighted kappa reaches 0.702, approaching the human–human agreement of 0.754. Tier Macro-F1 reaches 0.766, further supporting the reliability of the automatically assigned relevance categories.
These results provide independent human support for INSPIRE's graded relevance labels. Combined with the unchanged system rankings under human rescoring in Q1, they offer complementary evidence for both annotation reliability and evaluation robustness.
We will add the human annotation protocol, agreement statistics, and human-rescored evaluation to the revised paper.
Q3: Does INSPIRE generalize beyond computer science?
We extended INSPIRE to lattice quantum chromodynamics (Lattice QCD), constructing 100 research problems in physics and evaluating the same six search agents.
The results demonstrate INSPIRE's applicability beyond computer science and its ability to differentiate scientific-search agents in a distinct research domain. Claude Opus 5 leads with 0.253 nDCG@10 and 0.700 AnyHit@10.
The multi-hit results further reveal the challenge of assembling complementary research antecedents. Claude Opus 5 achieves a 0.210 three-hit rate, compared with 0.700 AnyHit@10, reproducing the distinction between recovering individual references and constructing broader evidence portfolios.
This experiment demonstrates the portability of INSPIRE's construction and evaluation framework to physics. The exposure–selection–ranking formulation provides consistent diagnostics of candidate discovery, retention, and ordering across research domains.
We will incorporate the 100-target Lattice QCD benchmark, its construction protocol, and the complete cross-domain evaluation into the revised paper.