acceptodds
Under review as a conference paper at ICLR 2027

Inspire: Benchmarking Scientific Literature Search for Open Research Problems

11 views

Abstract

Scientific literature search often begins with an open research problem rather than a known target paper or a fixed candidate set. We introduce INSPIRE, a benchmark for evaluating agents that search prior literature to make progress on solution-redacted research problems. Each instance pairs a research brief with a target-specific cutoff three months before a later paper and evaluates ranked outputs against graded cited antecedents from that paper's realized research lineage. Search proceeds over an open corpus, while the identity of the target paper and membership of its cited antecedents remain hidden from the agent. Beyond end-to-end retrieval quality, INSPIRE uses logged search trajectories to distinguish three coupled stages: resource exposure, whether useful antecedents are surfaced during search; selection, whether exposed antecedents are retained; and ranking, how effectively retained papers are ordered. Across 476 computer-science targets under a shared search interface and budget, the strongest evaluated agent achieves 0.284 nDCG@10. Results show that current agents more readily recover an isolated antecedent than assemble a broader portfolio of relevant prior work. The stagewise analysis identifies resource exposure as the largest observed bottleneck, with further losses in selection and ranking. We additionally construct replay-valid hindsight demonstrations and show that they improve held-out search without changing test-time information, establishing that the benchmark provides an actionable learning signal. INSPIRE therefore enables both end-to-end comparison and stage-resolved diagnosis in a setting where the agent must construct its own working criterion of relevance.

Primary area:
datasets and benchmarks
Keywords:
Agentic Scientific Search, Open-Corpus Literature Retrieval, Hindsight Supervision

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ukupOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes INSPIRE, a benchmark designed to evaluate scientific literature search agents in an open-ended setting where the final solution and relevance criteria are hidden from the agent. The benchmark formulates literature search as a three-stage process consisting of resource exposure, selection, and ranking, and introduces stage-wise metrics to diagnose different sources of retrieval failure.
The authors construct the benchmark by collecting future computer science target papers and using their pre-cutoff cited antecedents as hidden relevance labels. Experiments on 476 evaluation targets compare several LLM-based search agents under a shared protocol, and a hindsight-SFT study demonstrates that benchmark-derived trajectories can improve search performance. Overall, the paper provides a systematic evaluation framework for scientific search agents, while the validity of using realized citation lineages as a proxy for scientific relevance remains an important consideration.

Critical strengths
  1. The paper proposes a carefully designed benchmark setting for an important but underexplored problem. The proposed INSPIRE benchmark provides a structured evaluation environment for scientific literature search agents and highlights the difficulty of searching for relevant prior work when the final solution and relevance criteria are hidden. This is a meaningful direction for evaluating AI systems for scientific discovery.
  2. The paper provides a valuable diagnostic perspective on search failures. The exposure-selection-ranking decomposition offers a more interpretable analysis than a single retrieval metric. This framework helps distinguish whether failures originate from missing relevant resources, inappropriate selection, or ineffective ranking.
Critical weaknesses
  1. The benchmark may not faithfully measure scientific literature search ability. The central evaluation assumption of INSPIRE is that the realized citation lineage of a future target paper provides an appropriate ground truth for evaluating scientific search agents. However, citation behavior does not necessarily capture all scientifically useful prior work. Important uncited papers, alternative solution paths, or interdisciplinary connections may provide valuable insights but receive no benchmark credit. Therefore, the proposed benchmark may measure the ability to recover future citation antecedents rather than the broader capability of scientific discovery or literature exploration. Although the authors acknowledge this limitation in Sec. 3.1 and Sec. 6, its impact on the validity of the benchmark conclusions remains substantial.
  2. The construction of hidden relevance labels relies heavily on automated judging and is insufficiently validated. The benchmark depends on LLM-based grading of cited references into relevance tiers and roles (e.g., T1/T2/T3 and conceptual/infrastructural categories). While the paper describes the grading pipeline and sensitivity analyses, there is limited evidence demonstrating that these automatically generated labels consistently agree with expert assessments across the benchmark. Since these labels directly determine the evaluation scores, potential annotation bias or judge-model inconsistency could substantially affect the reported comparisons between agents.
Questions for authors
  1. How do the authors justify that realized citation lineages are an adequate proxy for scientific literature search quality? The benchmark uses graded antecedents from future target papers as hidden relevance labels. However, useful prior work may not always be cited by the target paper, and alternative research directions may also provide valuable information. Could the authors provide additional validation (e.g., expert evaluation, comparison with independent relevance judgments, or analysis of uncited but potentially relevant works) to demonstrate that INSPIRE measures broader scientific search ability rather than citation-lineage recovery?
  2. How reliable are the LLM-generated relevance labels used for benchmark evaluation? The benchmark relies on judge models to assign citation roles and relevance tiers (T1/T2/T3). Could the authors provide more details or evidence regarding agreement between these automated labels and independent human expert judgments? Since these labels directly determine nDCG and stage-wise metrics, understanding their reliability is important for interpreting the benchmark results.
  3. To what extent do the reported results generalize beyond computer science literature? The current benchmark focuses on computer-science targets. Could the authors clarify whether they expect the proposed evaluation framework to transfer to other scientific domains with different citation practices and knowledge structures? Have any cross-domain validation experiments been considered or conducted?
1 comment

We thank the reviewer for the thoughtful review. We conducted additional human relevance evaluations, annotation studies, and cross-domain experiments to address the three questions.

Q1 / W1: Do citation-derived antecedents reflect scientific literature-search relevance?

We conducted a human relevance assessment of 350 candidates across 50 research problems, including 100 cited antecedents and 250 uncited agent recommendations. Two researchers assessed candidates using four relevance levels: directly actionable (3), substantively relevant (2), background (1), and irrelevant (0).

CandidateNScore 3Score 2Score 1Score 0
T3450.0%50.0%0%0%
T2316.5%61.3%32.3%0%
T1650%6.2%93.8%0%
Uncited2500%1.2%40.4%58.4%

Human judgments closely reflect the intended tier semantics. All sampled T3 references and 67.7% of T2 references were rated substantively relevant or actionable, while 93.8% of T1 references were classified as background, consistent with their intended role.

Among 250 uncited recommendations, only 3 (1.2%) received substantive-relevance scores, with none classified as directly actionable. These findings provide empirical support for the scientific relevance captured by citation-derived labels, with few high-relevance omissions in the audited recommendation pool.

We further recomputed nDCG@10 using human relevance judgments on the 50-problem subset:

ModelOriginal nDCGHuman nDCG
Claude Opus 50.22860.2117
GPT-5.6-Sol0.17570.1874
Claude Sonnet 50.17090.1815
Gemini 3.5 Flash0.12650.1431
Gemini 3.1 Pro0.07420.1069
Qwen Base0.06250.0700

All six model rankings remain unchanged under human reassessment, demonstrating the stability of INSPIRE's system comparisons across automated and human relevance judgments.

These results establish an empirical connection between citation-derived antecedents and independently assessed scientific relevance. We will incorporate the human evaluation and rescoring analysis into the revised paper.

Q2 / W2: How reliable are the LLM-generated relevance tiers?

INSPIRE uses three independent LLM judges to assign relevance tiers and citation roles, with consensus annotations recorded during construction. We further evaluated annotation reliability using two human researchers on 200 citation–target pairs.

Agreement metricValue
Human–Human Exact Agreement79.5%
Human–LLM Exact Agreement72.5%
Human–Human Weighted Kappa0.754
Human–LLM Weighted Kappa0.702
Tier Macro-F10.766

The results demonstrate substantial agreement between automated relevance grading and human assessments. Human–LLM weighted kappa reaches 0.702, approaching the human–human agreement of 0.754. Tier Macro-F1 reaches 0.766, further supporting the reliability of the automatically assigned relevance categories.

These results provide independent human support for INSPIRE's graded relevance labels. Combined with the unchanged system rankings under human rescoring in Q1, they offer complementary evidence for both annotation reliability and evaluation robustness.

We will add the human annotation protocol, agreement statistics, and human-rescored evaluation to the revised paper.

Q3: Does INSPIRE generalize beyond computer science?

We extended INSPIRE to lattice quantum chromodynamics (Lattice QCD), constructing 100 research problems in physics and evaluating the same six search agents.

ModelnDCG@10Recall@10AnyHit@10≥3 hits
Claude Opus 50.2530.2580.7000.210
Claude Sonnet 50.1630.1490.5400.110
GPT-5.6-Sol0.1680.1410.4900.080
Gemini 3.5 Flash0.1320.1240.4700.050
Gemini 3.1 Pro0.1120.0920.3500.030
Qwen Base0.0790.0740.2800.020

The results demonstrate INSPIRE's applicability beyond computer science and its ability to differentiate scientific-search agents in a distinct research domain. Claude Opus 5 leads with 0.253 nDCG@10 and 0.700 AnyHit@10.

The multi-hit results further reveal the challenge of assembling complementary research antecedents. Claude Opus 5 achieves a 0.210 three-hit rate, compared with 0.700 AnyHit@10, reproducing the distinction between recovering individual references and constructing broader evidence portfolios.

This experiment demonstrates the portability of INSPIRE's construction and evaluation framework to physics. The exposure–selection–ranking formulation provides consistent diagnostics of candidate discovery, retention, and ordering across research domains.

We will incorporate the 100-target Lattice QCD benchmark, its construction protocol, and the complete cross-domain evaluation into the revised paper.

Reviewer_EpzhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

INSPIRE evaluates literature-search agents by turning published papers into earlier-stage research briefs and asking agents to retrieve useful prior work. It scores the ranked results against references used by the eventual target paper, and it breaks performance down into exposure, selection, and ranking. The setup is interesting, and the three-stage analysis is easy to follow. My central reservation is that recovering one future paper's citation choices is narrower than finding useful literature for an open research problem.

Critical strengths

The paper studies a genuinely challenging search setting: researchers often begin with a problem, not a known target paper. The evaluation includes 476 test targets, multiple models, and a consistent tool interface. The Hindsight SFT result is also worth reporting: the Qwen model improves from 0.082 to 0.166 nDCG@10, suggesting that the benchmark supplies a learnable training signal. The exposure/selection/ranking breakdown offers a useful descriptive view of where scored references disappear from the pipeline.

Critical weaknesses

1.

The gold references capture one realized research path, not all useful paths. A problem may have several reasonable approaches, each drawing on different prior work. INSPIRE rewards the references cited by the particular future paper used to create the brief. A system could recommend strong papers supporting another approach and still score poorly. This is not merely an edge case; it is a mismatch between the broad task framing and the narrower automatic target. The four-year citation window, the limit of ten gold papers, and LLM-based importance labels narrow that target further. Without independent expert judgments of uncited but useful recommendations, it is difficult to know whether the nDCG ranking reflects actual research usefulness. The authors could either validate this connection or frame the task more explicitly as retrospective citation-lineage recovery.

Temporal filtering applies to the search tool, not necessarily to the models. Restricting retrieval to material published before the target paper prevents a straightforward tool-based leak, but evaluated models may already know about the target work or its method. Removing titles and methods from the brief helps, yet the audit in Appendix A.4 illustrates that a model may infer the likely mechanism without identifying the precise paper. I would like to see a clearer analysis of this risk before treating the setting as temporally isolated.

2.

The stage analysis is informative, but its bottleneck claims go too far. Exposure loss is measured against an ideal score of one, not against what the available backend and search budget can actually retrieve. Also, a paper appearing somewhere in a tool response counts as exposed even if the agent never meaningfully reads it. The decomposition therefore describes where score is lost under the chosen labels; it does not by itself show that search strategy is the dominant cause or the largest practical improvement opportunity. Basic one-shot and multi-query retrieval baselines would help put the agent results in context.

INSPIRE offers an appealing testbed, and the training result is encouraging. But the paper needs to show that its scoring target is a trustworthy stand-in for the research-search ability it claims to measure. I lean Weak rejection (2/4).

Questions for authors
  1. How well does citation-based nDCG agree with expert judgments when agents recommend useful papers not cited by the target paper? Are the rankings stable under a broader relevance definition?
  2. How much do the results change when relaxing the four-year window and ten-reference cap, and when separating targets by possible future-knowledge leakage?
  3. Can exposure be measured against a realistic backend/budget ceiling and distinguish results returned by the tool from papers actually inspected by the agent?
5 comments

We thank the reviewer for the thoughtful assessment. We conducted additional human evaluations, sensitivity analyses, retrieval baselines, and an exposure intervention to address the main concerns.

Q1 / W1: Does citation-based nDCG agree with expert relevance judgments?

We assessed 350 candidates from 50 research problems, including 100 cited references and 250 uncited agent recommendations. Two researchers assigned relevance scores from 3 (directly actionable) to 0 (irrelevant).

CandidateNScore 3Score 2Score 1Score 0
T3450.0%50.0%0%0%
T2316.5%61.3%32.3%0%
T1650%6.2%93.8%0%
Uncited2500%1.2%40.4%58.4%

Human judgments reflect the intended tier semantics: 67.7% of T2 references were substantively relevant or directly actionable, while 93.8% of T1 references were classified as background. Among 250 uncited recommendations, only 3 (1.2%) received substantive-relevance scores, with none classified as directly actionable. These findings demonstrate that citation-derived labels capture meaningful distinctions in scientific relevance, with few high-relevance omissions in the audited recommendation pool.

We further recomputed nDCG@10 using human relevance judgments:

ModelCitation nDCGHuman nDCG
Claude Opus 50.22860.2117
GPT-5.6-Sol0.17570.1874
Claude Sonnet 50.17090.1815
Gemini 3.5 Flash0.12650.1431
Gemini 3.1 Pro0.07420.1069
Qwen Base0.06250.0700

All six model rankings remain unchanged under human reassessment, demonstrating the stability of system comparisons under independent relevance judgments. This provides empirical support for citation-derived antecedents as an informative evaluation target for scientific literature search.

We will incorporate the human relevance assessment and rescoring analysis into the paper, further clarifying the connection between realized research lineages and scientific usefulness.

Q2 / W1: How sensitive are results to the citation window, reference cap, and future knowledge?

We evaluated sensitivity to the four-year citation window and conducted no-tool evaluations across all six models.

ModelOriginalNo 4-year limitNo tools
Claude Opus 50.2840.2700.023
Claude Sonnet 50.1890.1900.012
GPT-5.6-Sol0.1640.1740.014
Gemini 3.5 Flash0.1510.1460.012
Gemini 3.1 Pro0.1100.1100.008
Qwen Base0.0820.0780.005

Removing the four-year citation window preserves the ordering of all six models, demonstrating stable system comparisons under expanded antecedent coverage. This ablation isolates the effect of citation age while maintaining the remaining construction settings.

The no-tool evaluation reveals substantial performance reductions across all models. Claude Opus 5 decreases from 0.284 to 0.023 nDCG@10, with similarly pronounced reductions across other models. These results demonstrate the substantial contribution of external literature retrieval to successful antecedent recovery.

INSPIRE's target-specific cutoff controls the literature accessible during search, while the no-tool experiments quantify the additional information gained through retrieval. We will incorporate both analyses and clarify the temporal filtering and evaluation protocols in the revised paper.

Q3 / W2: Is exposure an actionable bottleneck, and how does candidate availability affect downstream performance?

INSPIRE's stage-wise decomposition quantifies the ranking potential after exposure, selection, and ranking. The exposure gap measures the difference between ideal antecedent recovery and the evidence surfaced during search.

To examine the impact of candidate availability, we conducted a gold-conditioned retrieval intervention. Claude Opus 5 received antecedent identities and retrieved them through the arXiv API. We then evaluated downstream selection and ranking with Gemini and Claude.

Search / DownstreamExposedSelectedFinal nDCG
Gemini / Gemini0.1840.1480.110
Gold-guided / Gemini0.8690.2970.187
Gemini / Claude0.1840.1580.130
Gold-guided / Claude0.8690.5900.406

The intervention increases exposed gain from 0.184 to 0.869, demonstrating substantial antecedent retrievability. Expanded exposure improves final nDCG under both downstream models.

Intervention effectGeminiClaude
Gain from expanded exposure+0.077+0.276

The downstream comparison reveals a further interaction:

Downstream modelOriginal exposureExpanded exposure
Gemini0.1100.187
Claude0.1300.406
Gain from stronger downstream+0.020+0.219

The benefit of stronger downstream processing increases from 0.020 to 0.219 with expanded exposure, yielding an interaction effect of +0.199. This demonstrates how candidate availability shapes the effectiveness of selection and ranking.

These findings support INSPIRE's central insight that exposure, selection, and ranking are coupled stages. They also demonstrate how stage-wise analysis identifies opportunities to improve candidate discovery and evidence utilization.

We will add the intervention results to the paper, clarify the exposure gap relative to ideal recovery, and distinguish tool-returned exposure from explicit paper inspection in the metric definitions.

W2: How do simple retrieval baselines compare with agentic search?

We additionally evaluated three conventional retrieval pipelines to establish performance references for open-problem antecedent discovery.

MethodnDCG@10Recall@10MRRAnyHit@10
BM250.1030.0990.2010.287
SPECTER20.0810.0870.1470.287
BM25 + MiniLM0.0930.0960.1690.308
Claude Opus 50.2840.2150.4760.723

Claude Opus 5 achieves 2.76 times BM25's nDCG@10, with substantially higher recall and antecedent-hit rates. Meanwhile, BM25 outperforms Qwen Base (0.103 versus 0.082), demonstrating that INSPIRE effectively differentiates scientific-search capabilities across conventional retrieval systems and interactive agents.

These results provide concrete reference points for benchmark difficulty and highlight the value of effective multi-stage literature discovery. We will incorporate the retrieval baselines and their evaluation protocols into the revised paper.

Thank you again for your thoughtful review and constructive feedback. We have provided detailed responses and additional experiments to address your concerns. As the discussion period is approaching its end, we would greatly appreciate it if you could take another look at our rebuttal and share any further thoughts, if time permits. We would be happy to promptly address any remaining questions.

Reviewer_PBvGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces INSPIRE, a benchmark for evaluating scientific literature search agents given open research problems without access to the eventual solutions. The benchmark is constructed retrospectively from published research papers, with solution-redacted problem briefs and target-specific temporal cutoffs. Agents search a date-filtered literature corpus and return ranked lists of potentially useful prior papers, which are evaluated against graded antecedents cited by the corresponding future publications.

A key contribution is a three-stage evaluation framework that decomposes search performance into resource exposure, selection, and ranking. The authors evaluate six language models on 476 research problems and find that even the strongest model achieves only 0.284 nDCG@10, with resource exposure accounting for the largest observed performance deficit. They also demonstrate that hindsight-supervised fine-tuning improves a Qwen model from 0.082 to 0.166 nDCG@10.

Critical strengths
  1. The benchmark setting is interesting and practically relevant.

  2. The decomposition into resource exposure, selection, and ranking provides more informative diagnostics than end-to-end retrieval metrics alone. In particular, distinguishing failures to discover relevant papers from failures to retain or rank already discovered papers can help guide the development of future research agents.

  3. The experimental analysis is comprehensive.

Critical weaknesses
  1. The validity and completeness of ground-truth relevance labels are questionable. The benchmark defines relevant literature based on references cited by a particular future paper, which may not reflect all useful literature for solving an open research problem. Multiple valid solution directions may exist, and highly relevant but uncited papers receive zero credit. This issue is further amplified by the construction choices: only cited references are considered, references older than four years are filtered out, and the gold set is capped at ten papers.

Consequently, the benchmark may primarily measure reconstruction of a specific paper's citation lineage rather than scientific literature-search quality. Although the authors acknowledge this limitation, I believe it fundamentally affects the benchmark's validity. A human evaluation of the gold labels and highly ranked but uncredited agent outputs would help assess the severity of this issue.

  1. The benchmark uses a cutoff three months before each target paper's publication, ensuring that the agent's search tools cannot retrieve the target or later literature. However, this does not guarantee that the model itself lacks knowledge of the target paper. The evaluation targets were published in 2025–2026, while several evaluated models were released later. These models may have encountered the target papers, their abstracts, or their citations during pretraining or post-training. This is particularly concerning because the problem briefs are generated retrospectively from the target papers' titles and abstracts. Even after removing explicit solutions, a sufficiently capable model may recognize the underlying research contribution or infer the intended mechanism from the problem formulation.

  2. The evaluation lacks important retrieval baselines, making the difficulty and significance of the results unclear. Table 2 compares six language models acting as search agents. However, I find the absence of conventional retrieval and simpler search baselines a significant limitation for a benchmark paper.

Questions for authors
  1. Have the authors conducted any human evaluation of the gold relevance labels? In particular, what fraction of highly ranked but uncited agent-retrieved papers would human experts consider genuinely useful for solving the research problem?

  2. How do the authors distinguish successful scientific search from recognition or memorization of the target publication? Have they tested whether agents can identify the hidden target or its antecedents without using search tools?

  3. How do standard BM25, dense retrieval, and simple retrieve-then-rerank pipelines perform on INSPIRE? How sensitive are the results to the search backend and the number of candidates returned per query?

4 comments

We thank the reviewer for the thoughtful reviews. We conducted additional human relevance assessments, robustness analyses, conventional retrieval baselines, and controlled retrieval experiments to address the three questions.

Q1 / W1: How valid are citation-based relevance labels, and how often are uncited recommendations genuinely useful?

INSPIRE evaluates the recovery of graded antecedents from realized research lineages. To examine their correspondence with scientific usefulness, we conducted a human relevance assessment of 350 candidate papers across 50 research problems, including 100 cited antecedents and 250 uncited agent recommendations. Two researchers assigned relevance scores using four levels: directly actionable (3), substantively relevant (2), background (1), and irrelevant (0).

CandidateNScore 3Score 2Score 1Score 0
T3450.0%50.0%0%0%
T2316.5%61.3%32.3%0%
T1650%6.2%93.8%0%
Uncited2500%1.2%40.4%58.4%

Human judgments closely reflect the intended tier semantics. All sampled T3 references and 67.7% of T2 references were rated substantively relevant or directly actionable, while 93.8% of T1 references were classified as background, consistent with their intended role.

Among 250 uncited agent recommendations, only 3 (1.2%) received substantive-relevance scores, with none classified as directly actionable. These results demonstrate that citation-derived labels capture meaningful distinctions in scientific relevance, with few high-relevance omissions in the audited recommendation pool.

We further examined whether human relevance judgments change the relative performance of scientific-search agents. We recomputed nDCG@10 using human assessments on the 50-problem subset.

ModelOriginal nDCGHuman nDCG
Claude Opus 50.22860.2117
GPT-5.6-Sol0.17570.1874
Claude Sonnet 50.17090.1815
Gemini 3.5 Flash0.12650.1431
Gemini 3.1 Pro0.07420.1069
Qwen Base0.06250.0700

All six model rankings remain unchanged under human reassessment. This demonstrates the stability of INSPIRE's system comparisons across automated and independent human relevance judgments, providing empirical support for citation-derived antecedents as an informative evaluation target.

We additionally evaluated sensitivity to the four-year citation window:

ModelOriginalNo 4-year limit
Claude Opus 50.2840.270
Claude Sonnet 50.1890.190
GPT-5.6-Sol0.1640.174
Gemini 3.5 Flash0.1510.146
Gemini 3.1 Pro0.1100.110
Qwen Base0.0820.078

Removing the four-year window preserves the ordering of all six models, demonstrating stable comparisons under expanded antecedent coverage. The human evaluation and citation-window analysis provide complementary evidence for the robustness of the benchmark's relevance labels and evaluation results.

We will incorporate the human relevance assessment, human-rescored evaluation, and citation-window ablation into the revised paper, with a clearer description of the benchmark's realized-lineage evaluation objective.

Q2 / W2: Can agents identify the hidden target or recover its antecedents without using search tools?

INSPIRE employs a target-specific temporal cutoff, ensuring that the search interface exposes literature available before the target publication. The benchmark also includes a no-tool target-identifiability screening procedure during problem-brief construction.

To further examine the contribution of external literature retrieval, we evaluated all six models without search tools while preserving their original research briefs and evaluation targets.

ModelWith toolsNo tools
Claude Opus 50.2840.023
Claude Sonnet 50.1890.012
GPT-5.6-Sol0.1640.014
Gemini 3.5 Flash0.1510.012
Gemini 3.1 Pro0.1100.008
Qwen Base0.0820.005

All evaluated models exhibit substantial performance reductions without retrieval. Claude Opus 5 decreases from 0.284 to 0.023 nDCG@10, a reduction of 91.9%, with similarly pronounced reductions across the remaining models.

These results demonstrate that tool-mediated literature discovery makes a substantial contribution to antecedent recovery. The combination of target-identifiability screening, date-filtered retrieval, and no-tool evaluation provides complementary evidence about the information sources used during search.

We will incorporate the no-tool results into the revised paper and expand the description of the target-identifiability audit, temporal filtering, and retrieval information boundary.

Q3 / W3: How do conventional retrieval baselines perform?

We conducted additional experiments examining conventional retrieval performance, antecedent retrievability through the arXiv API, and the interaction between candidate exposure and downstream selection.

First, we evaluated three conventional retrieval pipelines: BM25, SPECTER2 dense retrieval, and BM25 followed by MiniLM reranking.

MethodnDCG@10Recall@10MRRAnyHit@10
BM250.1030.0990.2010.287
SPECTER20.0810.0870.1470.287
BM25 + MiniLM0.0930.0960.1690.308
Claude Opus 50.2840.2150.4760.723

Claude Opus 5 achieves 2.76 times BM25's nDCG@10, with substantially higher recall and antecedent-hit rates. Meanwhile, BM25 outperforms Qwen Base (0.103 versus 0.082), demonstrating that INSPIRE meaningfully differentiates search effectiveness across conventional retrieval methods and interactive agents.

Q3 / W3: How do the search backend and per-query candidate counts affect retrieval?

To further examine the retrieval capability of the search backend, we conducted a controlled gold-conditioned retrieval experiment. Claude Opus 5 was provided with the identities of target antecedent papers and instructed to locate them through the arXiv search API. We then evaluated how Gemini 3.1 Pro Preview and Claude Opus 5 selected and ranked papers from the resulting candidate sets.

Search / DownstreamExposed GainSelected GainFinal nDCG
Gemini / Gemini0.1840.1480.110
Gold-guided Opus / Gemini0.8690.2970.187
Gemini / Opus0.1840.1580.130
Gold-guided Opus / Opus0.8690.5900.406

With target identities guiding retrieval, exposed gain increases from 0.184 to 0.869. This demonstrates that the arXiv search interface can retrieve a substantially larger portion of the graded antecedents when queries are directed toward the relevant papers.

The expanded candidate sets also yield consistent improvements in final performance across downstream models.

Intervention effectGemini downstreamOpus downstream
Gain from expanded exposure+0.077+0.276

The downstream comparison reveals an additional interaction between candidate availability and evidence utilization.

Downstream modelOriginal exposureExpanded exposure
Gemini0.1100.187
Opus0.1300.406
Gain from stronger downstream+0.020+0.219

The benefit of stronger downstream processing increases from 0.020 to 0.219 under expanded exposure, yielding an interaction effect of +0.199. This demonstrates that the effectiveness of selection and ranking depends strongly on the relevant evidence surfaced during search.

These results also provide insight into the search backend. The substantial increase in exposed gain under targeted queries demonstrates the retrievability of graded antecedents through the existing arXiv interface and highlights the importance of effective query formulation and candidate discovery.

Regarding the number of candidates returned per query, INSPIRE uses an agent-controlled search protocol. The LLM dynamically determines the number of candidate papers requested during each search interaction as part of its tool-use decisions. The requested candidate count therefore forms part of the agent's adaptive search policy. All evaluated agents operate with the same search interface, temporal filtering, and interaction budget.

This design allows agents to adapt their search breadth to the evolving research problem. The choice of how many papers to request is evaluated through the resulting search trajectory and final retrieval quality, alongside query formulation, exploration, selection, and ranking. The per-query requested candidate count is an agent decision, not a separately tuned benchmark hyperparameter.

We will incorporate the conventional retrieval baselines and gold-conditioned retrieval experiment into the revised paper. We will also clarify the arXiv search interface, agent-controlled candidate counts, shared search budget, and stage-wise exposure measurements to make the evaluation protocol and retrieval behavior more explicit.

Thanks for your response. I would increase the score to 3.

Thank you very much for raising your rating and for your positive assessment of our additional experiments and clarifications. We sincerely appreciate your constructive feedback and support.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Rebuttal Summary

We thank area chair and all reviewers for their constructive feedback. We have conducted additional experiments and analyses to address the key questions raised during the review process.

  1. Ground-truth validity: We conducted independent human relevance assessments of cited antecedents and uncited agent recommendations. The results confirm that citation-derived labels capture meaningful scientific relevance, and human-based rescoring preserves the original model rankings, supporting the validity of INSPIRE's evaluation.

  2. Annotation reliability and robustness: Human–LLM agreement analyses demonstrate the reliability of automatically assigned relevance tiers. Additional sensitivity experiments confirm that system comparisons remain stable when relaxing the citation-age restriction.

  3. Temporal leakage: We conducted no-tool evaluations across all models, observing substantial performance degradation without external retrieval. These findings, together with target-identifiability screening and temporal filtering, demonstrate the importance of tool-mediated literature discovery.

  4. Retrieval baselines: We introduced conventional sparse retrieval, dense retrieval, and reranking baselines, providing additional reference points for evaluating agentic scientific search and highlighting meaningful performance differences across retrieval approaches.

  5. Stage-wise analysis: Controlled retrieval interventions confirm that relevant antecedents are accessible through the search backend and demonstrate how improved resource exposure enhances downstream selection and ranking. The results further validate the coupled nature of the three stages and the diagnostic value of INSPIRE.

  6. Cross-domain generalization: We extended INSPIRE to lattice quantum chromodynamics, demonstrating the applicability of our benchmark framework to a distinct scientific domain and confirming consistent patterns in agent performance and multi-antecedent retrieval.

We will incorporate these additional experiments, analyses, and methodological clarifications into the revised manuscript to strengthen the presentation of INSPIRE's contributions and empirical findings.

AuthorsOpenReview ↗

Update on Reviewer Feedback and Follow-up Clarifications

We sincerely thank all reviewers for their continued engagement and constructive feedback. We would like to summarize their responses to our rebuttal and the additional clarifications we have provided.

  1. Reviewer PBvG: We are grateful that the reviewer acknowledged our additional experiments and raised their rating to 3. Our rebuttal provided further evidence for the validity of citation-derived relevance labels, the importance of tool-mediated retrieval, and the value of stage-wise evaluation.

  2. Reviewer Epzh: The reviewer acknowledged that our human reassessment strengthens citation-based scoring and that the stable system rankings are encouraging. Regarding the remaining concern about alternative research paths, we further clarified that our human evaluation explicitly examined uncited papers recommended by agents as potentially useful. Very few were judged substantively relevant, and none were directly actionable. Moreover, giving these papers relevance credit preserves the original model rankings. These findings provide empirical support for the relevance coverage and robustness of INSPIRE.

  3. Reviewer ukup: The reviewer recognized the encouraging human evaluation results and the improvements in annotation reliability and cross-domain applicability. To further address concerns about potentially missing valuable literature, we clarified the empirical support for citation-derived labels and the practical infeasibility of exhaustively annotating an open scientific corpus. We also explained the motivation for focusing on recent, directly contributing antecedents in fast-evolving fields such as computer science. Our additional experiment removing the four-year citation restriction confirms that model performance and relative rankings remain stable.

We appreciate the reviewers' valuable suggestions and the opportunity to further clarify our contributions. We will incorporate these additional findings and clarifications into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.