- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces LINEAGE, a benchmark for scientific literature retrieval based on empirical baseline relationships between papers. The main idea is to go beyond topical relevance and citation links by identifying which prior methods are explicitly used as baselines in experiments. To build the benchmark, the authors propose LINEAGE-Constructor, an automated pipeline that collects papers, extracts baseline relations using LLMs, and constructs calibrated lineage graphs. The resulting benchmark contains 121K papers, 165K lineage edges, and 80K retrieval queries. The authors evaluate 19 retrieval methods and find that existing systems struggle to retrieve lineage-related papers, especially those connected through multiple hops.
Critical strengths
- The paper addresses a problem of practical importance in scientific literature retrieval, particularly for research agents and automated literature reviews.
- The benchmark is relatively large-scale, and the authors provide a systematic data construction pipeline with human-annotated validation to assess extraction quality.
- The experiments cover a diverse set of retrieval methods and analyze performance across different query types and lineage distances.
Critical weaknesses
1. Insufficient justification for baseline-based multi-hop ground truth.
The paper assumes that empirical baseline relations provide meaningful ground truth for multi-hop scientific literature retrieval, but this assumption is not sufficiently validated. Baseline selection is inherently subjective and can be influenced by community conventions, computational constraints, code availability,and selective reporting of experimental comparisons. While the authors acknowledge the connectivity issues caused by widely used baselines and introduce graph calibration, these procedures do not establish the semantic validity of multi-hop relationships. In particular, if paper B compares against A and paper C compares against B, it does not necessarily follow that A is a meaningful retrieval target for C. The paper validates the acuracy of extracting direct baseline relations, but provides no independent human evaluation of the relevance of multi-hop lineage targets. This raises fundamental concerns about the validity of the benchmark's ground truth.
2. Limited novelty relative to closely related work.
The identification of experimental baselines and the construction of comparison-aware relationships between scientific papers have been studied extensively. For example, FeRoSA (Chakraborty et al., 2016) explored comparison-aware scientific paper recommendation, while Bedi et al. (2022) explicitly studied baseline identification and introduced a human-annotated dataset. More importantly, MasterSet (Ratul et al., 2026) already introduced a large-scale scientific literature retrieval benchmark incorporating experimental baseline annotations. Intern-Atlas (Wu et al., 2026) also constructed a large-scale scientific method graph with comparison and methodological lineage relations. These closely related works are not adequately discussed. Although LINEAGE introduces multi-hop retrieval and additional evaluation metrics, its main contributions appear largely incremental, and the paper does not sufficiently establish its novelty relative to existing research.
3. Misalignment between retrieval queries and the evaluation objective.
The retrieval queries primarily describe a paper's topic, background, or methodology, while the ground truth consists of papers connected through empirical baseline relationships. More importantly, the LLM Search Prompt in Appendix I.3 explicitly asks models to retrieve relevant methodology papers, without instructing them to identify experimental baselines or trace predecessor/successor relationships. Consequently, the evaluated systems are not explicitly asked to perform the task that the benchmark measures. A retrieval system may return scientifically relevant papers but receive low scores simply because these papers are not connected through the baseline graph. It remains unclear whether LLM-based research agents would perform substantially better if explicitly instructed to identify baseline relationships and trace their corresponding lineage. Without such a task-aware prompting baseline, the reported poor performance does not convincingly establish the limitations of existing research agents on empirical lineage retrieval.
Questions for authors
-
Have the authors conducted any human evaluation to verify that multi-hop baseline-connected papers are actually meaningful retrieval targets? How does their relevance change with increasing hop distance?
-
How does LINEAGE differ from closely related work such as Bedi et al. (2022), MasterSet (2026), and Intern-Atlas (2026), beyond dataset scale and the introduction of multi-hop evaluation?
-
Have the authors evaluated LLM-based agents with explicit instructions to retrieve baseline papers and trace predecessor/successor relationships? If not, how can the current results support conclusions about these agents' ability to perform lineage retrieval?
3 comments
Response to Reviewer's Questions 2&3
Q2. How does LINEAGE differ from closely related work such as Bedi et al. (2022), MasterSet (2026), and Intern-Atlas (2026), beyond dataset scale and the introduction of multi-hop evaluation?
Response: We appreciate the reviewer's effort to identify a wide range of potentially related works. Since the full titles and publication venues were not provided in the review, could the reviewer clarify whether the intended papers are the following:
- FeRoSA — FeRoSA: A Faceted Recommendation System for Scientific Articles (PAKDD 2016).
- Bedi et al. — Why Did You Not Compare With That? Identifying Papers for Use as Baselines (ECIR 2022).
- MasterSet — MasterSet: A Large-Scale Benchmark for Must-Cite Citation Recommendation in the AI/ML Literature (SDM 2026).
- Intern-Atlas — Intern-Atlas: A Methodological Evolution Graph as Research Infrastructure for AI Scientists (NeurIPS 2026).
We emphasize that LINEAGE introduces a distinct benchmark for empirical lineage retrieval, integrating evidence-grounded baseline relation extraction, multi-hop ancestor/successor retrieval ground truth, multi-facet queries, and structural evaluation metrics.
These works differ fundamentally in their objectives.
- FeRoSA proposes a paper recommendation algorithm that categorizes results by citation context and evaluates the quality of recommendations through expert judgments. LINEAGE instead provides a benchmark for systematically evaluating diverse retrievers against evidence-grounded, multi-hop empirical lineage ground truth.
- Bedi et al. study direct baseline identification as a classification task, whereas LINEAGE benchmarks empirical lineage retrieval, a critical capability for current automated research workflows.
- MasterSet benchmarks 1-hop must-cite retrieval over directly cited papers, while LINEAGE evaluates multi-hop ancestor and successor retrieval with structural metrics.
- Intern-Atlas constructs a methodological evolution graph as input to downstream research tasks, including idea evaluation and generation. In contrast, LINEAGE uses empirical baseline graphs as retrieval ground truth, without exposing them to retrievers. Moreover, Intern-Atlas excludes comparison relations from its methodological lineage.
Thus, these works address complementary problems and do not cover LINEAGE's core benchmark task and evaluation framework.
Besides, MasterSet and Intern-Atlas are contemporaneous works. As documented in our anonymous repository, we publicly released LINEAGE's benchmark dataset and code on May 6, 2026, close to the release date of these two papers. We will clarify these distinctions in the related work section in the revised version.
Q3. Have the authors evaluated LLM-based agents with explicit instructions to retrieve baseline papers and trace predecessor/successor relationships? If not, how can the current results support conclusions about these agents' ability to perform lineage retrieval?
LINEAGE evaluates the effectiveness of existing retrieval methods in recovering empirical lineage papers. These retrieval methods rely on lexical or semantic matching, which is designed for query-based retrieval rather than instruction-driven actions. Therefore, introducing explicit lineage-tracing instructions would be inconsistent with the retrieval mechanisms under evaluation and would compromise fair comparisons across retrieval paradigms.
Dear Reviewer ySSv,
Thank you for your constructive comments. We have made the following revisions to our manuscript and highlighted them in blue.
- Added a brief discussion of multi-hop lineage utility in Section 4.1 (p. 6) and a new evaluation in Appendix I (pp. 30–31).
- Discussed FeRoSA, Bedi et al. (2022), MasterSet (2026), and Intern-Atlas (2026) in the revised Related Work (Appendix B, p. 15), with corresponding references added to the bibliography.
- Revised the descriptions of agentic retrievers and LLM-based search settings in Appendix C.2 (p. 17). We also clarified the benchmark scope and future research directions for agents in Appendix A (p. 15).
Could you please let us know if our response has addressed your concerns or if you have any further questions?
Best,
Authors
Response to Reviewer's Question 1
We thank the reviewer for the questions and feedback on our evaluation and related work.
Q1. Have the authors conducted any human evaluation to verify that multi-hop baseline-connected papers are actually meaningful retrieval targets? How does their relevance change with increasing hop distance?
Response:
Meaningful. The meaning of multi-hop lineage targets lies in recovering a broader empirical comparison context that supports baseline discovery and positioning of contributions, as motivated in our Introduction. For example, in A → B → C, retrieving A helps researchers understand the empirical comparisons underlying B, which serves as the direct baseline for C.
Change against hops. To further assess whether multi-hop lineage papers are meaningful retrieval targets and how their utility changes with hop distance, we conduct a large-scale LLM-based evaluation. Specifically, we sample query–target pairs across hop distances, with topically matched non-lineage papers as controls. We adopt a blind pairwise evaluation on 1,600 query–target pairs, sampling 400 pairs at each of the 1-, 2-, 3-, and 4+-hop distances. Each target is paired with a non-lineage control matched by title–abstract TF-IDF similarity, publication year, and temporal direction without directed lineage connection to the query. An independent GPT-6 judge assesses their usefulness for baseline discovery and contribution positioning on a three-level scale (0 = not useful, 1 = partially useful, 2 = useful), based on paper content. Lineage labels and paths are hidden, and A/B order is randomized.
The results show that multi-hop targets show higher LLM-assessed retrieval utility than matched controls, demonstrating that the multi-hop baseline-connected papers are actually meaningful retrieval targets.