acceptodds
Under review as a conference paper at ICLR 2027

LINEAGE: A Benchmark for Empirical Lineage Retrieval in AI Scientific Literature

Abstract

Current research agents are moving beyond academic search toward literature synthesis and experiment planning. These workflows increasingly rely on literature retrieval systems, which serve as standalone systems or as components of research agents. However, existing scientific literature retrieval benchmarks mainly evaluate these methods using topical relevance or citation links as supervision, which cannot capture which prior methods are used for empirical comparison. To address this gap, we introduce Lineage, a benchmark for empirical lineage retrieval in AI scientific literature. An empirical lineage relation links a prior paper to a source paper that explicitly uses its method as an empirical comparison baseline. These relations form lineage graphs whose multi-hop neighborhoods provide retrieval ground truth for structural evaluation. To construct these graphs at scale, we develop Lineage-Constructor, which converts heterogeneous paper records and full-paper empirical baseline evidence into calibrated lineage graphs. Built from 121,048 papers across 17 top AI venues from 2018 to 2025, Lineage contains 3,950 graph components, 67,502 paper nodes, 164,948 lineage edges, and 80,000 multi-facet retrieval queries. Experiments with 19 representative retrieval systems show that empirical lineage retrieval remains challenging. Performance drops sharply with lineage distance, while structural metrics reveal that isolated retrieval hits often fail to recover closely connected empirical comparison neighborhoods.

Primary area:
datasets and benchmarks
Keywords:
Scientific Literature Retrieval, Information Retrieval, Deep Research, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ySSvOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces LINEAGE, a benchmark for scientific literature retrieval based on empirical baseline relationships between papers. The main idea is to go beyond topical relevance and citation links by identifying which prior methods are explicitly used as baselines in experiments. To build the benchmark, the authors propose LINEAGE-Constructor, an automated pipeline that collects papers, extracts baseline relations using LLMs, and constructs calibrated lineage graphs. The resulting benchmark contains 121K papers, 165K lineage edges, and 80K retrieval queries. The authors evaluate 19 retrieval methods and find that existing systems struggle to retrieve lineage-related papers, especially those connected through multiple hops.

Critical strengths
  1. The paper addresses a problem of practical importance in scientific literature retrieval, particularly for research agents and automated literature reviews.
  2. The benchmark is relatively large-scale, and the authors provide a systematic data construction pipeline with human-annotated validation to assess extraction quality.
  3. The experiments cover a diverse set of retrieval methods and analyze performance across different query types and lineage distances.
Critical weaknesses

1. Insufficient justification for baseline-based multi-hop ground truth.

The paper assumes that empirical baseline relations provide meaningful ground truth for multi-hop scientific literature retrieval, but this assumption is not sufficiently validated. Baseline selection is inherently subjective and can be influenced by community conventions, computational constraints, code availability,and selective reporting of experimental comparisons. While the authors acknowledge the connectivity issues caused by widely used baselines and introduce graph calibration, these procedures do not establish the semantic validity of multi-hop relationships. In particular, if paper B compares against A and paper C compares against B, it does not necessarily follow that A is a meaningful retrieval target for C. The paper validates the acuracy of extracting direct baseline relations, but provides no independent human evaluation of the relevance of multi-hop lineage targets. This raises fundamental concerns about the validity of the benchmark's ground truth.

2. Limited novelty relative to closely related work.

The identification of experimental baselines and the construction of comparison-aware relationships between scientific papers have been studied extensively. For example, FeRoSA (Chakraborty et al., 2016) explored comparison-aware scientific paper recommendation, while Bedi et al. (2022) explicitly studied baseline identification and introduced a human-annotated dataset. More importantly, MasterSet (Ratul et al., 2026) already introduced a large-scale scientific literature retrieval benchmark incorporating experimental baseline annotations. Intern-Atlas (Wu et al., 2026) also constructed a large-scale scientific method graph with comparison and methodological lineage relations. These closely related works are not adequately discussed. Although LINEAGE introduces multi-hop retrieval and additional evaluation metrics, its main contributions appear largely incremental, and the paper does not sufficiently establish its novelty relative to existing research.

3. Misalignment between retrieval queries and the evaluation objective.

The retrieval queries primarily describe a paper's topic, background, or methodology, while the ground truth consists of papers connected through empirical baseline relationships. More importantly, the LLM Search Prompt in Appendix I.3 explicitly asks models to retrieve relevant methodology papers, without instructing them to identify experimental baselines or trace predecessor/successor relationships. Consequently, the evaluated systems are not explicitly asked to perform the task that the benchmark measures. A retrieval system may return scientifically relevant papers but receive low scores simply because these papers are not connected through the baseline graph. It remains unclear whether LLM-based research agents would perform substantially better if explicitly instructed to identify baseline relationships and trace their corresponding lineage. Without such a task-aware prompting baseline, the reported poor performance does not convincingly establish the limitations of existing research agents on empirical lineage retrieval.

Questions for authors
  1. Have the authors conducted any human evaluation to verify that multi-hop baseline-connected papers are actually meaningful retrieval targets? How does their relevance change with increasing hop distance?

  2. How does LINEAGE differ from closely related work such as Bedi et al. (2022), MasterSet (2026), and Intern-Atlas (2026), beyond dataset scale and the introduction of multi-hop evaluation?

  3. Have the authors evaluated LLM-based agents with explicit instructions to retrieve baseline papers and trace predecessor/successor relationships? If not, how can the current results support conclusions about these agents' ability to perform lineage retrieval?

3 comments
AuthorsOpenReview ↗

Response to Reviewer's Question 1

We thank the reviewer for the questions and feedback on our evaluation and related work.

Q1. Have the authors conducted any human evaluation to verify that multi-hop baseline-connected papers are actually meaningful retrieval targets? How does their relevance change with increasing hop distance?

Response:

  • Meaningful. The meaning of multi-hop lineage targets lies in recovering a broader empirical comparison context that supports baseline discovery and positioning of contributions, as motivated in our Introduction. For example, in A → B → C, retrieving A helps researchers understand the empirical comparisons underlying B, which serves as the direct baseline for C.

  • Change against hops. To further assess whether multi-hop lineage papers are meaningful retrieval targets and how their utility changes with hop distance, we conduct a large-scale LLM-based evaluation. Specifically, we sample query–target pairs across hop distances, with topically matched non-lineage papers as controls. We adopt a blind pairwise evaluation on 1,600 query–target pairs, sampling 400 pairs at each of the 1-, 2-, 3-, and 4+-hop distances. Each target is paired with a non-lineage control matched by title–abstract TF-IDF similarity, publication year, and temporal direction without directed lineage connection to the query. An independent GPT-6 judge assesses their usefulness for baseline discovery and contribution positioning on a three-level scale (0 = not useful, 1 = partially useful, 2 = useful), based on paper content. Lineage labels and paths are hidden, and A/B order is randomized.

Hop distanceLineage Useful RateNon-Lineage Useful RatePairwise Win Rate
179.50%19.25%85.00%
256.25%14.25%81.00%
342.25%9.00%79.25%
4+31.75%7.25%75.50%

The results show that multi-hop targets show higher LLM-assessed retrieval utility than matched controls, demonstrating that the multi-hop baseline-connected papers are actually meaningful retrieval targets.

AuthorsOpenReview ↗

Response to Reviewer's Questions 2&3


Q2. How does LINEAGE differ from closely related work such as Bedi et al. (2022), MasterSet (2026), and Intern-Atlas (2026), beyond dataset scale and the introduction of multi-hop evaluation?

Response: We appreciate the reviewer's effort to identify a wide range of potentially related works. Since the full titles and publication venues were not provided in the review, could the reviewer clarify whether the intended papers are the following:

  1. FeRoSA — FeRoSA: A Faceted Recommendation System for Scientific Articles (PAKDD 2016).
  2. Bedi et al. — Why Did You Not Compare With That? Identifying Papers for Use as Baselines (ECIR 2022).
  3. MasterSet — MasterSet: A Large-Scale Benchmark for Must-Cite Citation Recommendation in the AI/ML Literature (SDM 2026).
  4. Intern-Atlas — Intern-Atlas: A Methodological Evolution Graph as Research Infrastructure for AI Scientists (NeurIPS 2026).

We emphasize that LINEAGE introduces a distinct benchmark for empirical lineage retrieval, integrating evidence-grounded baseline relation extraction, multi-hop ancestor/successor retrieval ground truth, multi-facet queries, and structural evaluation metrics.

These works differ fundamentally in their objectives.

  • FeRoSA proposes a paper recommendation algorithm that categorizes results by citation context and evaluates the quality of recommendations through expert judgments. LINEAGE instead provides a benchmark for systematically evaluating diverse retrievers against evidence-grounded, multi-hop empirical lineage ground truth.
  • Bedi et al. study direct baseline identification as a classification task, whereas LINEAGE benchmarks empirical lineage retrieval, a critical capability for current automated research workflows.
  • MasterSet benchmarks 1-hop must-cite retrieval over directly cited papers, while LINEAGE evaluates multi-hop ancestor and successor retrieval with structural metrics.
  • Intern-Atlas constructs a methodological evolution graph as input to downstream research tasks, including idea evaluation and generation. In contrast, LINEAGE uses empirical baseline graphs as retrieval ground truth, without exposing them to retrievers. Moreover, Intern-Atlas excludes comparison relations from its methodological lineage.

Thus, these works address complementary problems and do not cover LINEAGE's core benchmark task and evaluation framework.

Besides, MasterSet and Intern-Atlas are contemporaneous works. As documented in our anonymous repository, we publicly released LINEAGE's benchmark dataset and code on May 6, 2026, close to the release date of these two papers. We will clarify these distinctions in the related work section in the revised version.


Q3. Have the authors evaluated LLM-based agents with explicit instructions to retrieve baseline papers and trace predecessor/successor relationships? If not, how can the current results support conclusions about these agents' ability to perform lineage retrieval?

LINEAGE evaluates the effectiveness of existing retrieval methods in recovering empirical lineage papers. These retrieval methods rely on lexical or semantic matching, which is designed for query-based retrieval rather than instruction-driven actions. Therefore, introducing explicit lineage-tracing instructions would be inconsistent with the retrieval mechanisms under evaluation and would compromise fair comparisons across retrieval paradigms.

Dear Reviewer ySSv,

Thank you for your constructive comments. We have made the following revisions to our manuscript and highlighted them in blue.

  • Added a brief discussion of multi-hop lineage utility in Section 4.1 (p. 6) and a new evaluation in Appendix I (pp. 30–31).
  • Discussed FeRoSA, Bedi et al. (2022), MasterSet (2026), and Intern-Atlas (2026) in the revised Related Work (Appendix B, p. 15), with corresponding references added to the bibliography.
  • Revised the descriptions of agentic retrievers and LLM-based search settings in Appendix C.2 (p. 17). We also clarified the benchmark scope and future research directions for agents in Appendix A (p. 15).

Could you please let us know if our response has addressed your concerns or if you have any further questions?

Best,

Authors

Reviewer_LhwJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LINEAGE, defining the "experimental baseline comparison relationship" in scientific literature as a retrieval supervision signal independent of topic similarity and ordinary citation relationships. The author constructed an experience spectrum graph from 121,048 AI papers, generated 80,000 multi-perspective queries, and evaluated 19 retrieval systems.

Critical strengths
  1. Whether the method of a certain paper was actually used for experimental comparison in another paper is different from general citation or topic relevance. Clarifying this relationship helps support baseline discovery, method comparison, and research positioning.

  2. Manual annotation and quality verification are relatively solid. LINEAGE-HA contains type annotations of 230 papers and 970 positive example relationships in 150 method papers. Each paper is independently annotated by two annotators, who also review any differences.

  3. The baseline coverage is relatively wide. The experiments include sparse retrieval, multiple dense encoders, research retrieval agents, and online search systems.

Critical weaknesses
  1. The paper treats large connected components as unsuitable for evaluation and splits them via edge pruning at bridges and Louvain community detection. However, large component size and abundant cross-community links do not invalidate experimental comparisons. A widely adopted baseline connecting multiple method groups may represent the genuine comparison structure in this field.

  2. The eight query types only describe paper content and lack requirements for tracing empirical research lineage, such as multi-hop experimental comparisons or baseline citations. Identical query texts can map to different source papers with different lineage labels, so queries cannot uniquely define retrieval targets.

  3. HWR, DR, and GN denote distance-weighted recall, one-hop recall, and distance-weighted nDCG, respectively. These metrics capture distances between queries and target nodes, yet they do not directly evaluate the connectivity and path completeness among retrieved nodes.

Questions for authors

See Weakness

2 comments

We thank the reviewer for the thoughtful comments and constructive suggestions.

Q1. The paper treats large connected components as unsuitable for evaluation and splits them via edge pruning at bridges and Louvain community detection. However, large component size and abundant cross-community links do not invalidate experimental comparisons. A widely adopted baseline connecting multiple method groups may represent the genuine comparison structure in this field.

Response: The purpose of graph calibration is to prevent widely adopted baselines from creating giant connected components that make multi-hop retrieval ground truth overly broad. Specifically, we use the IQR rule to identify the few upper-tail component-size outliers. If two method communities are closely related, additional empirical lineage relations can also connect them, rather than their connectivity depending solely on a few widely adopted baselines. The effects of this calibration and the relevant graph distributions are reported in Appendices F.2 and G.2.


Q2. The eight query types only describe paper content and lack requirements for tracing empirical research lineage, such as multi-hop experimental comparisons or baseline citations. Identical query texts can map to different source papers with different lineage labels, so queries cannot uniquely define retrieval targets.

Response:

  • Regarding query design, LINEAGE aims to evaluate the effectiveness of existing retrieval methods in recovering empirical lineage papers. These retrieval methods rely on lexical or semantic matching, which are designed for query-based matching rather than instruction-driven actions. Therefore, introducing explicit lineage-tracing requirements in the queries would be inconsistent with the retrieval mechanisms.

  • Regarding retrieval targets, each query instance is constructed around a specific source paper. Both its query text and ground-truth labels are grounded in the same paper. The queries reflect potential user search inputs. This follows the standard IR setting in which queries express information needs and relevance judgments specify the associated retrieval targets.


Q3. HWR, DR, and GN denote distance-weighted recall, one-hop recall, and distance-weighted nDCG, respectively. These metrics capture distances between queries and target nodes, yet they do not directly evaluate the connectivity and path completeness among retrieved nodes.

Response: Our metrics capture key aspects of connectivity and path completeness through lineage node recovery. Recall@k measures overall lineage coverage, while DR@k directly measures the recovery of query-incident lineage edges. HWR@k and GN@k further emphasize nearby nodes that serve as intermediate steps in multi-hop paths. These metrics assess the recovery of graph-grounded lineage neighborhoods, consistent with our paper-level retrieval objective.

Dear Reviewer LhwJ,

Thank you for your constructive comments. We have made the following revisions to our manuscript and highlighted them in blue.

  • Revised the graph calibration description in Section 3.3 (p. 5) and its analysis in Appendix F.2 (pp. 23–24).
  • Revised the descriptions of ground-truth construction in Section 4 (p. 5) and Section 4.1 (p. 6), as well as multi-facet query generation in Section 4.2 (p. 6).
  • Added a summary of the structural metrics in Section 4.3 (p. 7), describing distance-weighted coverage, direct-neighbor recovery, and hop-aware ranking.

Could you please let us know if our response has addressed your concerns or if you have any further questions?

Thank you!

Best,

Authors

Reviewer_vZtEOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LINEAGE, a benchmark for empirical lineage retrieval in AI scientific literature. An empirical lineage relation is established when one paper uses another paper’s method as an experimental baseline. These relations form a lineage graph, and the papers connected to a query paper in this graph are treated as correct retrieval results. To construct these graphs, the authors develop Lineage-Constructor, which converts heterogeneous paper records and full-paper baseline evidence into calibrated lineage graphs. Built from 121,048 papers across 17 top AI venues from 2018 to 2025, LINEAGE contains 3,950 graph components, 67,502 paper nodes, 164,948 lineage edges, and 80,000 multi-facet retrieval queries. Experiments with 19 representative retrieval systems show that empirical lineage retrieval remains challenging. Also, citation-based methods are not compared in this benchmark. And the paper mentions licensing constraints but does not clarify what data will be released. Overall, I think this a very solid work.

Critical strengths
  1. Most existing literature retrieval benchmarks rely on topical relevance or citation relations. However, identifying which papers are actually used as experimental baselines reflects a relationship that is closer to real research practice. Therefore, this topic is valuable to study.

  2. The paper builds a complete pipeline. It collects papers from different open-access sources, aligns metadata, validates the PDFs, parses the full text, uses LLMs for paper type classification and dual-source lineage extraction. The pipeline is clearly described and includes human verification. The experiments are solid, using retrieval and structural metrics. The evaluation covers 19 retrieval methods, including sparse retrieval, dense retrieval, agentic retrieval, search engines, and LLM search.

Critical weaknesses
  1. The paper argues that empirical lineage relations differ from general citation relations, and that existing citation-oriented benchmarks cannot capture empirical comparisons. However, it does not compare against citation-based retrieval baselines, such as direct citation retrieval or random walks on the citation graph. In addition, the authors argue that benchmark and survey papers construct evaluation resources and summarize existing work, so their citation relations are not suitable for constructing lineage graphs. I think this is true for surveys, but it is not entirely true for benchmark papers. In practice, benchmark papers often systematically and comprehensively compare a large number of methods, and their experiment tables contain many empirical baselines. These relations are important for understanding method comparison in a field. Therefore, I find the current justification not fully convincing.

  2. The authors evaluate the extractor on 150 papers and report a micro-F1 of 92.95 for dual-source integration. However, this scale is still small for 121048 papers and 164948 edges in the final benchmark. Moreover, the current precision and recall are still not high enough, and such errors may influence all downstream evaluations.

Questions for authors
  1. Could multi-hop ground truth contain many outdated baselines? If so, might the evaluation reward retrieving outdated baselines? Could the authors analyze this by hop distance or publication year, and consider down-weighting older multi-hop targets?

  2. Could the authors clarify what will be publicly released and whether the release complies with the licenses of the original paper sources?

2 comments

We thank the reviewer for the positive assessment of our work and the constructive feedback.

Q1. Could multi-hop ground truth contain many outdated baselines? If so, might the evaluation reward retrieving outdated baselines? Could the authors analyze this by hop distance or publication year, and consider down-weighting older multi-hop targets?

As shown in Figure 11, multi-hop targets may include older baselines. However, HWR@k and GN@k already assign lower weights to more distant lineage targets, reducing their influence on evaluation.


Q2. Could the authors clarify what will be publicly released and whether the release complies with the licenses of the original paper sources?

The released LINEAGE data includes paper metadata, lineage graphs, retrieval queries, and evaluation code. All papers were collected from open-access sources. Their licenses include CC0 1.0 for DBLP, OpenReview, and arXiv metadata, CC BY 4.0 for ACL Anthology and PMLR, and CC BY-NC-ND 4.0 for PVLDB. Our release does not redistribute original paper PDFs or full texts and complies with the applicable licensing terms.

Thank you for your response. My concerns have been addressed, and I will keep my score unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.