acceptodds
Under review as a conference paper at ICLR 2027

When Does Graph-Based Retrieval Actually Help? A Benchmark for Scientific Literature Discovery

Abstract

Knowledge graph structure is widely expected to improve scientific literature retrieval, but existing evaluations either presuppose the graph or score a generated answer rather than the retrieved set, so the retrieval step itself is rarely the quantity measured. KGRAF-BENCH exposes one corpus through both a flat and a typed-graph view, retrieves with one encoder at one matched budget, and scores the retrieved paper sets against the citation decisions of survey authors. Its construction is automated: given a list of source-paper arXiv identifiers, the pipeline assembles a corpus, extracts a typed graph over it, and generates questions whose ground truth is the citations of the source papers. We release two ML/AI corpora of 8,219 and 89,476 papers with a graph over each, 647 questions, the per-question features, and a biomedical instantiation produced by the same pipeline. Our measurements identify the extraction schema as the component that determines which retrieval channel improves on flat retrieval. Two graphs over the same 8,219 papers at the same canonicalization threshold, differing only in what the extractor emits, place the advantage on different question sets, and the difference follows the paper-to-entity edges each schema provides. The advantage is consistent in direction across conditions and small in magnitude: over 26 configurations varying corpus scale, canonicalization threshold, encoder and construction method, entity-aware traversal is significantly ahead of flat retrieval in most and significantly behind in none. Selecting a strategy per question does not improve retrieval. A per-question oracle does not exceed its own permutation null, and a router trained on pre-retrieval features performs worse than the best fixed strategy. Tuning the extraction schema therefore matters more for a deployed system than increasing corpus size or adjusting the canonicalization threshold. We evaluate four retrieval strategies that isolate the contribution of entity embeddings, phrase extraction, and graph neighborhood expansion. The value of graph structure is conditional and predictable. Hybrid similarity over entity embeddings is the overall best strategy, winning on 80% of standard and 57% of relational questions. Graph neighborhood expansion is uniquely best on 24% of questions where strategies diverge. These are precisely the questions where the entity-paper gap, a computable pre-retrieval signal, is high. This signal distinguishes two retrieval regimes and predicts when graph structure adds value without any LLM calls. Per-question oracle selection yields R@50 gains of +0.09 over the best fixed strategy, demonstrating substantial headroom for learned adaptive routing. These findings are stable across 2K and 8K corpus scales. We release KGRAF-Bench to support research on retrieval for scientific discovery.

Primary area:
datasets and benchmarks
Keywords:
Language and multimodal language models (e.g., text generation, summarization, VQA), AI and network science (e.g., graph neural networks, communication)

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EyCTOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The work presents KGRAF-Bench, which uses text and typed knowledge graph views of the same scientific literature corpus. Under controlled settings of shared encoder, fixed retrieval count and citation labels, four retrieval strategies are compared to explore when graph structures improve scientific literature retrieval. It includes an automated building pipeline, cross-scale and cross-encoder experiments, and domain extensions for biomedicine and physics.

Critical strengths

The evaluation setup is well-defined. Separating retrieval quality from final answer generation prevents generative models from masking retrieval defects. The shared corpus, encoder and fixed output count provide a clean baseline for analyzing text-only and entity-augmented retrieval.

Critical weaknesses
  1. The experiments fail to sufficiently answer when graph structures are beneficial. The clearest comparison between S2 and S1 only verifies the usefulness of entity representations and entity-to-paper mapping, and cannot prove the value of complex graph topologies. Compared with S2, S3 changes neighborhood expansion, LLM phrase extraction and entity matching rules concurrently. The two graph construction pipelines also modify schema and merging processes together, so the effect of individual structural factors cannot be disentangled. Therefore, the current analysis only demonstrates minor improvements from entity-assisted strategies under several settings. It remains unclear whether these gains come from entity semantic representations or well-defined graph connections, and which scenarios require graph structures versus simple concept matching.

  2. Standard and Relational differ along multiple dimensions: provision of cited abstracts, input granularity (single paragraph vs. paragraph pairs/sections), average valid references (2.2 vs. 20.7), and reference filtering (Standard applies essential/background filtering, whereas Relational does not). Therefore, the observed performance difference cannot be clearly attributed to graph structures or cross-paper relations.

  3. This work constructs a controlled retrieval evaluation framework using existing paper abstracts, citation links from surveys, and LLM-generated questions. Although new entity graphs and question sets are built, it does not verify that these artifacts provide missing, reliable capability annotations. Citation relations are not independently judged for relevance to the generated questions, and the “Standard/Relational” split has not been validated as a meaningful distinction for retrieval abilities. Therefore, the core contribution remains strategy comparisons under fixed corpus, encoder, and output count settings. This controlled setup enables certain analysis, yet for a benchmark-focused paper, it must demonstrate that it can measure critical capabilities overlooked by existing benchmarks or yield findings that reshape community understanding. Current results only show minor performance gaps and configuration sensitivity across custom strategies. It lacks systematic evaluation of representative full methods and fails to fully explain when and how graph structures work. Its overall contribution is closer to a preliminary diagnostic study and does not yet justify its positioning as a benchmark.

  4. The authors openly report router failure, insignificant biomedical results, incomplete disentanglement of construction factors, and variations caused by hardware and index rebuilding. Such transparency is commendable. However, acknowledging limitations does not resolve confounding factors that undermine the core conclusions.

Questions for authors

See Weakness

Reviewer_D2wTOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary
  • This paper introduces KGRAF-BENCH to compare flat and graph-based methods for scientific literature retrieval.
  • Across multiple configurations, graph-based retrieval shows small but generally positive gains, while per-query routing fails to outperform the best fixed strategy.
Critical strengths
  • Addresses a relevant question with a controlled comparison of retrieval strategies on the same corpus and ground truth.
  • Provides extensive experiments across graph construction settings, encoders, corpus scales, and domains.
  • Releases the benchmark, graphs, and code, and discusses limitations and reproducibility issues transparently.
Critical weaknesses
  1. The need for a new benchmark is not fully justified, as existing retrieval benchmarks could potentially be extended with graph structure.
  2. The experiments do not isolate graph traversal from LLM-based phrase extraction and entity matching.
  3. The hybrid scoring weights are not tuned or tested for sensitivity, and the paper does not establish clear conditions for when graph retrieval is preferable to embedding-based retrieval.
  4. Entity-based retrieval is widely investigated for the scientific retrieval task. But there are some missing references.
  • Scientific paper retrieval with llm-guided semantic-based ranking, EMNLP'25
  • PairSem: LLM-Guided Pairwise Semantic Matching for Scientific Document Retrieval, WWW'26
Questions for authors

From my understanding, the paper says

  • graph-based retrieval is generally better than embedding-based retrieval.
  • In some cases, different retrieval can be better.
  • However, the routing approach failed.

From these observations, what is the new and unique findings of the paper? What can we learn from this paper?, and how should we go further from this paper?

Reviewer_p2D6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks at whether a knowledge graph really helps when searching for scientific papers, if everything else like the corpus, encoder and retrieval budget is kept the same. The authors build KGRAF-Bench, which is an 8K ML paper corpus that can be searched either as a normal embedding index or as a graph built by an LLM. There are 647 questions, and the right answers are the papers cited in seven source papers, mostly surveys. When they compare normal search with three graph based methods, the graph helps but only a little, around 2 to 3 points of Recall@50 overall. Graph traversal is ahead pretty consistently and no method is ever significantly worse than normal search in their 80 tests. They also report that picking a method per question doesn't beat always using the single best method. They release the pipeline and also try it on biomedical and physics papers.

Critical strengths
  1. The comparison is designed to be fair. Graph and flat retrieval use the same corpus and encoder with a fixed budget of 50, and the paper scores the retrieved papers themselves, not a generated answer. S2 vs S1 doesn't even involve an LLM call, so it's a clean design, and that kind of controlled test is rare in GraphRAG papers.

  2. The ground truth is good. The correct answers are the papers the source authors actually cited (six surveys and one methods paper). The LLM can only drop citations, never add them, and this only affects 59 of the 647 questions. Nice that the authors checked this and reported it.

  3. The statistics are careful. Paired bootstrap intervals are used throughout, the authors measured their own noise floor (around 0.015), and when comparing across configurations they only claim the direction of the effect. This is not common in this area, and it makes the results easier to trust.

  4. The negative results are useful. The paper shows that its cheap signal for predicting which questions need the graph doesn't work, and this holds on the biomedical and physics data too (Appendices F and J). It also shows that the cheap entity-embedding method can end up below flat retrieval. That's helpful to know in practice, and a lot of papers would have just left it out.

  5. Reproducibility is strong. The prebuilt graphs, the questions with ground truth, the per-question features and the code with all prompts are released, and the toolkit can rebuild a benchmark from a list of arXiv IDs. Easy for others to build on. (I looked through the anonymous repository and these are all there.)

Overall the scope is fairly narrow (one main corpus, and only entity-based graphs), but within that scope this is a careful and useful contribution.

Critical weaknesses
  1. Unclear where the gain comes from. My main concern is that the gain is small, about 2 to 3 points of Recall@50 overall, close to the authors' own 0.015 noise floor, and the paper doesn't show why it happens. In S2 and S3 the final ranking is plain cosine similarity, so the graph only decides which papers enter the pool. It could be finding new papers, or just filtering out near duplicates. S2 also cuts its list by Python set order (Appendix I.5), Appendix I.3 admits the explanation is untested, and it is unclear whether S1 always gets 50 papers (question 1). Since the paper's main claim is that graph structure helps, this needs to be clearer.

  2. The RQ4 results don't add up. Figure 2 shows 27 rescued questions, 104 flat wins and d = +1.27, but the text says 22 rescued questions with d = +0.23 (Section 4.4) and 48 flat wins with d = +0.89 (Appendix E). These are simple counts, so they should match. The router also only chooses between S1 and S4, even though S2 is the best fixed strategy on Relational questions, so the claim that routing doesn't work isn't really a fair test yet. This is presented as one of the main findings, so as it stands it's hard to trust.

  3. The two question sets are confounded. Standard and Relational are supposed to separate questions where similarity is enough from questions that need the graph. But they also differ a lot in how many correct papers each question has (2.2 vs 20.7 on average), and only the Standard set goes through the LLM filter. The authors do mention this in their limitations, but it means the benchmark can't yet tell 'needs the graph' apart from 'has many correct papers'.

  4. The conclusions go further than the evidence. The graph has no citation links (Table 7), so the results are about LLM-extracted entity graphs, not graph structure in general. The abstract says traversal wins 'in 52 of the 80 configurations tested', but there are 20 configurations and 80 tests. The other domains add little, since none of the 24 biomedical tests is significant and the physics questions were not checked by the authors.

Questions for authors

Clarification questions

  1. Does S1 always return 50 papers? Appendix B says S1 and S2 draw candidates from a nearest-neighbour search over the node index at depth 500, while the Table 15 caption says S1 queries the paper index alone. If the node index includes entities, then with about 9.6 entities per paper the top 500 nodes could contain fewer than 50 papers. If S1 sometimes returns fewer, the S2 vs S1 comparison is not at the same budget.

  2. Which RQ4 numbers are correct? Figure 2 shows 27 rescued questions, 104 flat wins and d = +1.27. The text reports 22 rescued (Section 4.4) and 48 flat wins with d = +0.89 (Appendix E).

  3. The router only chooses between S1 and S4, but S2 is the best fixed strategy on Relational questions. How does the router do if it can also choose S2? And which strategies does the oracle choose from?

  4. How is the Standard average of 2.2 correct papers possible? Each Standard question uses the papers cited in its source paragraph (Appendix C.8), every such paragraph cites at least 2 corpus papers, 179 cite at least 4 (Appendix C.7), and only 77 papers are removed in total. That gives an average of about 3.0 or more.

  5. The abstract says the physics data is released, but the reproducibility statement lists only four graphs without it. Will it be released?

Additional experiments to show where the gain comes from (usually intermediate steps might reveal something)

  1. Pool sizes and ranks. In S2 and S3 the graph only decides which papers enter the candidate pool, and the final ranking is plain cosine similarity (Section 3.2). Please report (a) how big the pool usually is, and (b) for correct papers that S3/S4 find but S1 misses, where they rank under cosine similarity. If they rank far down the cosine list, the graph really reaches new papers. If they sit just outside the top 50, it mostly filters out distractors.

  2. One part of S3 at a time. S3 adds three things at once (neighbourhood expansion, LLM phrases and a new matching rule), and Section 3.2 and Appendix I.3 note that no contrast separates them. Running S3 with only the expansion, and then with only the phrase-matched entities, would show which part gives the gain.

  3. S2 with a proper cut. S2 currently cuts its candidate list in Python set order (Appendix I.5), which is arbitrary. Does the Relational gain of +.032 still hold with a score-based cut?

Addressing questions 1 to 3 and experiments 6 to 8 would help me reconsider my score.

4 comments

Thanks for the thorough comments. Q1, Q2, Q4 and Q5 were real defects, and the error report shipped with the code records each one, the fix, and the value before and after.

Fixes to the code, in the attached zip

The anonymous repository linked in the paper is unchanged. The fixes below are in the zip attached to this response, which is the full release at its current state.

  • Added a script that checks that every strategy in every run uses the full budget (Q1). No strategies falls short anywhere.
  • Provide graph + json retrieval results + the code that regenerates every artifact in the paper (Q2).
  • Physics dehydrated graph is released in the updated zip attachment (Q5).

Clarifications

Q1. We identified this too and fixed it before the rebuttal. S1 returned 48.7 papers on average on Standard questions and 47.0 on Relational; it now returns 50 everywhere, which moved S2 −- S1 by .001 and .005 and changed no direction.

Q2. The figure was generated from an outdated version of the code. It is regenerated and now agrees with §4.4: 24 questions where flat retrieval returns no ground truth, 107 where it wins, d=+0.85d = +0.85.

Q4. Several questions are generated per paragraph, so a question's ground truth does not necessarily contain all the citations of the paragraph it comes from. The paragraph-to-question mapping is released, with each paragraph, its questions and their ground truth.

Q5. Yes, we added it to the archive.

Q8. We agree that S2 should not depend on the Python hash seed, so S2 now breaks ties using the paper's own cosine similarity, and we recomputed every configuration that way, including the biomedical and physics extensions. No S2 value moves by more than .013, below the 0.015 reproducibility floor we report, and the Relational gain stays at +.032 in the main table.

Added experiments

ExperimentSectionWhat it doesResult
Router over every action space (Q3)Appendix ERoutes over {S1,S4}, {S1,S2}, {S1,S2,S4}, {S2,S3}, all four and all seven strategiesAdding S2 does not help. On Relational questions the best fixed strategy is S2 and the router matches it exactly; no row beats its best fixed strategy on either question set
Pool size and cosine ranks (Q6)§4.1, Figure 1; Appendix I.4Reports how often the graph pool fills the budget, and the cosine rank of every retrieved paper, including the ground-truth ones traversal finds and flat retrieval missesThe pool fills 50 papers without padding on 92% of Standard and 96% of Relational questions, so the gain is not a padding artifact. The recovered papers sit far down the cosine list, median rank near 95 and as deep as 2,144, so the graph reaches new papers rather than filtering distractors
S3 ablation (Q7)§4.1, Table 2; Appendix I.4Only the neighbourhood expansion, then only the phrase-matched entities, then holds one fixed and adds the otherNeither half reaches S3. Expansion alone falls below flat retrieval and the phrase-matched entities alone do not separate from it, while the two together rise above it. Each increment is significant on both question sets, so both parts contribute

Remark on build-to-build variation

Two runs of the same batch construction differ slightly because canonicalization matches against a cache that parallel workers fill in a different order, and the two builds we hold of the primary graph (the one that serves both the canonicalization threshold sweep and the batch arm of the sequential/batch comparison) differ by 0.19% in entities and by at most .003 in R@50, a fifth of our reproducibility floor, with both shipped in the release for transparency.

Thanks for the thorough reply and the revised version. You’ve addressed the questions and experiments I asked for, and I’m planning to raise my score to weak accept once one thing is fixed.

The paper says the LLM narrows the ground truth on only 59 Standard questions, and that for the rest it’s the authors’ full citation list (Sections 2.2 and 2.3, Appendix C.10). I looked at the released questions and that doesn’t seem right. Global questions come from paragraphs that cite at least four corpus papers, but 160 of the 179 global questions have fewer than four ground-truth papers. Counting the local ones too, at least 174 of the 274 Standard questions have a shorter ground truth than their paragraph’s citations.

I think the reason is that the 59 only counts questions where the LLM marked some papers as 'background'. When it simply leaves a cited paper out of both lists, that paper is dropped too, but it isn’t counted. Could you check this and correct the 59, 9.1% and 90.9% figures in the paper?

You are correct. We amended Sections 2.2 and 2.3, Appendix C.10 and the AI use statement, replacing 59, 9.1% and 90.9% with 191, 29.5% and 70.5%.

We added a script and a table to the code that produce these numbers, building on the source-text mapping released with our previous response:

QuantityStandardRelationalBoth
Questions274373647
Ground truth smaller than its source text's citations1910191
Ground truth is the source text's citation list unchanged83373456
— as a share of the set30.3%100.0%70.5%
Ground truth is a subset of the source text's citations274373647

Thanks for catching this.

Thanks for checking this and correcting it quickly. The updated numbers match what I saw in the released data, so I’ve raised my score accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.