- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The work presents KGRAF-Bench, which uses text and typed knowledge graph views of the same scientific literature corpus. Under controlled settings of shared encoder, fixed retrieval count and citation labels, four retrieval strategies are compared to explore when graph structures improve scientific literature retrieval. It includes an automated building pipeline, cross-scale and cross-encoder experiments, and domain extensions for biomedicine and physics.
Critical strengths
The evaluation setup is well-defined. Separating retrieval quality from final answer generation prevents generative models from masking retrieval defects. The shared corpus, encoder and fixed output count provide a clean baseline for analyzing text-only and entity-augmented retrieval.
Critical weaknesses
-
The experiments fail to sufficiently answer when graph structures are beneficial. The clearest comparison between S2 and S1 only verifies the usefulness of entity representations and entity-to-paper mapping, and cannot prove the value of complex graph topologies. Compared with S2, S3 changes neighborhood expansion, LLM phrase extraction and entity matching rules concurrently. The two graph construction pipelines also modify schema and merging processes together, so the effect of individual structural factors cannot be disentangled. Therefore, the current analysis only demonstrates minor improvements from entity-assisted strategies under several settings. It remains unclear whether these gains come from entity semantic representations or well-defined graph connections, and which scenarios require graph structures versus simple concept matching.
-
Standard and Relational differ along multiple dimensions: provision of cited abstracts, input granularity (single paragraph vs. paragraph pairs/sections), average valid references (2.2 vs. 20.7), and reference filtering (Standard applies essential/background filtering, whereas Relational does not). Therefore, the observed performance difference cannot be clearly attributed to graph structures or cross-paper relations.
-
This work constructs a controlled retrieval evaluation framework using existing paper abstracts, citation links from surveys, and LLM-generated questions. Although new entity graphs and question sets are built, it does not verify that these artifacts provide missing, reliable capability annotations. Citation relations are not independently judged for relevance to the generated questions, and the “Standard/Relational” split has not been validated as a meaningful distinction for retrieval abilities. Therefore, the core contribution remains strategy comparisons under fixed corpus, encoder, and output count settings. This controlled setup enables certain analysis, yet for a benchmark-focused paper, it must demonstrate that it can measure critical capabilities overlooked by existing benchmarks or yield findings that reshape community understanding. Current results only show minor performance gaps and configuration sensitivity across custom strategies. It lacks systematic evaluation of representative full methods and fails to fully explain when and how graph structures work. Its overall contribution is closer to a preliminary diagnostic study and does not yet justify its positioning as a benchmark.
-
The authors openly report router failure, insignificant biomedical results, incomplete disentanglement of construction factors, and variations caused by hardware and index rebuilding. Such transparency is commendable. However, acknowledging limitations does not resolve confounding factors that undermine the core conclusions.
Questions for authors
See Weakness
Thanks for the thorough comments. Q1, Q2, Q4 and Q5 were real defects, and the error report shipped with the code records each one, the fix, and the value before and after.
Fixes to the code, in the attached zip
The anonymous repository linked in the paper is unchanged. The fixes below are in the zip attached to this response, which is the full release at its current state.
Clarifications
Q1. We identified this too and fixed it before the rebuttal. S1 returned 48.7 papers on average on Standard questions and 47.0 on Relational; it now returns 50 everywhere, which moved S2 − S1 by .001 and .005 and changed no direction.
Q2. The figure was generated from an outdated version of the code. It is regenerated and now agrees with §4.4: 24 questions where flat retrieval returns no ground truth, 107 where it wins, d=+0.85.
Q4. Several questions are generated per paragraph, so a question's ground truth does not necessarily contain all the citations of the paragraph it comes from. The paragraph-to-question mapping is released, with each paragraph, its questions and their ground truth.
Q5. Yes, we added it to the archive.
Q8. We agree that S2 should not depend on the Python hash seed, so S2 now breaks ties using the paper's own cosine similarity, and we recomputed every configuration that way, including the biomedical and physics extensions. No S2 value moves by more than .013, below the 0.015 reproducibility floor we report, and the Relational gain stays at +.032 in the main table.
Added experiments
Remark on build-to-build variation
Two runs of the same batch construction differ slightly because canonicalization matches against a cache that parallel workers fill in a different order, and the two builds we hold of the primary graph (the one that serves both the canonicalization threshold sweep and the batch arm of the sequential/batch comparison) differ by 0.19% in entities and by at most .003 in R@50, a fifth of our reproducibility floor, with both shipped in the release for transparency.