- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents ImmunoGraphBench, a biomedical knowledge graph containing 1.62 million triples, and about 75% of which are enriched with textual evidence, provenance, or structured metadata. The paper investigates whether relation-level context improves LLM reasoning when graph topology alone cannot distinguish competing candidates. Through three levels of QA and controlled input-visibility experiments across seven LLMs, the authors show that adding context to explicit candidate paths improves Medium-level accuracy, with further experiments examining context types, instance specificity, evidence quantity, and missing structural support.
Critical strengths
-
This paper constructs a useful and large-scale benchmark and explores how triple-based KGs and the relations can affect LLM reasoning.
-
The experiment design is comprehensive and well contraolled. The controls including random-context replacement, context swapping, and evidence-quantity debiasing are appropriately implemented.
Critical weaknesses
Limited human verification: The KG combines source-provided information with literature-derived evidence selected using programmatic filtering, LLM verification, and stratified human review. However, the reported manual audit of 300 records provides limited assurance about the quality of the full 1.62-million-triple graph.
Questions for authors
-
How does the proposed framework provide insights beyond existing evidence-bearing or hyper-relational KGs and controlled graph-reasoning benchmarks? Can the authors identify a concrete methodological advance beyond the construction of a new benchmark?
-
How often do structurally underdetermined, context-resolvable cases arise in naturally occurring biomedical QA or scientific discovery workflows? Can the main findings be validated on independently sourced questions without deliberately selecting cases where context identifies the answer?
-
The retrieval experiment holds the retrieved graph fixed when comparing inputs with and without context. How sensitive are the results to retrieval errors, incomplete evidence retrieval, and incorrect entity or relation linking in a fully end-to-end pipeline?