acceptodds
Under review as a conference paper at ICLR 2027

ImmunoGraphBench: A Large-Scale Context Graph Benchmark for Scientific Reasoning

Abstract

Biomedical knowledge is recorded as conditional claims: a relation between two entities is observed under particular conditions, supported by specific evidence, and associated with particular measurements. A graph interface that exposes only context-free triples can make distinct candidate hypotheses observationally equivalent: they satisfy the same topology, while the observation that distinguishes them is absent. This is an identifiability problem at the reasoning interface, not merely a reduction in text. ImmunoGraphBench makes the problem testable by constructing structurally underdetermined questions where relation-level context breaks the symmetry. We build a biomedical KG of 1.62M triples, 75% of which carry textual evidence or structured metadata, and vary structure and context independently on the same QA instances. Across seven LLMs, adding context to explicit candidate paths raises Medium-level accuracy from 61.35% to 89.56% (+28.20 pp), and the gain persists when the graph is retrieved from the question alone (+14.83 pp). Controls show that the gain depends on relation-matched content, survives evidence-count debiasing, and partially compensates for missing structural support. The results identify relation-level context as the interface signal that restores candidate identifiability beyond topology.

Primary area:
datasets and benchmarks
Keywords:
Biomedical Knowledge Graphs, Large Language Models, Scientific Reasoning, Relation-Level Context, Context-Aware Reasoning, Multi-Hop Reasoning, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_18FeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents ImmunoGraphBench, a biomedical knowledge graph containing 1.62 million triples, and about 75% of which are enriched with textual evidence, provenance, or structured metadata. The paper investigates whether relation-level context improves LLM reasoning when graph topology alone cannot distinguish competing candidates. Through three levels of QA and controlled input-visibility experiments across seven LLMs, the authors show that adding context to explicit candidate paths improves Medium-level accuracy, with further experiments examining context types, instance specificity, evidence quantity, and missing structural support.

Critical strengths
  1. This paper constructs a useful and large-scale benchmark and explores how triple-based KGs and the relations can affect LLM reasoning.

  2. The experiment design is comprehensive and well contraolled. The controls including random-context replacement, context swapping, and evidence-quantity debiasing are appropriately implemented.

Critical weaknesses

Limited human verification: The KG combines source-provided information with literature-derived evidence selected using programmatic filtering, LLM verification, and stratified human review. However, the reported manual audit of 300 records provides limited assurance about the quality of the full 1.62-million-triple graph.

Questions for authors
  1. How does the proposed framework provide insights beyond existing evidence-bearing or hyper-relational KGs and controlled graph-reasoning benchmarks? Can the authors identify a concrete methodological advance beyond the construction of a new benchmark?

  2. How often do structurally underdetermined, context-resolvable cases arise in naturally occurring biomedical QA or scientific discovery workflows? Can the main findings be validated on independently sourced questions without deliberately selecting cases where context identifies the answer?

  3. The retrieval experiment holds the retrieved graph fixed when comparing inputs with and without context. How sensitive are the results to retrieval errors, incomplete evidence retrieval, and incorrect entity or relation linking in a fully end-to-end pipeline?

Reviewer_PAB5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ImmunoGraphBench, a biomedical knowledge graph benchmark designed to evaluate whether relation-level context helps LLMs resolve structurally ambiguous reasoning paths. Experiments across seven models show substantial improvements when contextual evidence and metadata are provided.

Critical strengths
  1. Controlled comparisons effectively isolate the impact of relation-level context.

  2. The large-scale knowledge graph and seven-model evaluation provide useful empirical evidence.

Critical weaknesses
  1. Limited novelty. The findings largely confirm the expected benefits of restoring information deliberately omitted from graph triples.

  2. Limited validity and generalizability. Context-derived questions and imperfect evidence quality weaken the benchmark's broader scientific claims.

  3. The title overstates the scope of the work, as the benchmark primarily evaluates context-dependent biomedical graph QA rather than general scientific reasoning.

Questions for authors
  1. What new scientific insight does the benchmark provide beyond demonstrating the value of additional context?

  2. Can the findings be validated using independently constructed questions and verified biomedical evidence?

Reviewer_fcDKOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This is a benchmark study of ImmunoGraphBench that looks at whether biomedical knowledge-graph interfaces lose information needed to distinguish candidate answers when they expose only entity–relation–entity triplets. The project develops a KG of 1,618,645 normalized triplets, approximately 75% of which carry textual evidence or structured metadata, and its primary Medium-level benchmark constructs questions with multiple structurally compatible candidate paths and selects a contextual condition that distinguishes the intended answer. The project combines this resource with a controlled input-visibility protocol and elementary propositions of varying difficulty and use this to benchmark 7 LLMs and show that preserving relation-specific context improves candidate identification at the graph–LLM interface.

Critical strengths

The experiments and ablations are solid and seem to clearly demonstrate benefit for this task series.

he shortcut analyses identify a substantial subset whose performance is not explained by the tested simple selectors.

Critical weaknesses

The validation does not establish that benchmark gold answers are scientifically supported and uniquely warranted. A targeted expert audit of complete QA instances, followed by evaluation on the verified subset, would directly address this.

No comparison to frontier models or simple LLM + search seems an odd omission.

Questions for authors

My main reservation is whether the benchmark’s answers are independently established as valid and support the manuscript's conclusions. A convincing gold standard evaluation and a reconciled, controlled swap analysis would be appreciated.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.