acceptodds
Under review as a conference paper at ICLR 2027

IBGraphRAG: Enhancing Medical Knowledge Graph Retrieval Based on Semantic Consistency and Information Bottleneck

Abstract

Large Language Models (LLMs) have achieved remarkable progress in recent years, but they still struggle to generate reliable and precise responses in domains such as medicine that rely heavily on specialized knowledge. Retrieval-Augmented Generation (RAG) offers a scalable solution for integrating external knowledge into LLMs without additional training, and while it performs well in general domains, its effectiveness is limited in medical settings that demand high terminological precision and factual consistency. To address this, we propose a medical-domain-oriented RAG framework, IBGraphRAG, which integrates medical knowledge graphs with two key technical innovations: (1) Medical Semantic Consistency Alignment, which improves entity recognition and linking by enforcing semantic consistency with a structured medical knowledge base; and (2) Information Bottleneck-based Reasoning Path, which prioritizes retaining highly relevant contextual information during knowledge graph retrieval while avoiding irrelevant or superficial paths. Experimental results show that IBGraphRAG achieves state-of-the-art performance on multiple medical question-answering benchmark datasets, effectively improves the specialization and accuracy of the retriever in selecting reasoning paths over the knowledge graph, helping the LLM better identify relevant knowledge for reasoning. Here is our code.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Large Language Model, Medical LLM, Retrieval-Augmented Generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_B5viOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose IBGraphRAG pipeline for medical QA to do improve factual consistency, built on a medical knowledge graph (PrimeKG), and add medical semantic consistency alignment (MSCA) which combines similarity to the query with average similarity to the other candidates and keeps only entities based on a specific threshold. They also propose Information Bottleneck-based Reasoning Path (IBRP), which is a subgraph retriever trained with BCE against shortest-path supervision, plus an IB term that approximates target relevance with cosine similarity and compression with a reconstruction decoder. IBGraphRAG reports gains of roughly 1–2 points over MedGraphRAG and SubgraphRAG.

Critical strengths

1- medical semantic consistency alignment idea of combining query relevance with coherence among candidate entities is simple and looks practical.

2- broad test across llm backbones and benchmarks.

Critical weaknesses

1- The second main claims of the paper (theoretical claim) is not new and also is not mathematically sound as they presented it here. The derivation of Eq. (6) and also Appendix A.2 is closely follow (Zhu et al., “An Information Bottleneck Perspective for Effective Noise Filtering on Retrieval-Augmented Generation” ACL 2024): which are almost identical in the notation and final objective (see equation 9 of the paper and others). That paper is not cited here. Also the adaptations of the equations has a major issue here. Appendix A.2 calls the decoder based term a variational upper bound on I(X~; X | Q) although Appendix A.3 says training the decoder maximizes a lower bound on the same quantity. Eq. (13) minimizes an expectation of a probability rather than a log-likelihood, and the step from Eq. (12) to Eq. (13) does not follow.

2- MSCA has two main issues that arent discussed in the paper. First, it is defined on question and answer entities, so how can it be used during inference without any answer entities? Also, MSCA needs about 67 seconds per sample (Appendix B.9). The authors say it can run offline as preprocessing, but for a new user query at inference time it cannot be precomputed, and this cost is not included in the 6.6 s/it query time reported in Table 12. This is not realistic or practical and If it really needs that much time, it should be stated as one of the main limitations of the paper; the limitations paragraph in Section 4.4 and Appendix B.12 only mention the IB overhead, not the MSCA cost.

3- The LLM used for the research are quire outdated (GPT4 and llama 2, etc) and the paper is not using frontier models or comparing against them

Questions for authors

1- Please correct derivation of the compression term: is it an upper or a lower bound, and on which quantity? and clarify your novelity vs (Zhu et al. ACL, 2024) 2- How is MSCA applied at test time without answer entities? What is the true end-to-end latency per query, including MSCA? and why that takes so long?

3- Which LLM performs entity extraction for each backbone?

4- Can you compare and use frontier models and more recent models of LLMs since the GPT4 and llama 2 and other used hre are quite old based on the timeline of the ICLR submission

Reviewer_wQXrOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes IBGraphRAG, a knowledge graph–based retrieval-augmented generation framework for medical question answering and health-related fact verification. The framework comprises two components: Medical Semantic Consistency Alignment (MSCA), which combines UMLS-based entity enrichment with query relevance and inter-entity consistency scores; and Information Bottleneck-based Reasoning Path (IBRP), which augments shortest-path supervision with answer-relevance and reconstruction objectives to train a triple retriever. The retrieved triples are subsequently provided to an LLM as evidence for answer generation. Experiments cover multiple medical benchmarks and six LLM backbones, with reported improvements over several retrieval-based baselines. The application motivation is compelling, but the information bottleneck formulation, the separation between training and evaluation, and the accounting of computational costs require substantial clarification.

Critical strengths

(1) The paper addresses two meaningful challenges in medical KG-based RAG: inaccurate entity alignment and insufficient evidence retrieved through shortest-path supervision. Both can materially affect downstream answer quality.

(2)MSCA combines biomedical terminology enrichment, query relevance, and consistency among candidate entities through an explicit scoring function. This provides an interpretable approach to filtering candidate entities before retrieval.

(3)The observation that shortest paths may omit answer-relevant context motivates learning a more flexible evidence selection strategy. Incorporating target relevance into retriever training is a promising direction.

(4) The evaluation includes medical multiple-choice questions, health-related fact verification, and LLMs of different sizes. The reported improvements across backbones suggest that the approach may complement improvements in generator capacity.

(5) The paper provides component ablations and analyses of retrieval budgets, hyperparameters, training epochs, cross-dataset transfer, and subgraph expansion depths. These offer a useful starting point for examining the method’s behavior.

Critical weaknesses

(1)Appendix A.2 contains several problematic transitions. The Jensen-based expression in Eq. (11) does not directly yield the KL divergence stated in Eq. (12), including the orientation of the probability ratio. Eq. (13) then replaces a log-probability expression with an expected probability without establishing optimization equivalence. Appendix A.3 also alternates between describing the reconstruction objective as an upper-bound estimator and a lower-bound objective. Maximizing reconstruction likelihood alone does not establish the upper bound required for the claimed compression objective. Furthermore, Eq. (6) introduces conditioning on the target YY, whereas the decoder is primarily specified as rψ(x∣x~,q)r_\psi(x\mid\tilde{x},q). The authors should provide a consistent derivation or explicitly characterize the objective as an IB-inspired heuristic.

(2)Section 3.1 extracts entities from both questions and answers, which is reasonable during training if properly isolated from evaluation. However, Appendix B.8 describes constructing subgraphs around entities appearing in “questions and answers” without clearly specifying whether this applies only to training. If ground-truth answer entities are used to construct test-time candidate graphs, this would introduce privileged information and compromise the evaluation. The current description does not establish that leakage occurred, but it leaves a consequential ambiguity unresolved.

(3)The pipeline involves triple scoring, discrete selection, textual conversion, secondary encoding, and decoder training. It does not sufficiently explain how gradients from the proposed objectives reach the retriever through these operations. The path sampling distribution, decoder output space, and joint or alternating optimization schedule are also underspecified. Moreover, the existing ablations do not distinguish the effects of answer-similarity supervision, reconstruction regularization, and their combination. Comparisons against BCE plus semantic similarity, BCE plus reconstruction, standard sparsity regularization, and parameter-matched alternatives are needed to establish the value of the proposed formulation.

(4)Appendix B.9 reports a query time of 6.6 seconds, but subsequently states that MSCA requires approximately 67 seconds per sample. The claim that MSCA can be performed offline does not fully resolve this issue because its alignment scores depend on the user query. For previously unseen queries, query-dependent processing must either be included in online latency or supported by an explicitly described caching or preprocessing mechanism. A fair comparison should separately report static knowledge-base preprocessing, entity alignment, graph retrieval, and answer generation under consistent conditions.

(5)The main text reports a 2.6-percentage-point MedQA decrease for Llama3-70B after removing MSCA, whereas Table 2 reports 87.3% accuracy and a 2.4-point decrease. The heading “Baselines with MSCA” is also ambiguous relative to the removal experiment described in the text. In addition, the main text and Appendix B.1 provide substantially different descriptions of the PrimeKG resource. The qualitative evidence also has limited scope: Figure 3 uses a manually constructed question, an LLM-generated graph, and a separately retrained retriever. This is useful for illustration but cannot establish retrieval quality on held-out benchmark examples. Likewise, varying subgraph expansion depth does not establish robustness to missing entities, incorrect relations, or entity-linking errors.

Questions for authors

(1)Can the authors provide a corrected derivation of Eqs. (11)–(13)? Which precise mutual-information bound is optimized, and how does the implemented loss correspond to it?

(2)Are ground-truth answers or answer entities used at any stage of test-time candidate generation, subgraph construction, or retrieval? Please clarify the reference to “questions and answers” in Appendix B.8 and provide separate training and inference algorithms.

(3) What constitutes the target YY for fact verification? Is it a class label, a textual label description, or an evidence-based justification? How is target similarity defined, and how are test-time labels and justifications excluded from retrieval?

(4) How are gradients propagated through discrete triple selection and subsequent textual encoding? Please specify the sampling mechanism, gradient estimator, trainable and frozen modules, decoder architecture, and optimization schedule.

(5)How does the full objective compare with simpler answer-aware retrieval objectives? In particular, what are the results for BCE plus semantic similarity, BCE plus reconstruction, and parameter-matched regularization baselines?

(6)Does the reported 6.6-second query time include MSCA? What is the complete latency for an unseen, uncached query, including entity extraction, alignment, retrieval, and answer generation?

(7) What do the reported uncertainty estimates represent? Please specify the number of runs, sources of randomness, retriever training seeds, and whether the reported values are standard deviations or standard errors, given that the generation temperature and random seed are fixed to zero.

(8)Can the authors reconcile the resource descriptions and experimental records? Please report the exact PrimeKG version and preprocessing, clarify the Table 2 ablation labels, and identify which qualitative and dataset examples are authentic benchmark instances versus constructed illustrations.

Reviewer_95Z2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes IBGraphRAG, a medical knowledge graph retrieval framework that combines Medical Semantic Consistency Alignment (MSCA) and Information Bottleneck-based Reasoning Path selection (IBRP). MSCA uses biomedical knowledge to improve entity alignment, whereas IBRP aims to select more informative reasoning paths rather than relying primarily on shortest-path heuristics. The authors evaluate the framework on several medical QA and fact-verification benchmarks and report improvements over existing RAG and GraphRAG approaches.

Critical strengths
  1. Relevant problem in medical knowledge retrieval. The paper focuses on two important challenges in medical GraphRAG: accurate medical entity alignment and the identification of relevant evidence from knowledge graphs. Both are meaningful problems, particularly when closely related medical concepts can lead to different interpretations.

  2. Good empirical evaluation. The authors evaluate the approach across multiple medical benchmarks and LLM backbones. The results show consistent improvements over several baselines, and the ablation studies suggest that both proposed components contribute to the overall performance.

Critical weaknesses
  1. Limited methodological novelty. My main concern is the novelty of the proposed approach. Biomedical entity alignment and information-bottleneck-based optimization are established techniques. While their combination is reasonable, I am not fully convinced that the paper introduces a sufficiently distinct learning method. The authors should clarify what is fundamentally different about this approach compared with existing semantic retrieval and representation compression approaches.

  2. The benefit of the information-bottleneck objective is not clearly established. The ablation studies show that IBRP improves performance, but they do not fully explain why. In particular, it is difficult to determine whether the gains stem from the information-bottleneck formulation or from the additional supervised path-ranking and semantic-similarity objectives. A direct comparison with simpler relevance-based or contrastive retrieval methods would help clarify this.

I also found the evidence for generalization somewhat limited. Strong performance on medical QA benchmarks does not necessarily indicate that the retrieved paths are medically meaningful or reliable in less-structured settings.

Questions for authors
  1. What does the information-bottleneck objective contribute beyond standard relevance-based or contrastive retrieval? Can the authors provide a controlled comparison using the same retriever architecture?

  2. How does MSCA compare with simpler UMLS-based entity linking or biomedical embedding similarity methods?

  3. Can the authors report retrieval-level metrics, such as Recall@K or evidence-path precision, to evaluate whether IBRP actually selects better reasoning paths?

  4. How robust is the approach to unseen medical terminology, ambiguous entity mentions, or different medical knowledge graphs?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.