- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors propose IBGraphRAG pipeline for medical QA to do improve factual consistency, built on a medical knowledge graph (PrimeKG), and add medical semantic consistency alignment (MSCA) which combines similarity to the query with average similarity to the other candidates and keeps only entities based on a specific threshold. They also propose Information Bottleneck-based Reasoning Path (IBRP), which is a subgraph retriever trained with BCE against shortest-path supervision, plus an IB term that approximates target relevance with cosine similarity and compression with a reconstruction decoder. IBGraphRAG reports gains of roughly 1–2 points over MedGraphRAG and SubgraphRAG.
Critical strengths
1- medical semantic consistency alignment idea of combining query relevance with coherence among candidate entities is simple and looks practical.
2- broad test across llm backbones and benchmarks.
Critical weaknesses
1- The second main claims of the paper (theoretical claim) is not new and also is not mathematically sound as they presented it here. The derivation of Eq. (6) and also Appendix A.2 is closely follow (Zhu et al., “An Information Bottleneck Perspective for Effective Noise Filtering on Retrieval-Augmented Generation” ACL 2024): which are almost identical in the notation and final objective (see equation 9 of the paper and others). That paper is not cited here. Also the adaptations of the equations has a major issue here. Appendix A.2 calls the decoder based term a variational upper bound on I(X~; X | Q) although Appendix A.3 says training the decoder maximizes a lower bound on the same quantity. Eq. (13) minimizes an expectation of a probability rather than a log-likelihood, and the step from Eq. (12) to Eq. (13) does not follow.
2- MSCA has two main issues that arent discussed in the paper. First, it is defined on question and answer entities, so how can it be used during inference without any answer entities? Also, MSCA needs about 67 seconds per sample (Appendix B.9). The authors say it can run offline as preprocessing, but for a new user query at inference time it cannot be precomputed, and this cost is not included in the 6.6 s/it query time reported in Table 12. This is not realistic or practical and If it really needs that much time, it should be stated as one of the main limitations of the paper; the limitations paragraph in Section 4.4 and Appendix B.12 only mention the IB overhead, not the MSCA cost.
3- The LLM used for the research are quire outdated (GPT4 and llama 2, etc) and the paper is not using frontier models or comparing against them
Questions for authors
1- Please correct derivation of the compression term: is it an upper or a lower bound, and on which quantity? and clarify your novelity vs (Zhu et al. ACL, 2024) 2- How is MSCA applied at test time without answer entities? What is the true end-to-end latency per query, including MSCA? and why that takes so long?
3- Which LLM performs entity extraction for each backbone?
4- Can you compare and use frontier models and more recent models of LLMs since the GPT4 and llama 2 and other used hre are quite old based on the timeline of the ICLR submission