- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper addresses missing relations in constructed knowledge graphs (KGs) that limit access to question-relevant evidence in GraphRAG. The authors show that only 14.5% of annotated evidence triples are preserved in the Medical graph, while approximately 61% have both entities present but lack the connecting relation. To address this issue, they propose a unified GAT-based enhancement framework that combines relation prediction with Query-Adaptive Gating (QAG). Predicted relations are utilized through two complementary paths: structural augmentation selectively insets edges before retrieval, while verbal augmentation provides ranked relational hints to the answer generator after retrieval. Experiments show that combining both paths improves answer correctness by 7.94 and 5.54 points on Medical and Novel, respectively, compared with a reranked RAG baseline, with consistent improvements across multiple retrieval systems. The analysis further demonstrates that indiscriminate edge insertion can degrade performance, highlighting the importance of query-conditioned relation selection.
Critical strengths
- Well-motivated distinction between relation prediction and downstream utility. The paper highlights that accurately predicting missing relations does not necessarify imporve downstream QA performance. The results show that indiscriminately inserting predicted edges degrades performance on Medical ( points), while query-adaptive selection improves it ( points). This provides empirical motivation for selecting relations based on their usefulness to the current question.
- Complementary structual and verbal augmentation. The framework distinguishes between using predicted relations to improve evidence retrieval and providing relational hints to supports answer generation. The stage-level analysis supports this distinction, showing improvements in retrieval Recall@5 from to through structural augmentation and in answer correctness from to under fixed gold passages through verbal augmentation.
- Extensive experimental comparisons across retrieval systems. The paper includes several meaningful controls, such as unfiltered edge insertion, confidence-based filtering, existing-edge hints, and BM25-based source sentences. The combined approach improves answer correctness by and points on Medical and Novel, respectively, with consistent improvements across six retrieval configurations. These comparisons help demonstrate the practical utility of the proposed augmentation framework
Critical weaknesses
- Overlapping training and evaluation questions. The GNN is trained using question-specific annotated evidence triples, yet the main end-to-end QA evaluation includes questions from the GNN training partition (Section 4.1 and Appendix C). Consequently, the reported improvements may benefit from question-specific evidence supervision encountered during training, making it difficult to assess generalization to unseen questions. This evaluation protocol substantially weakens the reliability of the main empirical claims. An independent held-out QA evaluation is necessary.
- Limited novelty and insufficient comparisons with closely related work. Knowledge incompleteness and missing relations in GraphRAG have already been investigated by TCR-QF [1] and CrossAug [2]. Therefore, the first claimed contribution appears to primarily quantify an established problem on specific datasets rather than identify a fundamentally new failure mode. Furthermore, ILlama [3] incorporates textualized knowledge graph relations into RAG, closely relating to the proposed verbal augmentation. These relevant studies are not adequately discussed or compared, making it difficult to establish the novelty and practical advantages of the proposed framework over existing approaches.
- Unclear contribution of semantic relation completion. Although recovering missing relations is presented as a central objective, Appendix E reports only approximately 67% relation-type prediction accuracy, and generic hints ("A is related to B") outperform explicitly typed relations. This raises the question of whether the observed improvements actually stem from recovering semantically meaningful relations or merely from identifying useful entity connections. A binary link prediction baseline that ignores relation types would help isolate the contribution of semantic relation completion.
- Insufficient factual grounding of predicted relations. Verbal augmentation directly provides GNN-predicted relations to the generator without guaranteeing that they are supported by source evidence. Appendix H reports that 26.4% of incorrect hints are adopted in the final answers. However, the paper does not sufficiently investigate how these errors affect factual faithfulness or introduce unsupported claims. Especially in the medical domain, improvements in aggregate answer correctness are insufficient to establish reliability without a systematic hallucination and evidence-grounding evaluation.
- Limited isolated benefit of the QAG module. Although QAG is presented as a central component, it improves over the best fixed-confidence filtering baseline by only 0.66 and 0.49 ACC points on Medical and Novel, respectively (Appendix D). Similarly, QAG-based verbal ranking provides only a 0.40-point improvement over learned query-relevance ranking on Medical (Appendix E). Without direct statistical significance tests for these relatively small differences, it remains unclear whether the additional gating module offers a meaningful advantage over simpler relation selection strategies.
- Insufficient implementation details for reproducibility. While Appendix B provides the GNN architecture and major training hyperparameters, several essential details remain unclear, including candidate relation construction, negative sampling for link prediction, supervision targets for individual prediction heads, and the construction of QAG training signals. Since relation selection is central to the proposed method, these missing details make it difficult to reproduce the framework and independently validate its reported improvements.
[1] Huang et al. How to mitigate information loss in knowledge graphs for graphrag: leveraging triple context restoration and query-driven feedback. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence (IJCAI '25). Article 901, 8104–8112. https://doi.org/10.24963/ijcai.2025/901
[2] Zhang, Jiaming, et al. "Beyond Chunk-Local Extraction: Cross-Chunk Graph Augmentation for GraphRAG." arXiv preprint arXiv:2605.28004 (2026).
[3] Park et al. "Leveraging Knowledge Graph-Enhanced LLMs for Context-Aware Medical Consultation." Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.
Questions for authors
Q1: Since the main end-to-end QA evaluation includes questions used for GNN training, I would suggest re-evaluating the results in Tabels 2 and 3 exclusively on held-out questions. Reporting the performance gap between training and held-out questions would help extablish whether the reported improvements generalize to unseen questions.
Q2: Appendix E reports approximately 67% relation-type prediction accuracy, while generic relational hints outperform explicitly typed relations. I would suggest adding a binary link prediction baseline without relation types to determine whether the improvements stem from semantically accurate relation completion or simply identifying useful entity connections.
Q3: Appendix H reports that 26.4% of incorrect hints are adopted in generated answers. It would be helpful to quantify whether these adopted hints lead to incorrect answers or unsupported claims, and to analyze failure cases where predicted relations contradict the source evidence.
Q4: Given that QAG improves over the best fixed-confidence filtering baseline by only 0.66 points on Medical and 0.49 points on Novel, I would suggest reporting statistical significance tests for these differences. Comparing QAG with fixed-confidence filtering under matched edge-retention rates would also help isolate the benefit of query-adaptive selection.
Q5: It would improve reproducibility to clarify how candidate relation sets are constructed, how negative samples are generated, and how supervision targets for the prediction heads and QAG are defined. In particular, explaining how question-specific annotated evidence is used during candidate construction and training would help clarify the proposed methodology.
2 comments
Thank you for the detailed response and additional experiments. While my concerns regarding soundness have been largely addressed, I still have reservations about the limited methodological novelty and relatively modest improvements over strong baselines. In particular, the statistical significance of these improvements remains unclear. Nevertheless, I appreciate the author's efforts and have raised my score.
Response to Reviewer AXCX: Held-Out Evaluation, Related Work, Binary LP, Faithfulness & Reproducibility
We thank Reviewer AXCX for the thorough review. Below we address all concerns with held-out results, baselines, binary LP ablations, faithfulness audits, significance tests, and reproducibility details.
1. Held-Out QA Re-Evaluation of Tables 2 & 3 (W1, Q1)
With GNN weights, early stopping, tg, and K frozen on the 60%/20% train/val splits, we compare pooled (n=509 / 609) and untouched 20% held-out test (n=102 / 122) scores (Tables 2 & 6):
.3917.3824(—).3833.3852(—).3394(-5.23).3369(-4.55).3845(+0.12).3886(+0.34).4198(+2.81).4129(+3.05).4057(+2.24).4042(+1.90).4264(+3.47).4146(+3.22).4106(+2.73).4153(+3.01).4412(+4.95).4358(+5.34).4138(+3.05).4120(+2.68).4711(+7.94).4559(+7.35, p=.0069).4387(+5.54).4364(+5.12, p=.0358)Held-out 95% CIs are
[+2.02,+12.68](Med) and[+0.34,+9.90](Nov), with a0.74/0.53-pt train–test gap. Across all 6 systems (Table 3), held-out gains are+7.35, +5.54, +5.98, +5.31, +6.19, +5.02(Med) and+3.28to+5.12(Nov). On zero-overlap unseen triples (32.4%Med /39.3%Nov), Dual Path gains+6.02/+4.25(Table 19).2. Comparison with TCR-QF, CrossAug & ILlama (W2)
We revised §§1 & 6 and added direct comparisons (Table 2):
0.4spass (<0.1sCPU GNN, Table 18) with zero LLM calls, beating sentence restoration (BM25:+1.86/+1.31held-out).14.5%Med coverage),Existing-edge hintsgains only+0.39/+0.62held-out..4559/.4364) beats CrossAug (.4298/.4132), Relink (.4278/.4167), TCR-QF (.4108/.4048), and existing-edge hints (.3863/.3914).3. Relation Completion vs. Binary LP (W3, Q2)
Comparing Untyped Binary GAT (ze=Wϕ[hu∥hv∥∣hu−hv∣],
203Kparams) vs. Typed GAT+QAG (.841/.527Med,.925/.691Nov Hits@10/MRR; Tables 7 & 20):.789/.462(Med) and.887/.624(Nov)..4311(+4.87) /.4094(+2.42) vs. Typed.4559(+7.35) /.4364(+5.12) (-2.48/-2.70pts)."A is related to B"(+4.95vs.+3.92typed at ~67% type acc., Table 9) avoids injecting label errors into prompts.4. Grounding, Hallucinations & Failure Cases (W4, Q3)
In our 100-question / 1,500-hint Medical audit (κ=.81, Table 16 & App. H.2): Hq reaches
91.7%factual precision (1,375/1,500;125incorrect). Of33(26.4%) adopted incorrect hints,31hit peripheral clauses while2.0%(2/100) flip correct-to-incorrect (Q-052: Amoxicillin → influenza; Q-207: Spironolactone → hypokalemia), vs.9.0%(9/100) incorrect-to-correct recoveries. Claim faithfulness rises from84.6%(Base) to89.2%(Dual) via stronger retrieval (Recall@5: .412 -> .463).5. Significance & Matched Retention (W5, Q4)
At matched
25%retention (Table 8), QAG (+3.47/+2.73) beats fixed-confidence top-25% (+2.34/+2.18) by+1.13(p=.0126, Med) and+0.55(p=.0330, Nov). In verbal ranking (Table 9), QAG (.4452,+5.35) and query-relevance (.4412,+4.95; p=.2847) both far beat LP ranking (+1.70).6. Reproducibility Details (W6, Q5)
<0.1sCPU; zero annotations).1:4corruptions; Lconf: BCE on E+ vs. hard LP negatives Ccal; Lrel: BCE on Tq alignment; LQAG: BCE on net PPR gain >τppr (α=0.15) on Vq∗. Judge:gpt-4.1-2025-04-14(temp=0, App. C).