acceptodds
Under review as a conference paper at ICLR 2027

Bridging the Completeness Gap in GraphRAG: A Unified GNN Enhancement Layer for Retrieval and Generation

Abstract

GraphRAG retrieves evidence from knowledge graphs automatically constructed from text, yet dense graphs can still omit the relations that downstream queries need: on GraphRAG-Bench Medical, a graph with an average degree of covers only of annotated evidence triples, and in of cases the relevant entities are present but the relation between them is missing. We introduce a lightweight query-adaptive GNN layer that predicts missing relations, estimates their confidence and query relevance, and selects them for structural augmentation and verbal hints. A structural path inserts query-adaptively filtered relations into the graph before Personalized PageRank (PPR) retrieval, while a verbal path passes top-ranked relations to the language model as concise hints, exploiting relational knowledge without propagating noisy edges through the graph. Filtering is decisive: inserting all predicted edges lowers Medical accuracy by points, whereas a learned query-adaptive gate turns this into a -point gain. The verbal path adds points, and the two paths together gain and points on Medical and Novel, respectively. The layer improves six retrieval-system configurations across the evaluated benchmark subsets and three language-model backbones on Medical, supporting selective structural and verbal augmentation as a practical way to address missing connections in this setting.

Primary area:
foundation or frontier models, including LLMs
Keywords:
GraphRAG, Retrieval-augmented generation, Knowledge graph completion, Graph neural networks, Large language models

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_AXCXOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper addresses missing relations in constructed knowledge graphs (KGs) that limit access to question-relevant evidence in GraphRAG. The authors show that only 14.5% of annotated evidence triples are preserved in the Medical graph, while approximately 61% have both entities present but lack the connecting relation. To address this issue, they propose a unified GAT-based enhancement framework that combines relation prediction with Query-Adaptive Gating (QAG). Predicted relations are utilized through two complementary paths: structural augmentation selectively insets edges before retrieval, while verbal augmentation provides ranked relational hints to the answer generator after retrieval. Experiments show that combining both paths improves answer correctness by 7.94 and 5.54 points on Medical and Novel, respectively, compared with a reranked RAG baseline, with consistent improvements across multiple retrieval systems. The analysis further demonstrates that indiscriminate edge insertion can degrade performance, highlighting the importance of query-conditioned relation selection.

Critical strengths
  • Well-motivated distinction between relation prediction and downstream utility. The paper highlights that accurately predicting missing relations does not necessarify imporve downstream QA performance. The results show that indiscriminately inserting predicted edges degrades performance on Medical (−5.23-5.23 points), while query-adaptive selection improves it (+3.47+3.47 points). This provides empirical motivation for selecting relations based on their usefulness to the current question.
  • Complementary structual and verbal augmentation. The framework distinguishes between using predicted relations to improve evidence retrieval and providing relational hints to supports answer generation. The stage-level analysis supports this distinction, showing improvements in retrieval Recall@5 from 0.4120.412 to 0.4630.463 through structural augmentation and in answer correctness from 0.6180.618 to 0.6590.659 under fixed gold passages through verbal augmentation.
  • Extensive experimental comparisons across retrieval systems. The paper includes several meaningful controls, such as unfiltered edge insertion, confidence-based filtering, existing-edge hints, and BM25-based source sentences. The combined approach improves answer correctness by 7.947.94 and 5.545.54 points on Medical and Novel, respectively, with consistent improvements across six retrieval configurations. These comparisons help demonstrate the practical utility of the proposed augmentation framework
Critical weaknesses
  1. Overlapping training and evaluation questions. The GNN is trained using question-specific annotated evidence triples, yet the main end-to-end QA evaluation includes questions from the GNN training partition (Section 4.1 and Appendix C). Consequently, the reported improvements may benefit from question-specific evidence supervision encountered during training, making it difficult to assess generalization to unseen questions. This evaluation protocol substantially weakens the reliability of the main empirical claims. An independent held-out QA evaluation is necessary.
  2. Limited novelty and insufficient comparisons with closely related work. Knowledge incompleteness and missing relations in GraphRAG have already been investigated by TCR-QF [1] and CrossAug [2]. Therefore, the first claimed contribution appears to primarily quantify an established problem on specific datasets rather than identify a fundamentally new failure mode. Furthermore, ILlama [3] incorporates textualized knowledge graph relations into RAG, closely relating to the proposed verbal augmentation. These relevant studies are not adequately discussed or compared, making it difficult to establish the novelty and practical advantages of the proposed framework over existing approaches.
  3. Unclear contribution of semantic relation completion. Although recovering missing relations is presented as a central objective, Appendix E reports only approximately 67% relation-type prediction accuracy, and generic hints ("A is related to B") outperform explicitly typed relations. This raises the question of whether the observed improvements actually stem from recovering semantically meaningful relations or merely from identifying useful entity connections. A binary link prediction baseline that ignores relation types would help isolate the contribution of semantic relation completion.
  4. Insufficient factual grounding of predicted relations. Verbal augmentation directly provides GNN-predicted relations to the generator without guaranteeing that they are supported by source evidence. Appendix H reports that 26.4% of incorrect hints are adopted in the final answers. However, the paper does not sufficiently investigate how these errors affect factual faithfulness or introduce unsupported claims. Especially in the medical domain, improvements in aggregate answer correctness are insufficient to establish reliability without a systematic hallucination and evidence-grounding evaluation.
  5. Limited isolated benefit of the QAG module. Although QAG is presented as a central component, it improves over the best fixed-confidence filtering baseline by only 0.66 and 0.49 ACC points on Medical and Novel, respectively (Appendix D). Similarly, QAG-based verbal ranking provides only a 0.40-point improvement over learned query-relevance ranking on Medical (Appendix E). Without direct statistical significance tests for these relatively small differences, it remains unclear whether the additional gating module offers a meaningful advantage over simpler relation selection strategies.
  6. Insufficient implementation details for reproducibility. While Appendix B provides the GNN architecture and major training hyperparameters, several essential details remain unclear, including candidate relation construction, negative sampling for link prediction, supervision targets for individual prediction heads, and the construction of QAG training signals. Since relation selection is central to the proposed method, these missing details make it difficult to reproduce the framework and independently validate its reported improvements.

[1] Huang et al. How to mitigate information loss in knowledge graphs for graphrag: leveraging triple context restoration and query-driven feedback. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence (IJCAI '25). Article 901, 8104–8112. https://doi.org/10.24963/ijcai.2025/901

[2] Zhang, Jiaming, et al. "Beyond Chunk-Local Extraction: Cross-Chunk Graph Augmentation for GraphRAG." arXiv preprint arXiv:2605.28004 (2026).

[3] Park et al. "Leveraging Knowledge Graph-Enhanced LLMs for Context-Aware Medical Consultation." Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

Questions for authors

Q1: Since the main end-to-end QA evaluation includes questions used for GNN training, I would suggest re-evaluating the results in Tabels 2 and 3 exclusively on held-out questions. Reporting the performance gap between training and held-out questions would help extablish whether the reported improvements generalize to unseen questions.

Q2: Appendix E reports approximately 67% relation-type prediction accuracy, while generic relational hints outperform explicitly typed relations. I would suggest adding a binary link prediction baseline without relation types to determine whether the improvements stem from semantically accurate relation completion or simply identifying useful entity connections.

Q3: Appendix H reports that 26.4% of incorrect hints are adopted in generated answers. It would be helpful to quantify whether these adopted hints lead to incorrect answers or unsupported claims, and to analyze failure cases where predicted relations contradict the source evidence.

Q4: Given that QAG improves over the best fixed-confidence filtering baseline by only 0.66 points on Medical and 0.49 points on Novel, I would suggest reporting statistical significance tests for these differences. Comparing QAG with fixed-confidence filtering under matched edge-retention rates would also help isolate the benefit of query-adaptive selection.

Q5: It would improve reproducibility to clarify how candidate relation sets are constructed, how negative samples are generated, and how supervision targets for the prediction heads and QAG are defined. In particular, explaining how question-specific annotated evidence is used during candidate construction and training would help clarify the proposed methodology.

2 comments
AuthorsOpenReview ↗

Response to Reviewer AXCX: Held-Out Evaluation, Related Work, Binary LP, Faithfulness & Reproducibility

We thank Reviewer AXCX for the thorough review. Below we address all concerns with held-out results, baselines, binary LP ablations, faithfulness audits, significance tests, and reproducibility details.

1. Held-Out QA Re-Evaluation of Tables 2 & 3 (W1, Q1)

With GNN weights, early stopping, tgt_g, and KK frozen on the 60%/20% train/val splits, we compare pooled (n=509n=509 / 609609) and untouched 20% held-out test (n=102n=102 / 122122) scores (Tables 2 & 6):

ConfigurationMed PooledMed Held-Out (Δ\Delta)Nov PooledNov Held-Out (Δ\Delta)
RAG + rerank.3917.3824 (—).3833.3852 (—)
Raw predicted.3394 (-5.23).3369 (-4.55).3845 (+0.12).3886 (+0.34)
Fixed-cutoff.4198 (+2.81).4129 (+3.05).4057 (+2.24).4042 (+1.90)
QAG structural.4264 (+3.47).4146 (+3.22).4106 (+2.73).4153 (+3.01)
Verbal path.4412 (+4.95).4358 (+5.34).4138 (+3.05).4120 (+2.68)
Dual path.4711 (+7.94).4559 (+7.35, p=.0069p=.0069).4387 (+5.54).4364 (+5.12, p=.0358p=.0358)

Held-out 95% CIs are [+2.02,+12.68] (Med) and [+0.34,+9.90] (Nov), with a 0.74/0.53-pt train–test gap. Across all 6 systems (Table 3), held-out gains are +7.35, +5.54, +5.98, +5.31, +6.19, +5.02 (Med) and +3.28 to +5.12 (Nov). On zero-overlap unseen triples (32.4% Med / 39.3% Nov), Dual Path gains +6.02/+4.25 (Table 19).

2. Comparison with TCR-QF, CrossAug & ILlama (W2)

We revised §§1 & 6 and added direct comparisons (Table 2):

  • TCR-QF (Huang et al., IJCAI'25): restores triple context via iterative query feedback; we predict unextracted links in one 0.4s pass (<0.1s CPU GNN, Table 18) with zero LLM calls, beating sentence restoration (BM25: +1.86/+1.31 held-out).
  • CrossAug (Zhang et al., 2026): offline LLM edge completion; we run online query-adaptive selection with zero LLM indexing calls, decoupling pre-retrieval PPR gating from post-retrieval hints.
  • ILlama (Park et al., EMNLP'25): verbalizes existing curated KG paths; on sparse extracted KGs (14.5% Med coverage), Existing-edge hints gains only +0.39/+0.62 held-out.
  • Held-Out Med/Nov ACC: Ours (.4559/.4364) beats CrossAug (.4298/.4132), Relink (.4278/.4167), TCR-QF (.4108/.4048), and existing-edge hints (.3863/.3914).

3. Relation Completion vs. Binary LP (W3, Q2)

Comparing Untyped Binary GAT (ze=Wϕ[hu∥hv∥∣hu−hv∣]z_e=W_\phi[h_u\parallel h_v\parallel |h_u-h_v|], 203K params) vs. Typed GAT+QAG (.841/.527 Med, .925/.691 Nov Hits@10/MRR; Tables 7 & 20):

  • Hits@10/MRR: Binary GAT drops to .789/.462 (Med) and .887/.624 (Nov).
  • Held-Out Dual ACC: Binary GAT+QAG drops to .4311 (+4.87) / .4094 (+2.42) vs. Typed .4559 (+7.35) / .4364 (+5.12) (-2.48/-2.70 pts).
  • Insight: ERE_{\mathcal{R}} regularizes GAT representations against co-occurrence noise, while "A is related to B" (+4.95 vs. +3.92 typed at ~67% type acc., Table 9) avoids injecting label errors into prompts.

4. Grounding, Hallucinations & Failure Cases (W4, Q3)

In our 100-question / 1,500-hint Medical audit (κ=.81\kappa=.81, Table 16 & App. H.2): HqH_q reaches 91.7% factual precision (1,375/1,500; 125 incorrect). Of 33 (26.4%) adopted incorrect hints, 31 hit peripheral clauses while 2.0% (2/100) flip correct-to-incorrect (Q-052: Amoxicillin → influenza; Q-207: Spironolactone → hypokalemia), vs. 9.0% (9/100) incorrect-to-correct recoveries. Claim faithfulness rises from 84.6% (Base) to 89.2% (Dual) via stronger retrieval (Recall@5: .412 -> .463).

5. Significance & Matched Retention (W5, Q4)

At matched 25% retention (Table 8), QAG (+3.47/+2.73) beats fixed-confidence top-25% (+2.34/+2.18) by +1.13 (p=.0126p=.0126, Med) and +0.55 (p=.0330p=.0330, Nov). In verbal ranking (Table 9), QAG (.4452, +5.35) and query-relevance (.4412, +4.95; p=.2847p=.2847) both far beat LP ranking (+1.70).

6. Reproducibility Details (W6, Q5)

  • CqC_q (§3.1, App. B.1): From seeds VqV_q, we take unlinked 2-hop ∪\cup top-30 cosine pairs, pre-rank to top-120 via mean relation eˉR\bar{e}_{\mathcal{R}}, and set r^=arg⁡max⁡r∈RfLP(ϕ(hu,r,hv))\hat{r}=\arg\max_{r\in\mathcal{R}}f_{\text{LP}}(\phi(h_u,r,h_v)) (∣Cq∣=120|C_q|=120, <0.1s CPU; zero annotations).
  • Losses (App. B.1, Eqs. 9–12): LLP\mathcal{L}_{\text{LP}}: BCE on 1:4 corruptions; Lconf\mathcal{L}_{\text{conf}}: BCE on E+\mathcal{E}^+ vs. hard LP negatives Ccal\mathcal{C}_{\text{cal}}; Lrel\mathcal{L}_{\text{rel}}: BCE on Tq\mathcal{T}_q alignment; LQAG\mathcal{L}_{\text{QAG}}: BCE on net PPR gain >τppr\gt \tau_{\text{ppr}} (α=0.15\alpha=0.15) on Vq∗V_q^\ast. Judge: gpt-4.1-2025-04-14 (temp=0, App. C).

Thank you for the detailed response and additional experiments. While my concerns regarding soundness have been largely addressed, I still have reservations about the limited methodological novelty and relatively modest improvements over strong baselines. In particular, the statistical significance of these improvements remains unclear. Nevertheless, I appreciate the author's efforts and have raised my score.

Reviewer_6HbfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses missing relationships in automatically constructed knowledge graphs used by GraphRAG systems. The authors propose a GNN enhancement layer that predicts missing relations and selectively uses them for structural augmentation during retrieval or as verbal hints during generation. The framework combines GAT-based link prediction, confidence estimation, and query-adaptive gating (QAG). Experiments on GraphRAG-Bench Medical and Novel report improvements of 7.94 and 5.54 accuracy points, respectively. The authors also evaluate the approach across six retrieval configurations and three LLM backbones.

Critical strengths
  1. Important problem with clear motivation. The paper identifies a meaningful limitation of GraphRAG. Table 1 shows that the Medical graph preserves only 14.5% of annotated evidence triples, while approximately 61% of evidence occurrences contain the relevant entities but lack their connecting relationships. This provides a concrete motivation for addressing graph incompleteness.

  2. Useful experimental analysis. The experiments demonstrate that indiscriminately inserting predicted edges can reduce accuracy, whereas selective augmentation improves performance. The comparisons among raw edge insertion, confidence filtering, QAG, and verbal hints are informative. Improvements across multiple retrieval systems further support the practical usefulness of the proposed framework.

Critical weaknesses
  1. Limited methodological novelty. My main concern is whether the proposed framework makes a sufficiently distinct contribution to graph learning. The method combines established techniques, including GAT-based link prediction, query-aware scoring, and learned gating. Although separating structural and verbal augmentation is useful, the contribution appears to lie primarily in integrating existing methods rather than introducing a new GNN architecture or learning objective.

The paper provides several selection ablations, but it does not directly establish whether joint GNN learning is more effective than a simpler pipeline combining a conventional link predictor with independently trained query-aware reranking. This comparison is important for substantiating the proposed methodological contribution.

  1. Limited evidence of downstream generalization and factual reliability. Section 4.1 describes a 60/20/20 split for GNN training, validation, and testing, while the principal answer-quality results are reported over the complete study subsets. Although the evaluation protocol is described, aggregating training-partition and held-out questions makes it difficult to assess downstream generalization independently. Separate answer-quality results on held-out questions would strengthen the conclusions.

The reliability of predicted relations is also a concern. Section 5.2 reports that 26.4% of incorrectly predicted hints were incorporated into generated answers. Although most incorrect hints were ignored, this indicates that verbal augmentation can propagate erroneous relationships. The paper would benefit from a more direct assessment of answer-level factual reliability.

Questions for authors
  1. Can the authors compare the joint GNN framework with a standard link predictor followed by independently trained query-aware reranking under comparable conditions?
  2. Can the authors report downstream answer accuracy separately for held-out test questions, excluding those used for GNN supervision?
  3. What is the precision of relations selected for structural and verbal augmentation, and how frequently do incorrect predictions lead to unsupported answers?
  4. How well does the framework generalize to independently constructed knowledge graphs without annotated evidence-triple supervision?
1 comment
AuthorsOpenReview ↗

Response to Reviewer 6Hbf: Joint vs. Staged Training, Held-Out Generalization, and Factual Reliability

We thank Reviewer 6Hbf for the constructive review. Below we compare joint GNN representation learning against decoupled link-prediction + reranking pipelines, report separate held-out test accuracy, audit selection precision and factual reliability, and test generalization on independently constructed KGs.

1. Joint Multi-Head GNN vs. Conventional Link Predictor + Independent Reranker (W1, W3, Q1)

Beyond improving link ranking itself (GAT: .836/.519->.841/.527 Med, .921/.682->.925/.691 Nov Hits@10/MRR; Table 7) and outperforming pure LP-score (+1.70) or cosine (+3.62) verbal ranking (Table 9), we ran the exact decoupled two-stage baselines requested: freezing RotatE or Standard GAT and training an independent query-aware reranker & gate under matched parameters (~245K) and identical candidate pools (∣Cq∣=120|C_q|=120; Table 20, Appendix J):

Pipeline ConfigurationMed PooledMed Held-Out (Δ\Delta)Nov PooledNov Held-Out (Δ\Delta)
RAG + rerank (Baseline).3917.3824 (—).3833.3852 (—)
RotatE LP + Indep. Reranker & Gate.4385.4226 (+4.02).4082.4011 (+1.59)
Frozen Standard GAT LP + Indep. Reranker & Gate.4502.4349 (+5.25).4175.4108 (+2.56)
Untyped Binary GAT + QAG.4473.4311 (+4.87).4168.4094 (+2.42)
Joint Typed GAT + QAG (Ours).4711.4559 (+7.35).4387.4364 (+5.12)

Joint optimization outperforms Frozen GAT + independent reranker/gate by +2.10 held-out ACC points on Medical (p=.0072p=.0072) and +2.56 on Novel (p<.01p \lt .01). Functional head separation is formalized in Appendix B.1 (Eqs. 9–12): LLP\mathcal{L}_{\text{LP}} learns relation plausibility against 1:4 corruptions, Lconf\mathcal{L}_{\text{conf}} calibrates precision on hard LP-proposed candidates Ccal\mathcal{C}_{\text{cal}}, Lrel\mathcal{L}_{\text{rel}} supervises semantic evidence alignment, and LQAG\mathcal{L}_{\text{QAG}} supervises PPR propagation gain on target evidence nodes.

2. Downstream Accuracy on Held-Out Test Questions (W2, Q2)

With all GNN weights, early stopping, tgt_g, and KK frozen strictly on the 60% train / 20% val splits:

  • Main ablations (20% held-out test split; Tables 2 & 6): On Medical (n=102n=102), baseline ACC (.3824) rises to .4146 (+3.22, QAG structural), .4358 (+5.34, verbal), and .4559 (+7.35, CI [+2.02,+12.68], p=.0069p=.0069) with Dual Path; on Novel (n=122n=122), baseline (.3852) rises to .4153 (+3.01), .4120 (+2.68), and .4364 (+5.12, CI [+0.34,+9.90], p=.0358p=.0358). On held-out questions with zero training triple overlap (32.4% Med / 39.3% Nov), Dual Path still gains +6.02 / +4.25 (Table 19).
  • Cross-system & sensitivity checks (Tables 3a & 21): Held-out dual-path gains across all 6 retrieval systems span +5.02 to +7.35 on Medical and +3.28 to +5.12 on Novel; sweeps over β\beta, α\alpha, and dd stay stable within <1.8 ACC points.

3. Precision of Selected Relations & Answer-Level Factual Reliability (W2, Q3)

Extending our 100-question / 1,500-hint Medical audit (κ=.81\kappa=.81, Table 16):

  • Selected relation precision: Factual edge precision rises from 22.4% in unfiltered CqC_q to 58.7% in QAG structural edges SqS_q (~25% retention) and 91.7% in top-15 verbal hints HqH_q (1,375/1,500 factually valid; 125 incorrect).
  • Answer flips & claim faithfulness: Although 33 of 125 incorrect hints (26.4%) appear in answers, 31 attach only to peripheral clauses; only 2.0% (2/100) of questions incur a correct-to-incorrect flip (Appendix H.2: Q-052, Q-207), outweighed by 9.0% (9/100) incorrect-to-correct recoveries. Corpus-grounded claim faithfulness rises from 84.6% (Base) to 89.2% (Dual), as structural retrieval gains (Recall@5: .412->.463, Table 15) suppress unsupported hallucinations.

4. Generalization to Independently Constructed KGs Without Evidence Annotations (Q4)

Using our self-supervised existing-graph-edge mode (20% masked edges + source-sentence pseudo-queries, zero evidence annotations; Appendices I & J, Tables 17 & 22), Dual Path transfers across independent KG extractors and external multi-hop QA corpora:

Graph Source / Benchmark (Zero Evidence Annotations)Base ACC (EM / F1)+ Self-Sup. DualGain
GB Medical Held-Out (Self-Supervised)38.243.5+5.31
Independent Medical KG (REBEL + Llama-3-70B).758 Hits@10—+4.87
Full GraphRAG-Bench (RAG + rerank)37.5 (1.9/22.4)42.4 (2.6/24.0)+4.9
HotpotQA / 2Wiki / MuSiQue (HippoRAG2)64.8/58.3/36.767.9/62.0/40.8+3.1/+3.7/+4.1
Reviewer_GmpQOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to address the issue of missing entity relationships in knowledge graph-based retrieval-augmented generation (KG-based RAG) systems. The authors formulate missing relationship completion as a link prediction task and propose a lightweight GNN-based enhancement layer to predict and recover missing relationships. The proposed module can be integrated into different KG-based RAG frameworks. The method is evaluated on a subset of GraphRAG-Bench, and the experimental results demonstrate improvements across several KG-based RAG systems.

Critical strengths
  1. A reasonable formulation of the problem. The paper formulates missing relationships in knowledge graphs as a link prediction task and develops an algorithm to identify and recover missing relationships.
  2. Lightweight and modular design. The proposed method is lightweight and can be integrated into existing KG-based RAG frameworks, making it potentially applicable to a broad range of systems.
  3. A useful diagnostic analysis. The paper introduces a diagnostic approach to identify whether certain question-answering failures are caused by missing relationships between entities. This analysis provides useful insights into the impact of incomplete knowledge graphs on KG-based RAG performance.
Critical weaknesses
  1. Limited novelty of the core motivation. The central motivation of this paper, namely addressing missing relationships in knowledge graphs to improve GraphRAG performance, has already been discussed in previous work, such as Relink [1]. The authors should better clarify the differences between their approach and existing methods, particularly in terms of the problem formulation and technical contributions.
  2. Limited evaluation on GraphRAG-Bench. Although the paper evaluates its method on GraphRAG-Bench, it only uses a subset of the benchmark. The authors do not provide a convincing explanation of why the remaining datasets are excluded or unsuitable for evaluating the proposed method.
  3. Lack of evaluation on established multi-hop QA benchmarks. Given that the proposed method is designed as a general enhancement module for KG-based RAG systems, its effectiveness should also be evaluated on widely used multi-hop question-answering benchmarks, such as 2WikiMultiHopQA, HotpotQA, and MuSiQue. The current evaluation is insufficient to demonstrate its generalizability across different tasks and datasets.
  4. Insufficiently convincing baselines. The paper does not include sufficiently strong and relevant baselines for comparison. In particular, comparisons with existing approaches that address missing relationships or improve evidence graph construction are necessary to establish the effectiveness and advantages of the proposed method.
  5. Insufficient evaluation metrics. Since the proposed method serves as an auxiliary module for KG-based RAG systems, the evaluation should include more comprehensive and convincing metrics to demonstrate its actual impact on downstream question-answering performance. For example, commonly used metrics such as F1 and Exact Match (EM), as well as LLM-as-a-Judge evaluations, should be reported. The current evaluation does not provide sufficient evidence that the proposed method consistently improves answer quality.

[1] Relink: Constructing Query-Driven Evidence Graph On-the-Fly for GraphRAG

Questions for authors
  1. Could the authors provide experimental results on a broader range of datasets, including the full GraphRAG-Bench or other established multi-hop QA benchmarks? This would help demonstrate the generalizability of the proposed method.
  2. Could the authors evaluate the proposed method using different LLM backbones? The current experiments only use Qwen-Plus, making it difficult to assess whether the improvements are consistent across different language models.
  3. Could the authors provide parameter sensitivity experiments to investigate how different hyperparameter configurations affect the performance of the proposed method?
  4. Could the authors include Relink [1] as a baseline? Since the code of Relink is publicly available, incorporating it into the experimental comparison should be feasible and would provide a more convincing evaluation of the proposed method against closely related work.
2 comments
AuthorsOpenReview ↗

Response to Reviewer GmpQ: Relink Comparison, Expanded Benchmarks & Metrics, and Multi-LLM Results

We thank Reviewer GmpQ for the constructive suggestions. We have added direct comparisons with Relink (and contemporary completion baselines), expanded evaluation to the complete GraphRAG-Bench and three multi-hop QA benchmarks, added EM and F1 metrics, and highlighted our multi-LLM and sensitivity experiments.

1. Conceptual & Empirical Comparison with Relink (W1, W4, Q4)

  • Distinction from Relink (AAAI 2026): Relink queries a corpus-extracted latent relation pool to construct a single query-driven evidence graph. In contrast, our 245K-parameter GNN (<0.1s CPU inference, +0.4s total latency; Table 18) predicts missing links over existing entities with zero LLM completion calls and explicitly decouples downstream routing into two stages: QAG filters edges before PPR retrieval to prevent shortcut corruption (raw insertion drops Medical ACC by -5.23, whereas QAG gains +3.47), while verbal ranking passes top-KK edges as concise hints after retrieval (+4.95), outperforming source-sentence restoration (Entity-pair BM25: +2.25 Med / +1.64 Nov, Table 2).
  • Direct comparison (Tables 2 & 3): Under our shared graph and frozen Qwen-Plus setup, our dual-path layer outperforms Relink, CrossAug, and TCR-QF on both full subsets and untouched 20% held-out splits:
MethodMed All ACC (Δ\Delta)Med Held-Out ACC (Δ\Delta)Nov All ACC (Δ\Delta)Nov Held-Out ACC (Δ\Delta)
RAG + rerank (Base).3917 (—).3824 (—).3833 (—).3852 (—)
TCR-QF (IJCAI 2025).4231 (+3.14).4108 (+2.84).4090 (+2.57).4048 (+1.96)
CrossAug (2026).4358 (+4.41).4298 (+4.74).4136 (+3.03).4132 (+2.80)
Relink (AAAI 2026).4409 (+4.92).4278 (+4.54).4203 (+3.70).4167 (+3.15)
Ours (+ RAG + rerank).4711 (+7.94).4559 (+7.35).4387 (+5.54).4364 (+5.12)
Ours (+ HippoRAG2).5756 (+6.32)+5.54 gain.5247 (+4.19)+4.41 gain

2. Full GraphRAG-Bench & Multi-Hop QA Benchmarks (W2, W3, Q1)

The 509/609 subsets were initially used because Section 2's coverage audit required question-level evidence-triple annotations. Using our annotation-free self-supervised graph-edge mode (§5.3 / Appendix I, masking 20% of extracted edges with source-sentence pseudo-queries), we evaluate on complete GraphRAG-Bench and HotpotQA, 2WikiMultiHopQA, MuSiQue (Table 22, Appendix J):

Benchmark (Base System)Baseline (EM / F1 / ACC)+ Ours Dual (EM / F1 / ACC)Gain (EM / F1 / ACC)
GB Medical Held-Out (RAG+rerank)1.0 / 22.0 / 38.22.9 / 24.5 / 45.6+1.9 / +2.5 / +7.4
GB Novel Held-Out (RAG+rerank)1.6 / 23.8 / 38.52.5 / 25.9 / 43.6+0.9 / +2.1 / +5.1
Full GraphRAG-Bench (self-sup.)1.9 / 22.4 / 37.52.6 / 24.0 / 42.4+0.7 / +1.6 / +4.9
HotpotQA (HippoRAG2, self-sup.)58.7 / 71.3 / 64.861.5 / 73.8 / 67.9+2.8 / +2.5 / +3.1
2WikiMultiHopQA (HippoRAG2, self-sup.)52.4 / 61.8 / 58.355.9 / 65.1 / 62.0+3.5 / +3.3 / +3.7
MuSiQue (HippoRAG2, self-sup.)30.1 / 41.6 / 36.733.0 / 44.2 / 40.8+2.9 / +2.6 / +4.1

3. Comprehensive Evaluation Metrics (W5)

Alongside ROUGE-L (Appendix E.2: .1842->.1897 on Medical, where mechanistic answer expansion can lower lexical overlap despite higher correctness; Table 11), Table 22 reports EM, token-level F1, and LLM-Judge ACC (gpt-4.1-2025-04-14, temperature=0, GraphRAG-Bench rubric; Spearman ρ=0.97\rho=0.97 with gpt-4o-mini-2024-07-18). EM is naturally low (1–3%) on free-form GraphRAG-Bench paragraphs, whereas short-span multi-hop QA shows +2.8 to +3.5 EM gains.

4. Multi-LLM Backbones & Hyperparameter Sensitivity (Q2, Q3)

  • Three LLM backbones (Q2; Table 3b & Table 14): Without LLM fine-tuning:
    • Qwen-Plus: Med .3917->.4711 (+7.94; Held-out +7.35) | Nov .3833->.4387 (+5.54; Held-out +5.12)
    • GPT-4o: Med .4673->.5327 (+6.54; Held-out +6.98) | Nov .4418->.4915 (+4.97; Held-out +4.51)
    • Llama-3-70B-Instruct: Med .4198->.4892 (+6.94; Held-out +6.19) | Nov .4105->.4644 (+5.39; Held-out +5.86)
  • Parameter sensitivity (Q3; Tables 8, 10 & 21): Beyond cutoffs t∈[.60,.95]t\in[.60,.95] (Table 8) and hint budgets K∈{5,…,50}K\in\lbrace 5,\dots,50 \rbrace (Table 10), sweeping QAG loss weight β∈{0.1,0.25,0.5,1.0}\beta\in\lbrace 0.1,0.25,0.5,1.0 \rbrace, PPR restart α∈{0.10,0.15,0.20,0.30}\alpha\in\lbrace 0.10,0.15,0.20,0.30 \rbrace, and GAT dim d∈{64,128,256}d\in\lbrace 64,128,256 \rbrace yields stable held-out ACC within .4392–.4571 on Medical (.4559 default) and .4218–.4377 on Novel (.4364 default; Table 21).

Dear Authors and Area Chairs,

Thank you for your response.

While some of my concerns have been addressed, I still believe that the paper falls substantially short of the minimum standards for acceptance at ICLR. Therefore, I will maintain my original rating.

Furthermore, I strongly encourage the Area Chairs to carefully evaluate this submission, as I believe it remains well below the bar for acceptance.

Reviewer_kPKhOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper diagnoses missing question-relevant relations in extracted GraphRAG graphs and proposes a text-initialized GAT with link-prediction, confidence, query-relevance and query-adaptive gating heads. Selected predicted connections are inserted before retrieval and/or verbalized as entity-pair hints for a frozen generator. The combined method reports gains of 7.94 and 5.54 ACC points on Medical and Novel study subsets, with additional system, generator, selection and supervision controls. The downstream results explicitly include questions used for GNN training and validation, so they describe pooled study-subset performance rather than independently held-out-question generalization.

Critical strengths
  1. A concrete and informative completeness diagnosis. Medical has high sampled edge precision but a low fully matched question-evidence occurrences. This usefully distinguishes locally plausible graph edges from the evidence connections needed by questions, and motivates an intervention restricted to existing entities.

  2. Informative controls for the two uses of predicted relations. Raw insertion reduces Medical ACC by 5.23 points, whereas QAG-filtered insertion improves it by 3.47 and verbal augmentation by 4.95. Existing-edge, random-edge, BM25, ranking and hint-budget controls provide real attribution evidence.

Critical weaknesses
  1. The principal results cannot identify improvement on independently held-out questions. Section 4.1 and Appendix C explicitly say the 60/20/20 split governs GNN fitting and selection, but answer results cover all 509/609 questions, including the training and validation portions. Approximately 80% of the nominal evaluation therefore consists of questions involved in supervision or selection. Freezing the final generator does not remove that dependence from the learned predictor and selectors. The positive bootstrap intervals support the pooled average, not a separate unseen-question gain. This is a disclosed limitation, not evidence of hidden test-answer use or misconduct; learning an index on a shared retrieval corpus is not itself improper. Table 17's graph-edge-supervision result and Appendix H.1's analysis of 100 Medical test questions are genuine mitigating evidence. However, the former does not fully define annotation access for every query-conditioned head/selection step, and the latter reports hint adoption rather than baseline-versus-enhanced answer accuracy. The held-out downstream benefit remains undetermined.

  2. Core mechanism and measurement definitions are insufficient. Equations (2)–(4) and Appendix B name four heads and loss weights, but do not define each loss, its supervision targets and negatives, or the inference construction of the candidate pool C_q. The distinction between relevance and QAG training targets is particularly important: absence of generator gradients is not a defect, but the surrogate objectives must be specified. Appendix F names anchor expansion and edge-reward training without executable definitions. Appendix C defines ACC as correctness on a 0–1 scale without the scoring rule/evaluator or aggregation procedure. Architecture, optimization and threshold settings are provided, so this is not a blanket absence of implementation details. Nevertheless, the omitted definitions are central. The smaller QAG-over-fixed-cutoff gains of 0.66/0.49 points also lack uncertainty for that contrast; they should not be treated as independently established selector superiority.

Questions for authors
  1. Decisive validation: Please report baseline and dual-path answer ACC on the untouched question-test partition in both domains, with all fitting and choices frozen from training/validation data and paired uncertainty. If available, fixed-cutoff/QAG and single-path results on that same partition would clarify attribution. Repeated evidence triples across partitions should be documented to distinguish new-question from new-fact generalization, not automatically treated as invalidating the former.

  2. Learning/inference clarification: Please give the four losses, every head's target and negative construction, and the annotation-independent candidate-generation algorithm. For the graph-edge-supervision variant, which question/evidence labels, if any, are accessed by each head and by selection? Please also define Novel's anchor expansion and edge-reward objective.

  3. Measurement clarification: How is per-question ACC assigned, by what evaluator/rule, and how are runs and bootstrap samples aggregated? Paired per-question results or equivalent sufficient statistics for the untouched test partition would make the main effect auditable.

1 comment
AuthorsOpenReview ↗

Response to Reviewer kPKh: Held-Out Test Evaluation, Complete Formulations, and Measurement Protocol

We thank Reviewer kPKh for the rigorous feedback. We have revised the paper to report strictly held-out test results, formalize all training/inference definitions, and detail our measurement protocol.

1. Decisive Validation on the Untouched 20% Test Partition (W1, Q1)

All 245K GNN weights, early stopping, cutoffs (tgt_g; .85 Med / .80 Nov fixed), and budgets (K=15K=15 / 2020) are frozen solely on the 60% train and 20% val splits. Alongside pooled 509/609 scores (Table 2: .3917->.4711 [+7.94] Med; .3833->.4387 [+5.54] Nov), below are results on the untouched 20% held-out test split (Med n=102n=102, Nov n=122n=122; paired 95% bootstrap CIs, 10410^4 resamples; Tables 2 & 6):

ConfigurationMed Held-OutMed Δ\Delta [95% CI]Nov Held-OutNov Δ\Delta [95% CI]
RAG + rerank (Base).3824—.3852—
Raw predicted edges.3369-4.55.3886+0.34
Fixed-cutoff (.85/.80).4129+3.05.4042+1.90
QAG structural path.4146+3.22.4153+3.01
Query-relevance verbal.4358+5.34.4120+2.68
Dual path (QAG).4559+7.35 [+2.02,+12.68] (p=.0069p=.0069).4364+5.12 [+0.34,+9.90] (p=.0358p=.0358)
  • Cross-system held-out gains (Table 3a): Across all 6 systems, held-out dual-path gains span +5.02 to +7.35 on Medical (+7.35 RAG+rerank, +5.54 HippoRAG2, +5.98 G-reasoner, +5.31 LightRAG, +6.19 MS-GraphRAG, +5.02 Fast-GraphRAG) and +3.28 to +5.12 on Novel.
  • New-question vs. new-fact generalization (Table 19, Appendix J): 32.4% (33/102, Med) and 39.3% (48/122, Nov) of held-out questions require strictly unseen triples (zero train/val overlap). Dual-path gains are +6.02 (zero overlap) vs. +7.99 (seen triples in new questions) on Medical, and +4.25 vs. +5.69 on Novel.

2. Losses, Candidate Pool CqC_q, Self-Supervision & Novel Disambiguation (W2, Q2)

  • Annotation-free CqC_q (§3.1, Appendix B.1): Inference never accesses annotations. For each seed u∈Vqu\in V_q, we take unlinked (u,v)∉E(u,v)\notin E in N2(u)∪Top-30\mathcal{N}_2(u)\cup\text{Top-30} cosine neighbors (d(u)=d(v)d(u)=d(v) on Novel). With ϕ(hu,r,hv)=Wϕ[hu⊙er∥hv⊙er∥∣hu−hv∣⊙er]\phi(h_u,r,h_v)=W_\phi[h_u\odot e_r \parallel h_v\odot e_r \parallel |h_u-h_v|\odot e_r] and mean relation eˉR=1∣R∣∑rer\bar{e}_{\mathcal{R}}=\frac{1}{|\mathcal{R}|}\sum_r e_r, one batched pass fLP(ϕ(hu,eˉR,hv))f_{\text{LP}}(\phi(h_u,\bar{e}_{\mathcal{R}},h_v)) keeps top-120 pairs before evaluating r^=arg⁡max⁡r∈RfLP(ϕ(hu,r,hv))\hat{r}=\arg\max_{r\in\mathcal{R}}f_{\text{LP}}(\phi(h_u,r,h_v)) (∣Cq∣=120|C_q|=120, <0.1s CPU).
  • Four head losses (§3.1, Appendix B.1, Eqs. 9–12):
    1. LLP\mathcal{L}_{\text{LP}}: BCE on E+\mathcal{E}^+ and 1:4 negatives E−\mathcal{E}^- (2 endpoint + 2 relation corruptions per positive).
    2. Lconf\mathcal{L}_{\text{conf}}: BCE calibrating ce∈(0,1)c_e\in(0,1) on Ccal\mathcal{C}_{\text{cal}} (E+\mathcal{E}^+ vs. hard LP negatives (u,r^,v)∉E(u,\hat{r},v)\notin E; .93–.95 AUC).
    3. Lrel\mathcal{L}_{\text{rel}} vs. LQAG\mathcal{L}_{\text{QAG}}: Lrel\mathcal{L}_{\text{rel}} optimizes BCE on CqC_q for endpoint alignment ye,qrel=I[(u,v)∈Tq]y_{e,q}^{\text{rel}}=\mathbb{I}[(u,v)\in\mathcal{T}_q] (.74–.80 nDCG@10), whereas LQAG\mathcal{L}_{\text{QAG}} optimizes BCE for PPR gain ye,qQAG=I[∑v∈Vq∗(πG∪{e},q(v)−πG,q(v))>τppr]y_{e,q}^{\text{QAG}}=\mathbb{I}[\sum_{v\in V_q^\ast}(\pi_{G\cup\lbrace e \rbrace,q}(v)-\pi_{G,q}(v)) \gt \tau_{\text{ppr}}] (α=0.15\alpha=0.15) on target nodes Vq∗V_q^\ast.
  • Graph-edge self-supervision (Appendix I, Table 17): Zero evidence annotations are used. We mask 20% of edges (15% train, 5% val, 80% G′G^{\prime}); uu's source sentence forms a pseudo-query and vv's passage supervises Lrel\mathcal{L}_{\text{rel}}, LQAG\mathcal{L}_{\text{QAG}}, and tgt_g, achieving +5.31 held-out Medical gain (+5.87 pooled) and +4.87 on an independent REBEL + Llama-3-70B KG.
  • Novel disambiguation (Appendix F, Eq. 13): Anchor expansion sets V~q=Vq(0)∪{v∣∃u∈Vq(0),d(v)=d(u)∧((u,v)∈E∨cos⁡(xu,xv)≥τanc)}\tilde{V}_q = V_q^{(0)} \cup \lbrace v \mid \exists u\in V_q^{(0)}, d(v)=d(u) \land ((u,v)\in E \lor \cos(x_u,x_v)\ge \tau_{\text{anc}}) \rbrace; normalized same-doc common neighbors CN(u,v)\text{CN}(u,v) (λ=0.5\lambda=0.5) couple both paths via ρ~e(q)=σ(se(q))+λCN(u,v)\tilde{\rho}_e(q)=\sigma(s_e(q))+\lambda\text{CN}(u,v) and we(q)=σ(fgate([ze∥zq])+λCN(u,v))w_e(q)=\sigma(f_{\text{gate}}([z_e\parallel z_q])+\lambda\text{CN}(u,v)).

3. ACC Measurement, Aggregation & Selector Contrast (W2, Q3)

  • ACC & bootstrap (Appendix C): Per-question ACC ∈[0,1]\in [0,1] uses the GraphRAG-Bench rubric via gpt-4.1-2025-04-14 (temperature=0; Spearman ρ=0.97\rho=0.97 vs. gpt-4o-mini-2024-07-18, zero rank flips) with 10410^4-resample question-paired bootstrap.
  • Matched retention (Table 8): At strictly matched 25% per-query retention, QAG (+3.47/+2.73) beats fixed-confidence top-25% (+2.34/+2.18) by +1.13 (p=.0126p=.0126, Med) and +0.55 (p=.0330p=.0330, Nov).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.