- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies multi-hop relation inference: given two known endpoint entities, recover the bridge entities and the chain of links that connect them in a document corpus, and explain the connection. The proposed ConRAG mines word and phrase candidates from the corpus, filters them into an entity vocabulary with an LLM, and builds a relation-free entity-document graph. At query time it retrieves seed documents that mention the endpoints, enumerates document paths between anchor-side and target-side seeds, and ranks them by query-document similarity before generation. The authors also re-purpose MuSiQue and 2WikiMultiHopQA into relation inference benchmarks with gold bridges and chains. Validation compares ConRAG with four RAG baselines on bridge and chain recovery, document recall, LLM-judged explanation quality, and graph-indexing token cost.
Critical strengths
- Recovering the chain between two known entities is a useful framing, and the re-purposed MuSiQue and 2WikiMultiHopQA give it a benchmark.
- The method is simple, the code is released, and the limitations and the false-negative analysis are stated openly.
Critical weaknesses
- The experiments do not support the central claim
Neither the path step nor evidence grounding is tested, and the comparison is not matched.
- Novelty is limited
The retrieval core follows MR.COD, and the closest published methods are not compared.
Questions for authors
- Seeds are restricted to documents that string-match the anchor or the target (lines 237 to 244), and 2-document paths are excluded by the target lengths {3, 4, 5} (Table 5). In 2-hop questions (all of 2WikiMultiHopQA, 1,252 of the 2,066 MuSiQue questions), both gold documents mention an endpoint and can enter as seeds without any path. What are BCM, CR, CP, and document recall with seeds only, and for Naïve RAG with the same endpoint filter?
- At the default p_hub = 0.9, about 34% of MuSiQue gold bridges stay traversable, down from about 72% (Figure 5), and 4-hop recall is flat across p_hub (Figure 4). What share of the recovered gold bridges comes from path documents rather than seeds?
- The motivation is that existing RAG misses bridge documents with weak query similarity (lines 088 to 090), yet Eq. 3 ranks paths by query similarity. What is the recall of gold bridge documents stratified by their dense rank, for ConRAG and each baseline?
- What do ablations of the LLM vocabulary filter (document frequency only), canonicalization, and hub suppression show?
- "Evidence-grounded" (lines 175 to 176) is never measured, and the endpoints are Wikipedia entities that GPT-4o largely knows. Beyond string matching, the only checks are an LLM judge that does not state whether answer order is swapped (lines 1385 to 1386) and a "blind" validation (Table 9) that does not say who judged. What does closed-book GPT-4o with the same prompt achieve, how faithful are the chain and the explanation to R(q), and who judged Table 9 with what agreement?
- BCM counts a gold bridge whenever its string appears in the response (lines 300 to 301), and the shared prompt asks every system to "Be comprehensive" (Appendix E.3), so longer answers score higher. What is the response length of each method?
- Baselines get exactly 10 documents while ConRAG averages 10.8 (lines 321 to 323), and on 2WikiMultiHopQA ConRAG's prompt has 3,301 ± 1,707 tokens vs 2,542 ± 699 for Naïve RAG (Table 3). What is baseline recall at k = 15 and 20, or under a token-matched context?
- The only p_hub sweep (Figure 4) runs on the evaluation questions themselves, and no development split is described (lines 291 to 292). At p_hub = 0.5, MuSiQue document recall is about 64%, close to HippoRAG 2's 64.2 (Table 4). How were the values in Table 5 chosen?
- The "1.5 orders of magnitude" saving (lines 025 to 026) compares GPT-4o tokens with GPT-4o-mini tokens (lines 318 to 319), although GPT-4o is priced far higher per token, so the token ratio does not show that ConRAG's indexing is cheaper, and "CONRAG also clearly undercuts the comparatively lightweight HippoRAG 2" (lines 421 to 422) is not established. What are the dollar cost, indexing time, and query latency of each method, and how does ConRAG do with a GPT-4o-mini filter?
- Table 6 "full" MuSiQue (71.7, 28.4, 34.5) differs from Table 1 (71.2, 29.5, 34.3) for the same setting. Which run is Table 1? Line 106 says ConRAG "outperforms all evaluated RAG baselines", yet HippoRAG 2 has the higher MuSiQue CP (36.4 vs 34.3, Table 1).
- The retrieval core is MR.COD [1]: a passage graph with shared-entity edges, depth-first search from head to tail passages, and paths ranked by average query-passage similarity, which lines 246 to 256 follow. PathRAG, HiRAG [2], and GNN-RAG [3] already retrieve paths between entities as LLM context, so "the first graph-based RAG framework designed to recover connections between known entities" (lines 104 to 105) is not earned. What is new beyond MR.COD?
- LinearRAG [4], named as closest in lines 145 to 146, reports zero LLM tokens for indexing, and MR.COD [1] is the method's own retrieval core; neither is compared. How do they perform on both benchmarks under the same document budget?
- All four baselines are answer-oriented, and no published multi-hop retriever evaluated on MuSiQue and 2WikiMultiHopQA is compared, such as PropRAG [5], HopRAG [6], GFM-RAG [7], or IRCoT [8] (cited at lines 154 to 155). How does ConRAG compare with them?
[1] Keming Lu, I-Hung Hsu, Wenxuan Zhou, Mingyu Derek Ma, and Muhao Chen. Multi-hop evidence retrieval for cross-document relation extraction. In Findings of ACL, 2023.
[2] Haoyu Huang, Yongfeng Huang, Junjie Yang, Zhenyu Pan, Yongqiang Chen, Kaili Ma, Hongzhi Chen, and James Cheng. Retrieval-augmented generation with hierarchical knowledge. In Findings of EMNLP, 2025.
[3] Costas Mavromatis and George Karypis. GNN-RAG: Graph neural retrieval for efficient large language model reasoning on knowledge graphs. In Findings of ACL, 2025.
[4] Luyao Zhuang, Shengyuan Chen, Yilin Xiao, Huachi Zhou, Yujing Zhang, Hao Chen, Qinggang Zhang, and Xiao Huang. LinearRAG: Linear graph retrieval augmented generation on large-scale corpora. In ICLR, 2026.
[5] Jingjin Wang and Jiawei Han. PropRAG: Guiding retrieval with beam search over proposition paths. In EMNLP, 2025.
[6] Hao Liu, Zhengren Wang, Xi Chen, Zhiyu Li, Feiyu Xiong, Qinhan Yu, and Wentao Zhang. HopRAG: Multi-hop reasoning for logic-aware retrieval-augmented generation. In Findings of ACL, 2025.
[7] Linhao Luo, Zicheng Zhao, Gholamreza Haffari, Dinh Phung, Chen Gong, and Shirui Pan. GFM-RAG: Graph foundation model for retrieval augmented generation. In NeurIPS, 2025.
[8] Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In ACL, 2023.
8 comments
Answer to Reviewer xtks [2/N]
[W1] The comparison is not matched. [Q7] Baselines get exactly 10 documents while ConRAG averages 10.8 (lines 321 to 323), and on 2WikiMultiHopQA ConRAG's prompt has 3,301 ± 1,707 tokens vs 2,542 ± 699 for Naïve RAG (Table 3). What is baseline recall at higher k?
We agree that matching the number of retrieved documents does not necessarily imply matching the token budget, as document lengths vary across methods. To strengthen the comparison, we extended Naïve RAG to k=15 documents, exceeding ConRAG's average of 10.8 documents, and evaluated it on both benchmarks (updated Tables 1 and 4).
| Dataset | Method | Avg. Docs | Doc Recall (%) | Avg. Prompt Tokens |
|---|---|---|---|---|
| MuSiQue | Naïve RAG (k=10) | 10 | 61.1 ± 0.6 | 2,242 ± 344 |
| Naïve RAG (k=15) | 15 | 64.5 ± 0.4 | Not measured | |
| ConRAG | 10.8 | 70.2 ± 0.5 | 2,261 ± 983 | |
| 2WikiMultiHopQA | Naïve RAG (k=10) | 10 | 67.0 ± 0.3 | 2,542 ± 699 |
| Naïve RAG (k=15) | 15 | 69.4 ± 0.3 | Not measured | |
| ConRAG | 10.8 | 84.1 ± 0.6 | 3,301 ± 1,707 |
Increasing Naïve RAG's retrieval budget from 10 to 15 documents improves recall by 3.4 percentage points on MuSiQue and 2.4 points on 2WikiMultiHopQA. Nevertheless, ConRAG retains a substantial advantage (+5.7 and +14.7 points, respectively), despite retrieving fewer documents on average. The extended baseline also achieves lower BCM and chain recall than ConRAG on both benchmarks (updated Table 1). This experiment demonstrates that increasing the dense retrieval baseline's document budget does not close the performance gap.
[W2] Novelty is limited, The retrieval core follows MR.COD, and the closest published methods are not compared. [Q11] The retrieval core is MR.COD [1]: a passage graph with shared-entity edges, depth-first search from head to tail passages, and paths ranked by average query-passage similarity, which lines 246 to 256 follow. PathRAG, HiRAG [2], and GNN-RAG [3] already retrieve paths between entities as LLM context, so "the first graph-based RAG framework designed to recover connections between known entities" (lines 104 to 105) is not earned. What is new beyond MR.COD?
Thank you for the opportunity to clarify our contributions. Our novelty claim has never been about introducing graph traversal or path retrieval, which we explicitly acknowledge and build upon from prior work, including MR.COD. Rather, our contribution lies in formulating and evaluating multi-hop relation inference within RAG, together with an efficient, automatically constructed relation-free graph and an end-to-end framework specifically designed to recover intermediate entities, reasoning chains, and supporting explanations. We have refined the wording of our contributions to make this distinction clearer and avoid any ambiguity regarding prior path-based retrieval methods.
-
Differences from MR.COD. We acknowledge that our path retrieval builds on MR.COD, which we explicitly cite. However, ConRAG differs in three important respects: Multi-document paths: MR.COD restricts candidate paths to passages within two endpoint-associated documents, whereas ConRAG permits paths spanning multiple documents across the corpus. Automatic indexing: MR.COD relies on annotated entity mentions, whereas ConRAG automatically mines, filters, and canonicalizes entities from raw documents without requiring relation extraction or task-specific retriever training. Different objective: MR.COD retrieves evidence to predict predefined relation labels, whereas ConRAG directly generates intermediate entities, reasoning chains, and explanations.
-
Differences from other graph-based RAG methods. We agree that PathRAG, HiRAG, and GNN-RAG already use paths as retrieval context. However, PathRAG uses LLM-extracted relation graphs for general QA, HiRAG focuses on hierarchical knowledge retrieval, and GNN-RAG uses a trained GNN over an existing structured KG to retrieve paths toward answer candidates. Unlike these methods, ConRAG explicitly targets reconstructing connections between two known endpoints directly from unstructured documents, without relation triples or a trained retriever.
To expand our baseline suite, we have added new results comparing with LinearRAG, a closely related relation-free graph RAG method, and component ablations that clarify all the components of ConRAG (Tables 1, 19–20). We have clarified these distinctions and narrowed the original "first graph-based RAG …" claim in the revised manuscript. Our novelty lies in the combination of a dedicated relation-inference formulation, lightweight automatic indexing, and endpoint-constrained evidence retrieval, with explicit evaluation of bridge and chain recovery.
Answer to Reviewer xtks [3/N]
[Q2] At the default p_hub = 0.9, about 34% of MuSiQue gold bridges stay traversable, down from about 72% (Figure 5), and 4-hop recall is flat across p_hub (Figure 4). What share of the recovered gold bridges comes from path documents rather than seeds?
We now explicitly quantify the provenance of recovered gold bridge entities in new Table 17, distinguishing whether they occur in seed documents, path-selected documents, or both. While most recovered bridges are already mentioned in seed documents, identifying a bridge entity is not equivalent to recovering the evidence needed to reconstruct its relationships. Importantly, 43.97% of recovered bridges on MuSiQue and 45.62% on 2WikiMultiHopQA also appear in path-selected documents. Our controlled seeds-only ablation further demonstrates that adding path retrieval consistently improves bridge recovery, document recall, and chain recall. These results show that path traversal contributes meaningfully beyond endpoint-constrained seeding, even when bridge entities are already present in the seed documents.
| Analysis | Metric | MuSiQue (%) | 2WikiMultiHopQA (%) |
|---|---|---|---|
| Bridge provenance | Seed documents only | 55.37 ± 1.36 | 54.38 ± 2.11 |
| Path documents only | 0.49 ± 0.06 | 0.16 ± 0.07 | |
| Both seeds and paths | 43.48 ± 1.19 | 45.46 ± 2.08 | |
| Neither | 0.66 ± 0.12 | 0.00 ± 0.00 | |
| Path contribution | BCM (seeds → full) | 68.5 → 71.2 (+2.7) | 76.6 → 81.0 (+4.4) |
| CR (seeds → full) | 26.3 → 29.5 (+3.2) | 46.2 → 49.4 (+3.2) | |
| Doc Recall (seeds → full) | 66.2 → 70.2 (+4.0) | 74.5 → 84.1 (+9.6) |
[Q3] The motivation is that existing RAG misses bridge documents with weak query similarity (lines 088 to 090), yet Eq. 3 ranks paths by query similarity. What is the recall of gold bridge documents stratified by their dense rank, for ConRAG and each baseline?
We clarify that ConRAG does not rely solely on query similarity to retrieve documents. Instead, graph connectivity first identifies candidate paths between the two endpoints, and Equation 3 subsequently ranks these paths by their average semantic similarity. This allows to surface supporting documents that would otherwise rank poorly under standard dense retrieval. To investigate this, we added a dense-rank analysis (new Table 18) comparing the original dense-retrieval ranks of gold documents with their positions in ConRAG's final context.
| Gold documents | Dense Rank (Mean / Median) | Context Rank (Mean / Median) | Rank Uplift (Mean / Median) |
|---|---|---|---|
| On retrieved paths | 181.66 / 2 | 1.99 / 2 | +179.67 / 0 |
| Path-only | 696 / 32 | 4.20 / 4 | +691.80 / 26 |
This shows that graph connectivity can promote relevant evidence that would otherwise be missed by top-k dense retrieval, despite using semantic similarity for path ranking. Moreover, our seeds-only ablation (Tables 1 and 4) confirms that path traversal improves document recall and chain recovery beyond endpoint-constrained dense retrieval.
[Q4] What do ablations of the LLM vocabulary filter (document frequency only), canonicalization, and hub suppression show?
We have expanded our ablation study (new Tables 19–20), which complement our existing analyses of hub suppression (Figure 4) and entity vocabulary filtering (Figure 5):
-
We attempted to remove the LLM filter while retaining document-frequency filtering. However, even on a reduced corpus, retrieval exceeded 5 minutes per query, compared with approximately 2–3 seconds with filtering enabled. This prevented completing the recall evaluation, but demonstrates the importance of LLM vocabulary filtering for computational tractability. As already shown in Figure 5, this filtering substantially reduces the entity vocabulary while largely preserving gold bridge coverage.
-
Canonicalization, endpoint filtering, and hub suppression (new Tables 19–20). We systematically vary these components and measure document recall and retrieval time on MuSiQue. Results show that endpoint-aware filtering improves document recall, with modest additional retrieval time; hub suppression provides a clear efficiency–recall trade-off, and canonicalization has a relatively small effect on recall, but helps reduce graph size and redundant entity noise by merging equivalent entity nodes.
| Hub Threshold | Endpoint Filter | Canonicalization | Doc Recall (%) | Retrieval Time (s) |
|---|---|---|---|---|
| 80% | On | On | 75.67 ± 29.39 | 2.44 ± 0.76 |
| 80% | On | Off | 75.92 ± 29.28 | 2.43 ± 0.60 |
| 90% | On | On | 79.33 ± 27.94 | 3.03 ± 1.63 |
| 90% | On | Off | 79.08 ± 28.49 | 2.79 ± 1.08 |
| 80% | Off | On | 69.75 ± 30.28 | 2.35 ± 0.65 |
| 80% | Off | Off | 70.33 ± 30.00 | 2.17 ± 0.49 |
| 90% | Off | On | 72.83 ± 29.60 | 2.90 ± 1.70 |
| 90% | Off | Off | 73.08 ± 30.10 | 2.60 ± 1.00 |
Answer to Reviewer xtks [1/N]
Thank you for recognizing these strengths and for your detailed and constructive feedback! We have carefully considered your concerns and conducted additional experiments to address them. We provide a point-by-point response below and would be very happy to discuss any remaining questions or concerns.
Thank you for highlighting this concern. We have added a task-adapted seeds-only baseline that retains ConRAG's endpoint-constrained dense retrieval but removes graph-based path traversal. This isolates the contribution of path retrieval and provides a comparison with endpoint-filtered dense RAG.
Thank you for raising these concerns. We clarify that BCM is intentionally a bridge-identification metric, rather than a measure of evidence grounding. To address your questions, we have expanded our evaluation as follows:
ConRAG substantially improves all metrics over closed-book GPT-4o. This demonstrates that retrieval contributes substantially beyond the model's parametric knowledge alone. Moreover, since all methods use the same GPT-4o generator, the potential influence of prior knowledge applies across baselines.
To directly measure evidence grounding, we additionally evaluate generated responses against their retrieved passages using GPT-4o-mini (new Table 12, 100 examples per dataset). For ConRAG, 71% of responses on 2WikiMultiHopQA are fully grounded (6% partially grounded), while 40% on MuSiQue are fully grounded (12% partially grounded); we further verify that 71.9 ± 11.4% of correctly recovered chain links on the MuSiQue sample are supported by retrieved evidence (Appendix C.4).
Blind validation (Table 9). The validation of alternative paths was performed by GPT-4o-mini, the same model used for our original qualitative evaluation. The judge evaluates each alternative path against its respective retrieved evidence. We have clarified this protocol in the manuscript. As this was a single-judge evaluation, inter-judge agreement was not measured.
Evaluation robustness. We have clarified that answer presentation order was randomized to mitigate position bias in the pairwise evaluation (Table 2). Additionally, we repeated the qualitative evaluation with Qwen3-32B, a judge from a different model family (new Table 14). ConRAG maintains majority overall win rates against all evaluated baselines on both datasets with both judges, supporting the robustness of our qualitative findings.