acceptodds
Withdrawn from ICLR 2027

MultiRAG-Bench: Benchmarking Joint Retrieval versus Knowledge Graph Fusion for Multi-Domain RAG

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Graph Retrieval-Augmented Generation (RAG) systems are widely deployed, yet a fundamental question remains unexamined: when multiple domain-specific Graph RAGs are independently constructed over the same corpus using different extraction models and prompts, should their knowledge graphs be fused into one unified structure or queried jointly while preserving domain separation? We present MultiRAG-Bench, the first benchmark for this problem. Our dataset comprises five domain-specific knowledge graphs built from a common document corpus, a multi-model heterogeneity analysis demonstrating that different LLM architectures produce substantially divergent graph structures even under identical prompts, and a stratified question set whose cross-domain requirements are each verified via direct Neo4j graph traversal, ensuring that cross-domain questions genuinely require evidence from multiple graphs. We establish baselines for both graph fusion and joint retrieval paradigms, finding that fusion incurs a hub noise cost that grows with retrieval depth, that our cache-based joint retrieval surpasses mean single-domain performance on moderate cross-domain questions, and that a substantial performance gap persists on harder multi-hop scenarios. We release all code, data, and baselines.

Primary area:
datasets and benchmarks
Keywords:
Retrieval-Augmented Generation, Knowledge Graph, Multi-Domain Retrieval, Graph Fusion, Joint Retrieval, Benchmark, Multi-Hop QA

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Ne8SOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes MultiRAG-Bench, a benchmark comparing graph fusion and joint retrieval over multiple domain KGs. Five KGs are extracted from the same MuSiQue corpus with different prompts, and 2,348 Neo4j-verified cross-graph questions are generated. Joint retrieval achieves the highest path hit rate.

Critical strengths

S1: The paper studies a practical question of whether multiple KGs should be merged or queried jointly. S2: Every gold chain is verified by Neo4j, and the union control and leakage analysis are reported honestly.

Critical weaknesses

W1: The cross-domain requirement is artificial. All graphs come from the same corpus and differ only in prompts, and the questions are selected to need edges from different graphs, so the finding that merging helps here but not on MuSiQue follows by construction. W2: The comparison is unfair. Joint pools about 39 entities from five graphs vs. about 8 for a single graph, and when the pooled result is cut to 25 entities, PHR drops from 54.64 to 23.08. No end-to-end QA is reported.

Questions for authors

Refer to Weakness.

Reviewer_39zDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MULTIRAG-BENCH, a benchmark for evaluating retrieval across multiple knowledge graphs. The authors point out that in real-world applications, different knowledge graphs may be built from the same documents using different models or extraction settings. This creates a challenge when a question requires information from multiple graphs. To study this problem, the authors build five domain-specific knowledge graphs from around 21K Wikipedia documents and generate 2,348 cross-domain questions. They compare two main approaches: Graph Fusion, which merges multiple graphs into one, and Joint Retrieval, which searches the graphs separately and combines the retrieved results. The authors also propose an entity-cache mechanism to help share information across retrieval steps. For validation, the authors evaluate different retrieval methods using LightRAG and PG-Direct, and analyze performance across different question difficulties. The results show that searching a single graph is often insufficient for cross-domain questions, while combining information from multiple graphs improves retrieval. However, they also find that most of the gains from Joint Retrieval come from simply combining search results, rather than the proposed cache mechanism. The paper also shows that merging graphs can introduce noisy connections, but remains helpful for questions that genuinely require cross-domain reasoning.

Critical strengths

I think the main strength of this paper is that it studies a practical but relatively underexplored problem: how to retrieve information from multiple independently constructed knowledge graphs. Most existing Graph RAG methods focus on retrieval from a single graph, while real-world applications may involve multiple graphs with different structures. The authors also use graph traversal to verify that the generated questions require information from multiple graphs. This makes the benchmark more convincing and provides a useful evaluation setting for future multi-graph RAG research. The paper compares graph fusion and joint retrieval across different retrieval backends and question difficulties. I especially appreciate the no-cache union baseline, which shows that much of the improvement from Joint Retrieval actually comes from combining search results rather than the entity-cache mechanism. The analysis of graph heterogeneity and fusion penalties is also useful for understanding when merging graphs helps or hurts retrieval. These experiments provide practical insights beyond simply reporting benchmark scores.

Critical weaknesses
  1. Poor Graph Quality: Low retrieval accuracy on hard questions may be caused by noisy or incomplete LLM-generated graphs rather than limitations of retrieval algorithms, making the benchmark results less reliable. I think for those multi-hop questions where the bridge entity is clean but the performance is super low, we definitely need humans to evaluate whether this is caused by the incomplete graph.

  2. Lack of Human Validation: Both questions and graph paths are generated by LLMs without human review, and 33% of sampled questions are ambiguous. This raises concerns about the quality and reliability of the benchmark. Human evaluation is very important because the graph accuracy is not high enough, and also the question quality is very important for benchmarking.

Questions for authors
  1. Did you make any entity ambiguous? The paper reports that 89.3% of questions contain a gold-chain entity, and 50.6% of two-hop questions expose the bridge entity. This could make the questions much easier to solve through simple entity matching, without requiring the intended cross-domain reasoning.
  2. Why does the method rely on naive exact-name matching, which may introduce noisy connections and artificially inflate hub nodes, rather than using or comparing against stronger, state-of-the-art entity alignment methods?
Reviewer_oSPLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether several knowledge graphs (KGs) should be merged before retrieval or searched separately and have their results combined. The authors build five "domain" KGs from the MuSiQue corpus. They then generate 2,348 questions in total: 1,598 whose answer path crosses two or three graphs, plus 750 single-graph controls. The metric is path hit rate (PHR): the share of questions where every entity on the answer path is retrieved. The paper compares:

  • single graphs: the average graph and a per-question best graph;
  • a merged graph;
  • "Joint": query each graph, pass found entities to the next query, and pool the results.
Critical strengths
  1. The benchmark defines a clear and checkable setting. It checks that the designated evidence chain cannot be completed in one graph, rather than assuming that a multi-hop question requires multiple graphs.
  2. The baseline is a useful negative result.Searching each of the five graphs with the original question and pooling the results scores almost as well as the paper's cache method: 54.64% vs 56.52% PHR.
Critical weaknesses
  1. The metric doesn't measure what's needed to answer, and it rewards bigger result sets.
  • PHR counts a question as solved when the path's entity names appear anywhere in the returned context. This doesn’t guarantee the connections required to answer the question is retrieved.
  • There is no e2e QA accuracy. The graph with high PHR may also retrieve more noise, hurting final QA accuracy.
  • Joint pools five searches (about 39 entities) versus about 8 for a single graph (Table 4 caption). Because PHR only checks presence, larger sets score higher automatically. The merged graph's size isn't reported.
  1. The main finding is built into the benchmark.
  • The merged graph must win here. Questions are selected so that each answer path exists only when two graphs are combined, so only the merged graph contains the full answer paths, and it is expected to beat any single graph.
  • The fusion-vs-joint conclusion depends on the configuration, but it is stated generally. All results come from one corpus (MuSiQue), one graph framework and one extraction model. The paper's own numbers show the verdict is sensitive to these choices. The reader is never tested. It is therefore unclear whether the conclusion holds for stronger fusion methods; stronger retrievers; stronger readers; or other corpora, particularly separate corpora owned by different teams, which motivate the paper.
  • The setup does not match the scenario that motivates the problem. The paper motivates un-mergeable graphs by separate owners. However, all five graphs built by the paper are from the same corpus by the same model.
  • The abstract's claim that merging adds "hub noise" on MuSiQue does not appear to be supported by the experiments.
Questions for authors
  1. What is the average result size for the merged graph? How do Merged and Joint compare at the same ranked top-k?
  2. Can you report edge recall (both gold edges retrieved) for each method? Does Joint still beat Merged?
  3. Using one fixed reader at an equal context budget, what is the QA accuracy of Merged vs Joint on the questions judged answerable?
  4. Does the model-vs-prompt gap survive once relation names are normalized?
  5. Would the authors expect the fusion-vs-joint ranking to hold when the domain graphs are built from different corpora, when a stronger entity-alignment method is used for fusion, or with a stronger retriever and a strong long-context reader?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.