acceptodds
Under review as a conference paper at ICLR 2027

Semantic Gain Graphs for Complementarity-Guided Retrieval-Augmented Generation

Abstract

Multi-hop Retrieval-Augmented Generation (RAG) requires discovering evidence whose value emerges through combination across documents. Dense retrievers rank chunks by isolated query relevance, while graph-based retrievers traverse connections that do not indicate whether neighboring documents contribute useful evidence. We propose SGRAG, which reframes graph retrieval around semantic gain, defined as the useful and nonredundant evidence that one document chunk contributes in the context of another. SGRAG aggregates these assessments into document-pair strengths in a Semantic Gain Graph, turning cross-document complementarity into an explicit retrieval relation. Starting from query-relevant seeds, controlled Personalized PageRank composes local gain relations into a global multi-hop ranking, following complementary evidence chains while maintaining query focus. This design moves evidence discovery beyond similarity ranking and topology-only traversal. Experiments on three multi-hop benchmarks show that SGRAG achieves the best overall performance. Under identical candidate support and propagation, semantic-gain weighting consistently outperforms similarity weighting. Gold-support analysis further validates the gain signal, while improvements across iterative retrieval frameworks demonstrate its modularity.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Retrieval-Augmented Generation, Multi-hop Question Answering, Graph-based Retrieval, Semantic Gain, Cross-document Complementarity, Personalized PageRank

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kJm9OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SG2RAG, motivated by the limitations of conventional RAG and Graph RAG in retrieving critical evidence for multi-hop question answering (QA), where information from multiple documents must be combined in a complementary manner. Specifically, conventional RAG relies on query–document similarity, while Graph RAG primarily considers inter-document connectivity, potentially overlooking the complementary information contributed by individual documents. To address these limitations, SG2RAG uses an LLM to evaluate the Semantic Gain between document pairs, quantifying the additional information contributed by one document relative to another. These scores serve as edge weights in a Semantic Gain Graph, which is constructed offline. During retrieval, SG2RAG selects documents similar to the query as seed nodes and applies Controlled Personalized PageRank (PPR) to identify complementary supporting evidence. This approach improves multi-hop QA performance compared with conventional Dense RAG and Graph RAG methods.

Critical strengths
  • Clear and intuitive problem formulation. The paper identifies a key limitation of existing RAG methods: they do not explicitly model how much useful, non-redundant information a document adds to existing evidence. The proposed concept of semantic gain directly addresses this issue in multi-hop QA.
  • Well-designed ablation studies. By comparing cosine-similarity and semantic-gain edges under identical retrieval settings, the paper demonstrates approximately 4-point improvements in EM, F1, and Recall, validating the contribution of semantic gain beyond graph expansion.
  • Comprehensive evaluation. The paper evaluates SG²RAG across three benchmarks, two LLM backbones, and multiple RAG baselines. Improvements in iterative frameworks further demonstrate its modularity and generalizability.
Critical weaknesses
  1. Misalignment between the semantic gain formulation and graph construction. The paper defines semantic gain as a directional conditional contribution (wij≠wjiw_{ij}\neq w_{ji}), but symmetrizes these scores before PPR, losing the directionality central to its motivation. The justification for symmetric traversal conflates directionality with sparse candidate connectivity, without evaluating a directed graph with both edge directions explicitly scored. Moreover, cosine-similarity-based Top-nn candidate selection may exclude highly complementary but semantically dissimilar document pairs.

  2. Performance improvements are not consistently statistically significant. Although SG²RAG achieves the best average performance, paired permutation tests show that its improvements over strong baselines are not consistently significant. For example, improvements over HippoRAG2 on HotpotQA and KG2RAG on 2Wiki and HotpotQA are insignificant under Llama-70B. The strongest evidence comes from MuSiQue, and significance testing is limited to Llama-70B. Thus, the results do not fully establish the consistent superiority of semantic gain over existing graph retrieval methods.

  3. Limited methodological novelty and heavy reliance on LLM judgments. SG²RAG essentially constructs a similarity-based sparse graph, uses an LLM to score pairwise document complementarity, and applies existing PPR-based retrieval. Its main novelty therefore lies in semantic-gain edge weighting rather than graph construction or propagation. However, semantic gain is implemented through a fixed natural-language rubric evaluated by a powerful LLM, rather than a novel learning objective or estimation mechanism. Consequently, the results primarily demonstrate that LLM-generated complementarity scores outperform cosine-similarity edges, rather than validating semantic gain as a distinct retrieval principle. Comparisons against simpler LLM-based edge scorers, such as relevance-only or usefulness-only scoring, are needed to isolate the proposed formulation's contribution.

  4. High offline indexing cost and insufficient scalability evaluation. SG²RAG shifts substantial computational overhead to offline graph construction. On MuSiQue, it requires approximately 36.81M LLM tokens, compared with 14.47M for HippoRAG2 and 8.10M for KG2RAG. Although the smaller storage footprint and lower online latency are advantages, experiments are limited to corpora of approximately 6K–12K documents. Without larger-scale evaluations, it remains unclear whether this indexing–retrieval trade-off is practical for real-world deployment.

  5. Insufficient explanation of discrepancies between retrieval and downstream QA performance. SG²RAG frequently achieves higher EM/F1 despite lower Recall@10 than competing graph retrievers. For example, on MuSiQue with Qwen-72B, SG²RAG achieves lower Recall@10 than HippoRAG2 (71.63 vs. 74.02) but higher QA accuracy. Similar inconsistencies appear in edge-treatment ablations, where larger recall degradation does not necessarily lead to larger EM/F1 drops. While Recall@10 may not capture complete evidence chains, ranking quality, or context composition, the paper does not sufficiently investigate these discrepancies. Complete-support recall, gold-pair recall, rank-sensitive metrics, and retrieval–answer correctness correlations would help establish whether QA improvements genuinely result from better complementary evidence retrieval.

Questions for authors

See Weakness.

Additionaly, could the authors include a comparison with LinearRAG, a recent and competitive RAG method? Although LinearRAG is cited in the Related Work section, its omission from the experiments leaves out an important baseline and weakens the empirical evaluation.

Reviewer_jnuoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SG2RAG builds a chunk graph whose edge weights are LLM-scored "semantic gain" between each chunk and its nearest cosine neighbors, i.e., how much useful new information one chunk adds given the other. At query time, personalized PageRank from the most query-similar chunks ranks the evidence. On the HippoRAG2 subsets of MuSiQue, 2Wiki and HotpotQA, SG2RAG achieves the best average EM and F1, with modest gains over HippoRAG2. Used as the retriever inside iterative RAG methods, it also improves them.

Critical strengths
  1. Comprehensive experiments and clean ablations. Keeping the candidate edges and PPR fixed and swapping cosine weights for gain weights clearly improves EM, F1 and recall. The gold-pair AUC analysis further supports the gain signal.
  2. Practical. Indexing needs one LLM call per chunk, the unfiltered variant needs no LLM call for retrieval at query time, and the method improves three iterative frameworks as a drop-in retriever.
Critical weaknesses
  1. Incremental method. It is HippoRAG-style PPR on a kNN chunk graph, and passage graphs have been used before (e.g., KGP). The new part is the LLM-scored edge weight.
  2. Recall barely changes against HippoRAG2. The small average gain comes entirely from 2Wiki, and recall is lower on MuSiQue and HotpotQA, so the central claim of reaching complementary evidence is not clearly supported.
  3. Modest and partly non-significant gains. The differences are not significant against KG2RAG on 2Wiki and HotpotQA, or against HippoRAG2 on HotpotQA. Without the LLM filter, the margin over HippoRAG2 shrinks further.
  4. "Beyond similarity" is limited by construction. Edges exist only between cosine neighbors, so only a small fraction of gold-gold pairs on MuSiQue are directly connected. The gain score is also query-independent.
Questions for authors
  1. HippoRAG2's scores here are noticeably lower than those reported in the HippoRAG2 paper on the same subsets with the same reader model. Does SG2RAG still win under HippoRAG2's original setting?
  2. Why do EM and F1 improve while Recall@10 does not?
  3. How much is lost by making the gain score query-independent?
Reviewer_PhQfOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper (SG²RAG) is about multi-hop question answering with retrieval. Offline, it links each text chunk to its 5 nearest neighbours by embedding similarity. An LLM then scores how much new, useful evidence each neighbour adds to the chunk ("semantic gain"), and the two directions of each pair are summed into one edge weight. At query time, the 5 chunks most similar to the query seed a Personalized PageRank (PPR) walk over this graph, and the top-10 chunks go to the generator. It reports the best average EM/F1/Recall on MuSiQue, 2Wiki and HotpotQA with two LLMs, and gains when plugged into three iterative frameworks.

Critical strengths
  1. Practical design. All LLM cost is offline, so queries are fast (0.66 s) and the index is small (198 MB vs 4.2 GB for HippoRAG2). New chunks can be added without rebuilding.
  2. The right core ablation. Swapping only the edge weights (cosine vs semantic gain) on the same candidate pairs and the same PPR gives about +4 EM.
  3. Useful validity check. Gain scores separate gold–gold from gold–distractor pairs (AUC 0.64–0.76). Cosine is below chance on the same pairs (AUC 0.45).
  4. Careful direction analysis, plus query-level significance tests, including the results that were not significant.
Critical weaknesses
  1. The margins are small and mostly not significant. Against KG2RAG, only MuSiQue is significant (2Wiki p = 0.52, HotpotQA p = 0.21). There are no tests for the Qwen-72B setting, for the version without the LLM filter, or for the ablations.

  2. The significant MuSiQue result comes from the LLM filter. Without it, SG²RAG with Llama reaches EM 29.40, slightly below HippoRAG2's 29.60. The filter isn't described in the paper, and with Qwen it lowers EM.

  3. The control that would test "complementarity" is missing. The only comparison is cosine, which is nearly flat among a chunk's 5 nearest neighbours. The paper's own results point the same way:

Binary scores retrieve almost the same chunks as graded ones (Recall@10 within ±0.41). The ordered, one-directional gain is averaged into a symmetric weight (pair mean ≈ pair sum). The scoring prompt rewards topical relatedness. An LLM relatedness score, cross-encoder pair scores, or entity-overlap weights on the same pairs are needed.

  1. The reranker baseline looks misconfigured. Adding it lowers Recall@10, and on 2Wiki recall drops from 84.95 to 77.68. Its model, candidate pool size and prompt are never given. The paper's argument that "reranking can't reach missing support" depends on this baseline.

  2. The graph covers only about a fifth of MuSiQue gold pairs (16.3% at 3 neighbours, 22.5% at 7). Recall@10 is not better than HippoRAG2 (70.57 vs 74.02 on MuSiQue), so the EM/F1 gains come from which chunks are chosen, not from reaching more evidence.

  3. The size claim contradicts the paper's own figure. The text says the recall gain is "between 4.85 and 5.52" and that bigger models help. Figure 5(a) shows +4.86 / +4.85 / +5.01 / +2.61, and the 72B value is the smallest. A cheap 7B scorer works at least as well, which is good news for cost.

  4. Generation noise is as large as the claimed gains. Coarse vs binary scoring moves 2Wiki F1 by 4.74 while Recall changes by only 0.08.

  5. The evaluation is narrow. It uses three curated 1,000-question subsets with 6–12k chunks, and has no single-hop "do-no-harm" check. There is no scalability analysis, even though construction takes about 3,000 tokens per chunk. Recent benchmarks like SiReRAG and BrowseNet are missing.

Questions for authors

Address all the weakness points mentioned

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.