- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SG2RAG, motivated by the limitations of conventional RAG and Graph RAG in retrieving critical evidence for multi-hop question answering (QA), where information from multiple documents must be combined in a complementary manner. Specifically, conventional RAG relies on query–document similarity, while Graph RAG primarily considers inter-document connectivity, potentially overlooking the complementary information contributed by individual documents. To address these limitations, SG2RAG uses an LLM to evaluate the Semantic Gain between document pairs, quantifying the additional information contributed by one document relative to another. These scores serve as edge weights in a Semantic Gain Graph, which is constructed offline. During retrieval, SG2RAG selects documents similar to the query as seed nodes and applies Controlled Personalized PageRank (PPR) to identify complementary supporting evidence. This approach improves multi-hop QA performance compared with conventional Dense RAG and Graph RAG methods.
Critical strengths
- Clear and intuitive problem formulation. The paper identifies a key limitation of existing RAG methods: they do not explicitly model how much useful, non-redundant information a document adds to existing evidence. The proposed concept of semantic gain directly addresses this issue in multi-hop QA.
- Well-designed ablation studies. By comparing cosine-similarity and semantic-gain edges under identical retrieval settings, the paper demonstrates approximately 4-point improvements in EM, F1, and Recall, validating the contribution of semantic gain beyond graph expansion.
- Comprehensive evaluation. The paper evaluates SG²RAG across three benchmarks, two LLM backbones, and multiple RAG baselines. Improvements in iterative frameworks further demonstrate its modularity and generalizability.
Critical weaknesses
-
Misalignment between the semantic gain formulation and graph construction. The paper defines semantic gain as a directional conditional contribution (), but symmetrizes these scores before PPR, losing the directionality central to its motivation. The justification for symmetric traversal conflates directionality with sparse candidate connectivity, without evaluating a directed graph with both edge directions explicitly scored. Moreover, cosine-similarity-based Top- candidate selection may exclude highly complementary but semantically dissimilar document pairs.
-
Performance improvements are not consistently statistically significant. Although SG²RAG achieves the best average performance, paired permutation tests show that its improvements over strong baselines are not consistently significant. For example, improvements over HippoRAG2 on HotpotQA and KG2RAG on 2Wiki and HotpotQA are insignificant under Llama-70B. The strongest evidence comes from MuSiQue, and significance testing is limited to Llama-70B. Thus, the results do not fully establish the consistent superiority of semantic gain over existing graph retrieval methods.
-
Limited methodological novelty and heavy reliance on LLM judgments. SG²RAG essentially constructs a similarity-based sparse graph, uses an LLM to score pairwise document complementarity, and applies existing PPR-based retrieval. Its main novelty therefore lies in semantic-gain edge weighting rather than graph construction or propagation. However, semantic gain is implemented through a fixed natural-language rubric evaluated by a powerful LLM, rather than a novel learning objective or estimation mechanism. Consequently, the results primarily demonstrate that LLM-generated complementarity scores outperform cosine-similarity edges, rather than validating semantic gain as a distinct retrieval principle. Comparisons against simpler LLM-based edge scorers, such as relevance-only or usefulness-only scoring, are needed to isolate the proposed formulation's contribution.
-
High offline indexing cost and insufficient scalability evaluation. SG²RAG shifts substantial computational overhead to offline graph construction. On MuSiQue, it requires approximately 36.81M LLM tokens, compared with 14.47M for HippoRAG2 and 8.10M for KG2RAG. Although the smaller storage footprint and lower online latency are advantages, experiments are limited to corpora of approximately 6K–12K documents. Without larger-scale evaluations, it remains unclear whether this indexing–retrieval trade-off is practical for real-world deployment.
-
Insufficient explanation of discrepancies between retrieval and downstream QA performance. SG²RAG frequently achieves higher EM/F1 despite lower Recall@10 than competing graph retrievers. For example, on MuSiQue with Qwen-72B, SG²RAG achieves lower Recall@10 than HippoRAG2 (71.63 vs. 74.02) but higher QA accuracy. Similar inconsistencies appear in edge-treatment ablations, where larger recall degradation does not necessarily lead to larger EM/F1 drops. While Recall@10 may not capture complete evidence chains, ranking quality, or context composition, the paper does not sufficiently investigate these discrepancies. Complete-support recall, gold-pair recall, rank-sensitive metrics, and retrieval–answer correctness correlations would help establish whether QA improvements genuinely result from better complementary evidence retrieval.
Questions for authors
See Weakness.
Additionaly, could the authors include a comparison with LinearRAG, a recent and competitive RAG method? Although LinearRAG is cited in the Related Work section, its omission from the experiments leaves out an important baseline and weakens the empirical evaluation.