acceptodds
Under review as a conference paper at ICLR 2027

InterveneRAG: Diagnosing Where GraphRAG Answers Can Be Recovered

Abstract

abstract Retrieval-augmented generation (RAG) helps large language models answer knowledge-intensive questions by supplying external evidence. GraphRAG organizes this evidence into graphs for questions requiring facts from multiple sources. Incorrect answers may arise from missing graph information, retrieval failures, or ineffective evidence use, but component metrics do not directly show whether a stage repair would improve the answer. We introduce InterveneRAG, a diagnostic framework measuring recoverability: the paired change in answer quality under a specified stage intervention. Annotation-guided construction and retrieval replacements and evidence-ordering probes quantify remaining repair potential with downstream operators fixed. Individual and joint interventions, together with graph-size and context-budget controls, examine how gains depend on stage interactions and evidence selection and presentation. Across two multi-hop question-answering datasets and three generator families, annotation-guided selection within existing automatic graphs improves answer F1 by –, despite those graphs containing about 95% of annotated supporting facts. Extensions with two generators yield F1 gains of – on an equal-hop MuSiQue cohort and – through selection restricted to existing passages in bounded shared HippoRAG indices. Construction–retrieval interactions vary across comparison protocols; on 2WikiMultiHopQA, a new factorial protocol combining common budget ceilings with a revised retrieval intervention yields interaction estimates whose confidence intervals cross zero. These findings reveal substantial selection headroom within existing graphs and indices while showing that stage interactions require protocol-specific interpretation. The measured gains characterize annotation-guided repair potential rather than guaranteed deployable improvements. Code: https://anonymous.4open.science/r/InterveneRAG-1E85

Primary area:
foundation or frontier models, including LLMs
Keywords:
retrieval-augmented generation, GraphRAG, multi-hop question answering, diagnostic evaluation, intervention analysis, recoverability, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wWdzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This article studies a problem that is to diagnose which intervention is effective by observing the results of the intervention. They divide interventions into three categories: the first category is replacing the detection results, the second category is replacing evidence selection, and the third category is changing the arrangement. Then they find a lot of interesting things.

Critical strengths

The diagnostic part is very detailed. It analyzes which part has problems and breaks down the process of Graph RAG very rigorously.

Critical weaknesses

Only did the diagnosis, without making further improvements based on the diagnostic results to improve GraphRAG. Therefore, the contribution of the article is relatively small.

Questions for authors

The author mainly tested these models: DeepSeek V4 Flash, Qwen3-8B, GLM-5.2. In fact, these 3 models differ greatly. Can the author provide the scale law of the same model, such as the performance of Qwen of different sizes?

Reviewer_799nOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes InterveneRAG, a diagnostic framework that evaluates the effects of graph construction, retrieval, and evidence ordering on GraphRAG performance. The authors replace different pipeline components with annotation-guided alternatives and measure changes in downstream QA performance. Ezxperiments are conducted mainly on 2WikiMultiHopQA and HotpotQA with several language models, with additional evaluations on MuSiQue and HippoRAG 2.

Critical strengths

Strengths

  1. The paper addresses an important problem in understanding the limitations of GraphRAG systems.

  2. The controlled interventions provide some interesting observations about evidence selection and downstream QA performance.

Critical weaknesses
  1. Very limited novelty. The proposed framework mainly evaluates GraphRAG components by replacing their outputs with oracle or annotation-guided alternatives and measuring downstream performance differences. This is a straightforward evaluation strategy, and similar component-level analyses have already been explored in previous RAG and GraphRAG studies [1, 2, 3]. The paper does not introduce a fundamentally new evaluation methodology or provide sufficiently novel insights beyond existing work.

  2. The framework does not adequately account for differences among GraphRAG architectures. Different GraphRAG methods have substantially different graph construction, retrieval, and generation mechanisms. For example, Microsoft GraphRAG uses entity relationships and community summaries, HippoRAG 2 integrates entities and passages, while RAPTOR constructs a hierarchical tree of document representations. It is hard to get the oracle results. The general applicability of the proposed framework remains unclear.

  3. Limited experimental generalizability. The primary experiments are conducted on only two traditional multi-hop QA datasets, 2WikiMultiHopQA and HotpotQA, with 300 questions sampled from each. Although the authors provide additional experiments on MuSiQue and shared HippoRAG indices, these evaluations remain within similar benchmark settings.

  4. The practical value of oracle interventions is unclear. The main improvements are obtained by selecting evidence using ground-truth supporting-fact annotations. However, such annotations are unavailable in practical retrieval scenarios. Therefore, the reported improvements mainly demonstrate the potential benefits of having access to gold evidence rather than showing how the identified bottlenecks can be addressed in real systems.

[1] Han et al. RAG vs. GraphRAG: A Systematic Evaluation and Key Insights.

[2] Xiang et al. When to Use Graphs in RAG: A Comprehensive Analysis for Graph Retrieval-Augmented Generation.

[3] Ru et al. RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation.

Questions for authors

Please refer to the weaknesses.

Reviewer_L75MOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper measures how much a GraphRAG answer improves when one pipeline stage (graph construction, retrieval or evidence ordering) is replaced with an oracle version.

Critical strengths
  1. The comparisons control for graph size and context length.
  2. The paper includes a check on a real system, HippoRAG 2.
Critical weaknesses
  1. The framework does not appear to go beyond standard RAG error-analysis practice, and its main finding has been reported before.
  • Method. The approach follows the usual index → retrieval → generation error analysis, carried out as gold swaps.
  • Prior work. FRAMES (Krishna et al., NAACL 2025; https://arxiv.org/abs/2409.12941) and Sufficient Context (Joren et al., ICLR 2025; https://arxiv.org/abs/2411.06037) both show that accuracy and abstention shift when the retrieved context is sufficient. The conclusion therefore does not seem new.
  1. It is unclear what conclusion the interaction analysis supports.
Questions for authors
  1. What diagnostic decision does the framework support that component metrics (coverage, recall, answer accuracy) do not?
  2. Does the framework include any method designed specifically for GraphRAG?
  3. What conclusions do the authors draw from the interaction analysis?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.