acceptodds
Under review as a conference paper at ICLR 2027

CompassRAG: Navigating Multi-Turn GraphRAG with Dynamic Evidence Memory and Retrieval

Abstract

Graph Retrieval-Augmented Generation (GraphRAG) has emerged as an effective paradigm for multi-hop reasoning by exploiting relational structures among entities and passages. Recent agentic retrieval methods further demonstrate that iterative retrieval and reasoning can improve complex knowledge tasks, motivating the extension of GraphRAG to multi-turn settings. However, existing multi-turn GraphRAG methods face two limitations: evidence from earlier turns is often left implicit and unstructured in the reasoning context, and graph retrieval adapts only weakly to evolving evidence needs across turns. Thus, we introduce CompassRAG, which serves as a compass for multi-turn GraphRAG by maintaining dynamic evidence memory and guiding subsequent retrieval. CompassRAG first maintains a Dynamic Evidence Graph (DEG) that records answer-relevant facts as source-linked triples, providing an explicit and traceable working memory across turns. It further adapts graph retrieval to the evolving reasoning objective through a query-aware relation graph, where edge weights are conditioned on the current query to guide propagation toward more relevant evidence. Together, these components enable multi-turn GraphRAG to accumulate verifiable evidence while adaptively navigating the retrieval space. Experiments on three multi-hop question answering benchmarks demonstrate that CompassRAG consistently outperforms existing baselines. Further analysis shows that dynamic evidence memory enables reliable reasoning with reduced dependence on raw retrieval history, while dynamic retrieval improves evidence acquisition.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Retrieval-Augmented Generation, Graph Retrieval-Augmented Generation, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wAcpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes CompassRAG, a training-free multi-turn GraphRAG framework that keeps a source-linked triple memory (DEG) across turns, folds old retrieval text into it, and reweights graph edges by the current query before PPR, with gains reported on MuSiQue, 2Wiki and HotpotQA.

Critical strengths

The method beats a fairly broad set of single-turn, RL-based and learning-free multi-turn baselines on all three benchmarks, and the ablations cover most components.

Critical weaknesses

The “w/o DEG” variant already beats the strongest multi-turn baseline, so a good part of the gain seems to come from the authors’ own reasoning loop and prompt rather than from the DEG or the retriever, and this isn’t separated out. Most component gains are around one point or less, there are no variance or significance numbers, and each test set has only 1,000 questions, so it’s hard to tell which gains are real. Questions take only about 1.5 to 2.4 turns on average, so the context growth problem that memory folding is meant to solve barely shows up, and there are no end-to-end token or cost numbers to back the efficiency story. The novelty is moderate: query-conditioned edge weighting is close to QAFD-RAG, structured working memory for agentic RAG isn’t new, and everything is tested with a single 70B reasoner.

Questions for authors

ightRAG gets close to zero EM on all three datasets, which looks like a setup problem. How was it configured, and is the comparison fair?

1 comment

Dear Reviewer wAcp,

Thank you again for your thoughtful review. As the discussion period ends today, we would greatly appreciate it if you could take a moment to review our detailed rebuttal and revised manuscript.

We have carefully addressed your concerns through additional same-scaffold comparisons across three retrieval backbones, repeated runs demonstrating low performance variance, and end-to-end token-efficiency analysis. We have also clarified our methodological differences from QAFD-RAG, the role of DEG in evidence management, and the source and evaluation settings of the LightRAG results. We hope these new results help resolve your concerns and would be glad to discuss any remaining questions before the deadline.

Thank you very much for your time and consideration.

Reviewer_SgEnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CompassRAG, a dynamic evidence memory and retrieval framework designed for multi-turn GraphRAG. It addresses the limitations of existing methods, which struggle to effectively manage historical evidence and fail to adapt their retrieval processes to evolving evidence requirements. Specifically, the authors propose a "Dynamic Evidence Graph" (DEG) that stores key facts, extracted during multi-turn reasoning as structured triples, complete with provenance information; a "Memory Folding" technique is then employed to compress the history of retrieved content. Furthermore, the framework utilizes "Query-Aware Graph Retrieval," which dynamically adjusts the edge weights within the knowledge graph based on the current query. By combining this with Personalized PageRank (PPR) and a re-ranking mechanism focused on the "evidence frontier," the system achieves adaptive evidence retrieval. Experiments conducted on three multi-hop question-answering datasets(MuSiQue, 2WikiMultiHopQA, and HotpotQA), demonstrate that CompassRAG achieves a 2.97% improvement over the strongest multi-turn baseline for F1 score and 3.17% improvement for EM, while simultaneously significantly reducing the length of the retrieved historical context.

Critical strengths

1. Well-motivated framework for multi-turn evidence management and retrieval.

The paper addresses two important challenges in multi-turn GraphRAG: maintaining structured, source-grounded evidence across reasoning turns and adapting retrieval to evolving information needs. By combining Dynamic Evidence Graphs (DEG), memory folding, and query-aware graph retrieval, CompassRAG provides a coherent framework for improving evidence utilization and reducing redundant retrieval context.

2. Strong empirical performance and comprehensive evaluation. CompassRAG consistently outperforms the evaluated baselines across three multi-hop QA benchmarks, achieving an average F1 improvement of 2.97 points and an average EM improvement of 3.17 points over the strongest multi-turn baseline. The experiments include recent GraphRAG methods, multi-turn retrieval frameworks, and component ablations. Notably, heavy memory folding reduces historical retrieval context to approximately 5% of its original length while largely preserving answer quality, demonstrating the practical potential of structured evidence memory.

Critical weaknesses

1. Limited methodological novelty and insufficient evidence for the independent contributions of the proposed components.

CompassRAG primarily combines existing techniques, including structured graph-based memory, query-conditioned PPR, and evidence-aware reranking. Although the overall framework is meaningful, its novelty remains limited. Moreover, removing DEG reduces the average F1 by only 0.97 points, while the relation-aware weighting ablations are conducted only in the single-turn setting. Comparisons with simpler memory representations and controlled evaluations of the retrieval components within the full multi-turn framework are needed to establish their independent effectiveness.

2. Concerns about experimental reliability and incomplete efficiency evaluation.

The reported performance of LightRAG is unusually low (5.20 average F1), substantially below even Vanilla RAG, raising questions about baseline implementation and evaluation fairness. Furthermore, the efficiency analysis primarily measures graph retrieval latency while excluding the computational overhead of multi-turn LLM reasoning, DEG updates, and memory management. Without end-to-end latency and token-consumption comparisons against strong multi-turn baselines, the practical benefits of CompassRAG remain insufficiently demonstrated.

Questions for authors
  1. Independent contribution of Dynamic Evidence Graphs (Clarification and additional evidence request): The removal of the DEG results in only a marginal decline in the average F1 score. Could the authors clarify what advantages DEG provides over simpler memory representations, such as structured fact lists or evidence summaries? Comparisons under matched context and token budgets would help establish whether the graph-based memory structure is essential to the reported improvements.

  2. Novelty and effectiveness of query-aware graph retrieval (Clarification and additional evidence request): The proposed query-conditioned edge weighting is closely related to existing query-aware graph retrieval methods, particularly QAFD-RAG. Could the authors clarify the key methodological differences and evaluate QAFD-RAG within the same multi-turn reasoning framework? Additionally, could the authors provide full-system ablations of relation-aware fact weighting and adaptive synonym weighting to better isolate their contributions?

  3. Baseline reliability and computational efficiency (Clarification and additional evidence request): The reported LightRAG performance is unusually low, with an average F1 of only 5.20, substantially below Vanilla RAG. Could the authors explain this discrepancy and verify the baseline implementation? Furthermore, could the authors report end-to-end latency and token consumption, including multi-turn LLM reasoning and DEG updates, to assess whether the performance improvements justify the additional computational costs?

Reviewer_NBHvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CompassRAG, a learning-free multi-turn GraphRAG framework built on top of a HippoRAG 2-style PPR retriever. It has two parts. The first is a query-specific Dynamic Evidence Graph (DEG): at each turn the reasoner LLM (Llama-3.3-70B) emits ADD_TRIPLE and MERGE_ENTITIES operations that record source-linked (s, r, o, doc) facts, and "memory folding" (weak/medium/heavy) then shrinks or drops the raw passages from earlier turns. The second is query-aware graph retrieval: fact edges are weighted by query–fact cosine similarity, entity–passage edges by query–passage similarity, synonym edges by the mean of their endpoints' incident fact weights, and the resulting PPR scores are reranked with a "DEG-frontier" boost for uncited passages that contain recently added DEG entities. On the standard 1,000-question subsets of MuSiQue, 2Wiki and HotpotQA, the best variant reaches 60.77 EM / 71.83 F1 on average, against 57.60 / 68.86 for the strongest baseline (MemGraphRAG + IRCoT). The paper also reports ablations, folding token ratios, DEG quality metrics, retrieval latency and a multi-turn recall analysis.

Critical strengths
  1. The design is clean and easy to inspect. Each DEG triple has to cite a chunk ID from the latest observation (Eq. 3, Tables 4–5), so the working memory can be checked against its sources. The case study in Tables 10–11 shows clearly what each folding level keeps. I also think the separation in §3.2.2 between "what the model has established" and "what raw text it still sees" is a useful way to frame context management in agentic RAG.

  2. The retrieval change is lightweight and the cost is reported honestly. Query-conditioned weights reuse query–fact and query–passage scores that HippoRAG 2 already computes (§3.3.2), and Table 3 shows total latency is essentially unchanged (at most 0.6% difference, which I checked: 396.4 vs. 394.0 ms on 2Wiki). The synonym-edge analysis in A.3.2 makes a real point. Static synonym weights of at least δ_syn = 0.8 are on a different scale from query–fact cosine scores, so they can soak up transition mass. Rescaling them to the mean of the endpoints' fact weights is a sensible fix.

  3. The analysis section is more candid than usual. The authors say outright that multi-turn interaction hurts on HotpotQA (§4.2, Table 2 "w/o multi-turn" gets 63.50 EM, above every full variant). Figure 5 / C.2 stratifies recall by the number of gold passages, which explains why the gains concentrate on MuSiQue. The CR gains I checked against Figure 3 (45.7→69.1 and 84.8→98.9) match the text.

Critical weaknesses
  1. The gains are hard to attribute to the DEG. The "w/o DEG" scaffold in Table 2 already reaches 59.43 EM and beats every baseline, including HippoRAG 2 + Search-R1 by 5.3 EM (10.3 on MuSiQue). Yet the retriever change alone is worth only about 1.4 EM single-turn, and the DEG plus frontier reranking add roughly 1.3 EM, so baselines should be rerun inside the same scaffold.

  2. There are no significance tests or repeated runs, and many differences look like noise. The folding variants differ by 0.17 average EM. Removing relation-aware weighting costs 0.21 F1 and raises MuSiQue F1, and "w/o frontier reranking (Medium)" beats the full model on HotpotQA (Table 2). One greedy run on 1,000 questions can't support "consistently outperforms."

  3. The folding claims are tested where folding barely matters. With initial retrieval the reasoner averages only 1.4–2.4 turns (Table 8). Algorithm 1 line 15 also passes all raw observations to FORCEANSWER, so heavy folding doesn't remove raw text in exactly the long, budget-exhausting cases where it should matter.

Questions for authors
  1. Can you report HippoRAG 2 and MemGraphRAG plugged into your exact multi-turn scaffold (same prompt, initial retrieval, force-answer, 5×5 budget) without the DEG and frontier reranking? Can you also explain why HippoRAG 2 + Search-R1 on MuSiQue (34.70 EM) falls below single-turn HippoRAG 2 (37.20)?

  2. How often is FORCEANSWER triggered on each dataset? Since it receives ∪O_i, what happens to the heavy-folding results if it only gets the DEG?

  3. Figure 3 compares single-turn recall at top-5 with the union of up to roughly 25 passages over turns. What is single-turn R/CR at matched passage budgets (top-10, top-15, top-25)? And how does single-turn QA with top-10/15 passages compare to the multi-turn system?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.