- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MemTrace, a graph-based retrieval framework for long-term conversational memory. It addresses two limitations of existing methods: incomplete evidence discovery for complex queries and insufficient source-level evidence ranking. MemTrace uses adaptive query decomposition and independent PCST-based subgraph retrieval to improve evidence coverage, followed by provenance-aware source scoring that combines relation support with contextual semantic similarity. Experiments on LongMemEval-S and LoCoMo demonstrate improvements in retrieval and downstream QA, supported by ablation studies and controlled ranking comparisons.
Critical strengths
1. Clear problem formulation. The distinction between evidence discovery and source attribution is meaningful. It highlights an important limitation of graph-based retrieval: retrieving relevant graph structures does not necessarily identify the original memories containing the required evidence.
2. Convincing controlled experiments. The improvements on two benchmarks are encouraging. In particular, the fixed-candidate-pool experiments provide useful evidence that provenance-aware scoring improves source ranking beyond semantic similarity and PPR-based methods.
Critical weaknesses
1. Limited evidence of generalization across memory graphs.
MemTrace assumes a preconstructed memory graph, but the main comparisons involve different memory systems with their own graph construction pipelines. Consequently, the reported gains may partly reflect differences in graph quality or representation rather than the retrieval method itself. Although the controlled experiments isolate individual components within the proposed graph, they do not establish whether the complete retrieval framework remains effective across different graph construction approaches. This limits the generality of the claimed contribution.
2. Insufficient differentiation from existing GraphRAG techniques.
The proposed framework largely combines established techniques, including query decomposition, local graph retrieval, and source reranking. Similar mechanisms already exist in GraphRAG literature. For example, DRIFT Search [1] generates follow-up questions and performs iterative local retrieval, while Microsoft GraphRAG's local-search implementation [2] prioritizes source text units using their associated graph relationships. The paper also adopts the PCST retrieval formulation from G-Retriever.
While MemTrace introduces query-dependent weighted relation provenance scoring, its incremental contribution over existing graph-to-source ranking strategies remains insufficiently established.
[1] Whiting et al., Introducing DRIFT Search: Combining Global and Local Search Methods to Improve Quality and Efficiency. 2024, Microsoft Research.
[2] Edge et al., From Local to Global: A Graph RAG Approach to Query-Focused Summarization. 2024, arXiv:2404.16130.
Questions for authors
1. Cross-graph generalization: Can the authors evaluate MemTrace using a memory graph constructed by a different method? This would help establish whether the proposed retrieval and ranking mechanisms generalize beyond the particular graph representation used in this paper.
2. Incremental novelty: How does the proposed source evidence scoring differ substantively from existing GraphRAG source-ranking mechanisms? In particular, can the authors compare against an unweighted relation-count baseline under the same retrieved subgraphs and candidate pool to isolate the benefit of query-dependent provenance weighting?
4 comments
[W2, Q2] What distinguishes the source score from existing GraphRAG source ranking, and does weighting outperform counting?
Thank you for highlighting existing source-ranking mechanisms in GraphRAG. We agree that provenance links and source prioritization are not new concepts, and do not claim query decomposition or PCST as techniques unique to our work.
The fundamental difference lies in how retrieved graph relationships are transformed into source-level relevance scores. In GraphRAG Local Search, retrieved entities serve as anchors for collecting associated relationships and source TextUnits. Each candidate TextUnit is prioritized according to the number of relationships associated with the retrieved entities. However, this counting strategy treats each qualifying relationship equally, without explicitly distinguishing its contribution to the current query. Consequently, a relationship-dense TextUnit may receive a higher ranking simply because it contains more entity-associated relationships, even when only a small subset provides direct evidence for the query. This makes relationship counts an imperfect proxy for the actual evidential relevance of a source.
MemTrace adopts a fundamentally different scoring procedure. For each decomposed query, PCST selects a query-conditioned relational subgraph, where individual edges retain their respective query-dependent prizes. Instead of counting the relationships associated with a source text, MemTrace traces each selected edge back to its originating dialogue turn through provenance mappings and accumulates its prize as evidence support for that turn. The resulting turn-level provenance score is normalized and combined with semantic similarity to determine source relevance. At the final stage, turn-level scores are aggregated at the session level to retrieve the corresponding conversational context.
Thus, while GraphRAG ranks TextUnits based on the number of relationships associated with retrieved entities, MemTrace explicitly models the individual evidential contributions of retrieved relations and attributes their query-dependent importance to their original dialogue turns. This allows relationships associated with the same entity to contribute differently to source ranking, enabling a turn containing fewer but more query-relevant relations to be prioritized over one containing a larger number of less relevant relations.To isolate weighting, we fix the PCST-selected relations, candidate turns, semantic scores, gold labels, normalization, and 0.8/0.2 fusion, replacing only positive-prize weights with 1:
| Graph setting | Source support | R@3 | R@5 | NDCG@5 | R@10 | NDCG@10 |
|---|---|---|---|---|---|---|
| Original setting | Query-dependent prize weighting | 67.28 | 74.53 | 69.67 | 83.74 | 72.79 |
| Original setting | Unweighted relation counting | 65.64 | 73.72 | 68.70 | 83.34 | 72.13 |
| GPT-constructed graph | Query-dependent prize weighting | 69.66 | 76.56 | 72.80 | 86.39 | 75.95 |
| GPT-constructed graph | Unweighted relation counting | 68.16 | 75.46 | 71.83 | 85.94 | 75.16 |
Weighting improves all five reported metrics in both settings, with R@3 gains of 1.64 and 1.50 percentage points. This supports an incremental benefit from query-dependent weighting after retrieval. Without uncertainty estimates, we do not describe these differences as statistically significant. This counting control also does not reproduce every detail of native GraphRAG source ranking, and we will limit the comparison accordingly.
Dear Reviewer dz3x,
Thank you for your feedback. We hope our response has clarified the points you raised. If any concerns remain, we would be happy to provide further details.
Best regards,
The Authors
Thank you for the author's response. I will consider maintaining my assessment. I hope the author will incorporate these comments into future revisions.
[W1, Q1] Does MemTrace generalize across graph construction methods?
Thank you for pointing out that the original evaluation does not sufficiently separate retrieval from graph construction. We additionally evaluate MemTrace on graphs constructed by GraphRAG, LightRAG, and MAGMA:
The corresponding QA improvements are 14.22, 23.52, and 2.57 percentage points. These results support applicability across the tested graph constructions, while the varying gains and increased MAGMA token usage preclude a graph-independent quality or cost claim.
These are downstream QA pipeline comparisons, rather than isolated retrieval comparisons under fully matched context budgets. Our supplementary evaluation uses binary correctness judgments, which should not be directly mixed with published scores using partial credit. We will specify the reader, judge, evaluation set, and Top-k settings and limit the conclusions to the evaluated configurations.