- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper is to tackle the problem of how to construct long-term memory in a conversation to LLM agent. There are two major challenges when building the long-term memory in the existing systems: (1) the loss of important details and contextual information during memory construction, and (2) the difficulty of efficiently retrieving relevant evidence scattered across multiple dialogue turns and sessions. The authors aim to build a memory system that preserves the context necessary to understand individual facts while enabling efficient retrieval of related evidence across sessions.
To achieve this goal, the authors propose ContextFact framework, a two-view memory framework. The main contributions include: (1) conducting diagnostic experiments to investigate how memory construction and retrieval granularity affect answerability and evidence accessibility; (2) proposing context-preserving fact memories that retain relevant information across nonadjacent dialogue turns; and (3) introducing source-grounded entity representations that connect related facts across sessions and support entity-linked retrieval and reasoning.
The authors evaluate ContextFact on two long-term conversational memory benchmarks, LoCoMo and LongMemEval-S, against several existing memory systems. They also conduct controlled experiments and ablation studies to validate the effectiveness of the proposed components. Overall, the experimental results show that ContextFact achieves better QA accuracy than the evaluated baselines while using significantly fewer evidence tokens.a
Critical strengths
-
The comprehensiveexperimental evaluation is one of the strengths of this paper. The authors compare ContextFact with many existing baselines and evaluate it using different LLM backbones. They also test on both LoCoMo and LongMemEval-S, and include detailed ablation studies to validate different components.
-
The proposed method achieves the best or second-best results on many tasks in Table 1, which demonstrates its effectiveness. What's more, the method uses significantly fewer evidence tokens than the strongest baseline.
-
I like the idea of preserving the context of individual facts and connecting related information across sessions through entity representations. The combination of these two memory views is a good design choice, and the experimental results show that it works well in practice. For example, the entity view helps improve multi-hop QA accuracy from 76.95% to 84.04% on LoCoMo with Qwen3.5-27B.
Critical weaknesses
-
The technical novelty of this paper is somewhat limited. The proposed framework mainly combines existing techniques, such as context-aware memory extraction, entity linking, graph-based memory construction, and hybrid retrieval with reranking. While the combination works well, I do not see a fundamental difference from existing structured or hierarchical memory systems, such as Memora and Zep.
-
Although the proposed method achieves the best overall results, the improvements over the strongest baseline are relatively small. For example, the improvement is 2.20 percentage points on LoCoMo and only 1.80 points on LongMemEval-S with Qwen3.5-27B. I would like to see statistical significance tests or confidence intervals to better understand how reliable these improvements are.
-
The entity view does not always improve performance. For example, on open-domain questions in LoCoMo, ContextFact-E performs worse than ContextFact-F with both GPT-4.1-mini (59.38% vs. 64.58%) and Qwen3.5-27B (51.04% vs. 54.17%). I find this somewhat surprising because the entity view is expected to provide additional evidence rather than hurt the results.
Questions for authors
-
How would ContextFact handle new conversation sessions in a continuously running agent? In particular, does the source-grounded entity reconstruction need to be repeated when new information arrives? I am curious about the efficiency and scalability of this process.
-
Why does adding the entity view sometimes lead to worse results, especially on open-domain questions? If the additional entity information does not conflict with the original ContextFacts, I would expect the performance to at least remain similar. Could this be caused by irrelevant information, redundancy, or changes in the retrieved facts? It would be helpful to see some failure-case analysis.
5 comments
W2. Reliability of the reported improvements
"Although the proposed method achieves the best overall results, the improvements over the strongest baseline are relatively small. ... I would like to see statistical significance tests or confidence intervals to better understand how reliable these improvements are."
To assess variability in the reported results, we conducted three runs for each method and dataset. All accuracy columns report the percentage of answers judged correct, summarized as mean ± sample standard deviation across three runs. The following results use Qwen for both memory construction and answer generation, with GPT-4o-mini as the judge.
1. LoCoMo: full-pipeline reruns.
We conducted three independent runs of the full pipeline for ContextFact and Memora (P), rebuilding the memory bank and repeating answer generation and LLM judging in each run. We used the same configuration as in our main experiments. These results capture variation across memory construction, answering, and judging.
| Method | Overall | Single-hop | Temporal | Multi-hop | Open-domain |
|---|---|---|---|---|---|
| Memora(P) | 84.68 ± 0.68 | 90.57 ± 0.56 | 82.04 ± 1.26 | 81.32 ± 1.43 | 51.74 ± 1.59 |
| ContextFact-F | 84.70 ± 0.80 | 90.05 ± 0.54 | 87.23 ± 1.36 | 76.48 ± 1.08 | 53.47 ± 0.60 |
| ContextFact-E | 86.41 ± 0.10 | 90.41 ± 0.25 | 89.62 ± 0.36 | 82.86 ± 1.14 | 51.04 ± 1.04 |
2. LongMemEval-S.
We kept each method's constructed memory bank fixed and performed three runs of answer generation and LLM judging. These results capture variation in answering and judging conditional on the constructed memories.
| Method | Overall | Single-session User | Single-session Assistant | Multi-session | Temporal Reasoning | Knowledge Update | Single-session Preference |
|---|---|---|---|---|---|---|---|
| Memora(P) | 89.20 ± 0.20 | 98.10 ± 0.82 | 98.21 ± 0.00 | 80.95 ± 0.43 | 85.96 ± 1.57 | 94.44 ± 0.74 | 88.89 ± 5.09 |
| ContextFact-F | 87.53 ± 0.50 | 96.19 ± 0.82 | 96.43 ± 0.00 | 80.45 ± 0.00 | 84.71 ± 1.57 | 91.03 ± 0.00 | 85.56 ± 1.92 |
| ContextFact-E | 91.20 ± 0.20 | 99.05 ± 1.65 | 98.81 ± 2.06 | 83.46 ± 0.00 | 89.72 ± 1.15 | 94.87 ± 2.22 | 90.00 ± 0.00 |
On LongMemEval-S, ContextFact-E's mean overall accuracy exceeds Memora (P)'s by 2.00 percentage points, with a run-to-run standard deviation of 0.20 points for each method.
Q1. Online memory updates and reconstruction costs
"How would ContextFact handle new conversation sessions in a continuously running agent? In particular, does the source-grounded entity reconstruction need to be repeated when new information arrives? I am curious about the efficiency and scalability of this process."
We agree that online memory updating is important for long-running agents, which must incorporate new information while maintaining coherent memories across sessions. ContextFact already provides a starting point through incremental fact insertion and cross-session entity binding. For each new session, it extracts ContextFacts and entity entries, appends the facts to the fact bank, and matches the entries to existing provisional entities or creates new ones. This process does not require repeating fact extraction over earlier sessions (lines 245-252; Algorithm 1, steps 3-8).
Regarding whether source-grounded reconstruction needs to be repeated, yes—in the current method, incorporating new sessions into the final entity view requires running reconstruction again using the updated source evidence. An online extension could reduce both how often reconstruction runs and how much source evidence it revisits.
To reduce reconstruction overhead while keeping new information accessible, we could separate immediate incorporation of new facts from deferred reconstruction of entity records. Newly extracted facts could become available for retrieval while their source turns accumulate in buffers associated with the affected entities. Reconstruction could then be triggered by accumulated evidence or a detected inconsistency, allowing multiple updates to be processed together.
A second consideration is to localize reconstruction while preserving access to source evidence. An update could reconsider the affected entity and, when identity decisions require it, its local review group. Reconstruction could use the current records and newly buffered turns, with provenance links providing access to earlier dialogue when needed. This would allow previous interpretations to be reconsidered as new information arrives. Batching updates and restricting their review scope could reduce repeated reconstruction work while keeping recent facts accessible.
Scalability also calls for compression and pruning to control the growth of active memory. An online extension could consolidate overlapping records, remove redundant entries, and archive older records while retaining their provenance for later retrieval. Each reconstruction call could select relevant records and source evidence within an explicit token budget. This would keep update inputs manageable while preserving access to historical evidence needed for temporal questions or identity correction.
Evaluating this extension would require studying reconstruction triggers, compression and retention policies, and consistency after entity merges or splits. These choices must balance long-term answer accuracy, memory growth, and update cost. We regard reliable online memory updating as an important future direction, building on the incremental binding and source provenance already present in ContextFact.
We thank you for recognizing the experimental evaluation, evidence efficiency, and complementary roles of the two memory views. We address your concerns and questions below point by point, along with the corresponding revisions in the manuscript (highlighted in blue).
W1. Contribution beyond existing memory approaches
We would like to clarify how ContextFact differs from Memora, Zep, and related memory systems. The distinction concerns the criterion for defining independently retrievable facts and the source-grounded reconstruction of cross-session evidence for the reader.
First, we formulate and validate an explicit principle for memory-unit boundaries: memory boundaries should follow interpretive dependencies while keeping distinct facts independently retrievable. ContextFact implements this principle by grouping each focal fact with the context needed to interpret it, including information from nonadjacent turns. This specifies which information should remain together to balance selective retrieval and contextual completeness.
Our diagnostic experiments motivate this principle (lines 179-184). Under matched evidence-token budgets, ContextFact-F achieves the highest overall judge accuracy among the evaluated memory systems at every tested budget on LoCoMo. Comparisons with sentence-level and atomic decompositions of the same memory bank further support the resulting unit design (lines 366-397; Tables 9-10).
Second, we reconstruct the entity view from the source dialogue after cross-session alignment. Memora connects concrete memories through primary abstractions and cue anchors, while Zep represents episodes, entities, relations, and temporal facts in a knowledge graph. ContextFact revisits the dialogue associated with aligned referents to reassess provisional groupings and reconstruct coherent accounts of entities' aspects and developments. These accounts retain links to supporting facts and are supplied directly to the reader (lines 772-782).
The entity view therefore participates in both evidence discovery through linked facts and evidence organization through reconstructed accounts. Controlled ablations show that fact-plus-entity readout outperforms additional-fact controls on multi-hop questions, even when those controls supply more evidence tokens (Table 3; lines 422-431).