- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper addresses tracking factual changes and integrating distributed evidence across long-term interactions. The proposed method consists of two core components:
- a State-Evolving Memory Graph, which organizes events into temporal chains and maintains semantic states with historical versions and supporting evidence
- Intent-Guided Multi-Round Retrieval, which decomposes queries when needed, assesses retrieved evidence, and iteratively expands retrieval to fill evidence gaps before reranking the final candidates.
Experiments on LoCoMo with two LLM backbones show modest overall gains over MRAgent. On MemConflict, the method improves dynamic-conflict accuracy and several retrieval metrics, but underperforms the strongest baseline in overall answer accuracy, particularly on conditional conflicts.
Critical strengths
- State-Evolving Memory Graph makes sense: using a graph to make evidence traceable
- The evaluation is sufficienty: Beyond answer accuracy, the paper examines evidence retrieval, controller capacity, and graph-edge contributions, helping identify the method’s strengths and limitations.
Critical weaknesses
- The evaluation does not isolate the proposed memory mechanisms. Removing semantic nodes also removes summaries and retrieval targets, so the ablation cannot attribute gains specifically to historical versioning. Evidence coverage metrics also conflate construction omissions with retrieval failures, leaving the source of gains unclear.
- The practical conflict-handling advantage is limited. Average answer accuracy is 47.59%, below the strongest baseline’s 55.39%. Static-conflict accuracy improves by only 0.14 percentage points, while conditional-conflict performance drops substantially. Despite improved retrieval metrics, the answer-level benefits remain uneven.
Questions for authors
- With semantic content, retrieval, and budgets held fixed, what improvement comes specifically from historical versioning and state evolution?
- How often is supporting evidence missing from the constructed memory versus present but not retrieved? Have extraction omissions or incorrect entity merging been measured?
- Does memory construction respect each query’s history prefix? Is the detected question type inferred from the query or supplied by benchmark annotations?
- Were Table 1 baselines rerun or taken from MRAgent? How does Figure 3’s Full multi-hop score of 86.3 correspond to Table 1’s 84.40 ±1.23 across three runs, and are all ablations reported using the same statistical procedure?
10 comments
Response to Reviewer Feedback – Part 2 [ Weakness 1, Question 2 — Part 1 of 2 ]
The final evidence coverage is affected by both the memory construction and the retrieval processes. However, all configurations in Section 5.5 share the same memory construction snapshot. Therefore, the annotated sources not retained during the construction phase remain consistent across all configurations. The differences in coverage across different retrieval rounds in Figure 4 reflect the changes in the number of retrieved annotated evidence items under fixed memory conditions.
We agree that the final evidence coverage itself cannot distinguish between the evidence not retained during the construction phase and the evidence stored in the memory but not retrieved. To address this, we further conducted a phased analysis of the constructed memory and the final retrieval results. We analyzed the logs of three runs for each of the two backbones in the main experiment, with each run containing 2360 valid question and evidence pairs. In our statistics, we directly used the gold evidence annotated in the LoCoMo dataset [1] to calculate the coverage. This means that if a specific annotated source is not retained, it does not imply that equivalent supporting facts expressed by other sources are absent from the memory. In the following table, we report five metrics separately: extraction omissions, extracted but not retained, absent from memory, present in memory but not retrieved, and retrieved. The values represent the mean plus or minus the sample standard deviation across three runs. The first two items are subdivisions of absent from memory. The standard deviation of the total item is calculated from the total proportions of each run.
Table 1: Analysis of evidence source retention and retrieval coverage.
| Backbone | Source not cited in extraction output | Source cited but not retained in final memory | Absent from memory (sum of the first two categories) | Present in memory but not retrieved | Retrieved |
|---|---|---|---|---|---|
| Gemini 2.5 Flash | 3.63 ± 0.13% | 0.18 ± 0.09% | 3.81 ± 0.18% | 23.38 ± 0.67% | 72.81 ± 0.63% |
| Claude Sonnet 4.5 | 2.94 ± 0.09% | 0.34 ± 0.07% | 3.28 ± 0.16% | 19.42 ± 0.25% | 77.30 ± 0.09% |
Table 1 further distinguishes the stages where construction omissions occur. The proportions of extraction outputs from Gemini and Claude that did not cite the annotated sources are 3.63% and 2.94%, respectively. Meanwhile, the proportions of sources that were cited in the extraction outputs but ultimately not retained in the final memory are 0.18% and 0.34%, respectively. This indicates that, when tracked by annotated sources, the construction omissions primarily occur during the extraction output stage. After further aggregating the unretrieved evidence pairs across the three runs, we found that 85.97% for Gemini and 85.56% for Claude of the annotated evidence sources were already present in the memory. Therefore, in terms of the availability of annotated sources, most of the unretrieved evidence still exists in the memory, indicating that there is still room for improvement in the retrieval phase. We will consider this issue in our future work to improve the retrieval coverage of existing evidence and reduce source omissions during the extraction phase.
[1] Maharana, Adyasha, et al. "Evaluating very long-term conversational memory of llm agents." Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.
Response to Reviewer Feedback – Part 3 [ Weakness 1, Question 2 — Part 2 of 2 ]
Regarding entity merging errors, referring to the research by Ravideshik and Kejriwal [2] on judgment bias in the evaluation of large language model entity alignment, we employed GPT-4o-mini to conduct a blind automated audit of the entity merging results. The judge determined whether the entities were identical based on the original dialogue context of the associated parties. The original model, the merging decision, and its rationale were hidden in the audit input. The audit covered a total of 2452 alias associations across three runs for each of the two backbones. The results are as follows.
Table 2: Automated audit results of entity alias associations. The proportions represent the mean plus or minus the sample standard deviation across three runs, with each run using all confirmed and effective model determined alias associations as the denominator.
| Backbone | Alias associations across three runs | Flagged as potentially incorrect | Insufficient evidence | Output validation failure |
|---|---|---|---|---|
| Gemini 2.5 Flash | 425 / 465 / 445 | 9.93 ± 1.27% | 25.89 ± 2.07% | 2.47 ± 0.97% |
| Claude Sonnet 4.5 | 363 / 370 / 384 | 15.83 ± 1.31% | 34.05 ± 1.52% | 2.32 ± 0.39% |
The above analysis supplements the source coverage analysis from the perspective of entity associations. Among the results, the suspected error label indicates that the automated judge determined the association might not hold based on the provided context. The insufficient evidence label indicates that the current audit input cannot support a definitive judgment. Therefore, we report these as automated diagnostic results rather than the true merging error rate verified by humans. Future work will incorporate more complete context and human calibration to improve the reliability of the association audit and further evaluate the actual impact of suspected erroneous associations on downstream retrieval and answering.
[2] Ravideshik, Vaibhava Lakshmi, and Mayank Kejriwal. “Reliability of LLM Judges for Evaluating Entity Alignment.” In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, 39175–39200. Budapest, Hungary: Association for Computational Linguistics. 2026.
Response to Reviewer Feedback – Part 1
Response to Reviewer Feedback:
We appreciate the reviewer’s time and effort in evaluating our submission and thank them for the valuable feedback. In the following, we provide a detailed response to the reviewer’s comment and clarify the relevant issues.
Response to Weakness 1 and Question 1:
We address the reviewer’s concerns about Weakness 1 in two parts. Here, we discuss the contribution of historical versioning and state evolution; the concerns regarding evidence coverage and construction versus retrieval errors are addressed in our response to Question 2 below.
We acknowledge that the original ablation variant without semantic components removes too many components simultaneously, making it difficult to isolate their specific contributions. To address this, we have introduced a new ablation variant without state transition, which removes the explicit semantic evolution edges while keeping all other components and settings unchanged, in order to analyze the contribution of historical state evolution. The experimental results show that, compared to the full model, the performance of this variant decreases by 3.2 pp and 15.6 pp on the Overall and Multi-hop metrics, respectively. In particular, the significant performance drop on the Multi-hop category indicates that, even with the complete semantic state content retained, relying solely on the retrieval mechanism with multiple rounds is insufficient to fully capture the evolution of different historical states. Explicit state transition relations can establish structured connections among historical states, thereby helping the model better acquire the multi-hop evidence required for complex questions. This result further verifies the independent value of state evolution relations in evolutionary memory modeling. The relevant experimental settings and result analysis have been added to Section 5.3 and Figure 3 of the manuscript.