- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This project proposes FactTrace, a memory framework for LLM agents that maintains current and historical facts while preserving their provenance. It uses LLM-based comprehension to resolve conflicting information and update memory without deleting previous facts. Experiments on AMemGym and AI-LieDar show improvements in state tracking and truthfulness.
Critical strengths
Meaningful problem: Maintaining accurate and trustworthy memory as user information changes is an important problem for long-term LLM agents.
Practical design: Separating current facts, historical facts, and inferred information while preserving provenance is a reasonable and useful approach.
Positive results: The method achieves improvements on both benchmarks. The authors also provide several ablation studies and statistical analyses.
Critical weaknesses
Limited methodological novelty: Dynamic state updates, conflict resolution, historical retention, and provenance tracking have already been explored in works such as Zep/Graphiti, Hindsight, Memanto, and CUPMem. The proposed framework appears to combine several existing ideas without clearly demonstrating what is fundamentally new. These closely related works are also missing from the discussion and experimental comparisons.
Gaps between benchmarks and claimed contributions: AMemGym mainly evaluates current-state tracking, while AI-LieDar focuses on truthfulness under social pressure rather than memory evolution. Neither directly evaluates historical rollback, provenance auditing, or recovery from incorrect updates. Therefore, the experiments only partially support the claimed capabilities of FactTrace.
Insufficient mechanism isolation: The truthfulness improvement may partly come from explicit response instructions requiring the assistant to disclose verified facts, rather than the proposed memory architecture. Although the authors provide some ablations, they do not sufficiently separate the effects of provenance labeling, memory updates, and response prompting. A simpler versioned memory baseline with the same response policy would be particularly useful.
Experimental reporting and robustness: The paper should clarify whether the 60 AI-LieDar scenarios include the original scenarios or their paraphrased variants. Additionally, the reported 50.9% truthfulness for Mem0-G does not directly correspond to an integer number of successful cases out of 60 under binary evaluation. The single-seed experiments and potential dependence among questions from the same user also limit the statistical conclusions.
Questions for authors
What are the key differences between FactTrace and existing temporal or versioned memory frameworks such as Zep and Memanto? Could the authors provide direct comparisons?
How much of the truthfulness improvement comes from memory management rather than the response prompt?
Can the authors provide evaluations of historical recovery, provenance correctness, and rollback after incorrect updates?
Could the authors clarify the AI-LieDar scenario selection and the Mem0-G result?
Reminder: the discussion period closes today
Dear Reviewers jB3B, ifCh and urRg,
As the discussion period closes today (Nov 18 AoE), we would be grateful to know whether our responses and the revised PDF address your concerns. The main points, in brief:
We are happy to clarify anything further before the discussion closes.