- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the issue of "overreliance" in LLM agents utilizing long-term memory. Even when historical memories are accurate and retrieved appropriately, they can still mislead the model’s reasoning if they are only partially relevant to the current task. Through experimental analysis, the authors identify that a partial overlap of evidence between the query and the retrieved memory is a primary driver of this phenomenon. To address this, the paper introduces the MEMTRIM framework. During the memory-writing phase, it parses historical information into structured "evidence" and organizes it using a Trie index. During the reasoning phase, it identifies and filters out redundant or conflicting evidence, while also determining whether to reuse past answers based on task shifts and answer dependencies. Experiments conducted across various LLM backbones, memory frameworks, and benchmarks demonstrate that this approach effectively mitigates memory overreliance—preserving the performance gains derived from useful memories—while incurring only minimal computational overhead.
Critical strengths
This paper addresses the issue of over-reliance on agent memory during the reasoning process. The research motivation is clear, and the work holds practical application value.
The authors conducted experiments across various LLM backbones, memory frameworks, and benchmarks, while also providing ablation studies and safety evaluations. The results demonstrate that the proposed method effectively mitigates the negative impact of erroneous historical memories while preserving those that are useful.
Critical weaknesses
-
Insufficient methodological innovation. The proposed MEMTRIM framework primarily functions by parsing historical memories into structured "evidence," then utilizing a Trie index and rule-based matching to filter out redundant, conflicting, or irrelevant information. This approach largely mirrors existing concepts in memory deduplication, conflict resolution, and RAG context filtering; it essentially combines and applies established techniques rather than introducing significant algorithmic innovation. Furthermore, the paper fails to clearly articulate the fundamental distinctions between MEMTRIM and existing memory management methods.
-
Lack of comprehensive comparison with mainstream memory methods. While the current experiments include frameworks such as Mem0, A-Mem, MemGPT, and Graphiti—alongside various retrieval-filtering and context-compression techniques—they omit systematic comparisons with other representative agent-memory methods, such as MemoryBank, LightMem, MemoryOS, MemP, and LangMem. Additionally, recent approaches that optimize memory based on task feedback (e.g., MemSkill, MemRL) or employ query-aware memory routing (e.g., BudgetMem) warrant inclusion or further discussion. It is recommended that the authors incorporate these baselines using a unified benchmark, backbone model, and experimental setup to more robustly demonstrate the effectiveness and novelty of their approach.
-
Absence of validation in realistic long-term memory scenarios. The evaluation is primarily based on task pairs derived from RippleEdits, CounterLogic, and CLADDER—datasets that align well with the specific mechanisms MEMTRIM employs for handling evidence conflicts and reusing historical answers. To better assess the framework's generalization capabilities in complex, multi-turn interactions and dynamic memory environments, the authors should conduct further evaluations using more realistic long-term memory benchmarks, such as LoCoMo or MemoryAgentBench.
-
Heavy reliance on structured information extraction, with inadequate analysis of errors and costs. The effectiveness of MEMTRIM hinges on the parser’s ability to accurately extract "evidence units," "task signatures," and "answer support sets"; however, the paper lacks a systematic assessment of the accuracy of these critical components. Moreover, the authors should provide details regarding the parser’s model configuration, additional computational/API costs, and end-to-end latency. Analyzing how parsing errors impact final performance is also essential to verify the reliability and efficiency of the method for practical deployment.
Questions for authors
Please see above.
9 comments
Weakness 2: Lack of Comprehensive Comparison With Mainstream Memory Methods
Lack of comprehensive comparison with mainstream memory methods. While the current experiments include frameworks such as Mem0, A-Mem, MemGPT, and Graphiti—alongside various retrieval-filtering and context-compression techniques—they omit systematic comparisons with other representative agent-memory methods, such as MemoryBank, LightMem, MemoryOS, MemP, and LangMem. Additionally, recent approaches that optimize memory based on task feedback (e.g., MemSkill, MemRL) or employ query-aware memory routing (e.g., BudgetMem) warrant inclusion or further discussion. It is recommended that the authors incorporate these baselines using a unified benchmark, backbone model, and experimental setup to more robustly demonstrate the effectiveness and novelty of their approach.
Thank you for these valuable suggestions and for highlighting additional developments in agent memory. We agree that these methods are relevant to the broader research landscape. However, we would like to clarify an important distinction in our experimental design: we evaluate memory backbones and inference-time mitigation strategies at two different levels, because they serve different purposes.
First, our four memory systems are used as underlying memory backbones. Mem0, A-Mem, MemGPT, and Graphiti employ different approaches to memory storage, representation, organization, and retrieval. MEMTRIM does not aim to replace their memory architectures. Instead, it operates after their original retrieval process and transforms the retrieved evidence before it is provided to the answering model.
Evaluating MEMTRIM across these four systems therefore directly examines whether the same inference-time mechanism can work with different underlying memory architectures. This is important for our plug-and-play objective because MEMTRIM is intended to complement existing memory systems rather than introduce another independent memory architecture.
Second, our nine mitigation baselines directly compare how retrieved memory is used during inference. These baselines cover different strategies, including:
- Memory selection: Top-1 and MMR.
- Similarity-based filtering: Similarity Filter.
- Context compression: RECOMP and LLMLingua-2.
- Historical-answer removal: Facts-only.
- Conflict-aware and answer-oriented reasoning: Context-faithful, Answer-first, and Micro-Act.
Within each memory backbone, these methods operate on the same underlying retrieved candidates. This allows us to examine how different inference-time strategies affect the model's predictions without mixing their effects with changes in memory storage or retrieval.
The two levels of comparison therefore address complementary research questions.
The comparison across memory backbones examines whether MEMTRIM can be applied consistently to different memory architectures. The comparison across mitigation strategies examines whether MEMTRIM provides an effective way to process the same retrieved information.
Table 2 presents the main mitigation comparison using Mem0, while Appendix B.3 extends the evaluation to A-Mem, MemGPT, and Graphiti. Together, these experiments demonstrate that the effectiveness of MEMTRIM is not restricted to one particular underlying memory system.
Regarding the additional methods suggested by the reviewer, MemoryBank, LightMem, MemoryOS, MemP, and LangMem provide relevant perspectives on memory organization, maintenance, and retrieval. Similarly, MemSkill, MemRL, and BudgetMem are important related approaches to memory learning, optimization, and query-aware memory use.
We agree that these methods deserve clearer discussion in our paper. We expanded Section 5 to position them more explicitly according to their objectives and intervention points.
At the same time, comparing different complete memory architectures is not equivalent to comparing different inference-time memory mitigation strategies under the same retrieval conditions. Our evaluation primarily focuses on the latter, while testing the proposed mechanism across four representative memory backbones.
We have clarified this baseline-selection rationale in Section 4 and strengthen the Related Work discussion. This makes the scope of our comparison more explicit without conflating memory-system design with inference-time memory-reuse decisions.
Weakness 3: Absence of Validation in Realistic Long-Term Memory Scenarios
Absence of validation in realistic long-term memory scenarios. The evaluation is primarily based on task pairs derived from RippleEdits, CounterLogic, and CLADDER—datasets that align well with the specific mechanisms MEMTRIM employs for handling evidence conflicts and reusing historical answers. To better assess the framework's generalization capabilities in complex, multi-turn interactions and dynamic memory environments, the authors should conduct further evaluations using more realistic long-term memory benchmarks, such as LoCoMo or MemoryAgentBench.
Thank you for this constructive suggestion. We agree that realistic long-horizon memory benchmarks are valuable for evaluating deployed memory-augmented agents. We would like to clarify that our current benchmarks are deliberately designed as controlled diagnostic settings to investigate a specific inference-time failure, rather than to evaluate every capability required for long-term conversational memory.
The central research question of our paper is whether a previously correct and appropriately retrieved memory can still mislead an LLM when the current task differs from the historical interaction.
Investigating this failure requires distinguishing incorrect historical-answer reuse from other sources of errors, such as retrieval failures, missing memories, or incorrect information stored during memory construction. In realistic multi-turn settings, these factors may change simultaneously, making it difficult to identify which component causes a particular prediction error.
Our benchmark construction addresses this challenge by using controlled historical and current query pairs. The three benchmark families examine complementary forms of task divergence.
RippleEdits considers changes in factual information, allowing us to study situations in which a previously correct answer becomes outdated.
CounterLogic examines changes in the question being asked. In particular, it distinguishes logical validity from real-world truth, allowing us to investigate whether a historical answer is incorrectly reused across different reasoning tasks.
CLADDER-Derived examines related causal scenarios in which answer-relevant evidence differs between the historical and current interactions.
These settings allow us to analyze memory behavior under task-aligned (TA) and task-divergent with duplicated evidence (TD-Dup) conditions. They also make it possible to measure memory-induced prediction changes, rather than relying only on overall task accuracy.
For example, Table 1 reports both correct-to-incorrect and incorrect-to-correct prediction changes after memory is introduced. Figures 2 and 3 further examine how query-memory evidence alignment relates to these outcomes, while Figure 5 evaluates the contributions of different MEMTRIM components.
These analyses are particularly useful because they provide a more direct understanding of when previously correct memory becomes harmful and which inference-time decisions help prevent its inappropriate reuse.
We agree that LoCoMo and MemoryAgentBench provide complementary settings for evaluating agents over longer interaction histories, where memory retrieval, updates, and user information may evolve across multiple turns.
However, such benchmarks examine a broader collection of memory capabilities than the particular inference-time failure studied here. Their value is complementary to, rather than a replacement for, controlled diagnostic evaluation.
Our current evaluation already spans two answering models, four memory architectures, three diagnostic benchmark families, nine mitigation strategies, and several ablation settings. This provides evidence of MEMTRIM's effectiveness across the tested conditions, although it does not establish performance in every possible long-horizon deployment scenario.
We have clarify this distinction in the Evaluation section and make the scope of our conclusions more explicit. We believe the controlled benchmark design is appropriate for identifying the failure mechanism and evaluating the proposed mitigation, while long-horizon benchmarks would provide an additional perspective on broader generalization.
We thank the reviewer for recognizing the practical motivation of memory overreliance and the effectiveness of MEMTRIM across different models, memory systems, and evaluation settings. We appreciate the opportunity to further clarify our methodological contribution, experimental design, evaluation scope, and implementation details.
Weakness 1: Insufficient Methodological Innovation
Thank you for raising this important point and giving us the opportunity to clarify the methodological contribution of MEMTRIM. We would like to emphasize that our main contribution is not trie indexing or evidence deduplication alone, but a support-aware and task-aware mechanism that determines how historical evidence and previous answers should be reused during inference.
First, MEMTRIM addresses a different problem from conventional memory filtering. Existing memory systems primarily focus on storing, organizing, updating, and retrieving historical information. However, retrieving a relevant memory does not necessarily mean that its previous answer remains applicable to the current query. A historical interaction can be entirely correct when stored, yet its conclusion may become inappropriate for a new task even when much of its evidence remains useful. This is the specific failure mode studied in our paper.
Second, MEMTRIM distinguishes historical evidence from the outcome supported by that evidence. As defined in Section 3.1, each memory is represented using evidence units, a task signature, a historical outcome, and an answer-support set. Rather than treating all information in a retrieved memory as equally relevant to its previous answer, MEMTRIM explicitly models which evidence supports that outcome.
This distinction directly affects inference-time decisions. Section 3.3 defines the support-aware condition$D_m(q) \cap H_m \neq \emptyset$. A conflict involving unrelated historical information does not necessarily invalidate the stored answer. In contrast, when the conflict affects the evidence supporting that answer, MEMTRIM suppresses the previous outcome. Therefore, MEMTRIM does more than detect or remove conflicting facts: it considers whether those conflicts actually invalidate the historical conclusion.
Third, MEMTRIM explicitly handles task changes. Even when the historical evidence remains consistent, the current query may ask a different question. In such cases, MEMTRIM can convert an applicable previous outcome into task-labeled contextual evidence instead of presenting it as the answer to the new question. This prevents the model from confusing a previously correct answer with the answer required by the current task.
These components jointly address a problem that cannot be resolved by simple deduplication. The trie provides an efficient structure for indexing and matching evidence, while the main methodological contribution lies in deciding which historical information and outcomes should influence the current inference.
Our empirical results further support the effectiveness of this combined mechanism. Table 2 reports the following results with Qwen3.8-27B and Mem0:
All values are accuracy percentages reported in Table 2.
These results show that MEMTRIM substantially improves performance when historical conclusions are no longer transferable, while preserving or improving performance when they remain applicable. Figure 5 provides complementary ablation evidence showing that duplicated evidence, conflicting information, outdated outcomes, and task changes affect memory overreliance in different ways.
We have also clarified the role of answer-support sets and the distinction between evidence-level filtering and outcome-level reuse in Sections 3.1 and 3.3. We further emphasize this contribution in the Method and Related Work discussions to distinguish our proposed inference-time memory-reuse mechanism from conventional memory deduplication and context filtering.