acceptodds
Withdrawn from ICLR 2027

Mem-Behave: Behavioral Stress Testing of Agent Memory Systems

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The growing adoption of persistent language-model agents makes reliable long-term memory increasingly important. Yet existing benchmarks typically evaluate memory at a single difficulty level, leaving unclear how performance changes as a capability is placed under increasing stress. We introduce Mem-Behave, a benchmark for stress-testing persistent agent memory across four capabilities: plasticity, interference robustness, contextual integrity, and forgetting. Each capability is paired with a stressor that can intensify over long-term use, namely reinforcement of outdated information, near-miss competing memories, ambiguity in disclosure context, and entanglement of information that should be forgotten. Built from long multi-session histories, Mem-Behave contains 767 scenarios each evaluated at three stress levels across four memory systems yielding 3,508 probes in total. Our results show that performance declines with increasing stress across all four capabilities and memory systems, but the magnitude and onset of degradation vary substantially across systems and capabilities. Further diagnostic analysis shows that many downstream failures are already apparent in the memories retrieved by the system. Together, these findings demonstrate that single-difficulty evaluation can obscure important differences in memory reliability, motivating stress-based evaluation of persistent memory systems. We will publicly release Mem-Behave upon acceptance.

Primary area:
datasets and benchmarks
Keywords:
agent memory, stress-testing, behavioral evaluation, persistent agents

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rQobOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The work introduces a benchmark with four types of tasks to test the abilities of conversational memory systems, plasticity, interference robustness, contextual integrity, and forgetting. The benchmark also adds 3 difficulty types for each task which add more distractors or stressors as the difficulty increases. Results across Mem0, Cognee, Letta, HippoRAG show that existing systems struggle to perform well on these tasks. The authors do some analysis demonstrating that the performance decreases as more distractors are added or the difficulty of the task is increased.

Critical strengths
  • The work introduces some interesting tasks that are highly relevant to building strong, scalable memory systems.
  • The benchmark seems sufficiently difficult with tested memory systems scoring well below 50% on most tasks, meaning it can still provide a useful signal for comparing memory systems.
Critical weaknesses
  • The novelty of the tasks seems limited; the tasks mostly repurpose existing benchmarks for each particular task. Table 1 also shows that all of the tasks exist in at least a handful prior works; the only novelty then is that this work packages variants of those tasks into one dataset
  • The results and insights from the paper are not surprising nor any different than what has been established by previous works. It is not surprising that reinforcing a point many times, or having similar distractor memories would cause it to pollute the memory (in the case of interference robustness and plasticity tasks). As another example, it's also unsurprising that an LLM might leak private information that is retrieved from memory. The paper does not do any analysis into what different performance differences across tasks or memory architectures means, or how a practitioner can use this benchmark to build better memory systems.
  • There are some concerns about the evaluation setup (see Question 3, 4 below), although its possible these could be mitigated with additional clarification
Questions for authors

Clarification questions

  1. I don't understand how the contextual integrity task is testing anything about a memory system; the tested memory systems provide memories to an agent during answer-time. Isn't it true that whether or not an agent leaks a private fact is a property of the model and its instruction-following/alignment capabilities rather than a property of a memory system?
  2. Could you provide more details on what is considered a "private fact" in the contextual integrity task, and some examples of those? Is the model explicitly told that this fact is private and should never be directly disclosed? Or is the assumption that the model should just be able to infer this?

Questions about soundness

  1. My understanding is that for Contextual Integrity, a memory system that is very bad at retrieval would score high because it would just fail to retrieve the private information; doesn't this confound the numerical results for this task?
  2. For Letta, Cognee, HippoRAG, the paper mentions that it uses the API to manually delete all memories that mention the target for the Forgetting task. However, by doing this the memory system is no longer making a decision on its own. Doesn't this confound the original motivation of the task which was to "forget everything about [the target] while preserving information about others" (Section 3.4)?
  3. Table 5 says that when evaluating Cognee the temporal pipeline, contradiction detection are turned off. Doesn't this invalidate the Cognee evaluation on the Plasticity task, as these features seem to be especially relevant?

Questions about novelty

  1. Are there any fine-grained, novel insights from the benchmark results in Table 3? Findings 1, 2, 3 mostly seem to illustrate that having more stressors or distractors degrades performance, which is a general properties of memory system that has already been established by prior work
Reviewer_zU8EOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Mem-Behave, a benchmark for evaluating the robustness of agent memory systems under increasing stress. It examines four memory capabilities: plasticity, interference robustness, contextual integrity, and forgetting, each evaluated at three stress levels. The benchmark contains 767 scenarios and 3,508 probes, evaluating four memory systems and an in-context baseline. Experiments show that performance generally degrades as stress increases, with different systems exhibiting distinct strengths and failure patterns. Further analysis highlights the role of memory retrieval in downstream failures.

Critical strengths
  1. The paper introduces a stress-based evaluation framework that goes beyond single-difficulty memory benchmarks. Evaluating four important memory capabilities under increasing stress provides a useful perspective on the robustness and reliability of long-term agent memory systems.

  2. The authors evaluate four representative memory systems and an in-context baseline across 767 scenarios and 3,508 probes. The experiments reveal meaningful differences in how memory systems degrade under stress, providing valuable insights into their strengths and limitations.

Critical weaknesses
  1. For contextual integrity and forgetting, different stress levels involve different targets and evaluation probes, rather than increasing stress on the same task. Although the interaction history is held fixed, variations in task content and inherent difficulty may contribute to the observed performance degradation. Additional controls or analysis would strengthen the validity of the stress-based comparisons.

  2. All memory systems are evaluated using a single LLM (Gemini 3.5 Flash Lite) for answer generation and LLM-based grading. Since some failures may originate from the answer model rather than the memory system itself, evaluating additional model families and validating judge reliability against human annotations would strengthen the conclusions.

Questions for authors
  1. The benchmark introduces manually constructed stress levels, such as reinforcing outdated information or adding 5–10 competing memories. How representative are these stressors and their intensity levels of real-world, long-term agent interactions? Could the authors provide further discussion or empirical evidence on how frequently such stress conditions arise in practice?
  2. The results show that no single memory system dominates across all four capabilities. Are there inherent trade-offs between these capabilities, such as stronger memory retention potentially making selective forgetting more difficult, or are the observed differences primarily due to the specific architectures and configurations evaluated?
Reviewer_bY45OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Mem-Behave, a benchmark for evaluating how agent memory systems perform under increasing stress. It covers four capabilities: plasticity, interference robustness, contextual integrity, and forgetting. The benchmark contains 767 scenarios and 3,508 probes. The authors evaluate four memory systems and an in-context baseline, showing that performance generally declines as stress increases, with different patterns across systems and capabilities. Additional analysis examines how retrieval behavior contributes to these failures.

Critical strengths
  1. The paper offers a useful evaluation perspective by measuring how memory performance changes with increasing stress, rather than relying on a single difficulty level.

  2. The benchmark covers four important memory capabilities. The plasticity and interference experiments are carefully designed to control unrelated factors, making their results more convincing.

  3. The evaluation includes several memory systems with different designs. The retrieval analysis provides useful insights into why systems fail, beyond simply reporting accuracy.

Critical weaknesses
  1. The contextual integrity experiments use different private facts and requests across stress levels. As a result, the observed increase in leakage may reflect differences in the tasks rather than contextual ambiguity alone.

  2. The forgetting experiments use different targets with varying numbers of associated entities. Moreover, the systems use different deletion methods, making it difficult to determine whether performance differences come from memory design or evaluation settings.

  3. The paper lacks quantitative human validation of the generated data and LLM-based judgments, as well as statistical uncertainty estimates. This weakens confidence in the reported differences, especially when the changes are small.

Questions for authors
  1. Can you show that contextual ambiguity, rather than differences in private facts or requests, is responsible for the increased leakage?

  2. How much do target complexity and different deletion methods affect the forgetting results? Can the authors provide additional controlled comparisons?

  3. Have the authors measured human agreement with the LLM judges? Can they provide confidence intervals or significance tests for the main results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.