- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The work introduces a benchmark with four types of tasks to test the abilities of conversational memory systems, plasticity, interference robustness, contextual integrity, and forgetting. The benchmark also adds 3 difficulty types for each task which add more distractors or stressors as the difficulty increases. Results across Mem0, Cognee, Letta, HippoRAG show that existing systems struggle to perform well on these tasks. The authors do some analysis demonstrating that the performance decreases as more distractors are added or the difficulty of the task is increased.
Critical strengths
- The work introduces some interesting tasks that are highly relevant to building strong, scalable memory systems.
- The benchmark seems sufficiently difficult with tested memory systems scoring well below 50% on most tasks, meaning it can still provide a useful signal for comparing memory systems.
Critical weaknesses
- The novelty of the tasks seems limited; the tasks mostly repurpose existing benchmarks for each particular task. Table 1 also shows that all of the tasks exist in at least a handful prior works; the only novelty then is that this work packages variants of those tasks into one dataset
- The results and insights from the paper are not surprising nor any different than what has been established by previous works. It is not surprising that reinforcing a point many times, or having similar distractor memories would cause it to pollute the memory (in the case of interference robustness and plasticity tasks). As another example, it's also unsurprising that an LLM might leak private information that is retrieved from memory. The paper does not do any analysis into what different performance differences across tasks or memory architectures means, or how a practitioner can use this benchmark to build better memory systems.
- There are some concerns about the evaluation setup (see Question 3, 4 below), although its possible these could be mitigated with additional clarification
Questions for authors
Clarification questions
- I don't understand how the contextual integrity task is testing anything about a memory system; the tested memory systems provide memories to an agent during answer-time. Isn't it true that whether or not an agent leaks a private fact is a property of the model and its instruction-following/alignment capabilities rather than a property of a memory system?
- Could you provide more details on what is considered a "private fact" in the contextual integrity task, and some examples of those? Is the model explicitly told that this fact is private and should never be directly disclosed? Or is the assumption that the model should just be able to infer this?
Questions about soundness
- My understanding is that for Contextual Integrity, a memory system that is very bad at retrieval would score high because it would just fail to retrieve the private information; doesn't this confound the numerical results for this task?
- For Letta, Cognee, HippoRAG, the paper mentions that it uses the API to manually delete all memories that mention the target for the Forgetting task. However, by doing this the memory system is no longer making a decision on its own. Doesn't this confound the original motivation of the task which was to "forget everything about [the target] while preserving information about others" (Section 3.4)?
- Table 5 says that when evaluating Cognee the temporal pipeline, contradiction detection are turned off. Doesn't this invalidate the Cognee evaluation on the Plasticity task, as these features seem to be especially relevant?
Questions about novelty
- Are there any fine-grained, novel insights from the benchmark results in Table 3? Findings 1, 2, 3 mostly seem to illustrate that having more stressors or distractors degrades performance, which is a general properties of memory system that has already been established by prior work