- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether selective deletion in agent memory systems effectively removes targeted user preferences while preserving unrelated information. Deletion is evaluated at three levels: stored memory, retrieved context, and downstream agent behavior, measuring behavioral residue against a NeverSeen baseline. The results show that deleting memory records does not necessarily eliminate all representations of a preference, while broader deletion may inadvertently remove unrelated preferences.
Critical strengths
- The paper investigates selective forgetting in persistent agent memory. Its distinction between record deletion and behavioral forgetting is well motivated.
- Evaluating deletion across stored memory, retrieved context, and downstream actions provides a more informative assessment than checking record removal alone. The inclusion of target removal, non-target retention, and the NeverSeen counterfactual strengthens the evaluation methodology.
- The experiments show failure modes, including delayed reappearance of deleted information in Zep and differences in behavioral residue across reading models. The subgroup analysis further demonstrates how near-zero aggregate residue can conceal systematic forgetting failures.
Critical weaknesses
- All experiments use synthetic dashboard deployment scenarios involving explicit preferences and template selection. This constrained environment does not capture the complexity of realistic conversational agents, including implicit preferences, conflicting memories, multi-hop retrieval, and long-horizon workflows. The evaluation should include more diverse tasks and real-world interaction histories.
- Prior studies like MemScope and MemLeak, already investigate residual information and behavioral effects following memory deletion. This paper's primary contribution is an empirical characterization of existing frameworks.
- Lexical matching may miss paraphrased or implicit representations of deleted preferences, potentially underestimating information retention. Moreover, comparison sets differ across frameworks, temporal experiments use only matched scenarios, and most behavioral experiments use a single reading model. These limitations weaken cross-framework comparisons and the generality of the conclusions. It would be useful to add semantic detection, larger temporal evaluations, and additional reading-model configurations.
Questions for authors
How to address the concerns mentioned in the Weaknesses section.
Correction to Zep results
While preparing our response, we re-verified our Zep results and found two that we can no longer support.
1. Stored memory. We reported that after targeted deletion in Zep the target preference "remains in stored memory in 66 of 75 scenarios" while being absent from all 75 returned contexts. We used this contrast to argue that absence from context does not establish removal from storage.
On re-verification, we found that 66/75 counts the scenarios in which Zep refused the request to delete the canonical user node. That observation stands: the node cannot be removed through the interface we tested. However, the persistence of the node does not show that the preference persisted in it, and the original experiment did not read the node's content back. In a re-run of 115 scenarios, the node was still present after deletion, but the target preference was no longer detected in it. Our Zep data therefore do not support the storage-context contrast, and we withdraw that claim.
2. Temporal results. We could not reproduce the temporal pattern in Table 3 and Appendix B. With the same client code, the target did not reappear after source-thread deletion in any of 30 scenarios. We cannot determine the cause of the difference from our original runs. Zep is a hosted service that may have changed between the runs, and we did not retain the full context text of the original runs, so we cannot re-examine them.
We found both issues during our own re-verification and are correcting the record here. These two results should not be relied upon. We regret the error and thank the reviewers for their careful reviews.