acceptodds
Withdrawn from ICLR 2027

From Deletion to Action: An Empirical Study of Selective Deletion in Agent Memory

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Persistent memory enables large language model (LLM) agents to draw on past interactions. Privacy and other practical needs motivate selective deletion: removing designated information and its influence on subsequent behavior while preserving other useful memories. Yet it remains unclear how well the record-level deletion operations available in existing memory frameworks satisfy this requirement. We empirically assess selective deletion of user preferences across four memory frameworks by tracing deletion operations through stored memory and context to task actions. We measure behavioral residue relative to a NeverSeen baseline with no prior exposure to the target preference. We find that a mismatch between preferences and records can leave alternative representations of the target preference intact or remove non-target preferences consolidated in the same record. Moreover, failure to detect a target preference in context neither establishes its removal from stored memory nor guarantees its absence from later contexts. At the behavioral level, near-zero overall mean residue can mask positive residue in subgroups where the target preference remains detectable. We find that post-deletion behavioral residue varies across reading model configurations within the same memory framework and on the same scenarios. These findings show that record deletion, absence from context, and the disappearance of behavioral influence need not coincide, motivating a joint assessment of stored memory, context visibility, and subsequent task actions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Agent Memory, Memory Deletion, Behavioral Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tb7mOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether selective deletion in agent memory systems effectively removes targeted user preferences while preserving unrelated information. Deletion is evaluated at three levels: stored memory, retrieved context, and downstream agent behavior, measuring behavioral residue against a NeverSeen baseline. The results show that deleting memory records does not necessarily eliminate all representations of a preference, while broader deletion may inadvertently remove unrelated preferences.

Critical strengths
  1. The paper investigates selective forgetting in persistent agent memory. Its distinction between record deletion and behavioral forgetting is well motivated.
  2. Evaluating deletion across stored memory, retrieved context, and downstream actions provides a more informative assessment than checking record removal alone. The inclusion of target removal, non-target retention, and the NeverSeen counterfactual strengthens the evaluation methodology.
  3. The experiments show failure modes, including delayed reappearance of deleted information in Zep and differences in behavioral residue across reading models. The subgroup analysis further demonstrates how near-zero aggregate residue can conceal systematic forgetting failures.
Critical weaknesses
  1. All experiments use synthetic dashboard deployment scenarios involving explicit preferences and template selection. This constrained environment does not capture the complexity of realistic conversational agents, including implicit preferences, conflicting memories, multi-hop retrieval, and long-horizon workflows. The evaluation should include more diverse tasks and real-world interaction histories.
  2. Prior studies like MemScope and MemLeak, already investigate residual information and behavioral effects following memory deletion. This paper's primary contribution is an empirical characterization of existing frameworks.
  3. Lexical matching may miss paraphrased or implicit representations of deleted preferences, potentially underestimating information retention. Moreover, comparison sets differ across frameworks, temporal experiments use only matched scenarios, and most behavioral experiments use a single reading model. These limitations weaken cross-framework comparisons and the generality of the conclusions. It would be useful to add semantic detection, larger temporal evaluations, and additional reading-model configurations.
Questions for authors

How to address the concerns mentioned in the Weaknesses section.

Reviewer_4zABOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether record-deletion operations in existing memory frameworks can remove a designated user preference and its behavioral influence while preserving another designated preference. It constructs 138 dashboard-deployment scenarios spanning 12 preference dimensions, including color, layout, and chart type, and writes interaction histories into Memobase, Mem0, LangMem, and Zep. After deletion, the reader uses tools to inspect four candidate templates and deploy one; exactly one matches the target preference. Each scenario and condition has six probes with five sampled trajectories each, all sharing that condition’s context. Qwen2.5-7B is the primary reader and forms memory in the first three frameworks; Zep uses hosted extraction.

Conditions include Seen, NeverSeen without exposure to the target preference, and different deletion scopes. Targeted deletion attempts to remove identified target records; Zep retains its canonical user node after the interface rejects its deletion. Broader deletion additionally removes related events in Memobase, clears user memory in Mem0, and removes the source thread in Zep’s separate temporal experiments. The targeted LangMem record is the sole entry, so deleting it empties the store. The authors check lexical matches in storage and delivered context, track retention of the designated other preference, and define behavioral residue as the post-deletion target-template selection rate minus the matched NeverSeen rate. The four framework comparisons contain 137, 137, 125, and 75 eligible scenarios, respectively, with paired scenario-bootstrap intervals.

Table 2 shows that deleting records can leave other expressions of the target or remove non-target preferences consolidated in the same entry. Figure 2 reports positive residue in small target-detected subgroups despite near-zero overall means. Separate Zep experiments read context from another thread of the same user and observe later reappearance after source-thread deletion, with additional results on 28 scenarios and 12 scenarios whose graph counts stabilized before deletion. A comparison on 30 Memobase scenarios fixes existing memory and varies three reader configurations: targeted-deletion residue ranges from 18.1 to 72.0 percentage points, while broader-deletion means are near zero.

Critical strengths
  1. Table 2 reports both target and designated non-target preference states, distinguishing post-deletion presence from retention with a pre-deletion baseline. The removal of 115 shared LangMem entries eliminates both preferences, concretely illustrating the mismatch between an information-level request and record operations. Inspecting storage separately from returned context also distinguishes Zep’s retained but temporarily undelivered information from LangMem’s empty stores.
  2. Behavioral residue uses a scenario-matched NeverSeen baseline, accounting for target-template choices that would occur without prior exposure. Conditions share probes and sampling-seed schedules, and inference resamples paired scenarios instead of treating the 30 trajectories sharing a context as independent scenarios. Table 5 also changes the reader while fixing the memory, allowing inspection of differences at the memory-use stage. These controls strengthen behavioral comparisons within the tested configurations.
Critical weaknesses
  1. Page 5 and Appendix A.3 use lexical rules to locate deletion targets and detect residual preferences, without false-negative or false-positive estimates against independent human or semantic annotation. The authors distinguish whole-word, dimension-anchored, and substring matching, and carefully describe detection rather than semantic absence. However, handling of paraphrases, negation, and background mentions remains unvalidated. Rule errors can affect both deletion coverage and Figure 2 subgroup membership. The positive subgroups contain only eight and seven scenarios, so membership changes could materially alter their means and intervals. Reproducible lexical rules alone do not establish how these detection results correspond to retained preference information.
  2. The operations in Table 1 directly determine part of the main findings: targeted Memobase deletion removes profiles while retaining events, broad-scope Mem0 deletion clears all user memory, and LangMem removes an entire consolidated entry. The cited Deployment-Time Memorization and MemLeak already study information remaining in other representations, so the additional value should come from explaining the specific deletion difficulties. The study does not test supported operation combinations under a common information-level objective, such as updating shared entries to preserve other content where the interface permits it. Zep does report that its canonical user node cannot be removed through the tested interface, but whether other residuals can be cleared through supported operation combinations remains unclear. Mem0 and Zep also build memory separately for Seen and post-deletion conditions rather than branching from one target-exposed initial store. The comparisons describe the chosen operations, but insufficiently separate operation choice, interface capability, and memory-construction differences to yield a clearer additional understanding.
  3. Appendix B.2 detects the target in user summaries at five and fifteen minutes after source-thread deletion in 12 Zep scenarios whose graph counts had stabilized before deletion, making the temporal observation worth investigating. However, stable counts do not establish that summary contents have finished updating. No corresponding completion criterion or state trace is reported, leaving the roles of retained derivatives and asynchronous updates unresolved. Figure 2’s NeverSeen pairing also lacks a control that holds other context fixed and removes or restores only the target content, so subgroup differences remain associational. The paper does not claim to prove these mechanisms, but this limited separation weakens guidance on how to complete selective deletion.
Questions for authors
  1. Can independent human or semantic annotation check post-deletion preferences in storage and context, covering both lexically positive and negative cases and reporting false negatives and false positives? Do Figure 2 means and intervals persist after revising subgroup membership?
  2. Starting from the same initial memory and a common requirement to remove the target while preserving the designated other preference, what can supported operation combinations achieve in each framework? Where shared entries can be updated, what happens; where they cannot, is the limitation the interface, missed target identification, or the chosen deletion scope?
  3. Can you track Zep user-summary contents and available update timestamps around source-thread deletion, and specify which observable states indicate completion? After that condition is met, does the target still reappear or persist?
  4. With other context fixed, how does behavioral residue change when only residual target content is removed or restored in the Figure 2 scenarios? This would help separate its effect from other scenario or memory-construction differences.
Reviewer_dEd2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether deleting records from persistent LLM agent memory removes a user preference while preserving unrelated preferences. It evaluates Memobase, Mem0, LangMem, and Zep on 138 constructed dashboard deployment scenarios, examining stored memory, the context supplied to the agent, and subsequent template choices. A NeverSeen baseline measures how much the deleted preference continues to influence behavior beyond choices the agent might make without prior exposure. The results show that targeted deletion can leave other representations of a preference, broader deletion can remove information meant to be retained, and information absent from context at one point can reappear later. The study provides a useful evaluation of selective deletion, although its synthetic task and lexical detection method limit the generality of its conclusions.

Critical strengths
  1. Comprehensive evaluation across stages. The paper separately examines stored records, the context returned to the model, and the final deployment action, helping identify where deletion fails.
  2. Practically relevant findings. The four frameworks reveal distinct failure modes, including residual copies, unintended deletion of preferences combined in one record, discrepancies between storage and retrieval, and the later reappearance of deleted information.
Critical weaknesses
  1. Narrow task setting. All scenarios involve constructed dashboard deployment preferences. It remains unclear whether the conclusions extend to real conversations, more complex tasks, or sensitive information.
  2. Reliance on lexical detection. Synonyms, implicit expressions, and semantically equivalent summaries may be missed. Consequently, “not detected” does not establish that the information is gone.
  3. Limited comparability across frameworks. The systems differ in memory structure, deletion scope, context source, and number of eligible scenarios. Their observed differences therefore cannot be attributed solely to deletion capability.
Questions for authors
  1. What is the false-negative rate of the lexical detection method? Could the authors manually annotate a sample or test paraphrased preferences to assess whether information classified as “not detected” is still present?
  2. Do the findings hold in more realistic tasks? For example, could the authors evaluate multi-turn interactions, free-form responses, histories with conflicting preferences, or workflows requiring multiple tool calls?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Correction to Zep results

While preparing our response, we re-verified our Zep results and found two that we can no longer support.

1. Stored memory. We reported that after targeted deletion in Zep the target preference "remains in stored memory in 66 of 75 scenarios" while being absent from all 75 returned contexts. We used this contrast to argue that absence from context does not establish removal from storage.

On re-verification, we found that 66/75 counts the scenarios in which Zep refused the request to delete the canonical user node. That observation stands: the node cannot be removed through the interface we tested. However, the persistence of the node does not show that the preference persisted in it, and the original experiment did not read the node's content back. In a re-run of 115 scenarios, the node was still present after deletion, but the target preference was no longer detected in it. Our Zep data therefore do not support the storage-context contrast, and we withdraw that claim.

2. Temporal results. We could not reproduce the temporal pattern in Table 3 and Appendix B. With the same client code, the target did not reappear after source-thread deletion in any of 30 scenarios. We cannot determine the cause of the difference from our original runs. Zep is a hosted service that may have changed between the runs, and we did not retain the full context text of the original runs, so we cannot re-examine them.

We found both issues during our own re-verification and are correcting the record here. These two results should not be relied upon. We regret the error and thank the reviewers for their careful reviews.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.