acceptodds
Under review as a conference paper at ICLR 2027

Agent Memory Can Only Forget What Its Deletion Path Can See

Abstract

Existing forgetting evaluations can report near-perfect deletion while the forbidden fact remains fully retrievable. Across three widely deployed agent memory systems, surface-form retrieval scoring gives 97–100% while direct queries for the same forbidden fact give 0%: the fact was rewritten rather than removed, and the old scoring credited the rewrite. What a memory system can forget is bounded by what its deletion path — the control plane responsible for memory mutation — can see; existing benchmarks evaluate the recall plane instead. We show this with a controlled ablation that holds the store, model and prompts fixed and moves only the call site. Write-time knowledge helps only when the deletion path can read it: granting the deletion step access to the same annotations, with no new information and no new authority, recovers part of the gap. But a model placed before the request cannot see what the user later asks to remove. On the 345 cases that measure forgetting, mutation-time placement raises the pass rate from 71.0% to 91.3% (+20.3 pt); including a separate 40-case partial-edit diagnostic, the full suite rises from 63.6% to 88.8%. The same contract takes a third-party store from 0/38 to 38/38 on the affected canonicalization categories. We release ForgetEval, a 1385-case benchmark with deterministic scoring and an Adapter Protocol for heterogeneous stores. Reliable forgetting depends on the control plane and cannot be inferred from retrieval quality alone.

Primary area:
datasets and benchmarks
Keywords:
agent memory, machine unlearning, selective forgetting, memory systems, retrieval-augmented generation, benchmark, LLM agents, control plane, evaluation methodology

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_un8rOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies forgetting in external memory systems for LLM agents. It distinguishes the recall plane from the control plane and argues that reliable forgetting depends on what information the deletion path can access. The paper introduces FORGETEVAL, a benchmark with 1,000 template cases and 385 adversarial cases, together with an Adapter Protocol for evaluating heterogeneous memory systems. The main empirical study keeps the memory store, model, and prompt family fixed, while changing where the LLM is called. The results show that write-time annotation brings little improvement, while placing the LLM at mutation time improves the forgetting success rate from 71.0% to 91.3% on the 345 forgetting cases. The paper further evaluates several existing memory systems and analyzes different forgetting failure modes.

Critical strengths
  1. The paper identifies an interesting evaluation problem in existing memory systems. The distinction between “a fact is not returned by the normal retrieval query” and “a fact is truly unreachable after deletion” is useful. The experiments show that some systems can appear to forget successfully under the former criterion while the deleted information can still be recovered by direct probing. This observation is clear and practically relevant.

  2. The controlled experiments are carefully designed. The paper fixes the memory store, model, and prompt family, and changes mainly the location where the LLM is used. This gives a relatively clean comparison between write-time and mutation-time processing. The additional experiments on LangGraph and Letta also provide evidence that the observation is not limited to one implementation.

Critical weaknesses
  1. The significance of the main finding is limited. The paper presents a seemingly surprising result: existing forgetting evaluation can report nearly perfect success, while direct probing shows that the forbidden information is still accessible. However, after redefining forgetting as making the targeted information truly unreachable, the main conclusion becomes rather direct: a deletion procedure needs access to the deletion request and relevant memory information in order to make a correct semantic deletion decision. The proposed architectural change, namely moving the LLM to mutation time, follows naturally from this formulation.

  2. The empirical support for the general claim is concentrated in a small number of failure categories. The reported improvement is highly non-uniform across the benchmark. After excluding compound_fact, 64 of the 72 fixed cases come from identifier_obfuscation and cross_lingual_identifier. These categories mainly test whether different surface forms of the same entity or identifier can be correctly canonicalized. It is therefore expected that an LLM with access to the deletion request can substantially improve these cases. The paper itself also notes that the gain is statistically localized and that mutation-time placement mainly improves canonicalization. This makes the broad claim about agent memory forgetting appear stronger than what the current evidence directly supports.

Questions for authors

For A-MEM, OpenMemory, and Letta, what exact content is returned by direct probing in the canonicalization cases that score 0/38 under reachability scoring? Are the forbidden facts still present in the retrieved memory text, in associated metadata or annotations, or in another form that remains accessible to the deletion path?

Reviewer_w4qTOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces ForgetEval, a benchmark for evaluating forgetting in LLM agent memory systems. It distinguishes memory retrieval from deletion capabilities and investigates how the placement of an LLM within the memory pipeline affects forgetting performance. The authors evaluate several memory systems and report improvements from using LLMs at memory-mutation time.

Critical strengths

The paper addresses a relevant problem in persistent agent memory and provides a fairly extensive empirical evaluation across multiple memory systems. The distinction between retrieval performance and actual forgetting is useful, and the controlled ablation provides some supporting evidence.

Critical weaknesses
  1. Unclear research gap and limited novelty. The paper does not clearly establish what is fundamentally missing from existing forgetting and memory-management research. The main observation that deletion effectiveness depends on what information the deletion mechanism can access seems relatively intuitive. The authors should better articulate the novel research insight beyond this observation.

  2. Presentation and writing quality. The paper is unnecessarily long and difficult to follow. The main contributions are buried under extensive experimental details, repeated arguments, and overly complicated explanations. Several sections read as heavily AI-polished, with repetitive and sometimes unnatural phrasing. The paper needs substantial rewriting to communicate its research questions, contributions, and findings more directly. To me it is far below the ICLR accepted paper level.

  3. Limited real-world generalizability. The evaluation relies heavily on synthetic and template-generated cases, which may not reflect the complexity of forgetting in real-world agent systems. The authors also report substantially lower performance on the externally authored test cases, raising concerns about benchmark representativeness. More realistic evaluations are needed to establish whether the reported improvements generalize to practical deployments.

Questions for authors
  1. What is the key research gap that existing forgetting benchmarks and memory-management methods fail to address, and what is the fundamental novelty of the proposed approach?
  2. Can the authors clarify why LLM placement itself is the central contribution, rather than the more general requirement that deletion mechanisms have access to relevant information?
  3. Can the paper be substantially shortened and reorganized to make the main research findings clearer?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.