- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies forgetting in external memory systems for LLM agents. It distinguishes the recall plane from the control plane and argues that reliable forgetting depends on what information the deletion path can access. The paper introduces FORGETEVAL, a benchmark with 1,000 template cases and 385 adversarial cases, together with an Adapter Protocol for evaluating heterogeneous memory systems. The main empirical study keeps the memory store, model, and prompt family fixed, while changing where the LLM is called. The results show that write-time annotation brings little improvement, while placing the LLM at mutation time improves the forgetting success rate from 71.0% to 91.3% on the 345 forgetting cases. The paper further evaluates several existing memory systems and analyzes different forgetting failure modes.
Critical strengths
-
The paper identifies an interesting evaluation problem in existing memory systems. The distinction between “a fact is not returned by the normal retrieval query” and “a fact is truly unreachable after deletion” is useful. The experiments show that some systems can appear to forget successfully under the former criterion while the deleted information can still be recovered by direct probing. This observation is clear and practically relevant.
-
The controlled experiments are carefully designed. The paper fixes the memory store, model, and prompt family, and changes mainly the location where the LLM is used. This gives a relatively clean comparison between write-time and mutation-time processing. The additional experiments on LangGraph and Letta also provide evidence that the observation is not limited to one implementation.
Critical weaknesses
-
The significance of the main finding is limited. The paper presents a seemingly surprising result: existing forgetting evaluation can report nearly perfect success, while direct probing shows that the forbidden information is still accessible. However, after redefining forgetting as making the targeted information truly unreachable, the main conclusion becomes rather direct: a deletion procedure needs access to the deletion request and relevant memory information in order to make a correct semantic deletion decision. The proposed architectural change, namely moving the LLM to mutation time, follows naturally from this formulation.
-
The empirical support for the general claim is concentrated in a small number of failure categories. The reported improvement is highly non-uniform across the benchmark. After excluding compound_fact, 64 of the 72 fixed cases come from identifier_obfuscation and cross_lingual_identifier. These categories mainly test whether different surface forms of the same entity or identifier can be correctly canonicalized. It is therefore expected that an LLM with access to the deletion request can substantially improve these cases. The paper itself also notes that the gain is statistically localized and that mutation-time placement mainly improves canonicalization. This makes the broad claim about agent memory forgetting appear stronger than what the current evidence directly supports.
Questions for authors
For A-MEM, OpenMemory, and Letta, what exact content is returned by direct probing in the canonicalization cases that score 0/38 under reachability scoring? Are the forbidden facts still present in the retrieved memory text, in associated metadata or annotations, or in another form that remains accessible to the deletion path?