- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces BEMULA, a benchmark for user-requested forgetting in LLM agents, with 50 multi-session dialogues, 135 forgetting requests, and 1,742 test questions. It pairs dialogues with and without forgetting requests and evaluates retention, forgetting, preservation, relearning, and resistance to extraction attacks. Across the evaluated systems, almost no request passes the complete suite. I find the evaluation framework useful, although some interpretations of the scores need clarification.
Critical strengths
- I find the paired-control design a meaningful contribution. A system that never retained a fact should not receive credit simply because it cannot answer a later question about it. The authors address this by checking performance on a corresponding history without forgetting requests. Preservation and relearning probes further discourage strategies such as clearing all memory or refusing everything. Together, these checks make the evaluation more informative than measuring non-disclosure alone.
- The benchmark tests several practically different aspects of forgetting rather than reducing the task to answering with an updated value. In particular, the replacement and attack suites distinguish using new information from preventing recovery of the old information. The evaluation across multiple systems and backbones, the documented deletion paths, and the instance-level bootstrap analysis make this a useful diagnostic resource. I also appreciate the human validation of the judge, which reports 98% agreement on 100 sampled items.
Critical weaknesses
- Some interpretations go beyond what the reported metrics establish. AR requires both successful extraction on the control version and successful defense on the forget version, while keeping all attack questions in the denominator. Its complement is therefore not the leakage rate. For example, Appendix H interprets LangMem’s AR of 35.7% as leakage in roughly two thirds of attacks, which does not follow from AR alone. Similarly, low PR can reflect facts that neither version answers correctly, rather than loss caused by forgetting. I think the joint scores are useful, but raw leakage rates and paired preservation outcomes are needed to support these explanations. This looks addressable through analysis of the existing results.
- The baseline suite leaves some uncertainty about where the difficulty lies. Appendix C explains that several systems lack an online removal path under this protocol. The authors do enable LangMem’s removal channel and use Mem0’s native update mechanism, so this is not simply a case of disabling deletion. Still, a baseline that explicitly recognizes a forgetting request and executes search-and-delete or rewrite operations would help separate missing functionality from difficulties in selective deletion. Without that reference, I would limit the broader conclusions to the evaluated integrations rather than agent memory systems in general.
Questions for authors
- Could you report the raw leakage rate on the forget version and the proportion of control attacks that successfully extract the target? For preservation, how often does a control-correct answer become incorrect after forgetting? Do the interpretations in Appendices H and J still hold after separating these outcomes?
- Have you tried a simple baseline that explicitly turns forgetting requests into memory search and deletion or rewriting? If not, could you clarify which conclusions concern the evaluated ingestion pipelines and which you expect to extend to agents with an explicit deletion workflow?
2 comments
Q2 & W2: A simple baseline that explicitly turns forgetting requests into memory search and deletion or rewriting
We implemented an agent-managed file-memory baseline following the OpenClaw-style memory mechanism. This baseline explicitly turns forgetting requests into memory deletion or rewriting by editing the MEMORY.md file. The file is capped at 20,000 characters. Once this limit is exceeded, the LLM compresses its content.
Due to time constraints, we evaluate this baseline on Bemula-Lite using DeepSeek-V4.1-Flash (0910 version) as the backbone, consistent with the paper. The results are shown below, with the new results in bold.
| System | RA | FE | PR | RLA | AR | RPR | Bemula Score |
|---|---|---|---|---|---|---|---|
| Full Context | 83.7 | 58.7 | 57.0 | 100.0 | 33.1 | 1.6 | 33.3 |
| Naive RAG | 79.8 | 14.4 | 75.3 | 84.6 | 12.9 | 0.0 | 10.9 |
| LangMem | 95.2 | 48.1 | 78.5 | 86.5 | 39.8 | 1.6 | 36.1 |
| Mem0 | 89.4 | 18.3 | 89.2 | 80.8 | 13.1 | 0.0 | 13.2 |
| A-MEM | 79.8 | 29.8 | 64.5 | 94.2 | 20.2 | 0.0 | 19.1 |
| LightMem | 96.2 | 15.4 | 96.8 | 80.8 | 10.5 | 0.0 | 11.2 |
| SimpleMem | 93.3 | 23.1 | 89.2 | 94.2 | 22.6 | 0.0 | 21.0 |
| Agent-Managed File Memory (new) | 94.23 | 74.04 | 88.17 | 46.15 | 66.87 | 11.29 (7/62) | 44.89 |
Notably, the file-memory baseline achieves an RPR of 11.29% (7/62), outperforming all previously evaluated memory systems, whose RPR scores do not exceed 1.6. This result suggests that explicitly turning forgetting requests into memory search and deletion or rewriting is effective. Nevertheless, its overall Bemula Score remains limited at 44.89, and only 7 out of 62 forgetting requests are successfully handled.
In addition, in the official LangChain implementation, LangMem can also turn forgetting requests into memory search and deletion or rewriting through the LLM tool calling mechanism. In our evaluation, LangMem demonstrates some forget-on-request capability, but its overall performance is inferior to the file-memory baseline. These results indicate that reliable and comprehensive memory unlearning remains a challenging problem.
We appreciate your careful reading of our paper and your constructive comments. Below, we address each of the concerns and questions in detail.
Q1 & W1: Raw Leakage, Control-Attack Success, and Preservation Degradation
We thank the reviewer for pointing out that some interpretations go beyond what the reported metrics establish. The interpretation in Appendix H could be more precise. LangMem's AR is 35.7%, and this relatively low score reflects both its failure to correctly retain memories in the control version and its incomplete forgetting in the forget version.
We report in the table below the raw leakage rate on the forget version and the proportion of control attacks that successfully extract the target. The two new metrics are named leakage rate (forget) and leakage rate (control), respectively.
Backbone: GPT-5.6-Luna
Backbone: DeepSeek-V4.1-Flash
Backbone: Qwen3-32B-AWQ
Using DeepSeek-V4.1-Flash as the backbone model, LangMem exhibits a leakage rate of 71.0% on the control version and 40.9% on the unlearned version, indicating that LangMem forgets specific memories to a certain extent. In contrast, the score of Mem0 decreases only from 60.9 to 56.9, suggesting that under attack questions, Mem0 has almost no ability to genuinely forget upon explicit forget requests.
For preservation, we report the Conditional Preservation Failure Rate (CPFR), defined as the proportion of preservation questions answered correctly on the control version that become incorrect on the forget version:
Backbone: GPT-5.6-Luna
Backbone: DeepSeek-V4.1-Flash
Backbone: Qwen3-32B-AWQ
The experimental results match our expectations. For Naive RAG, a memory system that entirely lacks deletion capability, CPFR remains 0.0% under GPT-5.6-Luna and DeepSeek-V4.1-Flash, indicating that it does not erroneously delete other memories. In contrast, when LangMem uses GPT-5.6-luna as the backbone, CPFR reaches 49.2%, which shows that it cannot precisely forget specific content and instead erroneously deletes other memories.
After separating these outcomes, we find in Appendix H that Full Context exhibits a lower AR score on jailbreak-type questions, which is affected by the difficulty of extracting the target from the attack itself in the control version. In Appendix J, the observation that attack robustness barely moves with length still holds, but the original leakage rate of the forget version generally decreases with dialogue length. The remaining conclusions are largely unchanged, and we will revise the wording in the appendix to provide a more rigorous explanation.