acceptodds
Under review as a conference paper at ICLR 2027

Can Agents Forget When Asked? A Benchmark For Memory Unlearning in LLM Agents

Abstract

Long-term memory has become a core component of large language model (LLM) agents. As agents accumulate personal information over long-term use, users increasingly ask them to delete sensitive or outdated facts. A memory system should honor such requests. It should remove exactly what was requested and keep the rest of its memory intact. However, existing memory benchmarks evaluate only memory writing, retrieval, and organization, so whether agents can forget at the user's explicit request remains under-evaluated. To address this gap, we propose Bemula, the first benchmark dedicated to memory unlearning. It comprises 50 multi-session dialogue instances, 135 forget requests of six types, and 1,742 test questions. Each dialogue has a control version without the requests, so that only systems that have stored the target can earn credit for forgetting. Across the paired versions, five test suites measure storage, removal of the requested information, preservation of unrelated memories, relearning of similar facts, and resistance to extraction attacks. On Bemula, we evaluate six mainstream memory systems under three backbone LLMs, together with the Full Context LLM on two of them. The results are negative. Across all evaluated systems and backbones, no system passes more than 1.5% of forget requests end to end, and most pass none at all. These systems either leave the requested information extractable under simple attacks or delete far more than requested. Memory unlearning therefore remains an open challenge for agent memory design. We release our data and code at https://anonymous.4open.science/r/Bemula-E321/.

Primary area:
datasets and benchmarks
Keywords:
Agent Memory, Memory Unlearning, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_kHEfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces BEMULA, a benchmark for user-requested forgetting in LLM agents, with 50 multi-session dialogues, 135 forgetting requests, and 1,742 test questions. It pairs dialogues with and without forgetting requests and evaluates retention, forgetting, preservation, relearning, and resistance to extraction attacks. Across the evaluated systems, almost no request passes the complete suite. I find the evaluation framework useful, although some interpretations of the scores need clarification.

Critical strengths
  1. I find the paired-control design a meaningful contribution. A system that never retained a fact should not receive credit simply because it cannot answer a later question about it. The authors address this by checking performance on a corresponding history without forgetting requests. Preservation and relearning probes further discourage strategies such as clearing all memory or refusing everything. Together, these checks make the evaluation more informative than measuring non-disclosure alone.
  2. The benchmark tests several practically different aspects of forgetting rather than reducing the task to answering with an updated value. In particular, the replacement and attack suites distinguish using new information from preventing recovery of the old information. The evaluation across multiple systems and backbones, the documented deletion paths, and the instance-level bootstrap analysis make this a useful diagnostic resource. I also appreciate the human validation of the judge, which reports 98% agreement on 100 sampled items.
Critical weaknesses
  1. Some interpretations go beyond what the reported metrics establish. AR requires both successful extraction on the control version and successful defense on the forget version, while keeping all attack questions in the denominator. Its complement is therefore not the leakage rate. For example, Appendix H interprets LangMem’s AR of 35.7% as leakage in roughly two thirds of attacks, which does not follow from AR alone. Similarly, low PR can reflect facts that neither version answers correctly, rather than loss caused by forgetting. I think the joint scores are useful, but raw leakage rates and paired preservation outcomes are needed to support these explanations. This looks addressable through analysis of the existing results.
  2. The baseline suite leaves some uncertainty about where the difficulty lies. Appendix C explains that several systems lack an online removal path under this protocol. The authors do enable LangMem’s removal channel and use Mem0’s native update mechanism, so this is not simply a case of disabling deletion. Still, a baseline that explicitly recognizes a forgetting request and executes search-and-delete or rewrite operations would help separate missing functionality from difficulties in selective deletion. Without that reference, I would limit the broader conclusions to the evaluated integrations rather than agent memory systems in general.
Questions for authors
  1. Could you report the raw leakage rate on the forget version and the proportion of control attacks that successfully extract the target? For preservation, how often does a control-correct answer become incorrect after forgetting? Do the interpretations in Appendices H and J still hold after separating these outcomes?
  2. Have you tried a simple baseline that explicitly turns forgetting requests into memory search and deletion or rewriting? If not, could you clarify which conclusions concern the evaluated ingestion pipelines and which you expect to extend to agents with an explicit deletion workflow?
2 comments

We appreciate your careful reading of our paper and your constructive comments. Below, we address each of the concerns and questions in detail.

Q1 & W1: Raw Leakage, Control-Attack Success, and Preservation Degradation

We thank the reviewer for pointing out that some interpretations go beyond what the reported metrics establish. The interpretation in Appendix H could be more precise. LangMem's AR is 35.7%, and this relatively low score reflects both its failure to correctly retain memories in the control version and its incomplete forgetting in the forget version.

We report in the table below the raw leakage rate on the forget version and the proportion of control attacks that successfully extract the target. The two new metrics are named leakage rate (forget) and leakage rate (control), respectively.

Backbone: GPT-5.6-Luna

SystemLeakage rate (forget)Leakage rate (control)
Full Context22.948.3
Naive RAG52.160.1
LangMem19.737.3
Mem072.271.4
A-MEM38.958.5
LightMem56.969.0
SimpleMem62.277.9

Backbone: DeepSeek-V4.1-Flash

SystemLeakage rate (forget)Leakage rate (control)
Full Context19.451.0
Naive RAG47.450.9
LangMem40.971.0
Mem056.960.9
A-MEM39.351.7
LightMem66.471.2
SimpleMem52.766.6

Backbone: Qwen3-32B-AWQ

SystemLeakage rate (forget)Leakage rate (control)
Naive RAG49.148.3
Mem054.755.8
LangMem16.820.8
LightMem63.760.6
SimpleMem48.654.2

Using DeepSeek-V4.1-Flash as the backbone model, LangMem exhibits a leakage rate of 71.0% on the control version and 40.9% on the unlearned version, indicating that LangMem forgets specific memories to a certain extent. In contrast, the score of Mem0 decreases only from 60.9 to 56.9, suggesting that under attack questions, Mem0 has almost no ability to genuinely forget upon explicit forget requests.

For preservation, we report the Conditional Preservation Failure Rate (CPFR), defined as the proportion of preservation questions answered correctly on the control version that become incorrect on the forget version:

Backbone: GPT-5.6-Luna

SystemControl-correct → Forget-incorrect / Control-correctCPFR (%)
Full Context8 / 1186.8
Naive RAG0 / 1010.0
LangMem29 / 5949.2
Mem07 / 1275.5
A-MEM10 / 9910.1
LightMem5 / 1373.6
SimpleMem11 / 1358.1

Backbone: DeepSeek-V4.1-Flash

SystemControl-correct → Forget-incorrect / Control-correctCPFR (%)
Full Context30 / 9930.3
Naive RAG0 / 970.0
LangMem15 / 12611.9
Mem05 / 1353.7
A-MEM7 / 888.0
LightMem3 / 1442.1
SimpleMem6 / 1334.5

Backbone: Qwen3-32B-AWQ

SystemControl-correct → Forget-incorrect / Control-correctCPFR (%)
Naive RAG1 / 961.0
Mem010 / 1089.3
LangMem19 / 2770.4
LightMem0 / 1290.0
SimpleMem21 / 10320.4

The experimental results match our expectations. For Naive RAG, a memory system that entirely lacks deletion capability, CPFR remains 0.0% under GPT-5.6-Luna and DeepSeek-V4.1-Flash, indicating that it does not erroneously delete other memories. In contrast, when LangMem uses GPT-5.6-luna as the backbone, CPFR reaches 49.2%, which shows that it cannot precisely forget specific content and instead erroneously deletes other memories.

After separating these outcomes, we find in Appendix H that Full Context exhibits a lower AR score on jailbreak-type questions, which is affected by the difficulty of extracting the target from the attack itself in the control version. In Appendix J, the observation that attack robustness barely moves with length still holds, but the original leakage rate of the forget version generally decreases with dialogue length. The remaining conclusions are largely unchanged, and we will revise the wording in the appendix to provide a more rigorous explanation.

Q2 & W2: A simple baseline that explicitly turns forgetting requests into memory search and deletion or rewriting

We implemented an agent-managed file-memory baseline following the OpenClaw-style memory mechanism. This baseline explicitly turns forgetting requests into memory deletion or rewriting by editing the MEMORY.md file. The file is capped at 20,000 characters. Once this limit is exceeded, the LLM compresses its content.

Due to time constraints, we evaluate this baseline on Bemula-Lite using DeepSeek-V4.1-Flash (0910 version) as the backbone, consistent with the paper. The results are shown below, with the new results in bold.

SystemRAFEPRRLAARRPRBemula Score
Full Context83.758.757.0100.033.11.633.3
Naive RAG79.814.475.384.612.90.010.9
LangMem95.248.178.586.539.81.636.1
Mem089.418.389.280.813.10.013.2
A-MEM79.829.864.594.220.20.019.1
LightMem96.215.496.880.810.50.011.2
SimpleMem93.323.189.294.222.60.021.0
Agent-Managed File Memory (new)94.2374.0488.1746.1566.8711.29 (7/62)44.89

Notably, the file-memory baseline achieves an RPR of 11.29% (7/62), outperforming all previously evaluated memory systems, whose RPR scores do not exceed 1.6. This result suggests that explicitly turning forgetting requests into memory search and deletion or rewriting is effective. Nevertheless, its overall Bemula Score remains limited at 44.89, and only 7 out of 62 forgetting requests are successfully handled.

In addition, in the official LangChain implementation, LangMem can also turn forgetting requests into memory search and deletion or rewriting through the LLM tool calling mechanism. In our evaluation, LangMem demonstrates some forget-on-request capability, but its overall performance is inferior to the file-memory baseline. These results indicate that reliable and comprehensive memory unlearning remains a challenging problem.

Reviewer_Sou7OpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper presents a benchmark which specifically tests how well an LLM equipped with user memories can 'forget'/delete specific memories when requested by the user. Existing benchmarks only test for storing, retrieval and usage of memories. This benchmark fills the gap and tests deleting memories.

Benchmark Creation

Benchmark has an intricate generation pipeline involving 5 stages. Authors generate 50 synthetic users, each with about 60 personal facts such as an address, health events, hobbies, finances, and career history. Some additional “derived” facts are never stated directly but can be inferred by combining two or more stated facts. Next comes a planning stage which decides (for each user) what each conversation session is going to be about, which facts come up in the conversation and where deletion requests should be made by the user. LLMs play the user and the assistant to write the conversation. A supervising LLM makes sure the planned facts are actually mentioned at appropriate places, and a checker rejects sessions that are inconsistent or unnatural. Deletion requests are then inserted at the planned points.

The deletion requests themselves come in 6 various forms: single fact, multiple facts, a whole broad category (ex. all financial information), an inferred fact (the fact itself is not stored in the memory but it can be deduced from multiple sored memories), everything within a particular time-period, and replacing an old memory with a new updated one.

The final benchmark is 50 conversations (each belonging to one of the 50 users), each between 100 and 3,500 turns long. The entire corpus contains 135 deletion requests.

Evaluation

Each of the 50 conversations exists in two forms, regular variant and control variant. In the regular variant, the user asks the assistant to forget certain facts. In control variant those turns are replaced with small-talk. A memory system reads one entire conversation turn-by-turn. Its memory bank is empty at the beginning and it adds required memories as facts come up in the conversation, and is expected to delete memories the user asks it to forget.

After processing the entire conversation, and after building the memory-bank, a memory system is put under test. An LLM equipped with the memory system is asked certain test questions in a fresh context window. The test questions can only be answered from the memory bank, since the context window holds nothing but the question. Each system is run on both variants, and a system gets credit for forgetting a fact only if it answered that fact correctly in the control run and can't answer in the regular run. This stops a system that never stored the fact from looking like it forgot it.

The questions check five things: whether the fact was stored (retention), whether it is gone after the request (efficacy), whether it can still be extracted through attacks such as role-play or jailbreak prompts (attack robustness), whether unrelated facts survive (preservation), and whether new facts can still be learned (relearning). An LLM judge grades the answers. The final score multiplies forgetting ability by usability, so deleting nothing or deleting everything both score near zero.

Findings

Across six memory systems and a full-context baseline on three backbone LLMs, no configuration handles more than 1.5% of deletion requests correctly. Systems either cannot delete at all, leaving facts extractable, or delete far more than was asked.

Critical strengths

S1. Careful dataset construction with multiple layers of error checking. A planner fixes which facts appear in which sessions, and the type, target, position, and phrasing of every request, before any dialogue is written. This keeps long multi-session dialogues consistent with their forget requests and test questions. Errors are then checked at each stage: a director records a fact only after confirming the text actually states it; a verifier regenerates sessions with temporal or naturalness problems; derived facts are checked to follow strictly from their sources; the assistant's confirmation of each request is constrained not to restate the target value; an LLM check confirms clear deletion intent for every request; and unanswerable questions are filtered out. The authors also manually verified 388 questions and validated the LLM judge against human annotation. This gives good confidence that the low scores reflect system failures rather than dataset errors.

S2. Rigorous evaluation design. Each dialogue is paired with a control version that is identical except that the forget requests are replaced by neutral small talk, and every forget-side probe is conditioned on its control-side counterpart. This prevents a system from earning forgetting credit for facts it never stored, for refusing to answer, or for wiping its memory. The other suites close the remaining gaps. Preservation catches over-deletion, relearning catches systems that stop working after a deletion, and the attack suite shows whether "forgotten" information can still be extracted. Because the overall score multiplies forgetting ability (FE, AR) by usability (PR, RLA), a system that deletes nothing or deletes everything cannot score well. As a result, the benchmark measures selective, robust deletion rather than the appearance of forgetting.

Critical weaknesses

W1 Four of the six evaluated memory systems (Naive RAG, LightMem, SimpleMem, and A-MEM, which can only overwrite) have no way to remove memories. The paper can be seen as showcasing this issue in majority of popular memory-backends which lack a pathway to delete memories. But it would be nice to include baselines which have the ability to delete memories freely. For example, OpenClaw's default memory allows agents to freely modify MEMORY.md, USER.md etc. and agents are directly responsible for adding and deleting memories. They can directly interact with those files, edit them freely. Many of the commercial systems like Muse, claude.ai's memory system, chatgpt.com's memory system work in a similar way to OpenClaw's default memory system. So the claim that the paper makes about current memory systems struggle to forget memories is incomplete. We don't yet know how a strong baseline like that of OpenClaw's default memory performs.

Questions for authors

Q1. Separating forgetting from storage in FE (and PR, AR).

The paired control design is a good way to ensure that a system gets no forgetting credit for a fact it never stored. For one efficacy item, the possible outcomes are:

Control run (no request)Forget run (after request)InterpretationEfficacy credit
Answers correctlySays it doesn't knowStored, then forgot1
Answers correctlyReveals the valueStored, did not forget0
Doesn't knowSays it doesn't knowNever stored0

The third row rightly earns no credit. However, as we read Eq. 2, these never-stored items stay in the denominator. As a result, they count as forgetting failures, and FE goes down when storage is poor, even for a system that forgets everything it actually stores. Consider two hypothetical systems, each tested on 10 target facts:

Facts storedStored facts forgottenFE as defined (Eq. 3)FE over stored facts only
System A1055/10 = 505/10 = 50%
System B222/10 = 202/2 = 100%

Under the current definition, B looks like the worse forgetter. But B forgets everything it stores; its real weakness is storage, which RA already captures. This seems to matter in practice. For example, LangMem on Qwen3-32B-AWQ has RA 21.9 and FE 14.7, which suggests it forgets most of the facts it actually stored.

Could you also report FE computed only over items whose control probe passed (restricting both numerator and denominator), alongside the current numbers? The same applies to PR, where a never-stored distractor currently looks like over-deletion, and to AR, where an attack that fails even on the control version currently counts as a leak. This would let readers separate forgetting ability from storage ability when comparing systems.

Q2 The control and forget versions are read in two separate runs with a reset in between. The pairing assumes that both runs stored the same facts before the request. For systems that store facts through LLM extraction (Mem0, A-MEM, LangMem, SimpleMem), this may vary from run to run. If a fact is stored in the control run but happens to be missed in the forget run, the system gets forgetting credit for nothing. Could you run the control version twice and report how often retention answers agree? That would give a noise floor for FE and AR.

Q3 Could the authors add an agent-managed file-memory baseline (e.g., OpenClaw-style), or a simple setup where the agent LLM detects the forget request and calls the backend’s own delete function?

4 comments

We appreciate your careful reading of our paper and your constructive comments. Below, we address each of the concerns and questions in detail.

Q1: Separating forgetting from storage in FE (and PR, AR).

We thank the reviewer for this insightful suggestion. Existing evaluations of metrics such as FE build forgetting ability on top of storage ability. To separate forgetting ability from storage ability, we additionally define FEcondFE_{cond}, PRcondPR_{cond}, and ARcondAR_{cond}, which are computed only over items whose control probe passed. The definitions are given below, with notation consistent with the paper:

$FE_{cond} = \frac{\sum_{i\in S_{eff}} z_i}{\sum_{i\in S_{eff}} c_i}, \qquad PR_{cond} = \frac{\sum_{i\in S_{pres}} z_i}{\sum_{i\in S_{pres}} c_i}, \qquad $$AR_{cond} = \frac{1}{m}\sum_{a=1}^{m} \frac{\sum_{i\in S_a} z_i}{\sum_{i\in S_a} c_i}$

Experimental results are as follows (all values are percentages and new metrics are in bold):

Backbone: GPT-5.6-Luna

SystemFE (orig.)FE_condPR (orig.)PR_condAR (orig.)AR_cond
Full Context26.831.773.393.227.665.5
Naive RAG14.719.267.3100.012.929.4
LangMem37.968.020.050.825.673.0
Mem08.59.580.094.54.77.2
A-MEM33.040.459.389.924.750.1
LightMem17.919.188.096.418.030.0
SimpleMem16.117.182.791.918.725.9

Backbone: DeepSeek-V4.1-Flash

SystemFE (orig.)FE_condPR (orig.)PR_condAR (orig.)AR_cond
Full Context53.669.846.069.732.971.2
Naive RAG12.517.064.7100.011.831.5
LangMem40.644.074.088.135.751.5
Mem017.019.086.796.312.623.7
A-MEM29.538.854.092.021.246.3
LightMem12.913.794.097.910.416.0
SimpleMem17.919.784.795.520.033.1

Backbone: Qwen3-32B-AWQ

SystemFE (orig.)FE_condPR (orig.)PR_condAR (orig.)AR_cond
Naive RAG7.19.663.399.05.016.7
Mem010.312.265.390.78.216.5
LangMem14.748.55.329.69.469.5
LightMem8.99.586.0100.05.610.1
SimpleMem14.317.954.779.614.029.3

The new metrics support the reviewer's interpretation. Most notably, for LangMem on Qwen3-32B-AWQ, FE is 14.7 while FEcondFE_{cond} reaches 48.5, and AR is 9.4 while ARcondAR_{cond} reaches 69.5. Thus, its low original FE substantially reflects poor storage rather than an inability to forget facts that were actually stored.

We agree that the new metrics better isolate forgetting ability from storage ability, while the original metrics capture end-to-end performance. We will include both sets of metrics in the revision to make this distinction explicit.

Q2: Retention-Answer Agreement Across Repeated Control Runs

Using DeepSeek-V4.1-Flash (0910 version, consistent with the paper) as the backbone model, we run the control version twice on the Bemula-Lite dataset and report how often the retention answers agree. All data in the table are percentages.

Memory SystemRA (Run 1)RA (Run 2)Correct-to-Incorrect Rate
Full Context83.781.78.0
Naive RAG79.878.81.2
LangMem95.296.24.0
SimpleMem93.395.20.0
Mem089.490.47.5
LightMem96.297.11.0
A-MEM79.883.73.6

Both control runs contain the same 104 retention questions across 31 Bemula-Lite instances. The Correct-to-Incorrect Rate is the proportion of questions answered correctly in Run 1 that were answered incorrectly in Run 2.

Experimental results show that the overall RA scores of the two runs are close. SimpleMem, LightMem, and Naive RAG exhibit relatively stable retention responses, whereas Full Context fluctuates more noticeably than the other systems, indicating that the answering capability of LLMs under long contexts still suffers from a certain degree of instability.


Q3 & W1: Baselines with Agent-Managed Memory Deletion

We implemented an agent-managed file-memory baseline following the OpenClaw-style memory mechanism. Specifically, the agent maintains a file named MEMORY.md, which the LLM autonomously updates after each dialogue turn. The file is capped at 20,000 characters. Once this limit is exceeded, the LLM compresses its content. Due to time constraints, we evaluate this baseline on Bemula-Lite using DeepSeek-V4.1-Flash (0910 version) as the backbone, consistent with the paper. The results are shown below, with the new results in bold.

SystemRAFEPRRLAARRPRBemula Score
Full Context83.758.757.0100.033.11.633.3
Naive RAG79.814.475.384.612.90.010.9
LangMem95.248.178.586.539.81.636.1
Mem089.418.389.280.813.10.013.2
A-MEM79.829.864.594.220.20.019.1
LightMem96.215.496.880.810.50.011.2
SimpleMem93.323.189.294.222.60.021.0
Agent-Managed File Memory (new)94.2374.0488.1746.1566.8711.29 (7/62)44.89

Notably, the file-memory baseline achieves an RPR of 11.29% (7/62), outperforming all previously evaluated memory systems, whose RPR scores do not exceed 1.6. This result suggests that agent-managed file memory can better support forget-on-request than existing memory mechanisms. Nevertheless, its overall Bemula Score remains limited at 44.89, and only 7 out of 62 forgetting requests are successfully handled. These results indicate that, while agent-managed file memory improves forget-on-request performance, reliable and comprehensive memory unlearning remains a challenging problem.

Thank you for the detailed responses and the new experiments. I am raising my score from 3 to 4. The benchmark is carefully designed and well built. I also feel the benchmark is a bit narrow, since it covers only forgetting in agent memory, but within that scope the execution is very strong.

I have a few other requests. It would be good if the authors could report some of this analysis before the end of the discussion period; otherwise, please include it in the camera-ready version:

  1. Conditional metrics for the file-memory baseline The response says FE_cond, PR_cond and AR_cond will be reported alongside the original metrics. However, they are not reported for the new agent-managed file-memory baseline. Please include them for this baseline as well.

  2. Failure analysis for all evaluated systems At the moment the paper shows that systems fail but gives little insight into why. A failure analysis for each system would add a lot to the paper. For example, it could sort failed requests into categories such as: • the system has no way to delete, • the forget request was not recognized, • only part of the target was deleted, • the target was deleted but can still be inferred from related memories, • more was deleted than requested, • etc. It could then report how often each category occurs for each system. The agent-managed file-memory baseline leads on forgetting by a wide margin (FE 74.04, AR 66.87). Yet it has the lowest relearning accuracy of all systems (RLA 46.15, against 80–100 for the rest), and only 7 of 62 requests pass. It is not clear why RLA drops so sharply here. Any failure analysis that helps developers build better memory systems would be valuable. A few qualitative examples of failures would also help.

  3. A stronger backbone For the camera-ready version, it would help to add a stronger frontier backbone (e.g., Claude Opus 5.5, or the strongest model available at that time).

I don't expect these to change the paper's core contributions; they would only strengthen them. I recommend accepting the paper.

We thank you for your thoughtful follow-up and for raising the score. We agree that the suggested additional analyses would make the benchmark substantially more diagnostic and useful for developers of agent memory systems. We address each request below.

  1. Conditional metrics for the agent-managed file-memory baseline.

We have now included FE_cond, PR_cond, and AR_cond for the new agent-managed file-memory baseline in the results table. All metric values are reported as percentages.

SystemRAFEFE_condPRPR_condRLAARAR_condRPRBemula Score
Full Context83.758.769.3257.079.10100.033.173.231.633.3
Naive RAG79.814.417.6575.398.5984.612.931.660.010.9
LangMem95.248.150.5178.591.2586.539.854.051.636.1
Mem089.418.320.0089.296.5180.813.122.720.013.2
A-MEM79.829.836.0564.593.7594.220.242.400.019.1
LightMem96.215.416.0096.898.9080.810.515.690.011.2
SimpleMem93.323.124.7489.295.4094.222.634.830.021.0
Agent-Managed File Memory94.2374.0478.5788.1798.8046.1566.8782.9911.29 (7/62)44.89

The conditional results further demonstrate the strong forgetting performance of the Agent-Managed File Memory baseline. In particular, it achieves the highest FE_cond and AR_cond among all evaluated systems, while its PR_cond remains comparable to the best-performing systems.

  1. Failure analysis for evaluated systems.

Due to time constraints, we conduct a preliminary error analysis covering a subset of the evaluated memory systems. Our main findings are as follows:

  • Full Context: "The system has no way to delete." Old information persists in the input, and the model can only choose not to mention it. Even when a request passes the FE check, the information may still be recoverable through adversarial queries.

  • Naive RAG: "The system has no way to delete." Both forget requests and outdated facts are stored as retrievable segments.

  • A-MEM: "The forget request was not recognized." A-MEM can rewrite relevant notes but lacks request-level deletion.

  • Agent-Managed File Memory: "Only part of the target is deleted." This memory system still exhibits residual leakage after editing.

  • SimpleMem: "Other cases." When processing subsequent requests, SimpleMem does not revisit or edit notes generated during earlier windows. Consequently, only information in the same window as the forget request can be deleted.

We will extend this analysis to all evaluated systems and provide a more in-depth analysis in the camera-ready version.

In addition, the low RLA in the agent-managed file-memory baseline is mainly associated with occasional failures to successfully write new information when approaching the capacity limit (20,000 characters).

  1. A stronger backbone for the camera-ready version.

We thank the reviewer for this suggestion. A stronger backbone model often exhibits better memory management capability. In terms of cost, the output price of the Opus 5.5 API is approximately 16.67 times that of GPT-5.6-Luna, which would incur substantially higher evaluation costs. We will evaluate these frontier backbones once we secure additional LLM API resources.


Thank you again for raising your score and for your constructive feedback. These suggestions have helped us improve both the empirical analysis and the presentation of our results.

Reviewer_9q3POpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces BEMULA, a benchmark for evaluating whether LLM agents can forget information from long-term memory when users explicitly ask them to. It contains 50 multi-session dialogues, 135 forgetting requests across six categories, and 1,742 evaluation questions.

A key part of the benchmark is its paired design: each dialogue has a version with forgetting requests and a control version without them. This helps distinguish genuine forgetting from cases where the system never stored the information in the first place. The evaluation also covers memory preservation, relearning, and resistance to information extraction attacks.

The authors evaluate six memory systems across three backbone LLMs. The results show that existing systems struggle with selective forgetting, with no system achieving a request pass rate above 1.5%.

Critical strengths
  1. Being able to forget specific information while keeping the rest of the memory intact is an important capability for long-term agents. I particularly like the paired-control design, which avoids counting information that was never stored as successfully forgotten.

  2. The benchmark goes beyond checking whether a forgotten fact can still be recalled. It also tests preservation of unrelated memories, relearning, and resistance to extraction attacks. The experiments cover several memory systems, and the authors include human validation of the judge and bootstrap confidence intervals.

Critical weaknesses

1.Lack of a baseline explicitly designed to handle forgetting requests.

As described in Appendix C, some of the evaluated systems do not support online deletion, or can only modify existing memories through limited update mechanisms. Their poor performance on forgetting tasks is therefore not entirely surprising.

Although the paper includes systems with some deletion capabilities, a baseline that explicitly translates natural-language forgetting requests into memory deletion operations would make the evaluation more convincing. It would help show how challenging BEMULA is for systems actually designed to support forgetting.

2.The evaluation does not fully distinguish behavioral forgetting from actual memory deletion.

BEMULA mainly uses question answering and extraction attacks to check whether information has been forgotten. However, a model refusing to reveal a fact does not necessarily mean the fact has been removed from its underlying memory store.

Also, the preservation suite checks only three unrelated facts per dialogue, which may miss broader cases of over-deletion. Some storage-level checks or more extensive preservation tests would make the benchmark stronger.

Questions for authors

1.Have the authors considered adding a baseline that explicitly supports memory deletion based on natural-language requests? This would help assess how well BEMULA evaluates systems with dedicated forgetting mechanisms.

2.Is there a way to check whether the target information has actually been removed from the memory store, rather than simply not appearing in the model's answers? Also, are three preservation questions per dialogue enough to catch unintended deletion of other memories?

2 comments

We appreciate your careful reading of our paper and your constructive comments. Below, we address each of the concerns and questions in detail.

Q1 & W1: Lack of a Natural-Language Memory Deletion Baseline

We implemented an agent-managed file-memory baseline based on the OpenClaw-style memory mechanism. The agent maintains a MEMORY.md file. After each dialogue turn, the LLM autonomously updates this file. The file is limited to 20,000 characters. When it exceeds this limit, the LLM compresses its contents. The agent can also interpret natural-language forgetting requests from users and remove or revise the corresponding entries in MEMORY.md. Therefore, this baseline explicitly supports memory deletion based on natural-language requests.

Due to time constraints, we evaluate this baseline on Bemula-Lite using DeepSeek-V4.1-Flash (0910 version) as the backbone, consistent with the paper. The results are shown below, with the new results in bold.

SystemRAFEPRRLAARRPRBemula Score
Full Context83.758.757.0100.033.11.633.3
Naive RAG79.814.475.384.612.90.010.9
LangMem95.248.178.586.539.81.636.1
Mem089.418.389.280.813.10.013.2
A-MEM79.829.864.594.220.20.019.1
LightMem96.215.496.880.810.50.011.2
SimpleMem93.323.189.294.222.60.021.0
File-Memory (new)94.2374.0488.1746.1566.8711.29 (7/62)44.89

Notably, the file-memory baseline achieves an RPR of 11.29% (7/62), outperforming all previously evaluated memory systems, whose RPR scores do not exceed 1.6. This result suggests that explicitly turning forgetting requests into memory search and deletion or rewriting is effective. Nevertheless, its overall Bemula Score remains limited at 44.89, and only 7 out of 62 forgetting requests are successfully handled. These results indicate that, while agent-managed file memory improves forget-on-request performance, reliable and comprehensive memory unlearning remains a challenging problem.

Q2 & W2: Verification of Memory Deletion and Preservation

Verification of Memory Deletion

We thank the reviewer for this insightful question. We attempt to audit whether memories are actually removed by probing with keywords in systems where memory is accessible. A target detection means that the complete target value can still be found in the memory storage, with the text normalized for case, punctuation, and similar variations.

We conduct a storage-level audit of three memory systems, A-MEM, SimpleMem, and LangMem, on the Bemula-Lite benchmark using DeepSeek-V4.1-Flash as the backbone model. The results are shown in the table below.

Memory SystemControl DetectedStill Detected in Forget
A-MEM105/135105/105 (100.0%)
SimpleMem95/13595/95 (100.0%)
LangMem71/13551/71 (71.8%)
  • Control Detected is the number of targets detected in the Control version.
  • Still Detected in Forget is the number of those Control-detected targets that are still detected in the Forget version.

This result indicates that, with DeepSeek-V4.1-Flash as the backbone, A-MEM and SimpleMem lack the ability to remove the corresponding memories upon forgetting requests, whereas LangMem possesses this ability but still achieves unsatisfactory performance.

However, this approach proves insufficient, and we do not find a suitable method that can rigorously determine whether the target information is actually removed from memory while also enabling fair comparison across different memory systems. The reasons are as follows.

First, the same memorized information may be expressed in different ways, and some memories can be inferred from other memories, which makes auditing methods based on keywords or semantic vectors not fully reliable. Second, different Agent Memory Systems differ substantially in their implementations: some are based on graph structures, some on text files, and some commercial memory systems only expose APIs (Memory-as-a-Service). Bemula cannot fairly confirm memory deletion through the underlying storage. Therefore, Bemula adopts a question-answering to detect whether a memory has been forgotten. In future work, we hope to further study more reliable methods for judging memory forgetting.

Preservation Questions

The preservation suite uses three unrelated facts per dialogue to check whether a memory system mistakenly deletes memories unrelated to forgetting requests. Although this number is relatively small, it provides a basic and reasonably accurate measure of whether a memory system erroneously deletes unrelated memories.

To measure preservation more reliably, we use the Question Generation component of the data generation pipeline again and expand the preservation suite to ten questions per dialogue as a supplement to the dataset. With DeepSeek-V4.1-Flash as the backbone model, we evaluate and report the PR performance of each memory system on the Bemula-Lite dataset.

Memory SystemPR (3 Questions/Instance)PR (10 Questions/Instance)
LightMem92.588.7
SimpleMem88.282.6
Mem079.679.4
LangMem79.674.8
Naive RAG76.372.6
Full Context57.063.5
A-MEM67.761.9

After expanding to ten questions, LightMem and SimpleMem still remain in the top two positions, and the overall ranking is largely consistent. Among the newly added questions, we indeed identify retention failures that the original three questions fail to detect, indicating that increasing question coverage helps uncover missed preservation failures. To account for evaluation cost and the randomness of LLMs, we use three preservation questions per instance as the default metric, and we also provide a supplementary dataset with ten preservation questions per instance for research purposes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.