- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper extends the Memora benchmark into MEMFORGET, a selective-forgetting benchmark with five scenarios: temporal updates, revoked preference decoupling, user-driven privacy erasure, value-aware discrimination, and localized targeted forgetting. It audits both the retrieved memory evidence and the final answer. The authors evaluate six memory agents and four full-context LLMs. I summarize their findings are:
- No system meets all forgetting requirements.
- Larger retrieval budgets increase both recall and leakage. 3.Refusing direct questions can coexist with leakage under indirect probes. 4.Excluding targets at retrieval works better than filtering them in the answer.
Critical strengths
- Auditing both the retrieval layer and the answer layer is useful. Figure 5 distinguishes whether target information is retrieved and whether it appears in the answer. This makes it clear that refusing to answer does not necessarily mean the information is no longer accessible, and helps separate retrieval exposure from output disclosure.
- The scenarios cover more than updating outdated facts. Revoked preferences, privacy erasure with indirect probes, and preservation of critical constraints are all relevant to deployment. Checking neighboring-fact retention also helps avoid rewarding systems that suppress targets by discarding useful information too broadly.
Critical weaknesses
- In §4.2.4, the WT condition "marks target memories as deleted and excludes them from retrieval." The paper does not say whether the system decides which memories to delete; it appears they are deleted directly using the annotations. GT, by contrast, requires the answer model to identify and filter the targets itself. Under this comparison, WT being close to 100 in Fig. 6 is inevitable. The genuinely hard part, mapping a natural-language request to the specific memory entries, is skipped.
- Several numbers in Table 1 cannot be interpreted as currently reported. For example:
- Full-context models have no retrieval step, yet ER/RL are reported for them without a definition.
- Target Forgetting is at most 17.8 for every system, including full-context models that see the forget instruction directly. This looks more like a problem with the probes or the judge than only with the systems.
- In Fig. 5(d), the value-aware scenario is classified as almost 100% "retrieval leak," but the goal of this scenario is to preserve critical constraints, so the four-way failure taxonomy does not apply.
- Table 1 uses top-k=30, while the top-k analysis in Fig. 4 only goes up to 20.
- The paper reports no judge–human agreement anywhere. The reproducibility statement says it is in the appendix, but it is not there.
Questions for authors
- In WT, are the memories to delete given by the annotations or chosen by the system? If the former, please add a condition where the system decides what to delete based on the forget request, and compare it with GT.
- How are ER/RL defined and computed for full-context models? Why is TF below 18 even for full-context models that can see the forget instruction? Please show some failure cases so we can tell judge errors from real leaks.
6 comments
C.1 Definition of ER and RL for Memory Agents and Full-Context Baselines
Full-context models have no retrieval step. In our implementation, we used two definitions of ER and RL: one for memory agents and one for full-context language model baselines. The main text reports the aggregate results, while this appendix specifies the two definitions used for ER and RL.
Memory agents. For memory agents, we inspect the memories retrieved by the agent and compute ER and RL from them. We do not rely on the final answer of the model.
Full-context language models. For LLMs, our code treats the full user history given to the LLM as its retrieved memory, and computes ER and RL based on the final answer. That is, an expected item counts as recalled if the final answer contains it, and a forbidden item counts as leaked if the final answer contains it. For memory agents, this definition evaluates the retrieval layer directly. For LLMs, this metric evaluates the model’s ability to reason over temporal updates in a long context through its final answer; it does not evaluate a separate retrieval layer.
Example. Probe: “What tasks remain on my todo list this week?”
Original session evidence:
- Session 101 (should be remembered). User: “I actually need to update the executive bio for work today.”
- Session 104 (should be remembered). Adds “Follow up with industry contacts”.
- Session 7 (should be forgotten). User: “I no longer need to track Prepare investor presentation on my to-do list.”
Model answer (Qwen3.7-Max): it lists follow-ups with industry contacts, budget approval, strategy offsite, personal training, and other items. It mentions neither “Update executive bio” nor “Prepare investor presentation”. Table 2 shows the evaluation.
Table 2: Answer-based ER/RL evaluation of the todo-list example for a full-context baseline.
| Check | Expected | Result |
|---|---|---|
| Mentions “Update executive bio” | yes | × |
| “Follow up with industry contacts” is in the list | yes | ✓ |
| Mentions the revoked “Prepare investor presentation” | no | ✓ |
For agent systems, the evaluation considers both the retrieval layer and the answer layer. For the LLM setting, it mainly checks the answer layer.
C.2 Target Forgetting Scores
Memory agents. For agents, this is an important phenomenon we found. Current agents often only reply verbally that the forgetting has been completed, while their memory store still contains the target memory that the user explicitly asked to forget. This does not count as truly completing the forgetting requested by the user.
Full-context models. For full-context models that can directly observe the forgetting instruction, their target forgetting scores are low. Our recheck confirms that this result is not due to judge errors; rather, the rule treats “restating the target while refusing” as exposure.
A case labeled as a forgetting failure. The model answered:
“Based on our conversation history, here are the details regarding your memories: NeurIPS Submission: I do not remember this as your current plan. While you initially mentioned planning to submit your federated learning paper to NeurIPS, you later explicitly asked me to forget that plan (and updated me that you are now targeting ICML instead).”
The answer shows that the model has in fact understood which target the user wants to forget. However, it restates the target to be forgotten, so it is labeled as a failure. Because an LLM has no ability to manage memory on its own, such as updating or forgetting memory, it keeps the facts the user asked to forget. We still label such cases as forgetting failures.
It should also be clarified that the evaluation targets of agents and LLMs differ. For agents, both the retrieval layer and the answer layer are considered. For LLMs, only the answer layer is considered, i.e., whether the forgetting is finally carried out.
C.3 Case Studies of Targeted Forgetting
The cases below come from actual API runs on our weekly synthetic dialogue data.
Case 1: Acknowledges the deletion instruction but still discloses the submission venue. Model: qwen/qwen3.7-max. Fact to delete: the user plans to submit the current federated learning paper to NeurIPS. Neighbors to keep: the research is on federated learning; experiments use Python; Alice works on the empirical evaluation.
Deletion request:
Please forget only that I plan to submit my current federated learning paper to NeurIPS, but keep the other research project details.
Dear Reviewer,
Thank you very much for your time and constructive comments.
We have submitted our point-by-point rebuttal and revised manuscript. As the rebuttal period is drawing to a close, we are writing to check whether you have had a chance to go through our responses.
Please feel free to let us know if you have any remaining questions or require further clarification, and we will reply promptly. If you find our responses adequately address your concerns, we would highly appreciate it if you could reconsider your score at your discretion.
We fully understand that you have a busy schedule. Thank you again for your careful review.
Best regards,
All authors
We thank the reviewer for these helpful suggestions. We have added the corresponding experiments and explanations to the relevant sections of the revised manuscript. The specific additions are provided below.
4.2.2 Top-k Retrieval Sensitivity
We evaluate the retrieval budget on a subset of the weekly data with k∈{5,10,15,20,25,30} (Figure 4). Table 1 uses top-k=30 and all data, so the final numbers differ.
The extended results show that the effect of the parameter depends on the metric. For both agents, Expected Recall and Retrieval Leakage usually rise as the retrieval budget grows, but FAMA does not improve steadily. In particular, the Expected Recall of A-Mem levels off between k=20 and 30, while its leakage keeps rising. A larger retrieval range does not necessarily bring more useful information, but it can increase the exposure of outdated information. The two systems also move in different directions on some metrics, which relates to how each organizes, updates, and retrieves memory.
Beyond this, we believe a robust memory framework must actively maintain the validity of its memories, including version management, state updates, and conflict resolution. When a memory is explicitly updated or judged outdated, the system should overwrite the old value, mark it as invalid, or link the old and new versions with a replacement relation, so that the old memory is no longer used as currently valid information. For example, after a meeting moves from Tuesday to Thursday, the system should set “Thursday” as the current state and remove the old time from the retrieval and use of valid facts. If the task needs to keep the history, the system should also mark the old time as invalid. The retrieval budget can affect how well valid information is covered, but it should not be the main safeguard against outdated information becoming active again. Selective forgetting should therefore rely more on continuous management of memory states, rather than only on shrinking top-k to lower the chance of retrieving outdated information. This is also the main problem that our planned new memory framework aims to solve.
4.2.4 Retrieval-Side Exclusion vs. Answer-Side Filtering
The preceding analysis reveals that forgetting failures stem from insufficient retrieval filtering or model inference leakage. To clarify the performance gap between target removal in the retrieval phase and post-processing filtering relying only on the answer LLM, we evaluate three forgetting strategies, as shown in Figure 6.
WT condition. The WT condition marks the target memories as deleted and excludes them from retrieval. The deletion targets in this condition are given by the annotations. The system does not decide on its own which content should be deleted. This setting skips a core difficulty: mapping a natural-language forgetting instruction to specific memory entries. In contrast, GT requires the answer model to identify and filter the target information by itself.
WT-system condition. WT is an oracle (upper-bound) reference condition, referred to as WT-oracle. We compare it with WT-system, in which the system selects the memory IDs to delete from the user’s original forgetting request, the previous dialogue, and the full text of the current memory store. It does not use the annotated targets to perform deletion. The selected memories are then excluded from retrieval. Figure 6 compares WT-oracle, WT-system, and GT with three answer models: Qwen3.8-Max, GLM-5.3, and Kimi-K2.6.
Results. On this controlled data, the system can locate the target memories from the natural-language request and evict them by itself, without directly using the annotations to specify deletion targets. It still achieves high forgetting performance. WT-system remains clearly above GT for all three answer models. Direct Inquiry and Hidden Inducement stay at or near the WT-oracle level.
WT-system does not reach the performance upper bound. The drop, mainly visible in Target Deletion, comes from errors in the instruction-to-memory mapping step, such as missing related memories or selecting the wrong memory entries.
Limitation of this setting. All forgetting instructions in our benchmark are clear and explicit. The mapping is therefore easier than in real use. For semantically ambiguous forgetting requests, we expect the gap between WT-system and WT-oracle to grow further. We consider this gap itself a measure of the ability to map a forgetting request to memory entries, and include it as part of the benchmark.