- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes DELETECERT, a counterfactual verification framework for memory deletion in persistent agents, designed to address the issue that simply deleting original records fails to eliminate their residual effects in summaries, user profiles, skills, and other derived memories. The authors define the deletion effect as the behavioral difference between the agent after deletion and a counterfactual agent that has never observed the target information, and measure privacy leakage using excess residual risk. The method achieves a trade-off between privacy risk and memory utility by constructing a memory trace graph, estimating the impact of information, executing a minimal deletion closure based on a greedy strategy, and generating a deletion certificate with a statistical upper bound on risk. The authors conducted experiments on structured simulation benchmarks and in small-scale language model memory environments, verifying the phenomenon of information residue after the deletion of original records, the deletion effectiveness of the proposed method, and the effectiveness of the risk audit mechanism under specified test distributions.
Critical strengths
The research question has practical significance. This paper addresses the issue that, even after raw records are deleted from persistent agents, sensitive information may still persist through derived memories such as summaries, user profiles, and skills. Compared to traditional deletion at the storage layer and forgetting at the parameter level, this paper extends the scope of research to heterogeneous agent memory systems, providing a new perspective for understanding and evaluating the residual privacy information within agents. A relatively systematic formal framework and verification mechanism are proposed. This paper defines the effects of memory deletion through a counterfactual world and establishes a relatively comprehensive research framework by integrating memory traceability graphs, impact estimation, and statistical risk auditing. Furthermore, the authors analyze the theoretical boundaries of deletion guarantees and validate the phenomenon of information residue as well as the effectiveness of the auditing mechanism through structured simulation experiments and language model experiments, while clearly discussing the method’s scope of applicability and limitations.
Critical weaknesses
- The rationality and practical applicability of the counterfactual reference (Never-seen Agent) have not been sufficiently substantiated. This paper defines memory erasure as a process that makes the agent, after erasure, behave similarly to a “Never-seen Agent”—one that has never observed the target information. However, my primary concern is whether this counterfactual reference is reasonable and constructible in real, task-driven, long-term agent interactions. In practical applications, users often provide private information to complete specific tasks; this information may directly influence the agent’s decision-making, tool invocation, task execution outcomes, and subsequent interactions. Therefore, simply removing the target information from historical inputs and re-executing the memory generation process may not necessarily construct a causally consistent counterfactual world. For example, if a task must rely on sensitive information provided by the user to be completed, the “Never-seen Agent” may be unable to complete the original task; the difference between its subsequent state and that of the real agent would stem not only from residual privacy information but also from changes in the entire task execution trajectory. Furthermore, a user’s request to delete sensitive information does not imply a desire to revoke completed tasks or lose related non-sensitive outcomes. The paper currently verifies counterfactual differences primarily by manually implanting secret information and constructing corresponding retrieval queries; it has not yet fully demonstrated the validity of the Never-seen Agent in real-world task-dependent scenarios, nor does it provide a systematic evaluation of task completion and long-term utility. Therefore, the current experiments are insufficient to support the universal applicability of this counterfactual deletion definition across the actual, persistent agent lifecycle.
- There is a lack of sufficient comparison with relevant memory deletion methods; the necessity and technical advantages of IWC have not yet been effectively validated. This paper proposes Influence-Weighted Closure (IWC), which aims to strike a balance between privacy protection and memory utility by selectively deleting, recomputing, or retaining derived memories. However, the experiments primarily compare basic strategies such as raw record deletion, TTL, full memory reset, and Reachability Purge, lacking direct comparisons with existing comprehensive agent memory deletion methods. Although the authors introduced the Sanitiser Baseline, its implementation across different memory layers is incomplete. More importantly, Table 15 in the paper itself shows that, for the same optimization problem, the exact solver not only achieved a 100% feasibility rate but also had a lower cost than IWC; a simple, search-free strategy also achieved a 100% feasibility rate, whereas IWC achieved only 76%. Furthermore, the main structured experiments in Table 14 did not measure utility on real tasks but instead used a manually designed utility surrogate; the simple Reachability Purge in Table 31 was already capable of achieving zero residual risk while preserving 98% of irrelevant memory utility. Although subsequent hybrid memory experiments indicate that recomputation can reduce the loss of irrelevant information, these benefits primarily stem from fine-grained deletion and recomputation operations; they have not yet sufficiently demonstrated that IWC’s greedy planning mechanism itself possesses irreplaceable advantages. I believe the authors need to further introduce competitive relevant methods and systematically compare privacy leakage, task performance, and computational overhead in real-world long-term agent tasks to fully demonstrate the necessity and practical value of the proposed method.
Questions for authors
On the Rationality of the “Never-seen Agent”: In real-world task-driven agents, sensitive information may directly affect task execution, tool invocation, and subsequent interactions. If this information is removed from historical inputs, subsequent task trajectories may no longer hold. How do the authors ensure that the “Never-seen Agent” constitutes a causally consistent and semantically reasonable counterfactual reference? In particular, when a user wishes to delete sensitive information while retaining the results of tasks that have already been completed, does the current definition of counterfactual deletion still apply? On the Necessity and Effectiveness of IWC: Table 15 shows that both the exact optimization method and a simple search-free strategy achieve a 100% feasibility rate, whereas IWC achieves only 76%, and the exact optimization method incurs lower costs. Furthermore, the current experiments lack direct comparisons with relevant agent memory deletion methods. Could the authors further explain the unique advantages of IWC compared to these simple strategies, and under what conditions its greedy planning mechanism can achieve a better trade-off between privacy protection and practical task utility?
Dear Reviewers and Area Chairs,
We sincerely thank you for your time, thoughtful evaluations, and constructive feedback throughout the review process.
Your comments have helped us identify important limitations and opportunities to strengthen our work. We appreciate the careful consideration given to our submission.
We will take your suggestions into account, further improve the manuscript, and prepare a stronger version for submission to a future conference.
Thank you once again for your valuable feedback and contributions to improving our research.