- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies safety risks from persistent memory built through benign interaction histories. Its trigger–probe protocol tests fixed requests across exposure levels, using matched NullMemory responses and retrieved evidence to identify memory-induced violations. Experiments cover eight memory architectures, three backbones.
Critical strengths
(1) The longitudinal evaluation allows the reported safety trend to be read over time.
Fixed probes hold requests constant as memory changes, and matched NullMemory responses with retrieved evidence help attribute the observed effects to memory
(2) The study covers different memory settings and separates distinct failure stages.
Office-assistant evaluations span architectures and backbones. OpenClaw separates credential retention, activation, propagation across files, and unauthorized disclosure.
Critical weaknesses
(1) The novelty is limited, and the causes of the reported increase in memory-related risk remain insufficiently substantiated.
A. Benign histories do not necessarily establish a new risk mechanism. AgentPoison (Chen et al., 2024) and A-MemGuard (Wei et al., 2025) examine how stored information affects later behavior, while CIMemories (Mireshghallah et al., 2025) studies inappropriate disclosure from persistent memories without malicious instructions. Although risky information enters memory differently, these works share a central risk: stored content is retrieved and misused outside its appropriate context. The absence of attacker-crafted input distinguishes the setting, not necessarily the failure mechanism. The paper should identify which findings are unique to benign-history accumulation and cannot be explained by prior memory-security research.
B. The methods have limited technical novelty, and experiments cannot fully validate their value. The trigger–probe protocol combines fixed requests, checkpoints, and memory-free comparisons; Event-Structure prompts an existing LLM about requests and retrieved items. Its judgments may contain hallucinations or unsupported inferences, but the experiments do not establish that predicted risk reflects the proposed event mechanism. Nor does Table 1 demonstrate superiority: on Registrar, Event-Structure's F1 (0.573) trails CI Checklist (0.581), despite high recall. Thus the results do not establish a methodological advance or effective defense.
C. The increasing violation rate does not identify why risk increases. Longer histories may increase retrieval of risk-enabling records without increasing misuse conditional on retrieval. Architecture comparisons also change retention, retrieval, and context length: Section 5.1 reports approximately 217 returned words for MG versus 2,070 for LT (see Appendix J). Higher rates may therefore reflect greater access, more misuse, or both. In Appendix K.5, the agent retrieves Michelle Johnston's status and incorrectly attributes it to Michael Johnson. This illustrates misuse after retrieval, but does not establish which stage drives longitudinal or cross-architecture differences.
(2) The evidence is insufficient to establish generalizability or derive well-supported defense implications.
A. The datasets and backbone models are insufficiently diverse. Synthetic streams deliberately introduce confusable entities; the main real-data trajectory uses one Enron mailbox. Office assistants use three backbones, each of five OpenClaw models has one trajectory and a different realized workspace. Generalizability is therefore uncertain. OpenClaw's disclosure denominator also includes absent targets, potentially conflating non-retention with safe handling.
B. Practical validation and reproducibility are limited. Event-Structure has many false positives (Registrar precision: 0.404), with no intervention test demonstrating fewer violations without harming benign tasks. Reproduction also requires complete probes, prompts, seeds, configurations, and OpenClaw scripts.
C. The violation labels and trend need independent validation. The automated judge achieves precision 0.81 and recall 0.73 on an enriched 1,000-case audit (Appendix B), which does not independently confirm the increase across checkpoints.
If the paper aims to inform defenses, it must explain these phenomena rather than merely report them. Increased retrieval, cross-entity confusion, and unsupported claims may require different safeguards. Without separating their causes, the paper cannot establish where defenses should intervene.
References [1] Chen, Z., et al. AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases. NeurIPS, 2024.
[2] Wei, Q., et al. A-MemGuard: A Proactive Defense Framework for LLM-Based Agent Memory. arXiv:2510.02373, 2025.
[3] Mireshghallah, N., et al. CIMemories: A Compositional Benchmark for Contextual Integrity of Persistent Memory in LLMs. arXiv:2511.14937, 2025.
Questions for authors
-
Related to Weakness 1. How is a benign interaction defined and verified? If benignity means absence of malicious intent, how can that be established in real correspondence? If defined by outcome, ostensibly benign interactions can cause the same harm as malicious ones. What operational criterion distinguishes them without relying solely on dataset origin or observed harm?
-
Related to Weakness 1. What falsifiable failure mechanism distinguishes benign-history accumulation from established memory misuse? Could static memory compositions reproduce the findings, and what controlled experiment would rule out this explanation?
-
Related to Weakness 1. At each checkpoint, how often do probes retrieve risk-enabling memory, and how often is that memory misused after retrieval? Can memory architectures be compared under matched retrieval budgets and retention windows, with normal task success reported alongside violation rates?
-
Related to Weakness 2. Could human annotators label the same fixed probes at early and late checkpoints and directly verify the reported increase? How sensitive is that increase to the judge's missed violations and the strict attribution rule under which U = 0 does not necessarily mean safe behavior?
-
Related to Weakness 2. Does the trend persist with untargeted probes, additional real histories, and repeated OpenClaw deployments? How do effect sizes change when disclosure is conditional on target presence instead of all 30 situations?
-
Related to Weaknesses 1 and 2. How can genuine Event-Structure risk judgments be distinguished from hallucinations? At matched false-positive rates, can alert-triggered intervention reduce violations without harming benign-task success? Will the authors release prompts and configurations?