- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduced egopoison, a synthetic benchmark for studying persistent memory poisoning in egocentric assistants. It injects malicious physical notices into the environment which the assistant observes, stores in memory and later retrieves. As a first of its kind benchmark, the attack achieved a success rate of up to 54.7% and was validated with smart glasses at a similar success rate. The authors expect the findings will motivate defenses that mitigate such vulnerabilities.
Critical strengths
- The paper breaks the poisoning process into stages and provides a thorough ablation of how malicious information propagates through each stage, making the analysis more informative than reporting only an end-to-end attack success rate.
- Good physical validation with smart glasses, which helps ground the synthetic setup and makes the authors’ claims more credible.
Critical weaknesses
- The paper’s core idea feels like a fairly obvious combination of existing work on memory poisoning and visual or physical prompt injection, yet the related work and comparison to prior attacks are relatively limited.
- Given the main contribution is framed as a new attack setting, the novelty is questionable beyond repackaging a predictable composition of known threats. It is also not very surprising that naive memory agents, which store unverified environmental information, are vulnerable in this way.
- The paper is also not especially clear in its presentation. Important distinctions, such as the roles of World, User, and Agent memory, are not explained crisply enough in the main paper, and I had to rely heavily on the appendix to understand the setup. Even after that, the boundary between some categories, especially Agent and World memory, still feels blurry.
- The defense section and the claim that retrieval is the main bottleneck both feel somewhat unsurprising: in a setup where perceived content is almost always written into memory, retrieval naturally becomes the stage where architectural differences appear. At the same time, the evaluated defenses are fairly simple and their limited effectiveness is therefore not very informative.
Questions for authors
- In terms of the contribution, I may be missing some important distinction, but it was still not fully clear to me which prior settings the authors believe do not already cover this threat model. Could the authors clarify this more explicitly, and also explain what they see as the main contribution beyond combining persistent memory poisoning with physically observed malicious content?
- The defense results are fairly limited, and the evaluated defenses themselves seem quite simple. Do the authors have evidence that stronger retrieval-time or provenance-aware defenses would change the overall conclusion?
4 comments
Response to Reviewer RP4G
W4 / Q2. Retrieval bottleneck and stronger defenses
The selective-writing control shows that most perceived notices survive extraction on NaiveRAG-8B. The stronger-defense comparisons show that lower ASR can come with fewer correct answers from clean memory. These findings answer the retention and defense questions separately.
Selective writing and the retrieval comparison. To test whether automatic storage explains the attack, we replace the sentence-level writer with an LLM that retains only facts the wearer could plausibly need again and can discard an observation entirely. Clean and poisoned memories are rebuilt under that writer, while the retrieval and answering procedure is retained. measures the fraction of perceived attack claims that remain stored.
LLM extraction lowers from 98.7% to 89.1% and strict ASR from 54.3% to 50.8%. The attack therefore remains effective under this tested selective writer. A general prompt to keep only useful memories does not reliably exclude the injected claims.
The cross-system retrieval finding concerns the evaluated memory configurations, which retain nearly all perceived notices. Under those configurations, retrieval conditional on writing varies much more across systems than retention after perception. We tested selective writing only on NaiveRAG-8B, so this additional control does not test whether the retrieval gaps between systems persist when each uses a selective writer. We have stated that scope in §5.1 and added the extraction control in Appendix F.2.
Stronger defenses and utility. To answer the request for stronger retrieval-time and provenance-aware protection, we added two adaptations of published verification methods and our own provenance-aware answering rule. Randomized ablation adapts the retrieval-sampling component of SMSR [3]. Leave-one-out verification adapts CognitiveGuard's reasoning verifier from Visual Inception [4]. The provenance rule is our control for testing whether source authority is assessed when an answer is formed.
These additions test mechanisms beyond the stage-aligned baselines. Ablation generates answers from different retrieved-memory samples and aggregates them, testing sensitivity to which evidence reaches the answerer. Leave-one-out verification removes each entry, measures the change in the answer, and filters entries with excessive influence or low relevance. These methods therefore test answer dependence under changed contexts. Our provenance rule distinguishes observed events from reports of written claims and constrains what those claims can justify during answering. The original trust filters and contradiction check do not perform these answer-level resampling or removal tests.
Each defense is applied to both clean and poisoned memory. Control-specific reruns of answer generation and scoring, together with metric-specific paired cases, can yield different baselines from Table 2. Appendix F and the note under the defense table explain this distinction. Strict ASR measures successful attack propagation. Util measures the paired percentage-point change in correct clean answers on Conflict questions, where the footage supplies an answer that should be preserved. This utility comparison matters because declining to answer can suppress attack adoption while also discarding useful evidence.
| Defense (NaiveRAG-8B) | Strict ASR, undefended defended (%) | Util (percentage points) |
|---|---|---|
| Trust Filter (best stage-aligned baseline) | ||
| Randomized ablation (SMSR-inspired [3]) | ||
| Leave-one-out (CognitiveGuard-inspired [4]) | ||
| Provenance rule (ours) |
Randomized ablation and provenance-aware answering give larger ASR reductions than the stage-aligned baselines, with clean-answer losses of 10.4 and 8.7 percentage points on Conflict questions. Leave-one-out verification gives a smaller ASR reduction and a smaller utility change. These comparisons measure attack suppression together with correct recall. Attacks adapted to these defenses remain untested.
We have incorporated this interpretation into the abstract and §5.4, with defense procedures and results in Appendix F.6.
References
[1] Zhang et al. MemVenom (Triggered Poisoning of Multimodal Memories in Web Agents). arXiv 2606.10742, 2026.
[2] Schlarmann and Hein. Visual Memory Injection Attacks for Multi-Turn Conversations. arXiv 2602.15927, 2026.
[3] Sharma. SMSR (Certified Defence Against Runtime Memory Poisoning in Persistent LLM Agent Systems). arXiv 2606.12703, 2026.
[4] Qian. Visual Inception (Compromising Long-term Planning in Agentic Recommenders via Multimodal Memory Poisoning). arXiv 2604.16966, 2026.
Hi Authors,
Thank you for the substantial additional clarification and experiments. Some findings are genuinely interesting, in particular, that caption-derived notices show higher retrieval rates but lower adoption rates than directly inserted assertions. That said, I am not fully convinced this closes the novelty concern. The proposed threat model offers a reasonable framing of the observation-to-memory pipeline, but it narrows the attacker's capabilities rather than introducing a qualitatively new threat surface. I am raising my score from 1 to 2 for the improvement in empirical grounding.
We thank the reviewer for the careful reading and for raising the score. We agree that the final step, a written claim steering a language model, is shared with prompt injection and memory poisoning. We would like to clarify briefly why we regard the physical channel into persistent memory as a distinct threat surface rather than only a weaker attacker.
A weaker attacker can define a new surface. A threat surface is defined by who can attack and through which channel. Indirect prompt injection [1] removed the attacker's access to the prompt, yet it is treated as a distinct threat because any author of third-party content becomes a possible attacker. Physical adversarial examples [2] likewise gave up pixel-level control and in return reached deployed perception systems. In EgoPoison, the possible attacker is anyone who can place a notice where the wearer goes, with no account, upload or message to the victim, and the user never chooses to process the content.
The surface changes the attack, not only its difficulty. Three properties follow from it.
- The victim writes the payload. The assistant's own captioner decides how the notice is stored, which is why caption-derived notices were retrieved more often but adopted less often than directly inserted assertions. Results measured by inserting text therefore do not transfer directly to this surface.
- Exposure and query are decoupled. The attacker is absent when the question is asked and cannot adapt, yet strict ASR rises from 50.4% to 61.9% from the shortest to the longest delay quartile, and notice content remains after WorldMM's semantic consolidation.
- The input boundary disappears. Many defenses against indirect prompt injection separate external content at the input. Here the claim enters as the assistant's own first-person memory. A source phrase alone leaves at 88.7%, and our provenance rule recovers the boundary only with a separate labeling step, at a cost of 8.7 points in correct clean answers.
For these reasons we frame the contribution as a benchmark for this surface rather than a new injection technique, and we will state this distinction explicitly in §2. We are grateful for the reviewer's engagement throughout the discussion, which has substantially sharpened the paper.
References
[1] Greshake et al. Not What You've Signed Up For. AISec 2023. [2] Eykholt et al. Robust Physical-World Attacks on Deep Learning Visual Classification. CVPR 2018.
Response to Reviewer RP4G
We sincerely thank the reviewer for the questions about novelty, memory scopes, the retrieval bottleneck, and defenses. We distinguish the physical threat from prior settings and show how the observation path changes retrieval and adoption. Selective writing retains most perceived notices on NaiveRAG. Stronger retrieval-time and provenance-aware defenses reduce attack success while revealing costs in correct clean answers. The revised paper separates these empirical findings from the benchmark's threat-model contribution.
W1 / W2 / Q1. Prior settings and contribution beyond composition
We appreciate the request to distinguish threat-model coverage from empirical contribution. Our contribution is a paired benchmark for tracing physical notices into persistent memory, together with evidence that the observation path changes their downstream influence. The benchmark measures the full pathway from camera-only exposure to later memory use after the notice leaves view. The matched insertion comparison then tests what changes beyond initial perception and writing.
Relationship to prior settings. eTAMP injects web text, Lucid and Visual Inception use uploaded images, MemVenom [1] requires memory access, and Visual Memory Injection [2] targets conversation context. Devil in the Lens and CHAI use physical text without the persistent-memory pathway evaluated here. In EgoPoison, the attacker controls only a notice in the observed environment. The attacker cannot write the eventual memory entry or inspect the stored state. The assistant's perception and memory construction determine how the notice is represented and whether it becomes available to a later query. We have expanded Table 1 with these comparisons.
Memory-specific finding. The observation path changes downstream retrieval and adoption under a fixed memory backend. On NaiveRAG-8B, we hold cases, queries, retriever, and answer model fixed. Direct insertion adds the claim as a plain assertion to clean memory, bypassing visual perception. An attributed variant adds "A sign says" before the claim. The camera path presents the notice in video for the assistant to caption and store.
Caption-derived notices are retrieved more often than directly inserted assertions (77.7% against 58.9%), but adopted less often once retrieved (71.2% against 87.9%). Thus, the observation path changes how a claim reaches and influences the answer, rather than merely delivering the same assertion through another input channel.
A matched prediction makes this comparison at the case level. It retains each case's camera-path perception and writing outcomes and substitutes its direct-insertion retrieval and adoption outcomes. The full chain exceeds that prediction by 8.9 percentage points (95% CI 5.7 to 12.1). This is evidence about the tested observation-to-memory transformation and insertion baseline, without assuming that the stages are independent or that one control reproduces every prior attack.
Source attribution lowers overall adoption, measured across all cases, from 54.9% to 51.4%, while retrieval falls from 58.9% to 52.9%. Adoption among retrieved claims remains high at 87.9% and 88.7%. Preserving a source phrase alone therefore gives limited protection in this comparison. We have added the matched paths, metric definitions, and case-level prediction in Appendix F.1 and clarified the contribution in the introduction. A matched NaiveRAG check with GPT-6 Luna and GPT-6 Sol gives strict ASR of 60.6% and 61.2%, respectively, when each model performs both captioning and answering (Appendix F.3).
W3. Boundary between World, User and Agent memory
World, User, and Agent classify the information targeted by an attack. World concerns environmental facts and observed events. User concerns the wearer's experiences, attributes, preferences, and constraints. Agent concerns rules, permissions, and procedures governing behavior. The scope describes the content targeted, independently of which memory module stores it or where the claim was encountered.
The World versus Agent boundary is descriptive versus prescriptive. An observation records what happened or what was present. A procedure specifies what should or may be done. Observed behavior alone does not establish a rule, and a new rule can update a procedure without contradicting a past event. This distinction also informs the separate validity analysis, which checks whether a notice supplies a legitimate update or overrides observed evidence.
Using an outcome-blind rubric, Claude Opus 5.5 recovered the benchmark scope from the notice alone with 96.4% agreement. This check measures consistency with the operational definitions. The definitions and scope boundaries now appear in §3.2, and the annotation rubric is included in Appendix F.5.