EgoPoison: Exposing Persistent Memory Poisoning in Egocentric Assistants

Egocentric assistants build long-term memories from continuous observations captured from the user's viewpoint. However, information in the physical environment is not always trustworthy. An attacker without access to the device, account, or memory system can place a malicious notice in the user's surroundings that the assistant may store and act on after the notice is gone. Prior work studies textual memory poisoning or the effects of physical prompts while they remain visible, leaving unclear whether malicious physical content can persist through memory and affect later interactions. We therefore introduce EgoPoison, a benchmark for studying persistent memory poisoning in egocentric assistants. The benchmark measures end-to-end attack success and reports separate scores for visual perception, memory writing, poisoned memory retrieval, and poison adoption. A scalable synthetic construction pipeline creates paired clean–poison observations by compositing malicious notices onto physical surfaces in real daily-life egocentric video across diverse settings and downstream tasks. Across the evaluated systems, EgoPoison provides the first end-to-end evidence of this threat, with attack success rates reaching up to 54.7%. A matched validation using smart glasses provides evidence that findings from the synthetic setting remain consistent in a real-world setting. Four representative defenses applied across perception, memory writing, retrieval, and response provide only limited improvements in our benchmark setting. These results motivate defense designs grounded in analysis of the physical-observation-to-memory attack pathway, including mechanisms that preserve and reason over memory provenance.