Local Cues, Global Misrouting: Poisoning Hierarchical Embodied Memory through Recursive Consolidation

Long-horizon embodied agents usually store extensive visual observations, motivating the use of hierarchical memory. Typically, local visual observations are recursively consolidated into higher-level summaries that guide coarse-to-fine retrieval. However, the same consolidation process can also preserve misleading visual cues in higher-level summaries. By exploiting these cues, an attacker biases subsequent retrieval and thereby misdirects the agent to a specified wrong location. Despite growing work on poisoning in RAG and agent memories, poisoning through recursive consolidation remains underexplored. To address this gap, we develop CueClimb, a visual poisoning method that operates during environment mapping. CueClimb constructs sparse, scene-grounded visual cues and searches for compact bundles whose semantic influence survives recursive consolidation. It uses only local observations and auxiliary consolidation contexts, without access to the full victim scene or subsequent queries. Across 99 target-query cases in 23 HM3D scenes, CueClimb achieves a 46.5% ASR, compared with 28.3% for the strongest baseline. The attacks transfer across different models, consolidation prompts and hierarchical memory constructions. These findings expose recursive consolidation as a critical attack surface, underscoring the need for trustworthy embodied memory.