Contextual Episodic Memory for Context-Aware Robot Navigation

Embodied navigation in partially observable environments requires agents to retain information acquired earlier in an episode, since relevant objects, rooms, and landmarks may become inaccessible from the current observation. We propose Contextual Episodic Memory (CEM), a framework that progressively constructs episodic memory by integrating previously accumulated context into newly encountered observations. CEM selectively retains informative visual observations and associates each with the agent's 3D pose. Each new visual-spatial representation attends to stored episodic representations to retrieve relevant context, which is integrated with the current representation before being stored as the next memory entry. This produces an evolving representation of navigation experience that complements the current observation during decision making. We evaluate CEM in the photorealistic environment on two complementary tasks that require episodic memory in distinct ways: multi-room object search, where an agent must use information about previously observed objects and their locations after they are no longer visible, and target-room navigation, where an agent must retain scene-level experience gathered along the trajectory to navigate toward a room from varying starting locations. We compare CEM with recurrent, external-memory, and Transformer-based navigation approaches and analyze the contributions of informative observation selection and contextual memory construction. Results demonstrate that progressively constructed episodic representations improve navigation under partial observability by enabling agents to make decisions using experience accumulated throughout the episode.