Have I Scene This Before? Spatially Grounded Conversational Memory for Complex Queries in Egocentric Assistants

Personal assistants are evolving from search and text interfaces toward interactive multimodal systems that retain spatial and conversational history to support ongoing tasks. Effective assistants must therefore reason jointly over observed scenes and past dialogue to address complex requests. We formalize this challenge as *Spatially grounded Conversational Reasoning* (SPACR), covering cross-scene, recall-oriented, and counterfactual spatial queries that require accurate geometric reasoning and the integration of dialogue with spatial evidence. Existing approaches are poorly suited to this setting. Direct VLMs incur high inference costs and context limits as multimodal histories grow. Keyframe selection and retrieval can omit evidence needed for complete recall, while memory systems must preserve both conversational facts and object geometry to support spatial reasoning. To bridge this gap, we propose *Spatially grounded Conversational Memory* (SPAC-MEM), an object-centric working memory that applies 3D reconstruction and segmentation to egocentric observations, thereby spatially grounding conversational context directly to physical objects. It compresses long multimodal histories while preserving spatial and conversational evidence and allowing object-specific information to be updated through later dialogue. We further introduce *EGO-SPACR*, an egocentric SPACR dataset comprising 620 video sessions from ScanNet, augmented with 95 conversations averaging 230 dialogue turns each, and 3,100 evaluation queries. Experiments show that SPAC-MEM achieves the highest overall answer accuracy among the evaluated methods and improves object recall, while requiring substantially fewer reference input tokens than native-video baselines. Analysis shows that removing 3D spatial information substantially degrades performance, highlighting the importance of preserving both spatial and conversational evidence for an egocentric assistant.