IGMem: Interaction-Grounded Memory for Multimodal Multi-Party Conversations

Agent memory for multimodal multi-party conversations aims to support recall and reasoning about past interactions and events, whose constituent information is often distributed across participants, turns, and modalities. Building such memory is challenging for two reasons: information about the same event can be scattered across different speakers, turns, and modalities, and different participants may attend to different parts of the same multimodal content. Existing approaches typically organize memory by temporal proximity, semantic similarity, or generic visual content, but overlook these interaction-dependent relationships, leaving event information fragmented or ignored. To address these challenges, we introduce IGMem, a novel Interaction-Grounded Memory framework for multimodal multi-party conversations. Our key insight is that each response provides a signal for connecting distributed sources and selecting the particular content relevant to it. Specifically, IGMem first recovers the interaction flow from multiple conversational cues, and then compiles the relevant textual and visual evidence into response-centered memory units. Such memory units brings information belonging to the same event together, making it easier to retrieve and reason over past information. Extensive experiments show that IGMem outperforms competitive baselines on two benchmarks across different backbones. Our code is available at https://anonymous.4open.science/r/IGMem.