DG-Mem: Dual-Graph Memory Retrieval for Multi-User Agents

Long-term memory enables LLM agents to build on past interactions, but most existing systems are designed around a single user. Shared assistants in enterprise channels, project discussions, and organizational workflows must instead serve multiple users through a common conversational history. This introduces three challenges: distributed and heterogeneous evidence, coexisting and evolving user states, and asker-dependent relevance. Content-oriented retrieval and memory consolidation can miss cross-user connections or obscure the attribution needed to distinguish valid information simultaneously. We propose , a dual-graph framework that couples shared content organization with user-conditioned evidence selection. A content-centric graph connects evidence through semantic, conversational, and temporal relations, while a user-context graph preserves speaker-grounded distinctions through authorship and interaction structure. Aligned message anchors connect these complementary views. Asker-conditioned cross-graph retrieval accesses relevant evidence across participants, and query-conditioned refinement selects supporting messages according to user context and the question's specific requirements. The framework also supports conventional single-user memory, providing a common foundation for personal and shared histories. Memory construction and retrieval do not require generative-LLM calls and incur zero generative-LLM token cost before answer generation. Experiments on GroupMemBench and LoCoMo demonstrate state-of-the-art overall performance in multi-user and conventional single-user settings, respectively. Our code is available at https://anonymous.4open.science/r/DG-Mem.