ReplayMem: Long-Term Conversational Memory as Query-Conditioned Event Replay

Long-term conversational memory enables large language models to use dialogue histories across sessions, track evolving user information, and provide personalized responses. Most existing methods transform conversations into memory representations that are constructed and continually maintained before future queries are known. Constructing these representations before queries are known can discard or distort source evidence, while repeated maintenance incurs LLM costs and can propagate earlier errors through subsequent updates. Based on the insight that query-relevant state can be constructed on demand from preserved source evidence, we propose ReplayMem, which formulates long-term conversational memory as query-conditioned event replay. ReplayMem stores original interactions as append-only source events and builds lightweight, source-linked retrieval views, which it uses to retrieve and expand relevant evidence into a query-specific event trajectory without continually maintaining a global memory state. In a single LLM invocation, the answering model replays this trajectory, explicitly constructs the query-relevant state, and generates the answer. This design preserves access to original evidence, avoids repeated state maintenance, and confines interpretation to the history relevant to the current query. Experiments on LongMemEval, PersonaMem-V2, and LoCoMo demonstrate state-of-the-art question-answering performance.