SaliMory: Orchestrating Cognitive Memory for Conversational Agents

Recall and personalization, enabled by memory, are pivotal to a lifelong conversational agent. However, simply expanding context windows with raw retrieval degrades reasoning quality, while training memory agents via standard reinforcement learning creates a severe credit assignment bottleneck in a multi-stage pipeline. To solve this, we introduce SaliMory, a framework that learns to manage and utilize a cognitively-structured memory—spanning user facts, preferences, and working memory. By introducing novel stage-wise process rewards and reward-decomposed contrastive refinement, SaliMory provides isolated supervision for distinct memory operations (selective filtering, consolidation, and cue-driven recall) end-to-end. SaliMory cuts memory-attributed failures by one-third, outperforms the state-of-the-art by over 10% in end-to-end recall accuracy, and more than doubles the Good Personalization rate across 4 datasets.