S-MEM: Specialized Memory Agents for Long-term Memory of LLM Agents

Long-term memory is essential for LLM agents operating over extended interactions, where answering a query may require connecting evidence scattered across sessions. Existing memory systems typically store past interactions in external structures, such as vector databases or knowledge graphs, which can fragment original interaction contexts and weaken contextual dependencies for long-horizon reasoning. Recent memory-agent approaches instead use dedicated agents whose contexts directly serve as memory, preserving interaction histories for local reasoning. However, they typically organize memory by individual interaction episodes, causing related experiences from different sessions to be stored in separate agents and making cross-session reasoning difficult. We propose Specialized Memory Agents (S-MEM), a long-term memory framework that organizes memory agents by topic rather than by interaction episode. Each Specialist is dedicated to a specific topic and maintains the interaction history related to that topic across sessions. Given a query, the Master Agent activates relevant Specialists, which reason over their accumulated histories, and synthesizes their evidence for global reasoning. By consolidating related experiences within the same memory agent, S-MEM enables cross-session reasoning while preserving the contextual richness of original interactions. Experiments on LoCoMo and LongMemEval show that S-MEM consistently improves F1 and BLEU-1 over strong baselines.