3D-Mem: Learning to Organize Evidence for Long-Horizon Conversational Memory

A long-horizon conversational agent can retain the information needed to answer a question yet fail to recover it within a compact context. Relevant details may remain buried in coarse summaries, disconnected across sessions, or obscured by redundant memories. We address this gap between information retention and evidence access with 3D-Mem, a framework that learns how to organize conversational evidence for future retrieval. Its central mechanism couples task-feedback-guided graph-operator induction with a provenance-linked evidence index: a PPO-trained controller selects graph edits, an LLM designer refines the operator library from training failures, and the resulting topology supplies retrieval signals grounded in source segments. Hierarchical reranking projects these signals back to the supporting text, while utility gating and diversity pruning assemble the answer context. Trained on LoCoMo with Qwen, the controller and operator library are frozen for evaluation across two benchmarks and two backbones. On LoCoMo with Qwen, 3D-Mem improves overall F1 over A-Mem by 5.58 points. With Qwen, it uses 24.2% fewer retrieved tokens than Mem0; holding graph architecture and PPO budget fixed, freezing the initial operator library lowers F1 by 1.11 points. These results support adapting memory organization to downstream evidence needs.