Which Memories Are Worth Consolidating? Selective Memory Consolidation for Language Model Agents

External memories enable large language model agents to reuse prior experience, improve exploration, and solve complex interactive tasks more effectively. However, relying on external memory makes agent performance dependent on retrieving appropriate memories, while noisy or conflicting guidance may lead to incorrect actions. Memory consolidation offers an alternative by selectively transferring experience that contributes to further policy improvement into model parameters, turning accumulated memories into reusable capabilities and reducing reliance on retrieval. The key challenge is to identify which memories remain useful for learning as training progresses, since a memory’s usefulness in guiding a rollout does not necessarily translate into further improvement of the memory-free policy. We propose $\text{Co}$nsolidation of $\text{M}$emory into $\text{P}$arameters (CoMP), a selective memory consolidation framework that identifies memories whose induced updates can persistently improve the memory-free policy. CoMP uses memories to guide trajectory generation and reinforcement learning to consolidate useful behaviors, selecting memories based on whether their induced updates are compatible with memory-free policy improvement. Experiments on ALFWorld, WebShop, and ScienceWorld show that CoMP consistently improves memory-free performance over GRPO and memory-augmented baselines. Specifically, CoMP achieves relative improvements of up to 20.3% on ALFWorld and 12.7% on WebShop over GRPO across model scales, while also demonstrating improved generalization to out-of-distribution tasks.