MemoWM: How World Models Change What Agents Need to Remember

World models capture regularities that can be reused across experiences, whereas episodic memories incur a new storage cost at every interaction. Making the same world model available at recall allows an agent to reconstruct predictable content and store only the additional information needed to preserve future task quality. We introduce ResidualMem, a task-aware conditional coding framework that turns this shared predictive knowledge into episodic storage savings. The world model provides a coding distribution for retained state components and reconstructs omitted ones. A state-dependent utility gate combines calibrated prediction-error probabilities with the task impact of an error to retain corrections according to their estimated risk and coding cost. Across six long-term agent-memory benchmarks, ResidualMem achieves the highest average QA accuracy and the lowest average persistent storage among the evaluated memory systems. We further find that stronger world models require fewer episodic bits at comparable task quality, but do not always minimize total storage once model parameters are included. The storage-minimizing model size increases with the number of remembered interactions, as savings on each experience offset the additional parameter cost.