MemPilot: Learning to Organize and Navigate Latent Memory

Large language models (LLMs) are increasingly capable of using long and complex histories, but processing the full history for every query remains expensive. Latent memory stores past information in reusable states, but repeatedly updating a single state can make earlier information difficult to recover as history grows. Storing history in multiple states can preserve earlier information, but introduces a new problem of retrieving the states relevant to each query. To this end, we propose MemPilot, a framework for organizing and navigating growing latent memory. First, we store history as independently written and restorable memory states, preventing later writes from modifying earlier ones. Each state is paired with a compact semantic key that supports candidate selection and briefly describes its content. Second, we retrieve evidence sequentially from memory: evidence from each read guides the next state selection, read instruction, and decision to stop. We supervise state readouts with page-local evidence to support navigation and train the access policy through RL post-training, using same-history counterfactual comparisons for more local credit assignment. Together, these components make growing latent memory easier to preserve and navigate adaptively. We evaluate MemPilot on five QA and long-term-memory benchmarks under two distinct latent-memory backends. Across both backends, MemPilot substantially improves over the corresponding native memory baselines. The five-task average improves from 13.51 to 39.99 with Metis and from 15.14 to 33.10 with $\delta$-Mem.