MemSteer: Self-Evolving Memory for Long-Horizon Vision and Language Reasoning

Self-evolving natural-language memory lets frozen models draw on experience during long-horizon visual and textual reasoning, but its value depends on selecting useful guidance at each step. Given the same memory and the same tools, a frozen solver that keeps every policy in context or picks one for itself falls below tool use without any memory in most of our settings. We propose MemSteer, a framework that couples state-conditioned memory selection with outcome-validated memory evolution across high-resolution images, long documents, and interactive environments. A lightweight controller selects at every decision the natural-language policies that the frozen solver acts on, learning from groups of rollouts whose advantages are discounted by how widely identical choices already scatter rewards. On a slower timescale, failed trajectories are clustered, a language model writes one abstract policy per cluster, and the update is evaluated through a paired comparison on unseen tasks. Extensive experiments on nine benchmarks spanning long-horizon visual search, long-document reasoning, and interactive task execution with four frozen solvers show MemSteer improves average performance over strong tool-use and experience-based baselines.