REMAP: RESTORING THE PERCEPTUAL CYCLE WITH REASONING-TIME LATENT VISUAL MEMORY

As multimodal large language models (MLLMs) reason for longer, attention to the initial visual input diminishes, weakening visual grounding. Visual memory reintroduces visual evidence during reasoning. We conduct a controlled analy- sis of visual memory along three axes: curation, organization, and access. We find that local evidence benefits from global context, compact latent representa- tions balance accuracy and visual-context cost, and the utility of memory access depends on the reasoning state. Guided by these findings, we propose ReMAP (Reasoning-Time Memory-Augmented Perception), which couples two comple- mentary latent memories: a static, question-conditioned Global memory that pre- serves scene and cross-image context, and a dynamic Local memory that uses this context as an anchor while selecting and re-encoding region-level evidence according to the current reasoning state and the question. Both memories re- turn compact latent tokens that are inserted into the reasoning sequence, and a reinforcement-learning access policy trained with branched rollouts decides when to continue reasoning or invoke Global or Local memory. On ten benchmark fam- ilies, ReMAP outperforms prior visual-memory methods on all four multi-image benchmarks, exceeding the strongest prior results on MuirBench and MIMIC by 8.38 and 14.84 percentage points, respectively. Across four backbone families, enabling memory access improves over the same trained model with memory dis- abled, and on shared V*Bench, CV-Bench-2D, and MuirBench questions ReMAP reduces the visual tokens entering the reasoning sequence by 51.0–76.8% relative to the native-resolution backbone. Further analyses show that Global and Local memory form distinct yet complementary latent representations and play coordi- nated roles in interleaved vision-language reasoning. Together, these components restore the perceptual cycle by letting the reasoning state trigger targeted visual retrieval, with the retrieved evidence guiding subsequent reasoning.