Visual Memory Fields: A Brain-Inspired Spatial Scaffold for Embodied Memory

Embodied agents need memory that can integrate growing streams of local, viewpoint-dependent observations into a persistent representation under finite capacity. Existing systems often trade off bounded storage against long-horizon retention and flexible reuse. Inspired by hippocampal-entorhinal systems, we propose **Visual Memory Fields**, which bind episode-specific sensory content to a reusable spatial scaffold represented as a continuously addressable field over position and heading. We instantiate this scaffold with a sparse Fourier basis shared across different maze layouts, and update episode-specific content online through compact sufficient statistics rather than storing observation history. A fixed Fourier support keeps the sufficient statistics—and therefore persistent memory—independent of experience length, while budget-constrained search allocates the available capacity across pose frequencies. This bounded representation can then be queried and reused after accumulation, rather than being tied to the objective under which experience was collected. We test this property in Memory Maze, first through view-centric recall over long histories and randomly withheld views, and then through scene-centric object localization using a separately supervised reader on the frozen field. We further show that the resulting scene belief can be re-expressed in candidate action frames for downstream control. Together, these results demonstrate **Visual Memory Fields** as a persistent, bounded, and reusable form of scaffolded embodied memory.