What Should a Navigator Remember? Spatiotemporal Memory for Zero-Shot Vision-Language Navigation

Zero-shot vision-and-language navigation (VLN) agents built on large proprietary models generalize well but collapse when the backbone is replaced with an open 8B-scale model that can run on a robot's compute budget. Post-trained agents achieve high success in simulation but tend to follow memorized routes rather than the given instruction, and degrade in the real world. We argue that the gap is not perception but memory: existing agents present the model with an unorganized stream of past observations, from which a small model cannot extract the few facts each decision requires. We ask what a navigator should remember, and answer it through a semantic communication view in which a navigation decision is a query and memory is a set of facts that must entail it. Memory therefore need not reproduce the observation stream, only preserve what may establish future queries, and the agent should act to gather evidence when memory is insufficient. We instantiate this view as a spatiotemporal memory agent with task memory over ordered subgoals, hierarchically compressed temporal memory anchored to poses, and a geometric spatial map with a committed navigation target. The agent selects waypoint intents executed by a geometric controller and issues a preview action when the current view cannot resolve the decision. To evaluate whether an agent follows instructions rather than routes, we introduce three instruction-perturbation protocols. On R2R-CE val-unseen, our method achieves the best SR and SPL among zero-shot methods on 8B-scale models, consumes roughly $4\times$ fewer tokens per decision than the strongest zero-shot baseline, and transfers to a Unitree Go2 quadruped in real-world environments where a post-trained baseline fails.