acceptodds
Under review as a conference paper at ICLR 2027

What Should a Navigator Remember? Spatiotemporal Memory for Zero-Shot Vision-Language Navigation

Abstract

Zero-shot vision-and-language navigation (VLN) agents built on large proprietary models generalize well but collapse when the backbone is replaced with an open 8B-scale model that can run on a robot's compute budget. Post-trained agents achieve high success in simulation but tend to follow memorized routes rather than the given instruction, and degrade in the real world. We argue that the gap is not perception but memory: existing agents present the model with an unorganized stream of past observations, from which a small model cannot extract the few facts each decision requires. We ask what a navigator should remember, and answer it through a semantic communication view in which a navigation decision is a query and memory is a set of facts that must entail it. Memory therefore need not reproduce the observation stream, only preserve what may establish future queries, and the agent should act to gather evidence when memory is insufficient. We instantiate this view as a spatiotemporal memory agent with task memory over ordered subgoals, hierarchically compressed temporal memory anchored to poses, and a geometric spatial map with a committed navigation target. The agent selects waypoint intents executed by a geometric controller and issues a preview action when the current view cannot resolve the decision. To evaluate whether an agent follows instructions rather than routes, we introduce three instruction-perturbation protocols. On R2R-CE val-unseen, our method achieves the best SR and SPL among zero-shot methods on 8B-scale models, consumes roughly fewer tokens per decision than the strongest zero-shot baseline, and transfers to a Unitree Go2 quadruped in real-world environments where a post-trained baseline fails.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-and-Language Navigation, Zero-Shot Navigation, Spatiotemporal Memory, Embodied Agents, Vision-Language Models, Instruction Following

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LpY4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies zero-shot vision-language navigation with smaller vision-language models and proposes a navigation system that combines task memory, hierarchical temporal memory, and geometric spatial memory. The system tracks progress through ordered subgoals, compresses past observations into summaries at multiple levels, and uses an occupancy map and a geometric controller to execute model-selected waypoints. When the current view provides insufficient information, the agent can acquire additional observations through a PREVIEW action. The authors use logical entailment from a semantic communication perspective to characterize the relationship between memory and decision-making. They also introduce three instruction-perturbation protocols: FLIP, GOAL-ONLY, and PARAPHRASE. Evaluation includes R2R-CE val-unseen, component ablations, and real-world robot experiments.

Critical strengths

S1. The system design aligns well with the requirements of navigation. Task memory maintains instruction order and execution progress, temporal memory preserves past observations, and spatial memory supports geometric planning. Pose information connects past experiences to spatial locations. The system also separates high-level waypoint selection from low-level motion control and includes an active observation action to supplement the current view. Together, these choices form a coherent perception–memory–decision–execution pipeline and provide a useful reference for navigation system design.

S2. The system achieves meaningful navigation improvements supported by component ablations. Among the zero-shot methods using 8B-scale models listed in Table 1, the proposed method achieves the highest SR and SPL, at 0.263 and 0.196, respectively. In Table 2, removing temporal memory, spatial memory, or PREVIEW reduces SR to 0.190, 0.205, and 0.190, respectively, supporting the utility of these components within the full system. Including temporal memory is also associated with substantial reductions in path length and action steps. Furthermore, adding temporal memory to CA-Nav improves SPL from 0.097 to 0.126, providing preliminary evidence that this component can benefit other navigation systems.

Critical weaknesses

W1. There is a gap between the central research question and the implemented method, and the methodological novelty is not sufficiently established. The paper centers on the question of what a navigator should remember, but the method primarily combines a fixed observation window, hierarchical summarization, and task and spatial memory. Although Section 3.2 provides an abstract criterion based on logical entailment, the paper does not translate it into an operational rule for selecting which facts to retain or directly evaluate whether summarization preserves the evidence needed for subsequent decisions. Consequently, the results primarily demonstrate the effectiveness of the integrated system rather than fully answer the central research question. Comparing different information-retention strategies under the same context budget would help substantiate the claimed novelty.

W2. The experiments do not establish the deployment advantage that motivates the work. A key motivation is navigation with smaller models that fit a robot's computational budget. Although the method consumes fewer tokens per decision, Table 3 reports 880.3k total tokens per episode, approximately 3.2 times the 273.3k reported for HSGM. Thus, a more compact input per decision does not necessarily imply lower resource consumption over an entire task. Moreover, the system uses both Qwen3-VL-8B and JoyAI, and the real-world experiments rely on remote inference on an A100 server. The results therefore support navigation effectiveness in real-world environments, but do not establish that the system fits an onboard computational budget.

Questions for authors
  1. Do you have controlled comparisons that vary only how historical information is retained and organized, while keeping the context budget and all other modules fixed? Such comparisons would help determine whether the gains arise from the proposed memory design rather than simply providing more historical information.

  2. Which specific memory design choices follow from the logical perspective in Section 3.2 and distinguish the method from generic hierarchical summarization? What concrete conclusions does the paper offer about what a navigator should remember, beyond the division into task, temporal, and spatial memory?

  3. The paper emphasizes navigation within a robot's computational budget, but the system uses both Qwen3-VL-8B and JoyAI, and the real-world experiments rely on remote A100 inference. Does the stated computational budget refer to the reasoning model alone or to the full system? Do you have measurements of end-to-end latency, peak GPU memory usage, and computational cost over a complete episode, accounting for both models, that demonstrate how fewer tokens per decision translate into practical deployment benefits?

3 comments
AuthorsOpenReview ↗

Thanks for your Great Point!!!

Thank you for pressing us to make “What should a navigator remember?” a concrete design question. The revision compares three ways to retain the same stream of navigation observations and explains how the retained information serves later decisions.

1. What changes when the system retains history differently?

We compare three temporal inputs to the same JoyAI observer. Sliding supplies the eight most recent archived observation descriptions. Full supplies raw descriptions in chronological order, truncated at the input-token limit. ST-VLN keeps recent observations in detail and organizes older observations into pose-associated hierarchical summaries. Task memory, spatial memory, and PREVIEW are held fixed. Input length follows each retention rule: Full and Sliding consume 1,093k and 114k JoyAI input tokens per episode, respectively.

Sliding, Full, and hierarchical summarization achieve 0.190/0.112, 0.167/0.102, and 0.263/0.196 in SR/SPL, respectively. Extending the raw history substantially increases observer input, yet both SR and SPL decline. The strongest results come from retaining recent detail while consolidating older observations into descriptions of landmarks and their spatial relationships.

2. What should a navigator remember, and how does the logical perspective inform the design?

Our answer is to retain recent observations in detail and preserve three things about earlier experience: which landmarks the agent encountered, where it saw them, and how far it had progressed through the instruction. Consider a robot approaching a doorway along a corridor. Several consecutive observations may describe the same door. After the robot turns, the door’s location relative to the corner and whether the agent has completed the “pass the doorway” subgoal can still affect its next waypoint choice. The repeated descriptions of the approach can be condensed.

The logical perspective gives the agent a concrete question to address: Does the available evidence support the next decision? Task memory tracks completed and pending subgoals, temporal summaries retain landmarks associated with poses, and spatial memory maintains the map and committed waypoint. When the current view and retained evidence leave the next direction unclear—for example, when a referenced landmark lies outside the field of view—the agent can request additional observations through PREVIEW.

3. How do input size, episode cost, and real-world deployment relate?

The 8B designation refers to the Qwen3-VL-8B reasoner. The complete system also uses a JoyAI observer. In the physical trials, the Go2 streams onboard observations to a remote A100 server and executes the returned actions.

Table 3 reports both normalized input size and total episode token use. ST-VLN averages approximately 6.6k input tokens per navigation step; HSGM averages approximately 23.0k per model call. Across a complete episode, ST-VLN uses 880.3k tokens across both models, compared with approximately 273.3k for HSGM. In the Qwen3-VL-8B comparison, ST-VLN also travels farther on average (16.98 vs. 6.27 m) and achieves higher SR (0.263 vs. 0.049). In this comparison, ST-VLN uses about 3.2 times as many tokens per episode and reaches the goal over five times as often.

Dear Reviewer LpY4,

As the discussion period closes later today, we will be online waiting for your feedback on our rebuttal, in which we respond to each of the concerns you raised. We have also added anonymized code and rollout videos to the supplementary material. We would highly appreciate it if you could take into account our response when updating the rating and having discussions with AC and other reviewers.

Thank you so much for your time and efforts.

Thank you for the clarifications. The comparisons still use different context budgets, the logical perspective does not provide an operational rule for selecting retained information, and full-system efficiency measurements remain unclear. I therefore maintain my current assessment.

Reviewer_zGVwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents a memory-based method for improving the performance of zero-shot vision-language navigation. Specifically, the method appears to remember evidence that may be relevant to future queries. Table 1 shows some improvement in a limited validation experiment.

Critical strengths

The idea of saving information influencing future decisions seems interesting, while not being entirely novel.

Critical weaknesses

This paper was difficult to read and understand, and the main contribution is obscured by poor presentation. I have the following comments that could help.

  1. The figure captions must actually describe the figure. This will make the figures stand-alone entities. This applies to both Figures 1 and 2.

  2. The notation is not consistent, complete, or well-defined. This is the primary reason making the paper difficult to read. For example, the functions in (4) are stated without definition. The same applies to the subgoals in (5). Equations (8), (9), (11), (13), and (15) also have issues.

  3. The validation is limited to one dataset. The authors are encouraged to conduct a much broader evaluation on popular datasets such as RxR, VLN-CE, etc.

  4. Further, the ablation study’s results are presented as supplementary material. This is unfair because the study is an important part of the work, and the corresponding results must be presented within the stipulated page limit.

  5. Most importantly, the conclusion that preserving facts related to future decisions is not adequately justified. The results in Table 1 do not inspire confidence in the proposed approach.

Questions for authors

Can you please help readers understand how the memory buffer saves information that could be helpful in future decisions (using the proposed method)?

2 comments
AuthorsOpenReview ↗

Thanks for your Great Point!!!

Thank you for asking what remains available to the navigator after an observation leaves its recent window. We revised the explanation of the memory buffer, moved the component ablation into the main paper, and clarified the figures and notation.

1. How does the buffer preserve information for later decisions?

Consider a doorway the robot passes before turning a corner. The eight most recent archived observations retain local detail. As the robot moves on, the original doorway description leaves that window. ST-VLN summarizes older observations and associates them with poses, so the summary can still record what the doorway looked like, where the robot saw it, and how it related to nearby landmarks. If a later subgoal refers to that doorway, the reasoner can use this evidence after the doorway has left the camera view.

The main paper now compares three retention strategies on R2R-CE val-unseen. Keeping the eight most recent raw descriptions yields 0.190 SR / 0.112 SPL. Keeping the chronological raw history up to the model’s input limit yields 0.167 / 0.102. Retaining recent detail and summarizing older observations yields 0.263 / 0.196. This gives a concrete answer to the paper’s central question: among the strategies tested, navigation works best when the agent keeps recent observations in detail and preserves earlier landmarks together with where they were observed.

2. How have the captions and notation changed?

The Figure 1 caption now walks the reader through a navigation decision. Task memory tracks instruction progress, temporal memory retains pose-associated observations and summaries, and spatial memory maintains the map and committed waypoint. The reasoner uses this information to select a waypoint or request additional views through PREVIEW. The controller executes the waypoint, and subsequent observations update memory. The Figure 2 caption identifies the R2R-CE simulation and Go2 rows, explains the camera, map, and trajectory panels in reading order, and identifies the surrounding views acquired through PREVIEW.

In the method section, we define the ordered subgoals and the current subgoal index, specify when that index advances, and describe how each memory component is updated. An explicit sequence-concatenation operator shows where each new observation or pose enters the history. We also distinguish waypoint selection from primitive action execution: once the reasoner commits to a waypoint, the controller can issue several successive actions toward it.

3. Which evaluation settings are covered? Thank you for suggesting an evaluation on RxR-CE. Given HSGM’s low success rate with Qwen3-VL-8B on R2R-CE (SR = 0.049), we selected SmartWay and Open-Nav as baselines for the additional RxR-CE evaluation. We evaluated these methods using Qwen-based 8B backbones. Our method achieved the highest success rate (0.203) and SPL (0.167), outperforming both SmartWay (0.087 SR, 0.064 SPL) and Open-Nav (0.027 SR, 0.021 SPL).

MethodNavigation backboneObs.SR ↑SPL ↑nDTW ↑OSR ↑NE ↓PLStepsRSR ↑
SmartWayQ3VL-8BPano0.0870.0640.2810.21314.725.334.700.408
Open-NavQwen3-8B-FP8Pano0.0270.0210.1890.04117.512.618.920.659
Ours (Intent Actions)Q3VL-8B + JoyAIMono0.2030.1670.2930.2869.4630.26226.970.709

Dear Reviewer zGVw,

As the discussion period closes later today, we will be online waiting for your feedback on our rebuttal, in which we respond to each of the concerns you raised. We have also added anonymized code and rollout videos to the supplementary material. We would highly appreciate it if you could take into account our response when updating the rating and having discussions with AC and other reviewers.

Thank you so much for your time and efforts.

Reviewer_nu4kOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a zero-shot navigation framework combining task memory over ordered subgoals, hierarchical temporal summaries associated with poses, and a geometric spatial memory containing an occupancy map, trajectory, and persistent waypoint target. A JoyAI observer summarizes observations, while a Qwen3-VL-8B reasoner selects navigation intents that a geometric controller executes. The framework is motivated through a semantic-communication perspective and evaluated using R2R-CE comparisons, component ablations, instruction perturbations, and real-robot experiments. The full system achieves 0.263 SR and 0.196 SPL on R2R-CE val-unseen.

Critical strengths
  1. The internal experiments support a useful navigation system. Removing temporal memory reduces SR from 0.263 to 0.190 and SPL from 0.196 to 0.112, with longer paths and more steps. Spatial memory and PREVIEW also help. These are informative component effects, particularly because Section 5.4 states that the removal variants retain minimal base memory, such as the eight most recent images.
  2. The paper probes behavior beyond endpoint success. FLIP, GOAL-ONLY, and PARAPHRASE are useful ways to examine instruction sensitivity. The action-interface ablation is also substantial: waypoint intents obtain 0.263 SR versus 0.125 for primitive actions. These experiments expose practical design choices that a headline SR comparison alone would miss.
Critical weaknesses
  1. The proposed explanations go beyond what the comparisons isolate. The temporal ablation is not a memoryless control: it retains recent observations, which is an important strength of the experiment. It still does not separate the effects of additional history, hierarchical compression, and pose association at comparable information or token budgets. The main system comparisons also vary the observer and action interface. I would interpret the results as supporting this integrated memory-equipped agent, with less certainty about whether the proposed organization is the decisive factor or whether memory explains the gap independently of perception and control.

The perturbations also do not uniquely diagnose route memorization. FLIP shows reduced compliance with an edited turn, but learned route priors, grounding mistakes, or inconsistency with the remaining instruction could produce that behavior. Table 6 reports GOAL-ONLY SR without the path-fidelity comparison used to motivate the test. AwareVLN's decline from 0.624 to 0.353 is compatible with relying on the route description that was removed. The results justify claims about perturbation robustness more directly than a causal claim that the baseline follows memorized routes.

  1. The deployment argument needs clearer cost and metric definitions. Table 3 reports approximately 880.3k tokens per episode for the proposed two-model system versus 273.3k for HSGM, despite shorter inputs per decision. Tokens across models are not a direct latency measure, but these totals do not establish lower end-to-end cost; the Go2 experiments use remote A100 inference. Separately, Table 4 labels SR as success within 3 m yet reports office NE/SR pairs of 0.91 m/0.6 and 1.07 m/0.4. If NE averages final distances over the same trials and success is determined solely by that threshold, mean NE is at least 3(1 - SR), giving 1.2 m and 1.8 m. A STOP requirement, other success conditions, or different averaging populations could explain the numbers. The paper needs to specify those conventions before the real-world comparison can be interpreted.
Questions for authors
  1. What exactly is retained in the minimal base-memory variants described in Section 5.4, including observer summaries and pose information? Do the current results distinguish hierarchy or pose association from access to a longer history? A matched-budget simpler history comparison would help if available.
  2. Were the edited FLIP routes checked for traversability and consistency with the remaining clauses? Can you provide the GOAL-ONLY path-fidelity results, or narrow the interpretation to the instruction-robustness effects directly measured?
  3. Please reconcile Table 4 by specifying the number of trials, STOP and other success requirements, NE averaging population, and treatment of timeouts. In Table 3, please distinguish decisions from model calls and clarify whether comparable complete-system latency is available.
2 comments
AuthorsOpenReview ↗

Thanks for your Great Point!!!

Thank you for your questions about the temporal-memory controls, instruction perturbations, and real-world evaluation. We added a raw-history comparison and path-fidelity results, and clarified how we measure token use and real-world navigation success.

1. What does the base memory retain, and what does summarization contribute?

The original w/o Temporal condition is now labeled Temporal/Sliding. Its results are unchanged. Sliding gives the JoyAI observer the eight most recent archived observation descriptions. The new Temporal/Full condition gives the same observer the raw descriptions in chronological order, truncated at the model’s input-token limit. Both retain history across subgoal transitions and clear it at the start of each episode. Neither generates historical summaries or filters observations by relevance. The window is indexed by archived observations, which need not correspond one-to-one to primitive action steps.

We keep the observer, task memory, spatial memory, and PREVIEW fixed. Each strategy uses the input length produced by its retention rule. ST-VLN’s pose-associated hierarchical summaries achieve 0.263 SR / 0.196 SPL, compared with 0.190 / 0.112 for Sliding and 0.167 / 0.102 for Full. Consider a landmark the agent passed earlier. Its description leaves Sliding after eight newer entries. Full retains it longer, alongside repeated descriptions from the same part of the route. A summary can retain the landmark and its spatial relationships while consolidating those repeated observations. Among the three strategies, summarization yields both the highest success rate and the most efficient successful paths.

In the separate w/o Spatial condition, the agent retains its pose trajectory and committed waypoint. It generates waypoint candidates from the current observation and moves toward the selected target without an accumulated occupancy map or map-based path planning.

2. What does FLIP validate, and how does GOAL-ONLY affect path fidelity?

FLIP reverses the first turn word while keeping the starting pose fixed. We check local navigability 2 m along the reversed direction and score whether the agent initially travels in the direction specified by the edit. The feasibility check covers this initial maneuver; subsequent clauses are outside its scope. After the edit, the initial-turn follow rate is 0.707 for ST-VLN, 0.439 for AwareVLN, and 0.268 for CA-Nav. This experiment exam the instruction following ability, because of the importance of "Follow the command of human" in society, so we design this experiment, and try to tell the generalization ability as well.

For GOAL-ONLY, we expected that removing route instructions might encourage exploration and lower nDTW. We now report nDTW against the original reference trajectory on the same 174 episodes. With only the final instruction segment, ST-VLN’s success rate fell from 0.293 to 0.181, and mean path length decreased from 19.37 to 11.25 m. Mean nDTW, however, rose slightly from 0.354 to 0.390. Thus, the agent reached the goal less often, even though its shorter trajectories remained, on average, slightly more closely aligned with the original routes.

3. How are system cost and real-world success measured?

Table 3 now specifies how token use is averaged. ST-VLN averages 131 navigation steps per episode and 6.6k input tokens per step. HSGM averages 11.7 model calls per episode and 23.0k input tokens per call. At the episode level, ST-VLN consumes 880.3k tokens across its observer and reasoner, compared with 273.3k for HSGM. This higher token cost accompanies longer average trajectories (16.98 m for ST-VLN versus 6.27 m for HSGM) and a 21.4-percentage-point gain in success rate (26.3% versus 4.9%). In this comparison, ST-VLN uses about 3.2 times as many tokens per episode and reaches the goal over five times as often.

In the Go2 evaluation, each distinct episode is run three times. Success requires the robot to issue STOP within 2 m of the goal center. Real-world NE is the shortest distance from the robot’s final position to the goal region, averaged over all trials. For example, a robot that enters the region, issuing STOP receives a success indicator of 1 and an NE of 0 m. This explains how a low mean NE can coexist with failed trials in Table 4.

Dear Reviewer nu4k,

As the discussion period closes later today, we will be online waiting for your feedback on our rebuttal, in which we respond to each of the concerns you raised. We have also added anonymized code and rollout videos to the supplementary material. We would highly appreciate it if you could take into account our response when updating the rating and having discussions with AC and other reviewers.

Thank you so much for your time and efforts.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Welcom to discuss your concerns!!!

Dear all reviewers,

We sincerely thank you for your helpful reviews. We have submitted our responses to address all your remaining questions. We are looking forward to your further feedback. Thank you again!

Best,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.