acceptodds
Withdrawn from ICLR 2027

VLN-MEM: Scene-Memory-Guided Vision-and-Language Navigation for Reliable Arrival

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-and-language navigation (VLN) asks an agent to follow instructions to a goal from egocentric observations. Recent VLN systems have improved grounding, yet they still fail in two distinct modes: they fail to reach the goal, and when they do reach it they fail to stop there. Both failures share one cause: the goal is specified as a destination, a pose and a view, which says where the goal is but not how to reach it nor whether it has been reached. In this paper, we propose VLN-MEM, a scene-memory-guided VLN for reliable arrival, which supplies both conditions from a persistent scene memory: a frozen per-scene graph of places, built offline from reference trajectories, from which retrieval returns next-hop geometry for the reachability and a stored goal view for the recognition. VLN-MEM adopts zero-initialized residual fusion to feed the reachability into the planner's visual stream, and training-free stop verification to confirm arrival against the goal's appearance and proximity. VLN-MEM reaches and SR on R2R-CE and RxR-CE val-unseen in a pre-mapped, goal-specified setting, gaining SR and SPL over a training-budget-matched backbone; against stronger full-data references it matches or exceeds their SR while navigating goal-directly. A component ablation confirms that each mechanism contributes.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-and-Language Navigation, Scene Memory, Reachability, Goal Recognition, Retrieval-Augmented Navigation, Embodied AI

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_uCF6OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes VLN-MEM, a scene-memory-guided vision-language navigation framework for reliable goal arrival. The method constructs an offline scene graph from reference trajectories, retrieves next-hop geometry through graph search, and integrates spatial information into an existing VLN policy using residual fusion. A training-free stop verification module further improves termination decisions. Experiments on R2R-CE and RxR-CE demonstrate improved navigation success rates.

Critical strengths
  1. The paper identifies two practical navigation failure modes: failing to reach the goal and failing to stop after arrival.
  2. The framework demonstrates improvements over its training-budget-matched baseline, with component-wise ablations supporting the reported gains.
Critical weaknesses

W1. Questionable problem formulation and insufficient justification for using VLN. My primary concern is the practical and scientific motivation of the proposed problem setting. Unlike standard VLN, the method assumes that the environment has been mapped offline and that the goal pose and reference image are explicitly provided. The scene memory is constructed using reference trajectories, including those from evaluation scenes. Under these assumptions, conventional mapping, localization, and graph-based planning methods could already provide a straightforward navigation solution. In fact, the proposed method itself uses A* to retrieve next-hop geometry. The paper does not demonstrate why integrating a VLM-based navigation policy offers meaningful advantages over conventional navigation pipelines. Moreover, the reliance on preconstructed maps and explicit goal information substantially reduces the need for language grounding and limits generalization to previously unseen environments.

W2. Limited methodological novelty. The proposed framework primarily combines established components, including scene graph construction, CLIP-based retrieval, A* planning, residual feature fusion, and DINOv2-based visual matching. The stop verification mechanism is essentially a rule-based combination of visual similarity, spatial proximity, and temporal consistency. These components are reasonable engineering choices but do not constitute substantial methodological innovations.

W3. Questionable effectiveness of stop verification under stricter arrival criteria. A considerable portion of the reported improvement comes from stop verification rather than improved navigation capabilities. On R2R-CE, stop verification contributes approximately 9 percentage points to the total 14.5-point success-rate improvement. More importantly, Figure 4(b) shows that stop verification significantly reduces arrival precision under stricter distance thresholds. At a 1m threshold, the success proportion decreases from 39.9% to 14.6%. This suggests that the method primarily benefits from the benchmark’s relatively permissive 3m success threshold rather than achieving more accurate goal arrival.

W4. Limited consideration of obstacle avoidance and lack of real-world validation. The proposed framework introduces no dedicated obstacle-avoidance mechanism or systematic evaluation of collision handling. Furthermore, all experiments are conducted in simulation, without real-robot deployment. Given its dependence on accurate localization and preconstructed scene memory, its robustness and practical applicability remain unclear.

Questions for authors
  1. What advantages does VLN-MEM offer over conventional SLAM-based localization and graph-based navigation under the same mapping assumptions?
  2. How does the method perform under stricter arrival thresholds and with imperfect localization or incomplete maps?
  3. How is collision avoidance handled, and can the authors provide quantitative evaluations of obstacle-related failures?
Reviewer_NXsoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VLN-MEM augments a DualVLN-based navigation policy with a frozen scene memory constructed offline from reference trajectories. Given an explicitly supplied goal pose and image, the system retrieves a goal node and uses graph search to obtain next-hop geometry. Retrieved geometry and goal appearance are integrated into the planner through zero-initialized residual branches. A training-free stopping mechanism supplements the policy’s STOP signal using appearance and proximity checks. The method reports 65.0% SR on R2R-CE and 59.8% on RxR-CE in a pre-mapped, goal-specified setting, together with an analysis of reaching versus stopping failures.

Critical strengths
  1. The stopping analysis explains a useful and nontrivial trade-off. The paired comparison converts 166 failures to successes and 14 successes to failures at the 3 m threshold. At the same time, success within 1 m falls from 39.9% to 14.6%. Reporting this curve makes clear that the mechanism changes the stopping operating point rather than uniformly improving localization.
  2. The system addresses an explicit pre-mapped, goal-specified use case. Retrieval, next-hop geometry, residual fusion, and stopping verification form a concrete pipeline evaluated on R2R-CE and RxR-CE. The paper discloses the extra goal inputs and reference-based memory, labels the language-only comparisons accordingly, and reports cumulative component ablations. I regard this setting as legitimate and assess the method on those terms.
Critical weaknesses
  1. The strongest missing control is inside the proposed setting. The language-only comparisons are explicitly contextual, so unequal inputs there are not by themselves an experimental violation. However, Table 3's training-budget-matched Base has no memory bank or retrieval; adding retrieval simultaneously introduces goal binding, graph guidance, and the proposed fusion. The cumulative ablations show useful system changes, but do not establish whether the learned fusion is preferable to straightforward use of the same goal pose/image and scene graph. A simple graph-guided alternative with the same controller and a goal-dependent stopping rule would be particularly informative. The point is not to withhold inputs that define the task, but to understand which design is needed once those inputs are available.
  2. Reference-route coverage limits the interpretation of persistent scene memory. Appendix A.1 builds frozen scene memories from the evaluation split's reference episodes, including 1,223 deduplicated RxR trajectories. This is disclosed and consistent with the chosen pre-mapped setup; I am not treating it as hidden leakage. Still, the evaluation does not show whether the same improvement holds for a new route or goal not used in preparing that memory. Since A* uses the resulting connections, route overlap and scene coverage matter to the practical conclusion. The paper acknowledges this limitation, but does not quantify it.
Questions for authors
  1. Under the same supplied goal information and scene memory, how much performance requires the residual-fusion design rather than simpler graph guidance? Is there an existing control or analysis that addresses this comparison?
  2. Does each evaluated episode's reference trajectory, or a duplicate of it, contribute to memory construction? What goal and connectivity coverage would remain if those contributions
Reviewer_wzoNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VLN-MEM, a scene-memory-guided VLN framework designed to improve goal arrival and termination.Beyond in-episode observations, VLN-Mem explicitly incorporates scene priors, including a scene graph and next-hop geometry. Each node of the scene graph stores a 6-DoF pose, an RGB view, a CLIP embedding, and a dense point map. The next-hop geometry is derived from the shortest path between the agent’s current pose and the goal node within that graph.Termination is triggered when the visual similarity between the current observation frame and the goal frame exceeds a predefined threshold.

Critical strengths

The paper empirically validates that integrating scene priors improves navigation success rate, and provides an effective method for integrating scene priors.

Critical weaknesses
  1. Unfair comparison. The evaluation setting substantially differs from standard VLN. For VLN tasks, at least for the R2R and RXR benchmarks, the premise is that the agent has not seen the test scenes before, thereby assessing their capacity to interpret natural language instructions and execute reliable navigation in unfamiliar scenes.Since VLN-Mem relies on scene priors—information inherently unavailable during standard R2R/RXR evaluation—it would be unfair to assess it on these benchmarks.Authors should evaluate on lifelong or continual learning setups, where agents may accumulate and leverage scene knowledge in previous episodes, enabling principled assessment of how prior information is stored, retrieved, and applied for improved navigation performance.

  2. Confusing motivation.Lines 14-15 state that failures are caused by navigation goals not specifying how to reach them. But both R2R and RXR inherently provide step-by-step instructions telling the agent how to proceed.

  3. The improvement on RxR is quite marginal.

  4. The overall navigation performance of VLN-Mem is critically dependent on the retrieval accuracy of goal view and next-hop. Authors should report the accuracy / precision of these two retrievals.

Questions for authors
  1. The scene graph is constructed based on reference trajectories. How exactly are these reference trajectories obtained?

  2. Are any observations from the evaluated episodes themselves used to construct the scene memory? If not, how are the memory-construction and evaluation trajectories separated, and how much do they overlap in spatial coverage?

  3. As illustrated in Figure 2a, VLN-MEM assumes access to both the goal pose and the agent’s current pose. Why can't we directly determine whether the destination has been reached based on these poses?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.