- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes SMLA-Nav, an MLLM-based navigation framework integrating implicit spatial memory, localization, and action prediction. It combines point-goal navigation pretraining with scene-specific post-training using image-pose observations and sparse demonstrations. The method is evaluated on two simulation benchmarks and a real quadruped robot.
Critical strengths
- The framework provides a unified approach to spatial localization, multimodal goal grounding, and navigation.
- Extensive simulation experiments demonstrate competitive performance across text, image, and coordinate goals.
- Real-world quadruped deployment provides evidence of practical feasibility.
Critical weaknesses
W1. Limited methodological novelty and unclear advantages over conventional navigation pipelines. My primary concern is that the proposed approach essentially replaces explicit spatial maps and localization with scene-specific parametric memory learned through MLLM fine-tuning. However, it still requires extensive posed observations and navigation demonstrations before deployment. Given that the environment has already been observed, it is unclear why conventional mapping, localization, and planning pipelines would not provide a simpler and more practical solution. Moreover, changes to the environment may require additional model adaptation, potentially introducing higher maintenance costs than incrementally updating an explicit map. The paper does not sufficiently demonstrate the advantages of this design in terms of navigation capability, computational efficiency, scalability, or adaptability. The core components largely build upon established techniques, making the methodological novelty relatively limited. W2. Restricted goal grounding and generalization. The framework requires scene-specific post-training and does not demonstrate zero-shot navigation in unseen environments. More importantly, text-goal navigation relies on predefined landmark names associated with locations during training. It remains unclear whether the system can navigate to previously unseen semantic goals or destinations without established name-to-location associations. This limits the demonstrated vision-language understanding and generalization capabilities. W3. Unclear autonomous obstacle-avoidance capability in real-world deployment. The real-world experiments rely on an additional onboard obstacle-avoidance module, but its contribution is not quantified. Therefore, the reported success rates cannot establish how effectively SMLA-Nav itself handles collision avoidance. Furthermore, evaluation is limited to one adapted environment, a fixed starting pose, and 20 trials per goal modality, providing limited evidence of robust real-world navigation.
Questions for authors
- What concrete advantages does parametric spatial memory offer over conventional SLAM-based navigation, particularly regarding scene updates and deployment costs?
- Can the model navigate to previously unseen textual destinations without explicit goal-location supervision?
- How much does the onboard obstacle-avoidance module contribute to real-world navigation success?
- Can the authors isolate the benefits of consistent-prompt training from point-goal pretraining and additional pose supervision?
7 comments
Response to Reviewer UFgA - Part II
2. Unseen textual destinations and semantic generalization (W2, Q2)
We thank the reviewer for this important question. We clarify that SMLA-Nav addresses navigation to specific destinations in previously observed environments, where scene-specific goal-to-location associations are acquired during adaptation. This is a practical setting for long-term robot deployment and differs from zero-shot navigation in entirely unseen environments. Nevertheless, we further investigate generalization beyond predefined destination names through compositional semantic goals combining objects, visual attributes, and spatial relations.
Examples of Compositional Semantic Goals.
| Semantic goal description |
|---|
| A bed and large water bottles stand at the end of an aisle between storage racks. |
| Shelves of yellow bins and cardboard boxes face a white enclosure with open doors. |
We consider both short descriptions and full four-view descriptions. For the latter, Front, Right, Back, and Left refer to directions relative to the target panorama rather than the robot's current heading.
Semantic Post-Training and Evaluation Settings.
We conduct a joint training experiment across seven scenes. Specifically, Scene A uses its original image–pose observations and navigation trajectories, augmented with semantic description–location pairs generated from its N posed observations. The other six scenes retain their original training data, consisting of image–pose observations and navigation trajectories, without additional semantic supervision. We mix the training data from all seven scenes to train a single model. All additional semantic supervision comes exclusively from Scene A.
We evaluate the resulting model under three settings:
-
Unseen paraphrases in Scene A: We replace descriptions of previously learned locations with unseen paraphrases, testing generalization to new language expressions.
-
Held-out goal views in Scene A: We generate semantic descriptions from goal views excluded from the N training observations, testing whether the model can ground destinations from previously unseen viewpoints.
-
Cross-scene semantic transfer: We evaluate semantic descriptions of held-out goal views in the other six scenes. Although their original spatial knowledge is included in joint training, they receive no semantic description–location supervision. This tests whether semantic grounding learned in Scene A transfers to spatial memories acquired in other scenes.
Localization Accuracy and Navigation SR for Unseen Semantic Goals (%).
| Evaluation setting | Original: Localization | Original: Navigation | Semantic post-training: Localization | Semantic post-training: Navigation |
|---|---|---|---|---|
| Scene A: Unseen paraphrases | 18.60 | 15.10 | 76.80 | 66.30 |
| Scene A: Held-out goal views | 8.20 | 7.00 | 46.50 | 38.40 |
| Six other scenes: Held-out goal views | 12.50 | 10.00 | 30.80 | 27.50 |
Semantic post-training substantially improves localization and navigation across all three settings. Notably, in the six other scenes, where no additional semantic supervision is provided, localization accuracy improves from 12.5% to 30.8% and navigation SR from 10.0% to 27.5%. These results provide initial evidence that semantic grounding learned in Scene A can transfer to unseen semantic descriptions and goal viewpoints in other adapted scenes, without additional semantic supervision in those scenes.
Response to Reviewer UFgA - Part III
3. Practical advantages and trade-offs over SLAM-based navigation (W1, Q1) - Part I
We thank the reviewer for asking us to substantiate the deployment benefits and scene-update procedure relative to conventional navigation systems.
We examine the value of incorporating scene-specific memory into a navigation MLLM through controlled executor and localizer comparisons, multi-scene experiments, and deployment measurements. We also examine compatibility with SLAM-based data collection and performance after scene changes.
3.1. Strong visual navigation and tolerance to localization errors
Our primary motivation for adopting an MLLM is its visual navigation capability. Manuscript Appendix Table 10 provides a controlled comparison: with the same HLoc localizer and A* planner, X-NavDP achieves 49.2% / 53.0% navigation SR on HM3D / Gauss, whereas our MLLM navigation executor, using only Stage 1 pretraining, already achieves 67.5% / 85.4%. Among the evaluated models, these results support the stronger transferable visual navigation capability of the MLLM executor.
Manuscript Table 2 further shows that, with the localization and planning modules fixed, replacing direct polyline tracking with our post-trained visual navigation executor improves navigation SR. Figure 4 illustrates one possible reason: when localization errors occur, following a path planned from an incorrect pose may lead to a collision, whereas our model can adjust its motion based on the current visual observations and subsequently relocalize. Navigation performance therefore depends on both localization accuracy and the execution policy's response to actual observations and intermediate errors.
Manuscript Table 4 also shows that scene adaptation improves navigation decisions in addition to localization. Even with ground-truth poses supplied to exclude localization errors, incorporating a small number of scene-specific navigation trajectories increases SR from 77.3% / 77.3% to 92.0% / 90.1%. This indicates that the model learns route selection and scene-specific navigation behavior from the trajectories, beyond image-to-coordinate correspondences.
3.2. Shared localization and navigation reduce additional module dependencies
Because the MLLM already performs navigation execution, we enable the same model to perform scene localization and goal grounding using its internal parametric memory, without querying an external localization database at inference.
Dedicated localizers remain strong alternatives. The added 2B localizer–executor comparison (Global Response §5; revised manuscript Table 23) fixes the SMLA-Nav executor and changes only the localization source. HLoc yields 89.7% / 85.3% navigation SR, while internal SMLA-Nav localization yields 84.8% / 84.0%, a difference of 4.9 / 1.3 percentage points. These results show that the existing navigation model can also provide competitive localization. Model sharing removes the need to deploy a separate localization model and its scene-specific reference database, and allows localization and action prediction to reuse intermediate computation. The resource measurements below evaluate parameter and computation sharing with the 2B implementation.
Response to Reviewer UFgA - Part I
We thank the reviewer for recognizing the unified framework, simulation results, and real-world deployment. Please see the Global Response for the shared clarifications and additional experiments. We address the specific concerns below; paired results are in HM3D / Gauss order unless otherwise stated.
1. Methodological contributions beyond existing navigation methods (W1)
We thank the reviewer for this constructive comment. We address a challenging and practically significant problem, and SMLA-Nav achieves good performance. Moreover, SMLA-Nav goes beyond simply connecting an off-the-shelf localizer to a navigation policy. We highlight three key methodological contributions below and will revise the manuscript to more clearly articulate our innovations.
1.1. Learning formulation: decoupled navigation-skill acquisition and scene-memory learning under sparse demonstrations.
We study how to adapt a navigation model to a deployment scene using sparse scene-specific trajectories. Motivated by NavGSim's behavior under sparse scene adaptation, Table 3 investigates its trajectory-based adaptation formulation through controlled experiments with a shared Qwen3-VL backbone. The evaluated VLN- and ObjectGoal-style pretraining does not effectively improve navigation performance under sparse scene demonstrations, with high collision rates and frequent incorrect stops at other registered destinations. We attribute these issues to two main factors: (1) difficulty in effectively transferring pretrained navigation capabilities to scene adaptation, and (2) insufficient goal-to-location grounding when relying solely on image-action supervision. We propose a problem-solving approach that decouples the acquisition of general navigation skills from scene-specific spatial memory, while enabling their effective integration through a unified spatial interface. PointGoal navigation is adopted for pretraining because its agent-centric relative-position guidance is independent of environments, providing strong generalization and transferability. During scene-specific post-training, sparse navigation trajectories help preserve and adapt the pretrained skills while supplying scene-specific layout knowledge; explicit pose supervision strengthens scene-specific spatial grounding.
1.2. Model design: a hierarchical spatial reasoning chain from parametric memory to navigation actions within a shared MLLM.
We implement this formulation through consistent-prompt multi-stage training that bridges navigation pretraining and scene-specific post-training through a common spatial structure, composing localization, tool-based relative-goal computation, and action prediction into a hierarchical spatial reasoning chain. This design enables a single MLLM to perform both localization and navigation, eliminating the need for separate models. The KV cache generated during localization can also be reused for navigation, reducing redundant computation and inference latency. Furthermore, parametric scene memory keeps localization latency nearly constant as scene scale increases within the evaluated range, while allowing a fixed-size model to accommodate spatial knowledge from multiple scenes.
Existing methods adopt different designs. Neural Topological SLAM, ULVN, and OpenNavMap rely on explicit scene representations and separate navigation execution policies. AstraNav-Memory predicts frontier points or target coordinates, but relies on external systems for navigation execution. NOLO directly predicts actions conditioned on scene-specific context videos, but has limited ability to generate optimal navigation trajectories. SMLA-Nav unifies these capabilities within a single MLLM, eliminating the need to query external scene maps, localization databases, or online SLAM at inference, thereby reducing inter-module dependencies and potential error propagation that could affect route execution.
1.3. Spatial capability: absolute metric pose generation from internalized scene knowledge through language decoding.
To the best of our knowledge, we are the first to demonstrate that a navigation MLLM can internalize scene-specific visual-spatial correspondences and directly predict absolute metric poses (x, y, yaw) through native autoregressive language-token decoding, without a dedicated pose-regression head or external scene maps and reference-image retrieval at inference.
Existing methods such as ACE also encode scene information in network weights, but rely on a dedicated network to predict image-to-scene correspondences, followed by RANSAC and PnP for pose estimation. VLM-Loc relies on external BEV maps and scene graphs for localization. In contrast, SMLA-Nav integrates localization and navigation within a shared MLLM, making localization an intrinsic capability of the navigation policy rather than a prerequisite provided by an external module.