acceptodds
Under review as a conference paper at ICLR 2027

SMLA-Nav: End-to-End Learning of Spatial Memory, Localization, and Action for Embodied Navigation

Abstract

Service robots in long-term operational environments are often expected to navigate from arbitrary starting positions to destinations specified by text, images, or coordinates. However, jointly learning scene-specific spatial memory and navigation behavior from sparse goal-directed demonstrations remains challenging. We propose SMLA-Nav, an end-to-end spatial memory navigation framework that complements sparse expert trajectories with readily obtainable image–pose observations to acquire scene knowledge and adapt pretrained navigation skills. A consistent-prompt multi-stage training strategy bridges navigation pretraining and scene-specific post-training through a common spatial representation. This connects scene-specific localization and goal grounding with transferable point-goal navigation, composing these capabilities into a hierarchical spatial reasoning chain within a shared multimodal large language model (MLLM). At inference, the policy uses implicit spatial memory encoded in its parameters and deterministic geometric computation to navigate without querying external maps or localization databases. We establish SMLA-Bench, comprising SMLA-HM3D and SMLA-Gauss, on which SMLA-Nav achieves success rates of 86.9% and 85.7%, respectively. Comparisons using the same sparse scene trajectories further show improved text-goal navigation over direct MLLM post-training, while deployment on a quadruped robot demonstrates real-world feasibility.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied Navigation, Spatial Memory, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_UFgAOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SMLA-Nav, an MLLM-based navigation framework integrating implicit spatial memory, localization, and action prediction. It combines point-goal navigation pretraining with scene-specific post-training using image-pose observations and sparse demonstrations. The method is evaluated on two simulation benchmarks and a real quadruped robot.

Critical strengths
  1. The framework provides a unified approach to spatial localization, multimodal goal grounding, and navigation.
  2. Extensive simulation experiments demonstrate competitive performance across text, image, and coordinate goals.
  3. Real-world quadruped deployment provides evidence of practical feasibility.
Critical weaknesses

W1. Limited methodological novelty and unclear advantages over conventional navigation pipelines. My primary concern is that the proposed approach essentially replaces explicit spatial maps and localization with scene-specific parametric memory learned through MLLM fine-tuning. However, it still requires extensive posed observations and navigation demonstrations before deployment. Given that the environment has already been observed, it is unclear why conventional mapping, localization, and planning pipelines would not provide a simpler and more practical solution. Moreover, changes to the environment may require additional model adaptation, potentially introducing higher maintenance costs than incrementally updating an explicit map. The paper does not sufficiently demonstrate the advantages of this design in terms of navigation capability, computational efficiency, scalability, or adaptability. The core components largely build upon established techniques, making the methodological novelty relatively limited. W2. Restricted goal grounding and generalization. The framework requires scene-specific post-training and does not demonstrate zero-shot navigation in unseen environments. More importantly, text-goal navigation relies on predefined landmark names associated with locations during training. It remains unclear whether the system can navigate to previously unseen semantic goals or destinations without established name-to-location associations. This limits the demonstrated vision-language understanding and generalization capabilities. W3. Unclear autonomous obstacle-avoidance capability in real-world deployment. The real-world experiments rely on an additional onboard obstacle-avoidance module, but its contribution is not quantified. Therefore, the reported success rates cannot establish how effectively SMLA-Nav itself handles collision avoidance. Furthermore, evaluation is limited to one adapted environment, a fixed starting pose, and 20 trials per goal modality, providing limited evidence of robust real-world navigation.

Questions for authors
  1. What concrete advantages does parametric spatial memory offer over conventional SLAM-based navigation, particularly regarding scene updates and deployment costs?
  2. Can the model navigate to previously unseen textual destinations without explicit goal-location supervision?
  3. How much does the onboard obstacle-avoidance module contribute to real-world navigation success?
  4. Can the authors isolate the benefits of consistent-prompt training from point-goal pretraining and additional pose supervision?
7 comments
AuthorsOpenReview ↗

Response to Reviewer UFgA - Part I

We thank the reviewer for recognizing the unified framework, simulation results, and real-world deployment. Please see the Global Response for the shared clarifications and additional experiments. We address the specific concerns below; paired results are in HM3D / Gauss order unless otherwise stated.

1. Methodological contributions beyond existing navigation methods (W1)

We thank the reviewer for this constructive comment. We address a challenging and practically significant problem, and SMLA-Nav achieves good performance. Moreover, SMLA-Nav goes beyond simply connecting an off-the-shelf localizer to a navigation policy. We highlight three key methodological contributions below and will revise the manuscript to more clearly articulate our innovations.

1.1. Learning formulation: decoupled navigation-skill acquisition and scene-memory learning under sparse demonstrations.

We study how to adapt a navigation model to a deployment scene using sparse scene-specific trajectories. Motivated by NavGSim's behavior under sparse scene adaptation, Table 3 investigates its trajectory-based adaptation formulation through controlled experiments with a shared Qwen3-VL backbone. The evaluated VLN- and ObjectGoal-style pretraining does not effectively improve navigation performance under sparse scene demonstrations, with high collision rates and frequent incorrect stops at other registered destinations. We attribute these issues to two main factors: (1) difficulty in effectively transferring pretrained navigation capabilities to scene adaptation, and (2) insufficient goal-to-location grounding when relying solely on image-action supervision. We propose a problem-solving approach that decouples the acquisition of general navigation skills from scene-specific spatial memory, while enabling their effective integration through a unified spatial interface. PointGoal navigation is adopted for pretraining because its agent-centric relative-position guidance is independent of environments, providing strong generalization and transferability. During scene-specific post-training, sparse navigation trajectories help preserve and adapt the pretrained skills while supplying scene-specific layout knowledge; explicit pose supervision strengthens scene-specific spatial grounding.

1.2. Model design: a hierarchical spatial reasoning chain from parametric memory to navigation actions within a shared MLLM.

We implement this formulation through consistent-prompt multi-stage training that bridges navigation pretraining and scene-specific post-training through a common spatial structure, composing localization, tool-based relative-goal computation, and action prediction into a hierarchical spatial reasoning chain. This design enables a single MLLM to perform both localization and navigation, eliminating the need for separate models. The KV cache generated during localization can also be reused for navigation, reducing redundant computation and inference latency. Furthermore, parametric scene memory keeps localization latency nearly constant as scene scale increases within the evaluated range, while allowing a fixed-size model to accommodate spatial knowledge from multiple scenes.

Existing methods adopt different designs. Neural Topological SLAM, ULVN, and OpenNavMap rely on explicit scene representations and separate navigation execution policies. AstraNav-Memory predicts frontier points or target coordinates, but relies on external systems for navigation execution. NOLO directly predicts actions conditioned on scene-specific context videos, but has limited ability to generate optimal navigation trajectories. SMLA-Nav unifies these capabilities within a single MLLM, eliminating the need to query external scene maps, localization databases, or online SLAM at inference, thereby reducing inter-module dependencies and potential error propagation that could affect route execution.

1.3. Spatial capability: absolute metric pose generation from internalized scene knowledge through language decoding.

To the best of our knowledge, we are the first to demonstrate that a navigation MLLM can internalize scene-specific visual-spatial correspondences and directly predict absolute metric poses (x, y, yaw) through native autoregressive language-token decoding, without a dedicated pose-regression head or external scene maps and reference-image retrieval at inference.

Existing methods such as ACE also encode scene information in network weights, but rely on a dedicated network to predict image-to-scene correspondences, followed by RANSAC and PnP for pose estimation. VLM-Loc relies on external BEV maps and scene graphs for localization. In contrast, SMLA-Nav integrates localization and navigation within a shared MLLM, making localization an intrinsic capability of the navigation policy rather than a prerequisite provided by an external module.

AuthorsOpenReview ↗

Response to Reviewer UFgA - Part II

2. Unseen textual destinations and semantic generalization (W2, Q2)

We thank the reviewer for this important question. We clarify that SMLA-Nav addresses navigation to specific destinations in previously observed environments, where scene-specific goal-to-location associations are acquired during adaptation. This is a practical setting for long-term robot deployment and differs from zero-shot navigation in entirely unseen environments. Nevertheless, we further investigate generalization beyond predefined destination names through compositional semantic goals combining objects, visual attributes, and spatial relations.

Examples of Compositional Semantic Goals.

Semantic goal description
A bed and large water bottles stand at the end of an aisle between storage racks.
Shelves of yellow bins and cardboard boxes face a white enclosure with open doors.

We consider both short descriptions and full four-view descriptions. For the latter, Front, Right, Back, and Left refer to directions relative to the target panorama rather than the robot's current heading.

Semantic Post-Training and Evaluation Settings.

We conduct a joint training experiment across seven scenes. Specifically, Scene A uses its original image–pose observations and navigation trajectories, augmented with semantic description–location pairs generated from its N posed observations. The other six scenes retain their original training data, consisting of image–pose observations and navigation trajectories, without additional semantic supervision. We mix the training data from all seven scenes to train a single model. All additional semantic supervision comes exclusively from Scene A.

We evaluate the resulting model under three settings:

  1. Unseen paraphrases in Scene A: We replace descriptions of previously learned locations with unseen paraphrases, testing generalization to new language expressions.

  2. Held-out goal views in Scene A: We generate semantic descriptions from goal views excluded from the N training observations, testing whether the model can ground destinations from previously unseen viewpoints.

  3. Cross-scene semantic transfer: We evaluate semantic descriptions of held-out goal views in the other six scenes. Although their original spatial knowledge is included in joint training, they receive no semantic description–location supervision. This tests whether semantic grounding learned in Scene A transfers to spatial memories acquired in other scenes.

Localization Accuracy and Navigation SR for Unseen Semantic Goals (%).

Evaluation settingOriginal: LocalizationOriginal: NavigationSemantic post-training: LocalizationSemantic post-training: Navigation
Scene A: Unseen paraphrases18.6015.1076.8066.30
Scene A: Held-out goal views8.207.0046.5038.40
Six other scenes: Held-out goal views12.5010.0030.8027.50

Semantic post-training substantially improves localization and navigation across all three settings. Notably, in the six other scenes, where no additional semantic supervision is provided, localization accuracy improves from 12.5% to 30.8% and navigation SR from 10.0% to 27.5%. These results provide initial evidence that semantic grounding learned in Scene A can transfer to unseen semantic descriptions and goal viewpoints in other adapted scenes, without additional semantic supervision in those scenes.

AuthorsOpenReview ↗

Response to Reviewer UFgA - Part III

3. Practical advantages and trade-offs over SLAM-based navigation (W1, Q1) - Part I

We thank the reviewer for asking us to substantiate the deployment benefits and scene-update procedure relative to conventional navigation systems.

We examine the value of incorporating scene-specific memory into a navigation MLLM through controlled executor and localizer comparisons, multi-scene experiments, and deployment measurements. We also examine compatibility with SLAM-based data collection and performance after scene changes.

3.1. Strong visual navigation and tolerance to localization errors

Our primary motivation for adopting an MLLM is its visual navigation capability. Manuscript Appendix Table 10 provides a controlled comparison: with the same HLoc localizer and A* planner, X-NavDP achieves 49.2% / 53.0% navigation SR on HM3D / Gauss, whereas our MLLM navigation executor, using only Stage 1 pretraining, already achieves 67.5% / 85.4%. Among the evaluated models, these results support the stronger transferable visual navigation capability of the MLLM executor.

Manuscript Table 2 further shows that, with the localization and planning modules fixed, replacing direct polyline tracking with our post-trained visual navigation executor improves navigation SR. Figure 4 illustrates one possible reason: when localization errors occur, following a path planned from an incorrect pose may lead to a collision, whereas our model can adjust its motion based on the current visual observations and subsequently relocalize. Navigation performance therefore depends on both localization accuracy and the execution policy's response to actual observations and intermediate errors.

Manuscript Table 4 also shows that scene adaptation improves navigation decisions in addition to localization. Even with ground-truth poses supplied to exclude localization errors, incorporating a small number of scene-specific navigation trajectories increases SR from 77.3% / 77.3% to 92.0% / 90.1%. This indicates that the model learns route selection and scene-specific navigation behavior from the trajectories, beyond image-to-coordinate correspondences.

3.2. Shared localization and navigation reduce additional module dependencies

Because the MLLM already performs navigation execution, we enable the same model to perform scene localization and goal grounding using its internal parametric memory, without querying an external localization database at inference.

Dedicated localizers remain strong alternatives. The added 2B localizer–executor comparison (Global Response §5; revised manuscript Table 23) fixes the SMLA-Nav executor and changes only the localization source. HLoc yields 89.7% / 85.3% navigation SR, while internal SMLA-Nav localization yields 84.8% / 84.0%, a difference of 4.9 / 1.3 percentage points. These results show that the existing navigation model can also provide competitive localization. Model sharing removes the need to deploy a separate localization model and its scene-specific reference database, and allows localization and action prediction to reuse intermediate computation. The resource measurements below evaluate parameter and computation sharing with the 2B implementation.

Reviewer_BR9EOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a navigation policy built on a VLM for previously observed indoor scenes. The paper proposes a framework where goals can be given as a scene-specific landmark name, a four-view goal panorama, or world coordinates. The model predicts the absolute poses of the agent and the goal as text tokens. A deterministic geometric adapter converts these into an agent-centric relative goal, and an MLP action head predicts a short-horizon action trajectory. The consistent-prompt two-stage training strategy first learns a transferable point-goal navigation policy from approximately 4M navigation samples, then performs scene-specific post-training using dense image–pose pairs and sparse expert trajectories to learn localization, goal grounding, and navigation behavior. The method is compared against VLA baselines and modular explicit-memory pipelines, and includes a single-site quadruped deployment.

Critical strengths
  1. The authors force the MLLM to commit to intermediate metric pose tokens during fine-tuning which acts as a achoring mechanism. This change has shown to mitigate landmark confusion and dropping wrong-destination stop failures.
  2. The consistent-prompt structure serializing poses as text creates a clean, modular interface that allows the internal localizer to be replaced with an external localization module without retraining the action policy.
  3. The authors present a clever conceptual decoupling by using a deterministic geometric adapter mid-prompt to translate global absolute poses into an egocentric relative goal vector.
Critical weaknesses
  1. The paper emphasizes that SMLA-Nav navigates "without querying external maps, localization databases, or online SLAM" at inference. However, spatial mapping and localization supervision are effectively shifted to the adaptation stage, where scene-specific spatial knowledge is distilled into the 8B VLM weights. The advantages of this parametric memory representation over conventional explicit memory are therefore not sufficiently established.
  2. The Geometric Goal Adapter only computes a straight-line Euclidean vector from the robot to the destination. In multi-room buildings with walls, cul-de-sacs, or U-shaped corridors, this Euclidean vector may point directly through obstacles rather than along a feasible route. Because the policy is Markovian and has no explicit map or route planner, its ability to route around non-convex obstacles must come from navigation behavior learned during pretraining and scene-specific route knowledge acquired from the 1–3 demonstrations per destination during Stage 2 post-training. The experiments do not sufficiently disentangle these contributions or establish whether the model can compose previously undemonstrated routes.
  3. The text goals are fixed landmark names shared between training and evaluation, with no evaluation of paraphrases or compositional descriptions. Text grounding therefore appears to be primarily a learned name-to-location lookup rather than a demonstration of general language grounding.
  4. All three modalities reduce to obtaining two poses and subtracting them. The near-parity of the 2B LoRA model with the 8B model is consistent with this. Together these points weaken the motivation for an MLLM over a lightweight localizer paired with a point-goal policy.
  5. The physical evaluation is limited relative to the paper's core motivation. Despite introducing the work as a solution for navigating from arbitrary starting positions, every physical trial was initiated from the same marked pose in the adapted environment, with an active onboard obstacle-avoidance safety layer. This leaves arbitrary-start robustness and the high-level policy's independent collision-avoidance capability insufficiently validated in real-world deployment.
  6. The evaluation includes information-asymmetric comparisons that inflate the perceived performance advantage of the proposed method. Highlighting comparisons against zero-shot baselines in the main text is potentially misleading, particularly for text goals, since these models lack exposure to the scene-specific landmark registry and cannot reasonably be expected to infer the coordinates of an arbitrary named landmark.
  7. Finally, the manuscript omits critical context by failing to compare against highly relevant contemporary literature, including general navigation VLAs, tour-based non-parametric site memories, and implicit weight-based scene coordinate regression methods.
Questions for authors
  1. Since adapting to a real building already requires teleoperating a robot, refining pose estimates offline using LiDAR and loop-closure constraints, and collecting posed panoramas, what is the concrete deployment advantage of distilling those observations into an 8B VLM rather than pairing the pretrained point-goal policy with a lightweight retrieval/relocalization module? In particular, how do these approaches compare in adaptation cost, inference latency, memory footprint, and scalability to additional buildings?
  2. In Table 3, SMLA-Nav (traj. only) benefits from Stage 1 point-goal pretraining (4M samples) and Stage 2 pose-token supervision, whereas the plain Qwen3-VL baseline does not have the same combination of components. How does Qwen3-VL, without the 4M point-goal pretraining, perform when post-trained with the proposed consistent prompt and explicit pose supervision on the same sparse trajectories? This would help isolate the contribution of point-goal pretraining from that of pose-based spatial supervision.
  3. Because evaluation episodes may share route segments with the post-training trajectories, can the authors report performance on a strict held-out route split where navigating from the start room to the goal room requires traversing a hallway connection never traversed in (D_nav)
  4. Given that classical localization and topological navigation systems can resolve ambiguous hallways using multi-hypothesis filtering or temporal observations, how does the strictly Markovian SMLA-Nav policy compare against such approaches in highly aliased environments? Appendix E already reports failures at visually similar intersections due to persistent localization errors. Can the authors quantify this limitation through direct comparisons with a temporally filtered localization baseline?
7 comments
AuthorsOpenReview ↗

Response to Reviewer BR9E - Part I

We thank the reviewer for the detailed assessment, including the suggestions on modular baselines, pretraining controls, and temporal localization. Please see the Global Response for the shared clarifications. We address the specific concerns below; paired results are in HM3D / Gauss order unless otherwise stated.

1. Practical value of the navigation policy and parametric scene memory (W1, W4, Q1) - Part I

We thank the reviewer for requesting a direct comparison with lightweight localization alternatives in both navigation performance and deployment cost.

Our primary motivation for using an MLLM is to learn reliable visual navigation behavior and enable the same model to acquire the scene knowledge needed for deployment. All SMLA-Nav experiments in this section use the 2B model. We examine the navigation executor first, then compare localization sources while keeping the executor fixed, and finally quantify adaptation cost, inference latency, memory usage, and multi-scene storage.

1.1. Navigation capability of the MLLM executor

We select X-NavDP, a dedicated RGB-D executor that performed strongly among the alternatives we tested, and additionally post-train it on the deployment scenes. With the same HLoc localization, scene-post-trained X-NavDP achieves 60.2% / 64.1% SR, whereas SMLA Stage 1, without scene post-training, already achieves 65.1% / 67.0%. Stage 2 further improves this to 89.7% / 85.3%. These results demonstrate stronger navigation and scene-transfer performance of the Qwen-based executor under the evaluated training conditions. This motivates our MLLM-centered approach, which aims to leverage the powerful navigation priors acquired through large-scale pretraining and adapt them to new environments.

Navigation performance with different localizers and executors: HM3D / Gauss SR (%)

LocalizerSMLA Stage 1SMLA Stage 2X-NavDPScene-post-trained X-NavDP
HLoc65.1 / 67.089.7 / 85.316.1 / 46.260.2 / 64.1
RAVEN61.6 / 62.081.6 / 81.314.7 / 41.857.2 / 63.5
ACE-G58.3 / 32.579.3 / 38.115.2 / 28.830.3 / 36.9
SMLA-84.8 / 84.0--

1.2. Localization within the navigation model

ACE-G and RAVEN yield lower navigation performance than internal localization, while HLoc performs slightly better, consistent with its strong localization accuracy reported in our paper. Nevertheless, SMLA-Nav remains highly competitive while using a single MLLM for both localization and navigation, without requiring a separate localization model or external reference-image retrieval. Improving localization accuracy remains an important direction for our future work.

1.3. Unified multimodal navigation and scalable parametric memory

The three goal modalities are unified through a common spatial representation: each is grounded into two absolute poses, which are then converted into an agent-centric relative goal. This demonstrates the advantage of our model in unifying multimodal navigation within a single task and policy. Furthermore, the comparable performance of 2B LoRA and 8B demonstrates the capacity of MLLMs to encode and reuse scene-specific spatial knowledge, even through a relatively small number of LoRA parameters, without substantial performance degradation. Notably, under the same scene-specific post-training supervision, X-NavDP has more parameters than our 2B LoRA model yet achieves lower navigation performance. This highlights that navigation performance depends not merely on parameter count, but also on the model architecture and the transferable navigation priors acquired through pretraining.

1.4. Shared data preparation and localizer adaptation cost

We compare scene adaptation time using the same image–pose data on one scene, with one H200 GPU allocated to each method.

Scene Adaptation Time.

MethodScene adaptation processTime (h)
RAVENFeature extraction and retrieval database construction0.43
HLocFeature extraction, matching, and mapping1.19
ACE-GScene map training1.81
SMLALocalization-only post-training, 10 epochs1.36

Although SMLA incurs a relatively higher training cost due to its large MLLM backbone, incorporating trajectory supervision allows localization and navigation execution to be jointly learned within the same training process. Furthermore, multi-scene joint training reaches a localization loss comparable to single-scene training in approximately 4 epochs, compared with 10 epochs for single-scene training.

AuthorsOpenReview ↗

Response to Reviewer BR9E - Part II

1. Practical value of the navigation policy and parametric scene memory (W1, W4, Q1) - Part II

1.5. Localization and action-prediction latency

We analyze localization and navigation latency separately on one NVIDIA H200 GPU. The localization measurements use coordinate-goal navigation, where the goal coordinates are supplied directly and SMLA predicts only the agent pose. We compare a small scene of 16m216\mathrm{m}^2 and a large scene of 800m2800\mathrm{m}^2.

HLoc and RAVEN rely on reference-image databases, and their mean localization latency increases between the two tested scene sizes. SMLA's mean latency remains nearly constant at 303.27 ms and 306.96 ms. RAVEN achieves the lowest latency in the small scene, while SMLA has the lowest reported mean in the large scene. ACE-G struggles to localize reliably in the large scene, so that entry is marked with a dash. These measurements show the practical benefit of directly predicting poses from a shared model as the reference scene becomes larger within the tested range.

Localization latency in small and large scenes (ms).

LocalizerSmall scene meanSmall scene medianLarge scene meanLarge scene median
HLoc994.591162.751049.101137.12
RAVEN194.94202.89307.50328.56
ACE-G287.60282.63--
SMLA303.27303.03306.96304.03

We also compare action-prediction latency for X-NavDP and SMLA-Nav. When SMLA is also used for localization, action prediction can reuse the KV cache of the image and prompt from the localization stage, further reducing redundant computation. The average action-prediction latency decreases from 55.75 ms to 22.10 ms with this reuse, compared with 229.78 ms for X-NavDP. This illustrates the computational benefit of using a unified model for both tasks.

Action-prediction latency (ms).

ExecutorMean (ms)Median (ms)
X-NavDP229.78226.93
SMLA55.7555.03
SMLA, with cache22.1022.44

1.6. CPU and GPU memory footprint

We evaluate four localizers running independently and in combination with either the SMLA or X-NavDP executor. Measuring both CPU and GPU memory allows us to compare deployment costs for localization alone and for the complete localization-and-execution configuration.

SMLA achieves the lowest CPU memory usage in these comparisons, as it shares the model for localization and action prediction and requires no separate external map. For localization alone, HLoc uses less GPU memory. With the SMLA executor held fixed, the unified SMLA configuration achieves the lowest combined GPU memory footprint, demonstrating the deployment benefit of shared parameters and computation.

CPU memory usage (GiB).

LocalizerLocalizer onlyWith SMLA executorWith X-NavDP executor
HLoc2.393.214.49
RAVEN2.774.374.82
ACE-G1.573.113.54
SMLA1.211.273.20

GPU memory usage (GiB).

LocalizerLocalizer onlyWith SMLA executorWith X-NavDP executor
HLoc2.047.534.24
RAVEN18.8124.2921.01
ACE-G3.589.075.78
SMLA5.475.487.66

1.7. Storage across multiple scenes

We evaluate the disk space required for all 56 scene units, including scene assets and shared model weights, with shared weights counted only once. The localization-only storage requirements of HLoc, RAVEN, and ACE-G are 25.35, 22.15, and 2.14 GiB, respectively. SMLA encodes the scene knowledge in one model without separate map files. The complete checkpoint, including the visual encoder, occupies 4.54 GiB.

ACE-G requires less storage for localization alone. When the same SMLA navigation executor is included, the total storage requirements of HLoc, RAVEN, and ACE-G increase to 29.89, 26.69, and 6.68 GiB, respectively. By sharing one model for localization and action prediction, SMLA retains a total storage footprint of 4.54 GiB. This comparison quantifies the storage advantage of shared deployment while accounting for both scene-specific assets and common model weights.

Disk storage across 56 scene units (GiB).

MethodScene mapsShared model weightsTotal storageTotal with SMLA executor
SMLA0.004.544.544.54
HLoc24.790.5625.3529.89
RAVEN6.7315.4222.1526.69
ACE-G0.831.312.146.68
AuthorsOpenReview ↗

Response to Reviewer BR9E - Part III

2. Independent contribution of PointGoal pretraining (Q2)

Thank you for identifying the missing control in Table 3. Following your suggestion, we retain the SMLA-Nav post-training protocol, including the proposed prompt and explicit pose supervision, and remove only PointGoal pretraining.

In the trajectory-only, text-goal setting, SR increases from 63.60% / 63.20% without pretraining to 71.0% / 72.6% with pretraining, a gain of 7.4 / 9.4 percentage points. With the full post-training dataset and evaluation across all three goal modalities, SR increases from 62.18% / 58.18% to 86.9% / 85.7%, a gain of approximately 24.7 / 27.5 percentage points. These comparisons isolate the contribution of the navigation prior while retaining the proposed spatial supervision during post-training.

Effect of PointGoal pretraining: HM3D / Gauss SR (%).

Scene post-training dataEvaluation goalsWithout pretraining: SR (%)With pretraining: SR (%)
Sparse trajectories onlyText goals63.60 / 63.2071.00 / 72.60
Image-pose observations + sparse trajectoriesText, image, and coordinate goals62.18 / 58.1886.90 / 85.70

3. Navigation along undemonstrated routes and the role of scene trajectories (W2, Q3)

We thank the reviewer for raising this important question. The Geometric Goal Adapter provides relative-goal conditioning but does not perform route planning. Instead, the visual policy predicts short-horizon actions based on current observations and the relative goal, with repeated closed-loop predictions composing longer routes. We conduct three complementary evaluations to examine the contributions of navigation pretraining, scene-specific trajectories, and the ability to navigate through undemonstrated connections.

Contribution of PointGoal Pretraining. As shown in our PointGoal pretraining ablation (Response §2), replacing the PointGoal-pretrained initialization with the original Qwen3-VL weights, while retaining the same scene-specific post-training formulation, reduces SR from 71.0% / 72.6% to 63.6% / 63.2% on HM3D / Gauss. This isolates the contribution of the transferable navigation prior to scene adaptation.

Contribution of Scene-Specific Trajectories. As shown in manuscript Table 4, when both configurations receive ground-truth evaluation poses, adding sparse scene-specific navigation trajectories increases SR from 77.3% / 77.3% to 92.0% / 90.1% on HM3D / Gauss. This demonstrates that scene trajectories improve navigation behavior beyond their contribution to localization.

Navigation through Held-Out Corridor Connections. To directly evaluate route composition beyond demonstrated connections, we construct a strict held-out corridor-connection evaluation subset of 64 episodes across all three goal modalities. Every episode requires traversing at least one corridor connection absent from all navigation trajectories in the scene-specific training dataset DnavD_{\mathrm{nav}}. SMLA-Nav achieves 81.25% SR and 62.06% SPL, with a collision rate of 1.56%.

Navigation Performance on Routes Requiring Undemonstrated Corridor Connections (%).

Evaluation subsetEpisodesSR ↑CR ↓SFR ↓TLR ↓SPL ↑
Three goal modalities6481.251.561.5615.6362.06

Together, these results show that PointGoal pretraining provides transferable navigation skills, while scene-specific trajectories further improve navigation behavior. The held-out connection evaluation shows that SMLA-Nav can successfully navigate through corridor connections absent from its navigation training trajectories.

Reviewer_ZFyGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SMLA-Nav, which integrates implicit spatial memory, visual localization, and action prediction within a shared MLLM. Through multi-stage training with image–pose observations and sparse navigation demonstrations, it supports text-, image-, and coordinate-goal navigation without external maps at inference. Experiments demonstrate strong simulation performance and preliminary real-world feasibility.

Critical strengths
  1. Strong empirical performance. The method achieves high success rates across multiple goal modalities, outperforming several existing navigation baselines.

  2. Practical framework. Combining image–pose supervision with sparse navigation demonstrations is appealing. Extensive ablations and real-world experiments further support its effectiveness.

Critical weaknesses
  1. Limited methodological novelty. The framework largely combines existing techniques, including visual localization, geometric transformation, and pretrained point-goal navigation. The contribution of consistent-prompt training is not sufficiently isolated from other components.

  2. Limited evaluation generalizability. The evaluation relies on a custom benchmark with shared training and test scenes, while baseline adaptation settings differ. Real-world experiments are also restricted to one site and a fixed starting pose.

Questions for authors
  1. Contribution of consistent-prompt training. Tables 3 and 4 vary multiple factors, including navigation pretraining and pose supervision. Could the authors isolate the contribution of consistent-prompt training under otherwise identical settings?

  2. Real-world navigation behavior. The demonstration shows nearly direct navigation with little apparent exploration or route correction. How much does this behavior depend on previously demonstrated routes, and are any external navigation or control mechanisms involved during evaluation? Could the authors provide results from more diverse starting positions?

  3. Benchmark novelty and generalizability. Given existing HM3D benchmarks (ObjectNav and InstanceImageNav) and Gaussian-splatting evaluation resources (NavGSim, GN-Bench and HabitatGS-Benchmark), what specific advantages do SMLA-HM3D and SMLA-Gauss offer over these existing benchmarks? Why is SMLA-Nav not evaluated on even a single established public benchmark to demonstrate its advantages over existing navigation policies under a common evaluation protocol?

8 comments
AuthorsOpenReview ↗

Response to Reviewer ZFyG - Part I

We thank the reviewer for the careful assessment and constructive suggestions. Please see the Global Response for the shared clarifications and additional experiments. We address the specific questions below; paired results are in HM3D / Gauss order.

1. Methodological contribution (W1)

We thank the reviewer for this constructive comment. We address a challenging and practically significant problem, and SMLA-Nav achieves good performance. Moreover, SMLA-Nav goes beyond simply connecting an off-the-shelf localizer to a navigation policy. We highlight three key methodological contributions below and will revise the manuscript to more clearly articulate our innovations.

1.1. Learning formulation: decoupled navigation-skill acquisition and scene-memory learning under sparse demonstrations.

We study how to adapt a navigation model to a deployment scene using sparse scene-specific trajectories. Motivated by NavGSim's behavior under sparse scene adaptation, Table 3 investigates its trajectory-based adaptation formulation through controlled experiments with a shared Qwen3-VL backbone. The evaluated VLN- and ObjectGoal-style pretraining does not effectively improve navigation performance under sparse scene demonstrations, with high collision rates and frequent incorrect stops at other registered destinations. We attribute these issues to two main factors: (1) difficulty in effectively transferring pretrained navigation capabilities to scene adaptation, and (2) insufficient goal-to-location grounding when relying solely on image-action supervision. We propose a problem-solving approach that decouples the acquisition of general navigation skills from scene-specific spatial memory, while enabling their effective integration through a unified spatial interface. PointGoal navigation is adopted for pretraining because its agent-centric relative-position guidance is independent of environments, providing strong generalization and transferability. During scene-specific post-training, sparse navigation trajectories help preserve and adapt the pretrained skills while supplying scene-specific layout knowledge; explicit pose supervision strengthens scene-specific spatial grounding.

1.2. Model design: a hierarchical spatial reasoning chain from parametric memory to navigation actions within a shared MLLM.

We implement this formulation through consistent-prompt multi-stage training that bridges navigation pretraining and scene-specific post-training through a common spatial structure, composing localization, tool-based relative-goal computation, and action prediction into a hierarchical spatial reasoning chain. This design enables a single MLLM to perform both localization and navigation, eliminating the need for separate models. The KV cache generated during localization can also be reused for navigation, reducing redundant computation and inference latency. Furthermore, parametric scene memory keeps localization latency nearly constant as scene scale increases within the evaluated range, while allowing a fixed-size model to accommodate spatial knowledge from multiple scenes.

Existing methods adopt different designs. Neural Topological SLAM, ULVN, and OpenNavMap rely on explicit scene representations and separate navigation execution policies. AstraNav-Memory predicts frontier points or target coordinates, but relies on external systems for navigation execution. NOLO directly predicts actions conditioned on scene-specific context videos, but has limited ability to generate optimal navigation trajectories. SMLA-Nav unifies these capabilities within a single MLLM, eliminating the need to query external scene maps, localization databases, or online SLAM at inference, thereby reducing inter-module dependencies and potential error propagation that could affect route execution.

1.3. Spatial capability: absolute metric pose generation from internalized scene knowledge through language decoding.

To the best of our knowledge, we are the first to demonstrate that a navigation MLLM can internalize scene-specific visual-spatial correspondences and directly predict absolute metric poses (x, y, yaw) through native autoregressive language-token decoding, without a dedicated pose-regression head or external scene maps and reference-image retrieval at inference.

Existing methods such as ACE also encode scene information in network weights, but rely on a dedicated network to predict image-to-scene correspondences, followed by RANSAC and PnP for pose estimation. VLM-Loc relies on external BEV maps and scene graphs for localization. In contrast, SMLA-Nav integrates localization and navigation within a shared MLLM, making localization an intrinsic capability of the navigation policy rather than a prerequisite provided by an external module.

AuthorsOpenReview ↗

Response to Reviewer ZFyG - Part II

2. Contribution of consistent-prompt training (W1, Q1)

Thank you for this constructive suggestion. We have added a controlled ablation to more directly assess the contribution of cross-stage prompt consistency.

Consistent prompts help preserve pretrained PointGoal skills while learning scene localization. We compare the Relative-only and Absolute + Relative pretrained checkpoints from Appendix B.2, Table 7. Both undergo the same post-training procedure as Table 4, row 2: scene image-pose supervision and 80,000 action steps sampled from the pretraining dataset, without scene navigation trajectories. Both use the Absolute + Relative prompt during post-training, so only the latter maintains a consistent spatial format across stages. All evaluations use ground-truth poses to exclude localization errors. Despite nearly identical pretraining performance, the consistent configuration retains 77.3% / 77.3% SR after post-training, whereas the inconsistent configuration drops to 51.98% / 50.57%. These results support the importance of maintaining a consistent spatial interface when learning localization while retaining pretrained navigation skills.

Contribution of consistent-prompt training

Pretraining → post-training promptHM3D pretraining SR (%)HM3D post-training SR (%)Gauss pretraining SR (%)Gauss post-training SR (%)
Consistent: Absolute + Relative → Absolute + Relative79.2077.3078.1077.30
Inconsistent: Relative only → Absolute + Relative79.3051.9877.9050.57
AuthorsOpenReview ↗

Response to Reviewer ZFyG - Part III

3. Benchmark design and fair comparison (W2, Q3)

We thank the reviewer for their interest in SMLA-Bench and their suggestion to further evaluate SMLA-Nav on established public benchmarks.

3.1. Why did the original evaluation prioritize a custom protocol?

Our objective is to study how robots acquire and use scene memory under controlled pre-deployment information and sparse demonstration budgets. Standard ObjectNav and InstanceImageNav primarily evaluate goal search in unseen environments, which differs from this setting. Memory-based methods such as MG-Nav also use different pre-collected scene-data budgets. We therefore initially established a common scene-adaptation protocol to avoid conflating differences in information access, demonstration budgets, and execution interfaces with methodological advantages—not because existing public benchmarks are inapplicable.

3.2. What does SMLA-Bench contribute?

SMLA-Bench builds on public HM3D scenes and independently scanned Gaussian scenes reconstructed using the NavGSim pipeline. Its contribution lies in providing scene-adaptation data and a unified evaluation protocol for destination-only navigation with text, image, and coordinate goals. By providing shared image–pose observations, destination bindings, and sparse navigation trajectories, SMLA-Bench supports both model post-training and explicit-memory construction, enabling controlled comparisons under consistent scene-information budgets. Tables 1–3 respectively disclose adaptation and execution conditions, align scene information while identifying additional resources, and provide controlled comparisons using the same backbone and trajectories.

3.3. We have added evaluations on public benchmarks.

To evaluate SMLA-Nav on an established benchmark, we conduct experiments on HM3D InstanceImageNav using its official evaluation episodes and metrics. Following our scene-adaptation setting, we collect randomly sampled navigation trajectories from the evaluation scenes for scene-specific post-training. Under this setting, SMLA-Nav achieves 62.5% SR and 51.3% SPL. For reference, UniGoal reports 60.2% SR and 23.7% SPL under its original zero-shot setting.

HM3D InstanceImageNav results (%).

MethodEvaluation contextSR (%)SPL (%)
SMLA-NavOfficial benchmark evaluation protocol; scene-specific post-training on sampled trajectories62.5051.30
UniGoalPublished result under its original evaluation protocol60.2023.70

We additionally conduct zero-shot evaluation on Habitat-GS PointNav, achieving 71.42% SR and 62.88% SPL. The original paper reports 70.70% SR and 58.49% SPL for a DD-PPO policy trained for 50 million steps across 100 3DGS scenes. Our Qwen-based PointGoal policy is learned through trajectory supervision, without benchmark-specific reinforcement learning.

Habitat-GS PointNav results (%).

PolicyTraining and evaluation contextSR (%)SPL (%)
SMLA PointGoal policyTrajectory-supervised pretraining; zero-shot benchmark evaluation71.4262.88
DD-PPO50 million training steps across 100 3DGS scenes70.7058.49

The result supports transfer of the pretrained navigation policy to unseen environments. Together, these evaluations examine both scene adaptation on a public image-goal task and zero-shot transfer of the navigation prior.

Reviewer_X2xXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SMLA-Nav, an end-to-end embodied navigation framework that integrates implicit spatial memory, visual localization, and action prediction within a shared multimodal large language model. The method introduces a consistent-prompt multi-stage training strategy that combines dense image-pose observations with sparse navigation trajectories. A hierarchical spatial reasoning chain is designed to connect absolute pose estimation, geometric relative-goal computation, and short-horizon action prediction. The authors also introduce SMLA-Bench, covering text-, image-, and coordinate-goal navigation. Experimental results demonstrate promising navigation performance across simulated environments, with additional validation on a quadruped robot.

Critical strengths
  1. The paper addresses a practically relevant problem of learning scene-specific spatial memory and navigation policies with limited expert demonstrations.
  2. The integration of visual localization, relative-goal computation, and action prediction provides a coherent framework for multimodal goal-directed navigation.
  3. The evaluation covers multiple goal modalities and includes comparisons with explicit-memory navigation methods, ablation studies, and real-world deployment.
Critical weaknesses
  1. The proposed framework primarily combines existing components, including MLLM-based visual localization, geometric coordinate transformation, point-goal navigation, and multi-stage fine-tuning. Although their integration is meaningful, the fundamental methodological contributions beyond existing navigation architectures are not sufficiently established. The authors should better distinguish the proposed approach from previous implicit-memory and hierarchical navigation methods.

  2. The paper claims that scene knowledge is encoded in the model parameters, but it remains unclear whether the model learns a structured and transferable spatial representation or primarily memorizes image-to-pose correspondences. More analysis is needed to investigate spatial memory capacity, robustness to visual changes, and behavior at previously unobserved viewpoints.

  3. Several baselines in Table 1 are evaluated in zero-shot settings, whereas SMLA-Nav benefits from scene-specific post-training. Differences in backbone architectures, pretraining data, and adaptation protocols may substantially influence the reported performance gaps. More controlled comparisons under matched training and supervision budgets would strengthen the conclusions.

Questions for authors
  1. How does the proposed implicit spatial memory differ fundamentally from conventional visual localization learned through image-pose supervision?
  2. How does navigation performance change as the number of scenes incorporated into the model increases?
  3. Can the authors provide comparisons using the same MLLM backbone, pretraining data, and scene-specific supervision budgets?
  4. What are the end-to-end inference latency and achievable navigation control frequency during real-world deployment?
  5. How robust is the learned spatial memory to environmental changes without additional post-training?
3 comments
AuthorsOpenReview ↗

Response to Reviewer X2xX - Part I

We thank the reviewer for the careful assessment and constructive suggestions. Please see the Global Response for the shared clarifications and additional experiments. We address the specific questions below; paired results are in HM3D / Gauss order.

1. Methodological contribution beyond existing navigation methods (W1)

We thank the reviewer for this constructive comment. We address a challenging and practically significant problem, and SMLA-Nav achieves good performance. Moreover, SMLA-Nav goes beyond simply connecting an off-the-shelf localizer to a navigation policy. We highlight three key methodological contributions below and will revise the manuscript to more clearly articulate our innovations.

1.1. Learning formulation: decoupled navigation-skill acquisition and scene-memory learning under sparse demonstrations.

We study how to adapt a navigation model to a deployment scene using sparse scene-specific trajectories. Motivated by NavGSim's behavior under sparse scene adaptation, Table 3 investigates its trajectory-based adaptation formulation through controlled experiments with a shared Qwen3-VL backbone. The evaluated VLN- and ObjectGoal-style pretraining does not effectively improve navigation performance under sparse scene demonstrations, with high collision rates and frequent incorrect stops at other registered destinations. We attribute these issues to two main factors: (1) difficulty in effectively transferring pretrained navigation capabilities to scene adaptation, and (2) insufficient goal-to-location grounding when relying solely on image-action supervision. We propose a problem-solving approach that decouples the acquisition of general navigation skills from scene-specific spatial memory, while enabling their effective integration through a unified spatial interface. PointGoal navigation is adopted for pretraining because its agent-centric relative-position guidance is independent of environments, providing strong generalization and transferability. During scene-specific post-training, sparse navigation trajectories help preserve and adapt the pretrained skills while supplying scene-specific layout knowledge; explicit pose supervision strengthens scene-specific spatial grounding.

1.2. Model design: a hierarchical spatial reasoning chain from parametric memory to navigation actions within a shared MLLM.

We implement this formulation through consistent-prompt multi-stage training that bridges navigation pretraining and scene-specific post-training through a common spatial structure, composing localization, tool-based relative-goal computation, and action prediction into a hierarchical spatial reasoning chain. This design enables a single MLLM to perform both localization and navigation, eliminating the need for separate models. The KV cache generated during localization can also be reused for navigation, reducing redundant computation and inference latency. Furthermore, parametric scene memory keeps localization latency nearly constant as scene scale increases within the evaluated range, while allowing a fixed-size model to accommodate spatial knowledge from multiple scenes.

Existing methods adopt different designs. Neural Topological SLAM, ULVN, and OpenNavMap rely on explicit scene representations and separate navigation execution policies. AstraNav-Memory predicts frontier points or target coordinates, but relies on external systems for navigation execution. NOLO directly predicts actions conditioned on scene-specific context videos, but has limited ability to generate optimal navigation trajectories. SMLA-Nav unifies these capabilities within a single MLLM, eliminating the need to query external scene maps, localization databases, or online SLAM at inference, thereby reducing inter-module dependencies and potential error propagation that could affect route execution.

1.3. Spatial capability: absolute metric pose generation from internalized scene knowledge through language decoding.

To the best of our knowledge, we are the first to demonstrate that a navigation MLLM can internalize scene-specific visual-spatial correspondences and directly predict absolute metric poses (x, y, yaw) through native autoregressive language-token decoding, without a dedicated pose-regression head or external scene maps and reference-image retrieval at inference.

Existing methods such as ACE also encode scene information in network weights, but rely on a dedicated network to predict image-to-scene correspondences, followed by RANSAC and PnP for pose estimation. VLM-Loc relies on external BEV maps and scene graphs for localization. In contrast, SMLA-Nav integrates localization and navigation within a shared MLLM, making localization an intrinsic capability of the navigation policy rather than a prerequisite provided by an external module.

AuthorsOpenReview ↗

Response to Reviewer X2xX - Part II

2. Expanded multi-scene capacity of parametric spatial memory (W2, Q2)

We thank the reviewer for asking how scene-memory capacity scales as additional environments are incorporated.

To further investigate the multi-scene capacity of SMLA-Nav, we extend the original 56-scene joint-training setup by incorporating 100 additional scenes, resulting in 156 jointly adapted scenes within a single 2B model. We evaluate the resulting models on both the original HM3D and Gauss benchmarks and the 100 additional scenes, examining both the retention of previously acquired navigation capabilities and navigation performance in the newly incorporated scenes.

Navigation Performance of 2B Models after Joint Adaptation to 156 Scenes.

ModelEvaluation Set56-scene checkpoint: SR / SPL (%)156-scene checkpoint: SR / SPL (%)
2B LoRAHM3D84.80 / 65.3083.15 / 66.22
2B LoRAGauss84.00 / 75.3083.80 / 77.93
2B LoRA100 additional scenes—82.15 / 76.42
2B Full FTHM3D78.60 / 56.3080.98 / 61.78
2B Full FTGauss80.50 / 68.9083.55 / 75.64
2B Full FT100 additional scenes—81.35 / 75.12

The results show that expanding the training set from 56 to 156 scenes does not lead to substantial degradation in navigation performance on the original benchmarks. For 2B LoRA, SR decreases by only 1.65 and 0.20 percentage points on HM3D and Gauss, respectively, while SPL improves by 0.92 and 2.63 percentage points. For 2B Full FT, joint adaptation improves SR by 2.38 and 3.05 percentage points and SPL by 5.48 and 6.74 percentage points on the two benchmarks, respectively.

Furthermore, the 156-scene 2B LoRA and Full FT models achieve 82.15% / 76.42% and 81.35% / 75.12% SR / SPL, respectively, on the 100 additional scenes. These results support the ability of a fixed-size model to acquire and reuse spatial knowledge across an expanded set of scenes while retaining navigation capabilities in previously adapted environments.

3. Robustness to visual changes and generalization to previously unobserved viewpoints (W2, Q5)

We appreciate the reviewer's questions about robustness to visual changes and navigation from previously unobserved viewpoints.

We evaluate pose noise, visual ambiguity, long-term scene changes, and starts distant from demonstrated trajectories within adapted environments.

3.1. Robustness to pose noise and visual ambiguity.

Appendix D.2 (Table 13) evaluates robustness to noisy post-training pose labels. Under Gaussian noise of (0.2 m, 15°), SMLA-Nav maintains SRs of 77.9% and 80.4% on SMLA-HM3D and SMLA-Gauss, respectively. Even with stronger noise (0.4 m, 30°), SRs remain at 70.6% and 65.0%. Furthermore, Appendix D.1 (Table 12) shows that replacing four-view localization with a single forward view reduces ImageGoal SR from 84.3% to 16.3% on HM3D and from 79.7% to 26.9% on Gauss, highlighting the importance of multi-view observations in mitigating visual ambiguity.

3.2. Robustness to long-term scene changes.

We evaluate the unchanged models on a rescan of the largest SMLA-Gauss office scene approximately one year later, without additional adaptation. The traversable regions differ by 43.22% of their union area, with 39.89% of the original traversable area becoming non-traversable.

Long-Term Navigation Evaluation on the Rescanned SMLA-Gauss Office Scene.

Goal modalitySMLA-Nav SR (%)HLoc + SMLA executor SR (%)
Text43.4331.31
Image33.3325.00
Coordinate42.0039.00

Unlike in the original evaluation, SMLA-Nav outperforms HLoc with the same SMLA executor across all three goal modalities. These results suggest that SMLA-Nav retains useful navigation capabilities under substantial long-term environmental changes, despite the degradation in overall performance.

3.3. Generalization beyond demonstrated navigation trajectories.

As reported in Appendix D.4 (Table 15), SMLA-Nav can navigate from starting positions beyond the demonstrated trajectories, achieving SRs of 43.0% on HM3D and 61.9% on Gauss for starts 2–4 m away from the nearest training trajectory. These results demonstrate partial generalization beyond the demonstrated routes, despite the limited trajectory supervision.

AuthorsOpenReview ↗

Response to Reviewer X2xX - Part III

4. Spatial memory beyond standalone visual localization (W2, Q1)

We thank the reviewer for asking us to clarify what spatial memory contributes beyond standalone visual localization.

We acknowledge that image–pose supervision itself is not new. Our implicit spatial memory extends beyond visual localization by incorporating both scene-specific spatial grounding and navigation-relevant knowledge. Image–pose observations establish visual–spatial correspondences, while sparse navigation trajectories additionally provide scene layout and route knowledge. Both capabilities are learned within a shared MLLM and integrated for navigation without querying external scene representations at inference. Table 4 provides supporting evidence: even with ground-truth poses, scene-trajectory adaptation improves SR from 77.3% / 77.3% to 92.0% / 90.1% on SMLA-HM3D / SMLA-Gauss.

Furthermore, our multi-scene experiment suggests that implicit spatial memory may enable knowledge sharing and positive transfer across scenes through joint training, potentially offering greater parameter efficiency than maintaining separate scene-specific representations.

5. End-to-End Inference Speed (Q4)

We appreciate the reviewer's request for end-to-end latency and policy-update measurements.

For the 2B SMLA-Nav model deployed on an NVIDIA RTX 4090 GPU in our real-world experiments, the measured average inference latency across text-, image-, and coordinate-goal inputs is 504.24 ms. Including 28 ms of system overhead, each policy step takes 532.24 ms, corresponding to an estimated update frequency of 1.88 Hz. The inference latency primarily comes from generating position coordinates in token form.

Measured end-to-end latency of the 2B real-world deployment on NVIDIA RTX 4090.

MeasurementValue
Model inference504.24 ms
System overhead28.00 ms
Complete policy step532.24 ms
Estimated policy update frequency1.88 Hz

6. Controlled comparisons under limited scene-specific supervision (W3, Q3)

We thank the reviewer for emphasizing controlled comparisons with matched backbones, data, and supervision.

Table 1 presents system-level comparisons using the released baseline interfaces, with their adaptation protocols detailed in Section 4.2 and Appendix C.1. To further ensure fair comparisons, we provide controlled experiments under matched training conditions.

First, Table 3 compares different navigation pretraining and spatial-memory formulations using the same Qwen3-VL backbone, the same underlying pretraining trajectories for navigation-pretrained variants, and identical scene-specific trajectories and evaluation episodes. The primary differences lie in the pretraining formulations and spatial supervision. Under the same sparse trajectory supervision, SMLA-Nav achieves 71.0% / 72.6% SR on HM3D / Gauss, compared with 59.8% / 44.2% for direct Qwen3-VL post-training. Appendix B.2 (Table 7) further isolates the effect of spatial conditioning under identical pretraining data and training schedules. Table 2 additionally evaluates different spatial-memory methods using shared scene-specific observations and trajectories.

The added prompt-consistency ablation further matches the backbone, pretraining trajectories, and scene post-training procedure, changing only the spatial fields used during pretraining. It yields 77.3% / 77.3% versus 51.98% / 50.57% SR after otherwise identical post-training, with ground-truth evaluation poses. Table 3 compares complete training formulations, whereas this ablation isolates cross-stage spatial-interface consistency.

Second, following the reviewer's suggestion, we further fine-tune MM-Nav and X-NavDP using the same sparse scene-specific navigation trajectories and evaluate them on CoordinateGoal navigation with ground-truth spatial inputs.

CoordinateGoal Navigation with Scene-Specific Fine-Tuning.

MethodDatasetSR (%)CR (%)SPL (%)
X-NavDP (FT)HM3D65.2034.8057.20
MM-Nav (FT)HM3D85.0013.8072.00
SMLA-Nav (GT)HM3D92.326.6576.30
X-NavDP (FT)Gauss63.3636.4160.02
MM-Nav (FT)Gauss86.007.9076.00
SMLA-Nav (GT)Gauss88.028.2982.85

FT denotes scene-specific fine-tuning. X-NavDP and MM-Nav receive ground-truth relative goals, while SMLA-Nav uses ground-truth absolute poses to compute relative goals.

SMLA-Nav achieves higher SR and SPL than both fine-tuned baselines on HM3D and Gauss. Together with the same-backbone experiments, these results further support the effectiveness of our navigation training formulation under limited scene-specific supervision. We will revise the manuscript to clarify the baseline adaptation protocols and highlight these controlled comparisons.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Global Response - Part I

We sincerely thank all reviewers for their careful evaluation, insightful comments, and constructive suggestions. We greatly appreciate the thoughtful questions raised throughout the reviews, which have helped us further clarify our methodology, strengthen our arguments, and improve the overall rigor of the paper. The Global Response addresses three common aspects of the reviews: the task definition, the methodological contributions of SMLA-Nav, and the real-world evaluation. The remaining questions and concerns are addressed individually below.

1. Task definition: multimodal goal navigation with scene memory

Reviewers: ZFyG, BR9E, UFgA.

We study robots that repeatedly perform navigation tasks in a designated environment, such as an office or a building, with the objective of reaching a specific user-designated destination reliably and efficiently. A destination can be specified by an image, a scene-specific name, or absolute coordinates. For example, a coffee-delivery robot may receive a photograph of a workstation, a place name such as "Ursa Major Meeting Room," or the absolute coordinates of the intended destination. All three forms of specification refer to a particular place in the environment. For scene-specific textual names, the robot must establish an association between the name and its corresponding physical location; reaching another place in the same semantic category does not constitute task completion. The user specifies only the destination, without providing step-by-step route instructions.

Our setting allows the robot to become familiar with the environment before deployment, acquiring scene experience through scene observations and sparse demonstrations. We investigate how such prior experience can help the robot reach user-designated destinations more directly and efficiently during subsequent navigation, reducing unnecessary searches and detours. Accordingly, our task differs from instruction-following vision-and-language navigation (VLN), which relies on route instructions, and from zero-shot exploration in entirely unseen environments. In our main comparisons, we also explicitly specify the scene information available to each method.

AuthorsOpenReview ↗

Global Response - Part II

2. Methodological contributions and related methods

Reviewers: X2xX, ZFyG, BR9E, UFgA.

SMLA-Nav introduces a learning and inference formulation that connects reusable scene memory with pretrained navigation skills within one MLLM. We discuss three contributions below.

2.1. Decoupling navigation-skill acquisition from scene adaptation.

We use PointGoal pretraining to learn navigation from agent-centric relative goals, then acquire scene-specific grounding and navigation behavior through posed observations and sparse trajectories. Table 3 shows that the evaluated VLN- and ObjectGoal-style pretraining formulations do not improve over direct post-training under the same sparse scene-trajectory budget. This motivates combining a transferable navigation prior with explicit spatial supervision, rather than relying on image-action supervision alone to learn both navigation and destination grounding.

2.2. Connecting localization and action prediction through a consistent spatial interface.

The training formulation uses a common spatial structure across stages: localization, tool-based relative-goal computation, and action prediction. A single MLLM performs localization and navigation, and the navigation stage can reuse the image and prompt KV cache from localization. This design supports both scene memory and navigation without a separate inference-time localization database.

Neural Topological SLAM, ULVN, and OpenNavMap use explicit scene representations with separate execution policies. AstraNav-Memory predicts frontier points or target coordinates and relies on external systems for navigation execution. NOLO conditions action prediction on scene-specific context videos. SMLA-Nav learns parametric scene memory, metric localization, and navigation within a shared model, connecting scene knowledge to action prediction through an explicit spatial interface.

2.3. Generating absolute metric poses through language-token decoding.

Our results show that an MLLM can internalize visual-spatial correspondences and generate absolute poses (x, y, yaw) in a scene coordinate frame without a dedicated pose-regression head or reference-image retrieval. ACE also encodes scene information in network weights, but predicts image-to-scene correspondences followed by RANSAC and PnP. VLM-Loc uses external BEV maps and scene graphs. SMLA-Nav learns localization within the MLLM and directly uses its decoded poses for subsequent navigation.

3. Clarifications on the real-world evaluation

Reviewers: ZFyG, BR9E, UFgA.

We thank the reviewers for identifying an error in the description of the starting poses. The common marked pose was used for data collection and coordinate alignment, not as the starting pose of every evaluation trial. Evaluation began at multiple locations, as illustrated in Figure 9 and the supplementary video. We will correct the relevant statements in the manuscript. At each step, SMLA-Nav uses current observations without relying on accumulated observation history.

The physical evaluation uses SMLA-Nav-2B on an NVIDIA RTX 4090 GPU and comprises 60 trials across text-, image-, and coordinate-goal navigation at an office site measuring approximately 86.2 m × 56.0 m, with 20 trials per goal modality.

Onboard LiDAR-based obstacle avoidance remained enabled. Among the 44 successful trials, five involved safety interventions: one trial had two interventions and four trials had one each, primarily at corners and during doorway entry. Counting any trial with an intervention as a failure leaves 39 successful trials out of 60, giving a success rate of 65.0%, compared with 73.3% under the original task-completion criterion. This stricter scoring measures successful completion without a safety intervention in the recorded trials.

AuthorsOpenReview ↗

Rebuttal Acknowledgment

Dear Reviewers and Area Chairs,

We sincerely thank all reviewers and the Area Chairs for their time, thoughtful consideration, and constructive feedback throughout the review process. We greatly appreciate their recognition of the practical relevance and potential of our work.

SMLA-Nav investigates how to develop MLLMs for practical embodied navigation. We propose an end-to-end framework built around a unified MLLM, trained through multiple stages to acquire transferable PointGoal skills and scene-specific spatial memory. A consistent spatial interface connects metric localization, goal grounding, and action prediction without relying on external resources at inference. We further demonstrate that an MLLM can directly generate absolute metric poses through native autoregressive language-token decoding. As recognized by the reviewers, SMLA-Nav offers a "practically relevant" and "coherent" framework for multimodal navigation, with "strong empirical performance" and real-world validation (X2xX, ZFyG, UFgA). Its metric-pose grounding provides an "anchoring mechanism", while its consistent-prompt design offers a "clean, modular interface" connecting localization and action prediction (BR9E).

During the rebuttal, we further strengthened our work through additional experiments and detailed responses to all questions and concerns raised by the four reviewers. We isolated the contributions of PointGoal pretraining, consistent-prompt training, and pose supervision, validating our multi-stage learning formulation (ZFyG, BR9E, UFgA). We further demonstrated semantic generalization to unseen descriptions and viewpoints across adapted scenes (BR9E, UFgA), strong MLLM navigation and competitive localization capabilities (X2xX, BR9E, UFgA), and the ability to retain navigation performance as scene memory expands (X2xX, UFgA). Additional results establish reduced joint deployment costs, more efficient multi-scene adaptation, and robustness to long-term environmental changes (BR9E, UFgA). We also added public benchmark evaluations, temporal localization experiments, and further evidence clarifying real-world navigation and safety interventions (ZFyG, BR9E, UFgA). We are particularly grateful to Reviewer ZFyG for the encouraging feedback and the opportunity for further discussion. Although we had limited opportunities for further exchanges with the other reviewers, we are confident that our detailed responses and additional experiments have addressed all questions and concerns raised and provided the requested clarifications and evidence. Together, these efforts further establish the effectiveness, generalization, scalability, and practical deployment potential of SMLA-Nav.

We believe that practical MLLM-based navigation should enable robots to efficiently reach user-specified destinations from different starting positions, without relying on step-by-step route instructions or inefficient exploration. SMLA-Nav takes a step toward this vision by enabling MLLMs to acquire and reuse scene-specific spatial knowledge for destination-driven navigation. We sincerely hope this work will help bring MLLM-based navigation closer to practical deployment and contribute to the continued development of embodied intelligence.

Once again, we sincerely thank the reviewers and Area Chairs for their time, effort, guidance, and support throughout the review process. Their thoughtful feedback and constructive discussions have been invaluable to us and have greatly enriched our work.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.