acceptodds
Under review as a conference paper at ICLR 2027

Geometry-Conditioned Latent World Models with Recurrent Adaptation in Dynamic Environments

Abstract

Autonomous navigation frequently relies on pre-mapped 3D spatial priors, but real-world deployments are inherently dynamic, resulting in inconsistencies between static maps and observations. Latent world models offer a promising, lightweight solution operating solely on RGB video, but current architectures lack the explicit spatial understanding and persistent memory to plan effectively in dynamical environments. To bridge this gap, we introduce GeoMamba-WM, a geometry-grounded recurrent world model that encodes a pre-mapped semantic 3D point cloud and utilizes a Mamba-based State Space Model (SSM) to dynamically correct a persistent hidden state using live visual observations. Prediction and planning are performed within a fused latent space derived via cross-attention between the observation and corrected geometry prior and stabilized by an auxiliary objective that encourages information captured by the SSM state to persist throughout the predictive architecture. To systematically evaluate this, we introduce GOI-NAV, a 3D navigation benchmark that features geometry-observation incongruences such as displaced and unseen obstacles, deformable surfaces, and visual noise. Empirical evaluations demonstrate that GeoMamba-WM significantly outperforms baseline latent world models across all incongruence axes, establishing a new standard for offline, reward-free predictive planning. Ultimately, our results show that latent world models are highly viable for planning in dynamic, pre-mapped 3D environments using depth-less RGB input, demonstrating that cross-modal inconsistencies can be resolved entirely in latent space without explicit mapping or online reconstruction objectives.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Latent World Models, State Space Models, 3D Navigation, Geometric Priors, Dynamic Environments, Cross-Modal Fusion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_oUA5OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper is a variation of the latent world model Dino-WM for navigation with some additional twists. An additional "prior" is supposed to be available on the environment in the form of a latent embedding of the geometric scene structure (an encoded point cloud) and the agent / WM receives visual observations, which can deviate from the prior in different ways, eg. dynamic objects or even changed env structure. The WM is based on frozen Dino-v2 features which are fused with the prior (and the obligatory action inputs) through gated cross-attention. A gated recurrent model maintains a hidden state and drives the predictor. Planning is done classically like other work in this field with CEM, a variant of MPC.

The work is evaluated on the authors own benchmark which they introduce, and which is itself based on a simple navigation environment introduced in 2021. The environment is extremely simple, consists of a single scene, and the distractors (dynamic objects) seem to be simple regular objects: cylinders, as suggested by Figure 2.

Critical strengths

The work adds some twist on Dino-WM (which itself has only ever been tested on toy problems) in cases where scene priors are available and in case one absolutely would want (for some reason I could not imagine) to solve navigation with world models and CEM and using geometric priors which are not maps.

Critical weaknesses

I would like to reassure the human co-authors of this paper that I did not use an LLM for my review and that thus their paper has not been uploaded to an LLM either. I see that as an advantage for the submitters, but it also comes with a downside: the paper needs to be understandable for a human.

SIMPLICITY OF TASK AND PROBLEM

This work builds on Dino-WM, a world model leveraging frozen Dino-v2 features to learn a predictor, ie. a forward model predicting the future state/features from the current state/features and actions at the current time step. It is thus a WM in a "strict" sense from the RL literature, as opposed to recent hi-jacked video prediction models which are also labelled with this term in recent literature. As a WM, it cannot be directly used for feature extraction in RL/IL setups since the current action is input. Rather, planning is done directly and explicitly with optimization at inference time (MPC/CEM) and this is where the problem lies. There is a sub community which claims very loudly that MPC is the way to go in robotics, as in that case planning is explicitely performed. However, in practise these methods simply do not work, at least not with the planning methods (eg CEM) and objective functions (difference to latent goal state) which these methods use. Case in point, Dino-WM is only applied to very simple toy problems and is far outperformed by models which directly train e2e with RL or IL.

This is also the main problem with this work: it targets an extremely simple navigation benchmark (which the authors created themselves) with an extremely simple task setup and it overfits to a single environment. This is extremely far from the SoTA in robotics or even from the Embodied AI sub field in the ML literatures which evaluates in simulation. PointGoal navigation is completely solved and saturated in simulation even in much more difficult setups, where the task requires the generalization to unseen environments and environments are photo realistic simulators.

The authors justify the choice of a single environment with "navigate with geometry-observation incongruence rather than standard pathfinding complexity" but simply do not understand this phrase.

There is no real reason that existing SoTA simulators for navigation cannot be used. The authors claim that those do not handle dynamic objects and even cite Habitat-Sim (L163) but this is not true. Since V3, Habitat Sim even handles people and can be used for social navigation. Other simulators like iGibson also handle humans. Placing cylinders into these simulators (at different poses per episode) has been done in Multi-Object Navigation since 2020 (and later extended to more complex shapes loaded from ShapeNet):

[a] Wany et al., MultiON: Benchmarking Semantic Map Memory using Multi-Object Navigation, NeurIPS 2020

For comparison on realistic work on navigation, work from 2022 trains model on realistic outdoor scenarios, deploys on real robots, and as this work provides prior information. But instead of requiring the knowledge of a 3D point cloud, it simple extracts features from satellite images:

[b] Dhruv Shah, Sergey Levine, ViKiNG: Vision-Based Kilometer-Scale Navigation with Geographic Hints, RSS 2022.

Similarly, realistic navigation agents can now be trained 100% in simulation on the HM3D dataset and they can be deployed as is on real robots to navigate in unseen environments, one example:

[c] Janny et al., Reasoning in visual navigation of end-to-end trained agents: a dynamical systems approach, CVPR 2025.

MOTIVATION, JUSTIFICATION, PRESENTATION

While the overall idea is clear (use a WM with a prior and handle deviations wrt to this prior), the motivation for many design choices and most modules is not as clear and lacks justification. The paper is written such that each individual sentence is correct and makes sense, but it is quite hard to see the forrest given all the trees we are presented with. In other words, the paper is written by going from detail to detail to detail and it is hard to follow the bigger picture.

For instance, it is not clear to me why Mamba was used for this model, and even whether this model actually is a Mamba model. What I can see from the equations is a recurrent network with gating functions, but the gates depend on the hidden state H_t, which makes this incompatible with Mamba (it cannot be trained in parallel). We also do not even know what the introduced blocks A, B, C are, this is not stated. What I do see looks like a GRU including update and forget gates, but they are not called this way. It is not clear to me why two different cross-attention operations are needed to fuse geometry and observations (L201 and L220).

The model makes a difference of the encode observations I_t and the encoded observations + state X_t, but the difference has not been motivated and I_t is actually never even explained.

Section 3.5 is unclear needs to be better explained.

The paper also makes use of quite some non-conventional terms and language and it is not clear where this comes from. While this language is not false per se, it takes some time to understand what is meant since it does not correspond to the established literature. Some Examples:

  • L211 this is not a factor but a gate. It corresponds to an udpate gate
  • L389 "Aligned" would probably be better named "In-distribution setting"
  • L390 "Variational" would probably be better named "Out-of-distribution setting". Variational has a quite clear term in ML.

It is not clear to me why inconsistent geometry-observation pairs can result in latent space collapse. To me this makes the learning problem harder (and this is what the proposed curriculum strategy tries to address), but this is not collapse. Furthermore, I think the fact that an anchor loss is necessary to link the model to the frozen in-distribution checkpoint raises some doubts on the training setup. This should not be necessary.

Beyond lacking motivation, they paper lacks details:

  • Geometric incoinsistency is not well enough explained
  • Deformable objects: what are they and why are they deformable? They look like cyclinders.
  • Visual noise: what is it?
  • The concepts of walkability and deformabiltity are mentioned by never discussed.

EXPERIMENTS

One of the biggest issues I have with the evaluation of WMs like Dino-v2 is that planning is performed on goal states which are not final goal states. Rather, the literature chooses intermediate goal states from expert trajectories for the test episode, which makes the method non-applicable in real life scenarios: these goal states are simple not available, and if we assume we have an expert plan, why do we even try to solve the problem?

One of the most important questions is thus how this is done in this paper: how are goal states chosen?

I did not understand Figure 3. I guess the green circle is the goal state? The model rarely seems to arrive at the goal? The baselines are just running in circles? This raises some doubts on the validity of the baselines.

The baselines should include e2e trained models. A SAC baseline is proposed, but SoTA models trained in simulation are base on on-policy methods like PPO. What are the frozen backbones mentioned there?

DETAILS

  • L391 3D continuous control is mentioned. Why 3D?
  • L92 this sentence talks about things which have not yet been introduced and actually never are.
Questions for authors
  • Do you assume knowledge of GT positions or do you handle noise?
  • What is the action space?
  • How are goal states chosen?
  • What is the planning horizon and how is it related to goal states?
  • Why SR and Soft-SPL but no SPL?
  • the points raised in "Weaknesses"
4 comments
AuthorsOpenReview ↗

Response to Reviewer oUA5 (1/4)

We thank the reviewer for an extremely detailed and thorough review. Several of the reviewer's points led to substantive changes, including the naming and description of the recurrent module, the clarity of the method section, the missing definitions, the SPL metric, the related-work positioning and Figure 3. We address every point below, in the order of the review.

S1 / W1

Why world models with CEM and a geometric prior; “in practise these methods simply do not work ... Dino-WM is only applied to very simple toy problems and is far outperformed by models which directly train e2e with RL or IL.”

Response: We agree that end-to-end RL and IL agents are currently the strongest navigation systems in simulation, and our paper does not claim otherwise. Our question is a different one: whether a reward-free world model, trained offline, can plan when its prior map is out of date. This setting matters when rewards are unavailable, when online interaction is expensive, or when a single model must serve many goals without retraining. Two results bear directly on the reviewer's concern (Table 3). First, under the planner-matched protocol, GeoMamba-WM with CEM reaches 84.4% aligned and 64.8% inconsistent success from offline data alone, close to NeDreamer, which is trained online with a distance-based reward and uses its own actor-critic (88.9% and 68.8%), and with higher SoftSPL (36.8 and 29.7 vs. 25.9 and 22.2). Second, the same frozen GeoMamba-WM representation with an RL controller (SAC) reaches 100% aligned and 86.4% inconsistent success, which shows that the representation is not the bottleneck.

W2

“it targets an extremely simple navigation benchmark (which the authors created themselves) with an extremely simple task setup and it overfits to a single environment.” ... “PointGoal navigation is completely solved and saturated in simulation”

Response: We would like to clarify two points. First, the task is image-goal navigation, not PointGoal navigation: the goal is specified by an RGB image and the map crop at the goal, and its coordinates are never given to the agent or used in the planning cost. We have replaced the phrase "simple point-to-point navigation", which invited this reading. Second, the 30 evaluation layouts are disjoint from the 80 training layouts, and the 2021 work cited for the environment is the ThreeDWorld simulator rather than a navigation task. The single-room topology is deliberate, so that differences between models can be attributed to how they handle map–observation inconsistency, as explained in our response to W3. We agree, however, that it limits the evidence, and we have therefore added Habitat ImageNav in 11 unseen Matterport3D houses. The results are in Section 6.4 and Appendix A.5.

W3

“The authors justify the choice of a single environment with "navigate with geometry-observation incongruence rather than standard pathfinding complexity" but simply do not understand this phrase.”

Response: We agree that this sentence was unclear, and we have rewritten it in Section 4: "We keep the room simple so that failures can be attributed to map-observation mismatch rather than to long-range route finding, which we test separately in multi-room houses in section 6.4."

W4

“There is no real reason that existing SoTA simulators for navigation cannot be used. The authors claim that those do not handle dynamic objects and even cite Habitat-Sim (L163) but this is not true.”

Response: We would like to clarify that the sentence at L162–164 referred to deformable obstacles rather than dynamic objects. However, it was poorly worded, and we have removed it. We built GOI-NAV in TDW because the benchmark requires a cloth curtain that the agent can push through, which TDW supports through its Obi cloth simulation. We now also evaluate the method in Habitat (Section 6.4).

W5

Positioning against Multi-Object Navigation [a], ViKiNG [b] and Janny et al. [c]

Response: We thank the reviewer for these references. We have added all three to the related work (Section 2). MultiON places objects in Habitat scenes for sequential object-goal navigation, ViKiNG performs long-range outdoor navigation with geographic hints, and Janny et al. show that agents trained entirely in simulation on HM3D transfer to real robots. Our setting differs from all three: the prior map is given but out of date, and the planner is a reward-free world model trained offline.

W6

“the paper is written by going from detail to detail to detail and it is hard to follow the bigger picture”

Response: Section 3 now opens with an overview paragraph that names each component, its role and its subsection, so that the details that follow can be placed within the overall design.

AuthorsOpenReview ↗

Response to Reviewer oUA5 (2/4)

W7

“it is not clear to me why Mamba was used for this model, and even whether this model actually is a Mamba model.” ... “What I do see looks like a GRU including update and forget gates”

Response: The reviewer is correct. The gates are computed from XtX_t, which contains the previous hidden state, so the recurrence is not linear in the state and cannot be computed with a parallel scan. Section 3.3 now states that the state is updated sequentially. What the module shares with Mamba is the selective discretization: a learned per-channel decay exp⁡(alog⁡)\exp(a_{\log}) whose step size Δt\Delta_t depends on the input, which gives input-dependent retention per token. Structurally, it is a token-wise gated recurrence close to a GRU, and we therefore describe it as "Mamba-inspired" rather than as a Mamba SSM. Because training sequences are short (three transitions), sequential computation does not affect training cost in practice.

W8

“We also do not even know what the introduced blocks A, B, C are, this is not stated.”

Response: The blocks were described in Section 3.3 and in full in Appendix A.2.1, but we agree that the main-text description was too brief. The revised Section 3.3 defines each block. Block A predicts a token-wise retention gate At=exp⁡(−Δt⊙exp⁡(alog⁡))A_t = \exp(-\Delta_t \odot \exp(a_{\log})), with a step size Δt\Delta_t computed from XtX_t. Block B predicts a candidate update whose content is computed from ItI_t and gated by XtX_t. Block C reads the updated state and produces a correction bounded to (−1,1)(-1, 1), which is scaled by γ=0.05\gamma = 0.05 and added to the geometry tokens.

W9

“It is not clear to me why two different cross-attention operations are needed to fuse geometry and observations (L201 and L220).”

Response: The two operations play different roles. The first, inside the recurrent module, lets the 16 geometry tokens query the image features to produce the evidence that updates the recurrent state; its output is not used for planning directly. The second builds the fused latent in which the predictor and the planner operate. Section 3.4 now explains both roles. The second operation is important: replacing it with concatenation lowers inconsistent success by 37.2 points, and replacing it with AdaLN lowers it by 61.9 points (Appendix A.6.1, Table 15). We would also like to clarify that only the first operation is gated, while the fusion is plain cross-attention.

W10

“The model makes a difference of the encode observations I_t and the encoded observations + state X_t, but the difference has not been motivated and I_t is actually never even explained.”

Response: ItI_t is the input to the candidate update: the geometry tokens, the action embedding and the observation evidence, without the hidden state. XtX_t additionally contains the previous hidden state and drives the gates. Keeping the hidden state out of the candidate's input means that the candidate is proposed from current evidence, while the gates decide how much of it to retain. The revised Section 3.3 now states this explicitly.

W11

“Section 3.5 is unclear needs to be better explained.”

Response: The revised Section 3.5 explains the auxiliary head as follows. It is a 2-layer MLP that predicts the normalized, mean-pooled recurrent state from a fused latent. It is trained only on encoded latents, while its loss on predicted latents is added to the world-model objective, so that the predictor's rollouts retain the information held in the recurrent state, which the planner sees only through those rollouts. Without the auxiliary head, inconsistent success drops by 17.9 points (Appendix A.7.1, Table 24).

W12

Terminology (“factor” vs. gate; “Aligned” vs. “In-distribution”; “Variational” vs. “Out-of-distribution”)

Response: We now refer to the update as a gate rather than a factor, and we have replaced "variational fine-tuning" with "inconsistent-data fine-tuning", since "variational" has a specific meaning in machine learning. We have kept "aligned" and "inconsistent" for the evaluation sets, because the inconsistent conditions appear in 10% of the fine-tuning data and are therefore not strictly out of distribution.

W13

“It is not clear to me why inconsistent geometry-observation pairs can result in latent space collapse.”

Response: We agree that "collapse" was the wrong word, and we have removed it. What we observe is that training directly on mixed aligned and inconsistent data yields a much weaker model, with 20.6 points lower inconsistent success than our curriculum (Appendix A.11, Table 35), rather than a collapsed latent space.

AuthorsOpenReview ↗

Response to Reviewer oUA5 (3/4)

W14

“the fact that an anchor loss is necessary to link the model to the frozen in-distribution checkpoint raises some doubts on the training setup. This should not be necessary.”

Response: We agree with the reviewer that the anchor should not be necessary, and our experiments confirm that it is not. A from-scratch curriculum without any teacher (starting first with aligned data, then followed by increasing fractions of inconsistent data) reaches 81.1% aligned and 65.6% inconsistent success, compared with 85.6% and 71.5% for our two-stage procedure with the teacher (Appendix A.11, Table 35). Appendix A.11 now states that the teacher is not essential. It acts as a standard distillation term that limits forgetting of aligned behavior during fine-tuning and improves results modestly.

W15

Definitions of geometric inconsistency, deformable objects, visual noise, walkability and deformability

Response: These definitions were given in Appendix A.1, and short definitions are now included in Sections 3.2 and 4. We would also like to clarify a misreading that our Figure 2 caused: the cylinder is the agent, not an obstacle. The deformable object is a cloth curtain that replaces the central barrier and that the agent can push through. Walkability labels mark points on traversable floor, and deformability labels mark points belonging to a deformable surface. The Figure 2 caption now identifies the agent and the obstacles.

W16 / Q3

“how are goal states chosen?”

Response: We do not use intermediate goals. Each episode has a single final goal, given as one RGB image rendered at the goal position together with the map crop at that position. The planner minimizes the distance between the predicted terminal latent and this fixed goal latent at every replanning step, and no expert trajectory is used at test time. Section 3.6 now states this explicitly.

W17

“I did not understand Figure 3. ... The baselines are just running in circles?”

Response: The poor baseline paths were due to the previous planning configuration. We have regenerated the figure from the paper models under the planner-matched protocol, with corrected labels, and the caption now explains every element (start, goal and its success radius, and success and failure markers). The panels show held-out tasks in which GeoMamba-WM reaches the goal and both baselines do not, and the caption now refers to Table 1 for success rates over all tasks.

W18

“The baselines should include e2e trained models. A SAC baseline is proposed, but SoTA models trained in simulation are base on on-policy methods like PPO. What are the frozen backbones mentioned there?”

Response: The paper includes three end-to-end systems in Section 6.3 and Appendix A.4. These are NeDreamer, trained with online RL; 3D-Belief, which uses an explicit map and planner; and SAC actor-critics trained on frozen PERSIST-S and GeoMamba-WM representations. "Frozen backbones" means that the pretrained world-model encoders are frozen, and only the SAC actor and critic are trained on top of them, with 25k online transitions. We chose SAC because it is off-policy and considerably more sample-efficient at this interaction budget than PPO, which typically requires orders of magnitude more interaction. We agree that PPO trained at scale would be the stronger reference for a pure RL comparison.

W19 / W20

“L391 3D continuous control is mentioned. Why 3D?” / “L92 this sentence talks about things which have not yet been introduced and actually never are.”

Response: The scenes and observations are three-dimensional, but the action is a planar displacement. We have rewritten the sentence that referred to 3D continuous control, and Section 3.6 now states that actions are planar. We have also replaced the sentence at L92 with one that introduces the per-point deformability label and the cloth curtain it marks.

Q1

“Do you assume knowledge of GT positions or do you handle noise?”

Response: The agent's position is used only to select the map crop around the agent and is never given to the planner as coordinates. To simulate how this would translate to real-life scenarios, our odometry-drift ablation (Appendix A.6.4, Table 20) offsets this position with an accumulated random walk of up to 0.1 m or 0.4 m; inconsistent success drops by only 2.5–5.0 points, whether or not drift is applied during training.

Q2

“What is the action space?”

Response: The action is a continuous planar displacement of at most 0.4 m in a world-aligned frame, with a fixed camera heading. This is now stated in Section 3.6 and detailed in Appendix A.1.1.

AuthorsOpenReview ↗

Response to Reviewer oUA5 (4/4)

Q4

“What is the planning horizon and how is it related to goal states?”

Response: Under the planner-matched protocol, CEM optimizes 5-step action sequences, executes all five actions, and replans. The cost is the distance between the final predicted latent and the fixed goal latent, so the horizon is not tied to the goal. The goal can lie beyond the horizon, and the planner moves toward it by repeated replanning.

Q5

“Why SR and Soft-SPL but no SPL?”

Response: We now report SPL for every column in Table 6 (Appendix A.3). GeoMamba-WM reaches 40.0% on aligned scenes and 32.3% on inconsistent scenes, compared with 37.1% and 31.4% for DINO-WM + Geometry and 27.9% and 22.5% for DINO-WM, so the ranking is the same as for SR and SoftSPL. We originally reported SoftSPL because it also credits progress in failed episodes, which separates models that nearly reach the goal from those that wander.

Summary

We thank the reviewer again for the particularly detailed and thorough review. In response, we have clarified the task and its positioning, corrected the description and naming of the recurrent module, defined all components and benchmark elements in the main text, regenerated Figure 3 and added SPL. We have also evaluated the method on Habitat ImageNav in real-world Matterport3D scans, alongside a planner-matched re-evaluation of all baselines with independently trained models. We believe these revisions address the reviewer's concerns, and we respectfully ask the reviewer to reconsider their assessment in light of them.

Reviewer_4dA4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how an agent can perform offline, reward-free planning and navigation in latent space using only RGB observations, when a static 3D representation is available but the environment has since changed dynamically. The authors propose GeoMamba-WM, which builds on DINO-WM by introducing a point-cloud geometry encoder. They also construct the GOI-NAV benchmark, which is based on the TDW simulator and covers three types of perturbations and their combinations: geometric inconsistencies, deformable obstacles, and visual corruptions. In the experiments, GeoMamba-WM achieves a substantially higher planning success rate in inconsistent scenarios than both DINO-WM with geometric priors and RGB-only DINO-WM. When combined with a SAC controller, it succeeds in all aligned scenarios and approaches the strongest baseline, PERSIST-S, in inconsistent scenarios, while having much lower inference latency. Based on these results, the authors conclude that inconsistencies between priors and observations can be resolved in latent space without online 3D reconstruction.

Critical strengths
  1. The paper addresses navigation in settings where a static prior map is available but the environment has changed at deployment time, and decomposes such changes into three dimensions: geometric inconsistencies, deformable objects, and visual noise. This is a genuine problem in real-world navigation.
  2. The paper provides a lightweight and effective way to incorporate static geometric priors into a latent world model. It samples a local point cloud centered on the agent and fuses it with visual features, improving navigation planning performance with almost no additional inference overhead.
  3. The paper conducts systematic ablations on how geometric priors are incorporated and on the training strategy, and honestly reports results that are unfavorable to the proposed method, which can serve as a reference for future designs. The paper also fully reports training and inference costs; its computational overhead is lower than that of generative 3D world models, suggesting potential for real-time applications.
Critical weaknesses
  1. The evaluation scope is narrow. All experiments are conducted in a simulated environment with a single room topology, and the geometry encoder is scene-specific. Even under this limited setting, the success rate of offline planning remains relatively low, which is insufficient to support the claim in the abstract that inconsistencies between priors and observations can be resolved entirely in latent space.
  2. A key contribution of the paper is enabling the model to continuously correct outdated map information during navigation based on its visual observations. The experiments show that adding the new module improves performance, but the paper's own analysis indicates that the model's retained historical memory has almost no effect on the corrected map information. Therefore, the current evidence cannot establish that the performance gain actually comes from correcting outdated information, which weakens the paper's core contribution.
  3. The settings of some baselines may have prevented them from performing at their normal level. LeWM and PLDM do not succeed even once in the simplest, unperturbed scenario, while the paper uniformly applies its own planning configuration and does not clarify whether this configuration caused them to fail. This limits how convincing the claimed superiority of the proposed method is.
Questions for authors
  1. Does the proposed method remain effective in scenarios with substantially different room layouts, such as more diverse environments with multiple complex rooms and various types of obstacles?
  2. Can the authors provide experimental evidence that the current performance gain indeed comes from correcting outdated map information?
  3. The settings of some baselines may have prevented them from performing at their normal level. Can the authors show experimentally whether their failure is due to the configuration or to the methods themselves? Can the authors provide comparisons under conditions in which these baselines function properly?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 4dA4 (1/3)

We thank the reviewer for a careful and precise review. The reviewer's questions led to three changes: all results now use the planning configuration released with the baselines, the main comparisons use three independently trained models with significance tests, and the role of the recurrent state is now described according to what our analysis supports. We address each point below.

W1a / Q1

“All experiments are conducted in a simulated environment with a single room topology.” / “Does the proposed method remain effective in scenarios with substantially different room layouts, such as more diverse environments with multiple complex rooms and various types of obstacles?”

Response: We have added Habitat ImageNav in 11 unseen Matterport3D houses, which contain multiple rooms and real-world clutter. The results, reported in Section 6.4, are mixed. GeoMamba-WM’s success rate of 6.0% matches DINO-WM at 5.9%, and it is blocked by obstacles on fewer steps (42.2% vs. 50.1% for DINO-WM). However, success is low for every model, because a CEM planner that compares predicted latents with a single goal image over a horizon of about 2 m has little signal about the direction of a goal that is far away or out of view. We interpret this as showing that the method transfers to realistic multi-room scenes, while long-range navigation requires a stronger planner. We now state this limitation in Section 6.4 and in the Limitations paragraph.

W1b

“the geometry encoder is scene-specific”

Response: In GOI-NAV, the geometry encoder is trained on crops from the benchmark's room family and evaluated on 30 held-out layouts, so it is environment-specific rather than layout-specific. To test whether it is specific to the scenes it was trained on, we evaluated the Matterport3D encoder on houses it has never seen in the new Appendix A.5.5, Table 14. Its reconstruction error on the 11 unseen houses is the same as on its training scenes, and about six times lower than a reference in which each crop is compared with the reconstruction of a different crop:

ScenesNumberReconstruction error (m$^2$)Shuffled reference (m$^2$)
Training scenes used to train the encoder550.0248 ± 0.00420.1554 ± 0.0129
Training scenes held out for validation60.0258 ± 0.00360.1482 ± 0.0112
Unseen evaluation houses110.0251 ± 0.00520.1384 ± 0.0080

The encoder therefore does not need to be retrained for new scenes. We have also replaced "scene-specific" with "environment-specific" in Section 3.2.

W1c

“the success rate of offline planning remains relatively low, which is insufficient to support the claim in the abstract that inconsistencies between priors and observations can be resolved entirely in latent space”

Response: We agree with the reviewer, and we have removed this claim from the abstract and softened the corresponding statements in the conclusion. Under the planner-matched protocol, GeoMamba-WM reaches 84.4% success on aligned scenes and 64.8% on inconsistent scenes, which is substantially higher than in the submission, but inconsistent scenes remain far from solved. The revised abstract states that the inconsistencies can be handled in latent space, not that they are resolved.

AuthorsOpenReview ↗

Response to Reviewer 4dA4 (2/3)

W2 / Q2

“the paper's own analysis indicates that the model's retained historical memory has almost no effect on the corrected map information” / “Can the authors provide experimental evidence that the current performance gain indeed comes from correcting outdated map information?”

Response: We ran a dedicated analysis, and its answer is that the gain does not come from correcting the map in the sense the submission described. We have revised the paper accordingly.

Is there a gain from the recurrent state? Yes, although it is smaller and less certain than the single-seed ablation suggested. Over three independently trained models, GeoMamba-WM improves on the same architecture without the recurrent state and auxiliary head by +17.0 points on aligned scenes and +7.3 points on inconsistent scenes.

Does the gain come from correcting the map? No. The correction that the state applies to the geometry tokens amounts to 3.1% of their norm, does not grow when the map is wrong, and setting it to zero leaves the prediction error unchanged (Appendix A.9.1, Table 32).

What does the recurrent state do? It accumulates evidence that the map disagrees with the observations. A binary linear probe on the hidden state, detecting whether the map is incorrect or not, has an AUC of 0.48 at the first step, which is chance, but grows to 0.72 by step 3, compared with 0.62 to 0.64 when the state is reset at every step (Table 33). This indicates that the hidden state is able to accumulate evidence of map-observation mismatch. However, the inputs of the last four steps decode the mismatch at least as well (AUC 0.74 to 0.86), so we do not claim that the long memory is necessary for detection. Rather, the state keeps this evidence available throughout the episode, and the auxiliary head is trained to carry it into the latent used for planning. Consistent with this, resetting the state at every step lowers inconsistent success by 3.9 points, while the reset model remains 12.8 points above the model without the SSM (Appendix A.9, Table 29), so part of the gain comes from how training with the recurrent state and auxiliary head shapes the fused latent.

Changes to the paper. We now describe the module as a recurrent geometry state that accumulates evidence of map-observation mismatch (Section 3 overview and Section 3.3, renamed "Recurrent Geometry State"), and report the analysis in Appendix A.9.1.

W3 / Q3

“LeWM and PLDM do not succeed even once in the simplest, unperturbed scenario, while the paper uniformly applies its own planning configuration and does not clarify whether this configuration caused them to fail.” / “Can the authors provide comparisons under conditions in which these baselines function properly?”

Response: The reviewer was right to raise this. Under our shared configuration, PLDM reached the goal in 0 of 30 aligned layouts, whereas under the navigation planner of stable-worldmodel, the released implementation of LeWM and PLDM, it reaches 9 of 30. We have therefore adopted this planner for every model, including ours, and re-run the full benchmark and all ablations. Under this protocol, PLDM reaches 27.0% success on aligned scenes and 16.9% on inconsistent scenes, DINO-WM 47.0% and 42.8%, and GeoMamba-WM 84.4% and 64.8%. The comparison is therefore planner-matched in both directions.

LeWM still fails under its own planner. To understand why, we evaluated each model's goal cost along 40 expert trajectories in held-out layouts (new Table 9, Appendix A.3.4):

ModelSpearman ρ\rhoExpert action rank
GeoMamba-WM0.800.14
DINO-WM + Geometry0.850.12
DINO-WM0.970.12
LeWM0.090.63
PLDM−0.030.63

Spearman ρ\rho is the correlation between the goal cost and the true distance to the goal; it is close to zero for LeWM and PLDM, so their latent distance does not direct the planner towards the goal. Expert action rank is the percentile of the expert's action sequence among 255 CEM candidates, where 0 is best and 0.5 is chance; all DINOv2-based models rank it close to the top. The weakness of PLDM and LeWM is therefore a property of their learned representations under a goal-image objective, rather than of our evaluation. We have also removed the statement that these baselines "fail completely".

AuthorsOpenReview ↗

Response to Reviewer 4dA4 (3/3)

Presentation

“Generally understandable, but requires substantial effort in places.”

Response: We have revised the presentation throughout. Section 3 now opens with an overview of the components and their roles, blocks A, B and C and the inputs ItI_t and XtX_t are defined in the main text, the missing benchmark definitions have been added to Section 4, and Figure 3 has been regenerated with a full legend.

Summary

We sincerely thank the reviewer for the careful and constructive review. Following the reviewer's questions, every result is now planner-matched, the main comparisons use three independently trained models with significance tests, the role of the recurrent state is reported as our analysis supports, and the method is evaluated in a second environment built from real-world scans. We believe these changes address the reviewer's concerns, and we respectfully ask the reviewer to consider them in their assessment.

Reviewer_GRCoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the author proposed a world model for 3D scene navigation which uses a static map(point cloud) as prior, but the static map may be different from the actual .observation. The whole pipeline of the method are combined of a point-cloud encoder (DINOv2), a Mamba recurrent state space model for correcting the geometric based on current RGB observation, and a predictor to estimate the future states. In the experiments. to validate performance of the proposed method, they show improvement on navigation task compare to other world-model-based baseline, specially on the scenes with dynamic changing. To evaluate the method, they design and build a synthetic navigation benchmark which could potentially contribute to future work.

Critical strengths
  1. The proposed idea is novelty and motivated. This paper is trying to solve an challenging problem that new observation may be different from the static map prior and a efficient way to update and preserve map-observation consistency is important for many tasks such as navigation and mapping. The proposed solution has potential contribution to related area.

  2. The improvement performance and synthetic benchmark are useful. In comparison experiments, the paper show an improvement compared to baselines on navigation task specially when environment is dynamic changing. Also, they build a synthetic benchmark to evaluate related task which will have broad contribution to the community and related areas.

Critical weaknesses
  1. The proposed method is only validated on synthetic dataset which is limited to prove their performance. More comparison experiments are need on diverse real world dataset.
Questions for authors

1 How does the recurrent hidden state performance for sequences longer than training? 2. How does the method keep correspondence between geometry tokens when agent moving, since the local pointcloud crop will change? 3. How sensitive is the method to quality of the pointcould map? For example if it has outliers, not complete.

1 comment
AuthorsOpenReview ↗

Response to Reviewer GRCo

We thank the reviewer for the positive assessment of the geometric prior and recurrent state, and for the questions, which led us to add three new analyses to the paper. We answer each point below.

W1

“The proposed method is only validated on synthetic dataset which is limited to prove their performance. More comparison experiments are need on diverse real world dataset.”

Response: We have added an evaluation on Habitat ImageNav in Matterport3D, which consists of 3D scans of real houses (Section 6.4). The geometry encoder is used unchanged in 11 houses unseen during training, with the same reconstruction error as on its training scenes. GeoMamba-WM matches DINO-WM in success rate (6.0% vs. 5.9%) and has the highest SPL (3.8% vs. 3.4%). The absolute success rates show that long-range navigation across multiple rooms from a single goal image remains difficult for every CEM-based world model we tested, which we now state as a limitation. Even so, our model is blocked by obstacles on about 8 percentage points fewer steps than DINO-WM (42.2% vs. 50.1%), which indicates that the geometry prior is useful for navigation. Real-robot experiments are beyond the scope of the rebuttal period.

Q1

“How does the recurrent hidden state performance for sequences longer than training?”

Response: Every evaluation episode is longer than the training sequences. The model is trained on clips of three transitions, whereas evaluation episodes run for up to 60 steps, and successful GeoMamba-WM episodes take a median of 20 steps, with an interquartile range of 15 to 29, so half of all successes come after more than five times the training length (Appendix A.9). Two results show that the recurrent state remains useful over these longer horizons. First, a linear probe on the hidden state detects whether the map is wrong with an AUC of 0.72 at 3 steps, with the same AUC at step 9, three times the training length (Appendix A.9.1, Table 33). Second, resetting the state at every step lowers success by 4 to 6 points in every evaluation column (Appendix A.9, Table 29), showing that its accumulated information improves planning success.

Q2

“How does the method keep correspondence between geometry tokens when agent moving, since the local pointcloud crop will change?”

Response: The method does not track point-level correspondence, and does not need to. The map is encoded by a frozen point-cloud autoencoder, separate from the DINOv2 encoder used for the RGB observation (Section 3.2). At every step, the frozen geometry encoder takes the 1 m crop of the map around the agent, re-sampled in a world-aligned frame, and maps it to 16 learned query tokens, each of which summarizes the local geometry rather than following individual points. Since each step moves the agent by at most 0.4 m, consecutive crops overlap substantially. Consistency over time is carried by the recurrent state, which is updated from the new crop, the observation and the executed action at every step. This is described in Sections 3.2 and 3.3.

Q3

“How sensitive is the method to quality of the pointcould map? For example if it has outliers, not complete.”

Response: The benchmark already contains structured map errors: obstacles that exist only in the map, and obstacles that are missing from it. We have additionally evaluated the trained model, unchanged, on degraded maps (Appendix A.6.6, Table 23), on the 122 benchmark jobs without the deformable curtain:

Prior mapAligned SR (%)Inconsistent SR (%)
Full map85.671.0
50% of points dropped76.769.6
90% of points dropped80.066.3
0.5 m hole on the route75.667.8
10% additional outlier points38.931.5
30% additional outlier points41.130.1

Missing map content costs between 1 and 5 points of inconsistent success. Spurious points in free space are more harmful, at about 40 points, because no training map contains them; in practice, the prior map would be filtered for outliers, which is a standard step when building point-cloud maps, as we note in Appendix A.6.6. The method also tolerates localization error: offsetting the map query from the true position by an accumulated drift of up to 0.4 m lowers inconsistent success by only 2.5 to 5.0 points. Full results are in Appendix A.6.4, Table 20.

Summary

We thank the reviewer again for the positive assessment and the helpful questions. In response, we have added an evaluation on real-world scans, an analysis of the recurrent state over long episodes and experiments with degraded maps, together with planner-matched baselines and independently trained models. We hope these additions further support the reviewer's positive assessment of our work.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers (1/2)

We thank all reviewers for their detailed and constructive feedback. The reviews raised four shared concerns: whether the baselines were evaluated under fair planning conditions, whether the results hold across independently trained models, whether the method works beyond our own benchmark, and whether the paper explains its method and claims clearly. We have addressed each of them with new experiments and a revised manuscript, summarized below. All section, table and appendix numbers refer to the revised PDF.

G1. Planner-matched evaluation of all baselines (Reviewer 4dA4)

Reviewer 4dA4 asked whether our shared CEM configuration caused LeWM and PLDM to fail. We checked, and it did hold PLDM back. Under our original configuration, PLDM reached the goal in 0 of 30 aligned layouts, whereas under the navigation planner of stable-worldmodel, the released implementation of LeWM and PLDM, it reaches 9 of 30. We have therefore re-run every model and every ablation in the paper with this planner. It has a horizon of 5, 300 samples, 30 elites, 30 iterations, 5 actions executed per replan, with more details in Appendix A.2.6.

Our conclusions still hold under this planner. GeoMamba-WM reaches 84.4% success on aligned scenes and 64.8% on inconsistent scenes, compared with 47.0% and 42.8% for DINO-WM and 27.0% and 16.9% for PLDM. LeWM remains close to zero. To understand why, we evaluated each model's planning cost along expert trajectories in the new Table 9 (Appendix A.3.4). The latent distance to the goal tracks the true distance to the goal for the DINOv2-based models with a Spearman ρ\rho of 0.80 to 0.97, but not for LeWM or PLDM, which have Spearman ρ\rho values near 0, even though both predict their own next latents accurately. Their weakness therefore lies in their latent spaces rather than in the evaluation protocol.

G2. Independent training runs (Reviewers 4dA4 and oUA5)

The submitted tables reported variation across CEM evaluation seeds only. GeoMamba-WM and all latent world-model baselines are now trained with three independent training seeds, each evaluated with three CEM seeds on all 218 benchmark jobs, and Table 1 reports the mean across training seeds, with standard deviations in Table 6 (Appendix A.3). We also report SPL for every column in Table 6. To test the main comparisons, we ran a paired hierarchical bootstrap that resamples training seeds and jobs (4,000 draws; 95% intervals in brackets):

ComparisonIsolatesAligned (30 jobs)Inconsistent (188 jobs)
DINO-WM + Geometry vs. DINO-WMGeometry prior+20.4 [+2.2, +37.8], p = 0.029+14.7 [+7.9, +21.0], p < 0.001
GeoMamba-WM vs. DINO-WM + GeometryRecurrent state and auxiliary head+17.0 [+0.7, +34.1], p = 0.044+7.3 [−0.7, +15.1], p = 0.078
GeoMamba-WM vs. DINO-WMFull model+37.4 [+22.2, +53.3], p < 0.001+22.0 [+12.9, +30.5], p < 0.001

The geometry prior accounts for most of the improvement over DINO-WM. The recurrent state and auxiliary head add a further gain, which is significant on aligned scenes but not, at the 5% level, on inconsistent scenes. Section 6.2 now reports this three-seed result alongside the single-seed ablation, and the full test is in Table 10 (Appendix A.3.4).

AuthorsOpenReview ↗

General Response to All Reviewers (2/2)

G3. A second environment: Habitat ImageNav in Matterport3D (all reviewers)

All three reviewers asked whether the method works beyond our benchmark. We have added an evaluation on Habitat ImageNav in Matterport3D in Section 6.4 and Appendix A.5. It consists of 495 episodes in 11 multi-room houses that are 3D scans of real buildings, and are unseen during training. A single geometry encoder is used for all houses, every model is trained with three training seeds, and the agent, action space and planner are the same as in GOI-NAV. Success rate and SPL are as follows:

MetricRandomPLDMDINO-WMGeoMamba-WM
SR (%)3.4 ± 0.63.1 ± 1.55.9 ± 0.96.0 ± 1.2
SPL (%)1.0 ± 0.12.1 ± 1.03.4 ± 0.83.8 ± 0.5

GeoMamba-WM matches DINO-WM in success, and both succeed more often than random actions, with three to four times their SPL. However, success is low for every model. When the goal is far away or out of view, a CEM planner that compares predicted latents with a single goal image over a horizon of about 2 m has little signal about the goal's direction.

The geometry pathway does change how the agent moves. GeoMamba-WM is blocked by obstacles on about 8 points fewer steps than DINO-WM, and its runs of consecutive blocked steps are shorter (Table 5):

MetricPLDMDINO-WMGeoMamba-WM
Blocked steps (%)53.5 ± 4.750.1 ± 2.242.2 ± 2.2
Average consecutive blocked steps7.6 ± 1.06.0 ± 0.54.4 ± 0.1

In addition, the geometry encoder transfers to unseen houses without retraining. Its reconstruction error on the 11 unseen houses is the same as on its training scenes and about six times lower than a shuffled-crop reference (new Appendix A.5.5, Table 14).

We interpret these results as showing that the method transfers to real multi-room scans, while long-range navigation from a single goal image requires a stronger planner than CEM. This is also why GOI-NAV keeps the room simple, so that it can measure how a world model handles map-observation mismatch within the planner's reach. Since the Matterport3D prior agrees with the scene, this experiment tests transfer rather than inconsistency. We state these points in Section 6.4, Appendix A.5 and the Limitations paragraph.

G4. Revised claims and presentation

Role of the recurrent state. Following Reviewer 4dA4's question, we analyzed what the recurrent state contributes in Appendix A.9.1. The correction it applies to the geometry tokens is small, at only 3.1% of their norm, and does not grow when the map is wrong, while the state itself accumulates evidence that the map disagrees with the observations. We now describe the module as a recurrent geometry state that accumulates this evidence rather than as a corrector of the map, and have renamed Section 3.3 accordingly.

Claims. We have removed "establishing a new standard" and "entirely in latent space" from the abstract, and softened the corresponding statements in Section 6.3 and the conclusion.

Clarity. Section 3 now opens with an overview of the components and their roles. Section 3.3 defines blocks A, B and C, the inputs ItI_t and XtX_t, and the learned decay alog⁡a_{\log}; Section 3.4 explains the roles of the two cross-attention operations; Section 3.6 states the goal specification and the action space; Section 4 defines the cloth curtain and the visual-noise types; and the Figure 2 caption identifies the agent.

Corrections. While preparing this rebuttal, we found that Figure 3 had been drawn from the wrong checkpoint with two legend labels swapped, and that the submitted odometry-drift table did not match the stored runs. Figure 3 has been regenerated, and all tables are now generated from stored runs under the planner-matched protocol.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.