- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper is a variation of the latent world model Dino-WM for navigation with some additional twists. An additional "prior" is supposed to be available on the environment in the form of a latent embedding of the geometric scene structure (an encoded point cloud) and the agent / WM receives visual observations, which can deviate from the prior in different ways, eg. dynamic objects or even changed env structure. The WM is based on frozen Dino-v2 features which are fused with the prior (and the obligatory action inputs) through gated cross-attention. A gated recurrent model maintains a hidden state and drives the predictor. Planning is done classically like other work in this field with CEM, a variant of MPC.
The work is evaluated on the authors own benchmark which they introduce, and which is itself based on a simple navigation environment introduced in 2021. The environment is extremely simple, consists of a single scene, and the distractors (dynamic objects) seem to be simple regular objects: cylinders, as suggested by Figure 2.
Critical strengths
The work adds some twist on Dino-WM (which itself has only ever been tested on toy problems) in cases where scene priors are available and in case one absolutely would want (for some reason I could not imagine) to solve navigation with world models and CEM and using geometric priors which are not maps.
Critical weaknesses
I would like to reassure the human co-authors of this paper that I did not use an LLM for my review and that thus their paper has not been uploaded to an LLM either. I see that as an advantage for the submitters, but it also comes with a downside: the paper needs to be understandable for a human.
SIMPLICITY OF TASK AND PROBLEM
This work builds on Dino-WM, a world model leveraging frozen Dino-v2 features to learn a predictor, ie. a forward model predicting the future state/features from the current state/features and actions at the current time step. It is thus a WM in a "strict" sense from the RL literature, as opposed to recent hi-jacked video prediction models which are also labelled with this term in recent literature. As a WM, it cannot be directly used for feature extraction in RL/IL setups since the current action is input. Rather, planning is done directly and explicitly with optimization at inference time (MPC/CEM) and this is where the problem lies. There is a sub community which claims very loudly that MPC is the way to go in robotics, as in that case planning is explicitely performed. However, in practise these methods simply do not work, at least not with the planning methods (eg CEM) and objective functions (difference to latent goal state) which these methods use. Case in point, Dino-WM is only applied to very simple toy problems and is far outperformed by models which directly train e2e with RL or IL.
This is also the main problem with this work: it targets an extremely simple navigation benchmark (which the authors created themselves) with an extremely simple task setup and it overfits to a single environment. This is extremely far from the SoTA in robotics or even from the Embodied AI sub field in the ML literatures which evaluates in simulation. PointGoal navigation is completely solved and saturated in simulation even in much more difficult setups, where the task requires the generalization to unseen environments and environments are photo realistic simulators.
The authors justify the choice of a single environment with "navigate with geometry-observation incongruence rather than standard pathfinding complexity" but simply do not understand this phrase.
There is no real reason that existing SoTA simulators for navigation cannot be used. The authors claim that those do not handle dynamic objects and even cite Habitat-Sim (L163) but this is not true. Since V3, Habitat Sim even handles people and can be used for social navigation. Other simulators like iGibson also handle humans. Placing cylinders into these simulators (at different poses per episode) has been done in Multi-Object Navigation since 2020 (and later extended to more complex shapes loaded from ShapeNet):
[a] Wany et al., MultiON: Benchmarking Semantic Map Memory using Multi-Object Navigation, NeurIPS 2020
For comparison on realistic work on navigation, work from 2022 trains model on realistic outdoor scenarios, deploys on real robots, and as this work provides prior information. But instead of requiring the knowledge of a 3D point cloud, it simple extracts features from satellite images:
[b] Dhruv Shah, Sergey Levine, ViKiNG: Vision-Based Kilometer-Scale Navigation with Geographic Hints, RSS 2022.
Similarly, realistic navigation agents can now be trained 100% in simulation on the HM3D dataset and they can be deployed as is on real robots to navigate in unseen environments, one example:
[c] Janny et al., Reasoning in visual navigation of end-to-end trained agents: a dynamical systems approach, CVPR 2025.
MOTIVATION, JUSTIFICATION, PRESENTATION
While the overall idea is clear (use a WM with a prior and handle deviations wrt to this prior), the motivation for many design choices and most modules is not as clear and lacks justification. The paper is written such that each individual sentence is correct and makes sense, but it is quite hard to see the forrest given all the trees we are presented with. In other words, the paper is written by going from detail to detail to detail and it is hard to follow the bigger picture.
For instance, it is not clear to me why Mamba was used for this model, and even whether this model actually is a Mamba model. What I can see from the equations is a recurrent network with gating functions, but the gates depend on the hidden state H_t, which makes this incompatible with Mamba (it cannot be trained in parallel). We also do not even know what the introduced blocks A, B, C are, this is not stated. What I do see looks like a GRU including update and forget gates, but they are not called this way. It is not clear to me why two different cross-attention operations are needed to fuse geometry and observations (L201 and L220).
The model makes a difference of the encode observations I_t and the encoded observations + state X_t, but the difference has not been motivated and I_t is actually never even explained.
Section 3.5 is unclear needs to be better explained.
The paper also makes use of quite some non-conventional terms and language and it is not clear where this comes from. While this language is not false per se, it takes some time to understand what is meant since it does not correspond to the established literature. Some Examples:
- L211 this is not a factor but a gate. It corresponds to an udpate gate
- L389 "Aligned" would probably be better named "In-distribution setting"
- L390 "Variational" would probably be better named "Out-of-distribution setting". Variational has a quite clear term in ML.
It is not clear to me why inconsistent geometry-observation pairs can result in latent space collapse. To me this makes the learning problem harder (and this is what the proposed curriculum strategy tries to address), but this is not collapse. Furthermore, I think the fact that an anchor loss is necessary to link the model to the frozen in-distribution checkpoint raises some doubts on the training setup. This should not be necessary.
Beyond lacking motivation, they paper lacks details:
- Geometric incoinsistency is not well enough explained
- Deformable objects: what are they and why are they deformable? They look like cyclinders.
- Visual noise: what is it?
- The concepts of walkability and deformabiltity are mentioned by never discussed.
EXPERIMENTS
One of the biggest issues I have with the evaluation of WMs like Dino-v2 is that planning is performed on goal states which are not final goal states. Rather, the literature chooses intermediate goal states from expert trajectories for the test episode, which makes the method non-applicable in real life scenarios: these goal states are simple not available, and if we assume we have an expert plan, why do we even try to solve the problem?
One of the most important questions is thus how this is done in this paper: how are goal states chosen?
I did not understand Figure 3. I guess the green circle is the goal state? The model rarely seems to arrive at the goal? The baselines are just running in circles? This raises some doubts on the validity of the baselines.
The baselines should include e2e trained models. A SAC baseline is proposed, but SoTA models trained in simulation are base on on-policy methods like PPO. What are the frozen backbones mentioned there?
DETAILS
- L391 3D continuous control is mentioned. Why 3D?
- L92 this sentence talks about things which have not yet been introduced and actually never are.
Questions for authors
- Do you assume knowledge of GT positions or do you handle noise?
- What is the action space?
- How are goal states chosen?
- What is the planning horizon and how is it related to goal states?
- Why SR and Soft-SPL but no SPL?
- the points raised in "Weaknesses"
4 comments
Response to Reviewer oUA5 (2/4)
W7
“it is not clear to me why Mamba was used for this model, and even whether this model actually is a Mamba model.” ... “What I do see looks like a GRU including update and forget gates”
Response: The reviewer is correct. The gates are computed from , which contains the previous hidden state, so the recurrence is not linear in the state and cannot be computed with a parallel scan. Section 3.3 now states that the state is updated sequentially. What the module shares with Mamba is the selective discretization: a learned per-channel decay whose step size depends on the input, which gives input-dependent retention per token. Structurally, it is a token-wise gated recurrence close to a GRU, and we therefore describe it as "Mamba-inspired" rather than as a Mamba SSM. Because training sequences are short (three transitions), sequential computation does not affect training cost in practice.
W8
“We also do not even know what the introduced blocks A, B, C are, this is not stated.”
Response: The blocks were described in Section 3.3 and in full in Appendix A.2.1, but we agree that the main-text description was too brief. The revised Section 3.3 defines each block. Block A predicts a token-wise retention gate , with a step size computed from . Block B predicts a candidate update whose content is computed from and gated by . Block C reads the updated state and produces a correction bounded to , which is scaled by and added to the geometry tokens.
W9
“It is not clear to me why two different cross-attention operations are needed to fuse geometry and observations (L201 and L220).”
Response: The two operations play different roles. The first, inside the recurrent module, lets the 16 geometry tokens query the image features to produce the evidence that updates the recurrent state; its output is not used for planning directly. The second builds the fused latent in which the predictor and the planner operate. Section 3.4 now explains both roles. The second operation is important: replacing it with concatenation lowers inconsistent success by 37.2 points, and replacing it with AdaLN lowers it by 61.9 points (Appendix A.6.1, Table 15). We would also like to clarify that only the first operation is gated, while the fusion is plain cross-attention.
W10
“The model makes a difference of the encode observations I_t and the encoded observations + state X_t, but the difference has not been motivated and I_t is actually never even explained.”
Response: is the input to the candidate update: the geometry tokens, the action embedding and the observation evidence, without the hidden state. additionally contains the previous hidden state and drives the gates. Keeping the hidden state out of the candidate's input means that the candidate is proposed from current evidence, while the gates decide how much of it to retain. The revised Section 3.3 now states this explicitly.
W11
“Section 3.5 is unclear needs to be better explained.”
Response: The revised Section 3.5 explains the auxiliary head as follows. It is a 2-layer MLP that predicts the normalized, mean-pooled recurrent state from a fused latent. It is trained only on encoded latents, while its loss on predicted latents is added to the world-model objective, so that the predictor's rollouts retain the information held in the recurrent state, which the planner sees only through those rollouts. Without the auxiliary head, inconsistent success drops by 17.9 points (Appendix A.7.1, Table 24).
W12
Terminology (“factor” vs. gate; “Aligned” vs. “In-distribution”; “Variational” vs. “Out-of-distribution”)
Response: We now refer to the update as a gate rather than a factor, and we have replaced "variational fine-tuning" with "inconsistent-data fine-tuning", since "variational" has a specific meaning in machine learning. We have kept "aligned" and "inconsistent" for the evaluation sets, because the inconsistent conditions appear in 10% of the fine-tuning data and are therefore not strictly out of distribution.
W13
“It is not clear to me why inconsistent geometry-observation pairs can result in latent space collapse.”
Response: We agree that "collapse" was the wrong word, and we have removed it. What we observe is that training directly on mixed aligned and inconsistent data yields a much weaker model, with 20.6 points lower inconsistent success than our curriculum (Appendix A.11, Table 35), rather than a collapsed latent space.
Response to Reviewer oUA5 (3/4)
W14
“the fact that an anchor loss is necessary to link the model to the frozen in-distribution checkpoint raises some doubts on the training setup. This should not be necessary.”
Response: We agree with the reviewer that the anchor should not be necessary, and our experiments confirm that it is not. A from-scratch curriculum without any teacher (starting first with aligned data, then followed by increasing fractions of inconsistent data) reaches 81.1% aligned and 65.6% inconsistent success, compared with 85.6% and 71.5% for our two-stage procedure with the teacher (Appendix A.11, Table 35). Appendix A.11 now states that the teacher is not essential. It acts as a standard distillation term that limits forgetting of aligned behavior during fine-tuning and improves results modestly.
W15
Definitions of geometric inconsistency, deformable objects, visual noise, walkability and deformability
Response: These definitions were given in Appendix A.1, and short definitions are now included in Sections 3.2 and 4. We would also like to clarify a misreading that our Figure 2 caused: the cylinder is the agent, not an obstacle. The deformable object is a cloth curtain that replaces the central barrier and that the agent can push through. Walkability labels mark points on traversable floor, and deformability labels mark points belonging to a deformable surface. The Figure 2 caption now identifies the agent and the obstacles.
W16 / Q3
“how are goal states chosen?”
Response: We do not use intermediate goals. Each episode has a single final goal, given as one RGB image rendered at the goal position together with the map crop at that position. The planner minimizes the distance between the predicted terminal latent and this fixed goal latent at every replanning step, and no expert trajectory is used at test time. Section 3.6 now states this explicitly.
W17
“I did not understand Figure 3. ... The baselines are just running in circles?”
Response: The poor baseline paths were due to the previous planning configuration. We have regenerated the figure from the paper models under the planner-matched protocol, with corrected labels, and the caption now explains every element (start, goal and its success radius, and success and failure markers). The panels show held-out tasks in which GeoMamba-WM reaches the goal and both baselines do not, and the caption now refers to Table 1 for success rates over all tasks.
W18
“The baselines should include e2e trained models. A SAC baseline is proposed, but SoTA models trained in simulation are base on on-policy methods like PPO. What are the frozen backbones mentioned there?”
Response: The paper includes three end-to-end systems in Section 6.3 and Appendix A.4. These are NeDreamer, trained with online RL; 3D-Belief, which uses an explicit map and planner; and SAC actor-critics trained on frozen PERSIST-S and GeoMamba-WM representations. "Frozen backbones" means that the pretrained world-model encoders are frozen, and only the SAC actor and critic are trained on top of them, with 25k online transitions. We chose SAC because it is off-policy and considerably more sample-efficient at this interaction budget than PPO, which typically requires orders of magnitude more interaction. We agree that PPO trained at scale would be the stronger reference for a pure RL comparison.
W19 / W20
“L391 3D continuous control is mentioned. Why 3D?” / “L92 this sentence talks about things which have not yet been introduced and actually never are.”
Response: The scenes and observations are three-dimensional, but the action is a planar displacement. We have rewritten the sentence that referred to 3D continuous control, and Section 3.6 now states that actions are planar. We have also replaced the sentence at L92 with one that introduces the per-point deformability label and the cloth curtain it marks.
Q1
“Do you assume knowledge of GT positions or do you handle noise?”
Response: The agent's position is used only to select the map crop around the agent and is never given to the planner as coordinates. To simulate how this would translate to real-life scenarios, our odometry-drift ablation (Appendix A.6.4, Table 20) offsets this position with an accumulated random walk of up to 0.1 m or 0.4 m; inconsistent success drops by only 2.5–5.0 points, whether or not drift is applied during training.
Q2
“What is the action space?”
Response: The action is a continuous planar displacement of at most 0.4 m in a world-aligned frame, with a fixed camera heading. This is now stated in Section 3.6 and detailed in Appendix A.1.1.
Response to Reviewer oUA5 (4/4)
Q4
“What is the planning horizon and how is it related to goal states?”
Response: Under the planner-matched protocol, CEM optimizes 5-step action sequences, executes all five actions, and replans. The cost is the distance between the final predicted latent and the fixed goal latent, so the horizon is not tied to the goal. The goal can lie beyond the horizon, and the planner moves toward it by repeated replanning.
Q5
“Why SR and Soft-SPL but no SPL?”
Response: We now report SPL for every column in Table 6 (Appendix A.3). GeoMamba-WM reaches 40.0% on aligned scenes and 32.3% on inconsistent scenes, compared with 37.1% and 31.4% for DINO-WM + Geometry and 27.9% and 22.5% for DINO-WM, so the ranking is the same as for SR and SoftSPL. We originally reported SoftSPL because it also credits progress in failed episodes, which separates models that nearly reach the goal from those that wander.
Summary
We thank the reviewer again for the particularly detailed and thorough review. In response, we have clarified the task and its positioning, corrected the description and naming of the recurrent module, defined all components and benchmark elements in the main text, regenerated Figure 3 and added SPL. We have also evaluated the method on Habitat ImageNav in real-world Matterport3D scans, alongside a planner-matched re-evaluation of all baselines with independently trained models. We believe these revisions address the reviewer's concerns, and we respectfully ask the reviewer to reconsider their assessment in light of them.
Response to Reviewer oUA5 (1/4)
We thank the reviewer for an extremely detailed and thorough review. Several of the reviewer's points led to substantive changes, including the naming and description of the recurrent module, the clarity of the method section, the missing definitions, the SPL metric, the related-work positioning and Figure 3. We address every point below, in the order of the review.
S1 / W1
Response: We agree that end-to-end RL and IL agents are currently the strongest navigation systems in simulation, and our paper does not claim otherwise. Our question is a different one: whether a reward-free world model, trained offline, can plan when its prior map is out of date. This setting matters when rewards are unavailable, when online interaction is expensive, or when a single model must serve many goals without retraining. Two results bear directly on the reviewer's concern (Table 3). First, under the planner-matched protocol, GeoMamba-WM with CEM reaches 84.4% aligned and 64.8% inconsistent success from offline data alone, close to NeDreamer, which is trained online with a distance-based reward and uses its own actor-critic (88.9% and 68.8%), and with higher SoftSPL (36.8 and 29.7 vs. 25.9 and 22.2). Second, the same frozen GeoMamba-WM representation with an RL controller (SAC) reaches 100% aligned and 86.4% inconsistent success, which shows that the representation is not the bottleneck.
W2
Response: We would like to clarify two points. First, the task is image-goal navigation, not PointGoal navigation: the goal is specified by an RGB image and the map crop at the goal, and its coordinates are never given to the agent or used in the planning cost. We have replaced the phrase "simple point-to-point navigation", which invited this reading. Second, the 30 evaluation layouts are disjoint from the 80 training layouts, and the 2021 work cited for the environment is the ThreeDWorld simulator rather than a navigation task. The single-room topology is deliberate, so that differences between models can be attributed to how they handle map–observation inconsistency, as explained in our response to W3. We agree, however, that it limits the evidence, and we have therefore added Habitat ImageNav in 11 unseen Matterport3D houses. The results are in Section 6.4 and Appendix A.5.
W3
Response: We agree that this sentence was unclear, and we have rewritten it in Section 4: "We keep the room simple so that failures can be attributed to map-observation mismatch rather than to long-range route finding, which we test separately in multi-room houses in section 6.4."
W4
Response: We would like to clarify that the sentence at L162–164 referred to deformable obstacles rather than dynamic objects. However, it was poorly worded, and we have removed it. We built GOI-NAV in TDW because the benchmark requires a cloth curtain that the agent can push through, which TDW supports through its Obi cloth simulation. We now also evaluate the method in Habitat (Section 6.4).
W5
Response: We thank the reviewer for these references. We have added all three to the related work (Section 2). MultiON places objects in Habitat scenes for sequential object-goal navigation, ViKiNG performs long-range outdoor navigation with geographic hints, and Janny et al. show that agents trained entirely in simulation on HM3D transfer to real robots. Our setting differs from all three: the prior map is given but out of date, and the planner is a reward-free world model trained offline.
W6
Response: Section 3 now opens with an overview paragraph that names each component, its role and its subsection, so that the details that follow can be placed within the overall design.