- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes OpenNWM, which uses action-free videos to train navigation world models. A latent action model trained with pixel reconstruction plus action supervision provides latent actions to pretrain a CDiT-B/2 world model on NavAnywhere ; the latent-action adapter is then replaced by a physical-action adapter for post-training and CEM planning. The paper claims state-of-the-art visual prediction and navigation on in-domain and out-of-domain benchmarks.
Critical strengths
- The problem is well motivated. Action-labeled navigation data is limited, and using action-free video is a sensible direction.
- The ablations on pretraining conditioning and LAM supervision (Tables 4-5) are informative.
Critical weaknesses
- The navigation gains are very small (RECON ATE 1.13 to 1.12, SCAND 1.01 to 0.98 in Table 2).
- Weak experimental comparison. It is not stated which baselines were re-run and which numbers were taken from prior papers (e.g., the NWM row is listed at 279M parameters, while its RECON numbers match those reported for the original larger model), and the ID-average advantage over RAE-NWM mostly comes from RAE-NWM's collapse on TartanDrive (LPIPS 0.662, Table 9). It would be better if the authors could unify all baselines under the same protocol or clarify the comparison.
- Missing controls for the main claim. There is no no-pretraining baseline, and no comparison with pseudo-action pretraining (an inverse dynamics model or VO/camera poses, which many NavAnywhere sources already have). Without these, it is unclear why latent actions are needed.
- Several claims are overstated. The claim that UNLAs "disentangle agent motion from environmental dynamics" is never tested, and the latent adapter is discarded at post-training, so there is no "shared latent-action interface" at test time. NavAnywhere is also about 75% Ego4D (13.09M of 17.5M frames), which weakens the diversity claim.
Questions for authors
As in weaknesses.
3 comments
W3. No-pretraining and pseudo-action controls.
R3: We have revised Table 4 to explicitly label the no-pretraining baseline and clarify how the geometry-based pseudo-actions are obtained. We present the two comparisons separately below.
1. Does latent-action pretraining improve downstream performance? The baseline skips world-model pretraining and trains the world model directly on physical actions. Both variants use the same architecture and downstream training data. Latent-action pretraining improves all six metrics, reducing direct LPIPS on Office Go2 from 0.628 to 0.587 and navigation ATE by 6.8% on RECON and 24.1% on SCAND (Response Table 2).
Table 2. Effect of latent-action pretraining on visual prediction and navigation.
| Pretraining setting | Office Go2 direct LPIPS ↓ | Go Stanford AR LPIPS ↓ | RECON ATE ↓ | RECON RPE ↓ | SCAND ATE ↓ | SCAND RPE ↓ |
|---|---|---|---|---|---|---|
| No pretraining | 0.628 | 0.667 | 1.198 | 0.367 | 1.252 | 0.316 |
| Latent-action pretraining | 0.587 | 0.658 | 1.117 | 0.356 | 0.950 | 0.267 |
2. How do learned latent actions compare with geometry-based pseudo-actions? We use VGGT-Omega to estimate camera trajectories from RGB videos and derive pseudo-actions from relative pose changes. The two variants use the same video subset, time conditioning, world-model training schedule, and downstream evaluation protocol. Learned latent actions perform better on five of six metrics, including both visual-prediction metrics and navigation ATE on both datasets. Geometry-based pseudo-actions yield slightly lower RECON RPE (0.353 vs. 0.356) (Response Table 3).
Response Table 3. Effect of the action signal used during world-model pretraining.
| Pretraining action signal | Office Go2 direct LPIPS ↓ | Go Stanford AR LPIPS ↓ | RECON ATE ↓ | RECON RPE ↓ | SCAND ATE ↓ | SCAND RPE ↓ |
|---|---|---|---|---|---|---|
| Geometry-based pseudo-actions (VGGT-Omega) | 0.594 | 0.668 | 1.152 | 0.353 | 1.039 | 0.283 |
| Learned latent actions | 0.587 | 0.658 | 1.117 | 0.356 | 0.950 | 0.267 |
Overall, these results demonstrate the effectiveness of latent action pretraining in leveraging diverse action-free visual experiences to improve world model learning and downstream navigation performance. Furthermore, our learned latent actions offer an advantage over geometry-based pseudo-actions by capturing more effective motion representations from visual experiences, providing stronger motion conditioning for world-model pretraining.
W4: Claims and dataset diversity.
R4: We agree that our experiments do not directly test whether agent motion is disentangled from environmental dynamics. We have revised the claim to state that physical-motion supervision encourages motion-related structure in the latent representation. We have also clarified that the shared latent-action interface is used only during world-model pretraining. During post-training, we replace the latent-action adapter with a physical-action adapter. At inference, the planner samples physical waypoint actions directly, without using the LAM.
We acknowledge that NavAnywhere is imbalanced in its raw data composition, with Ego4D accounting for 74.8% of the sampled frames (13.09M out of 17.50M, Appendix Table 6). Importantly, source-balanced sampling is already adopted in our training pipeline to prevent larger datasets from dominating the training process. With this strategy, Ego4D contributes only 27.1% of the samples drawn from NavAnywhere during pretraining. Moreover, Ego4D itself covers diverse activities and a wide range of indoor and outdoor environments. We have clarified the source-balanced sampling strategy in the revised PDF.
Dear Reviewer eXj8,
We sincerely appreciate your time and effort in reviewing our manuscript and offering valuable suggestions. As we're near the end of discussion period, we would like to confirm whether our responses have effectively addressed your concerns. We have provided detailed responses to your concerns, and we hope they have adequately addressed your issues.
Best regards.
Thank you for the constructive feedback. We address each weakness and question below.
W1: Small navigation gains.
R1: OpenNWM is designed to improve generalization to unseen environments by learning from videos without action labels. During pretraining, latent actions provide a common representation of motion across videos, allowing the world model to learn how motion affects future observations across diverse scenes. We therefore expect the benefit to be greater in unseen environments.
On RECON and SCAND, both models are trained and evaluated within the same domains. This places less demand on cross-domain generalization and may partly explain the modest ATE reductions from 1.13 to 1.12 on RECON and from 1.01 to 0.98 on SCAND. On the unseen Office Go2 dataset, OpenNWM reduces ATE from 3.03 to 2.89, a 4.6% reduction relative to NWM (Table 2 of the paper).
To further assess generalization, we evaluated both models offline on recordings collected with a Unitree Go2 quadruped in unseen real-world environments. We used the same image-goal navigation protocol for both models, without adapting either model to the recordings. Across all test recordings, OpenNWM reduced ATE by 8.1% and RPE by 8.8% relative to NWM (Table 1).
Table 1. Offline image-goal navigation on Unitree Go2 recordings from unseen real-world environments.
Side-by-side videos are available on our Real-world supplementary page.
W2: Baseline provenance and protocol.
R2: All baseline results in Table 1 of the paper are local evaluations of released checkpoints under the same visual-prediction benchmark protocol. In Table 2 of the paper, the results for GNM, NoMaD, CompACT, Garrido et al., and the RECON/SCAND entries of RAE-NWM are paper-reported; NWM/OpenNWM and all Office Go2 results are locally evaluated. We have marked every paper-reported number with a superscript asterisk (*) in the revised PDF. Both paper-reported and locally evaluated results follow the same benchmark tasks and metric definitions.
Regarding the reviewer’s concern about the NWM model size, we rechecked the checkpoint and evaluation outputs. The NWM baseline in Table 2 uses the released CDiT-B checkpoint with 279M parameters. We additionally evaluated the CDiT-XL checkpoint on with 1.096B parameters on RECON:
The CDiT-B results round to 1.13/0.35. Although these match the original paper’s XL results at two-decimal precision, our checkpoint verification confirms that they were obtained using CDiT-B, with no model-size mismatch.
Excluding TartanDrive, OpenNWM achieves comparable in-domain visual-prediction performance to RAE-NWM (Appendix Table 9 in the paper). It also achieves better visual-prediction results on the unseen Go Stanford, TUM RGB-D, Planetary Rover, and Office Go2 domains (Table 1 in the paper), and better navigation performance on RECON, SCAND, and Office Go2 (Table 2 in the paper). These results highlight OpenNWM's consistent advantages in navigation and its ability to generalize to unseen environments. We have revised the manuscript to clarify the provenance of the baseline results and the evaluation protocols used.