- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes WNM-3D, a geometry-conditioned generative world-action model for continuous vision-language navigation. The method uses a frozen VGGT-Ω geometry encoder and a trainable 3D Scene-to-Token Adapter to extract geometry-aware representations from historical RGB observations. These representations serve as persistent conditioning tokens for joint future-view and navigation-action generation through a diffusion transformer. The framework adopts a three-stage training procedure: supervised world-action learning, DAgger-style adaptation, and Counterfactual DanceGRPO refinement. Experiments on GN-Bench and VLN-CE show improvements over a backbone-matched WNM-2D baseline.
Critical strengths
-
A meaningful way to integrate 3D scene representations into generative navigation. The proposed framework brings history-derived geometric information into a joint world-action model, allowing future-view and action generation to share persistent spatial context.
-
Strong empirical improvements with a controlled backbone comparison. The paper includes a backbone-matched WNM-2D baseline that shares the world-action architecture, training curriculum, and inference procedure.
Critical weaknesses
-
The necessity of joint future-view generation is insufficiently validated. Although the model jointly generates future visual latents and navigation actions, the predicted visual latents are neither decoded nor fed back into the observation history during inference. Therefore, it remains unclear whether joint visual generation is actually necessary for effective navigation or mainly serves as auxiliary supervision during training. The paper lacks an action-only variant to isolate the contribution of future-view generation. This is a significant limitation because joint world-action modeling is a central motivation of the proposed framework.
-
Computational overhead and real-time feasibility are not adequately evaluated. WNM-3D introduces a relatively expensive geometry encoder that processes 33 historical frames at 512 × 512 resolution at each replanning step, in addition to the large world-action diffusion backbone. However, the paper does not report end-to-end inference latency, geometry encoding time, GPU memory consumption, or replanning frequency. Without these measurements, it is difficult to determine whether the performance improvements justify the additional computational cost, particularly for real-time robotic navigation.
-
Limited evidence of improved generalization and advantages over existing methods. Although WNM-3D consistently improves over WNM-2D, the gains are substantially larger on seen environments than on unseen environments. On GN-Bench, the success-rate improvement is 13.1 points on seen scenes but only 7.6 points on unseen scenes, and the seen–unseen performance gap remains substantial. Furthermore, WNM-3D does not outperform the strongest existing methods on VLN-CE, while comparisons with external baselines are not matched in terms of training data or model capacity. These results leave open questions about the generalization benefits and practical advantages of the proposed approach.
Questions for authors
Please refer to the weaknesses.
4 comments
Response to Reviewer aVhG [part 2]
Weakness 3: Generalization and External Comparisons
Seen–Unseen Performance. WNM-3D improves SR by 13.1 percentage points (pp) on Seen and 7.6 pp on Unseen. Relative to WNM-2D, these gains are comparable: (88.7 − 75.6)/75.6 = 17.3% and (53.5 − 45.9)/45.9 = 16.6%, respectively. Although the Seen–Unseen gap increases from 29.7 to 35.2 pp, geometry conditioning improves absolute performance on both splits.
External Baseline Comparisons. External methods are compared on the same benchmark splits and evaluation metrics using their respective reported configurations. Differences in model capacity and training data are inherent to cross-method comparisons; these results therefore provide benchmark-level performance references rather than controlled architectural comparisons.
Updated VLN-CE Results. Following submission, we retrained both WNM variants. The retrained WNM-3D achieves 68.2% SR and 74.9% OS, slightly surpassing GN-BAE's reported SR (67.7%), although its OS remains slightly lower (75.6%). We have updated the results and revised our comparative claims accordingly.
Thank you very much for your constructive feedback. We have carefully addressed your comments in our response and updated the revised manuscript accordingly. We hope these changes have helped clarify the issues you raised and would greatly appreciate any further feedback when you have a chance. We look forward to hearing your thoughts and continuing the discussion.
Gentle Follow-up on Our Rebuttal
Dear Reviewer aVhG,
Thank you again for your valuable feedback. With only two days remaining in the discussion period, we would sincerely appreciate it if you could take another look at our responses and share your thoughts on whether your concerns have been resolved.
We remain happy to address any outstanding questions or provide further clarification.
Thank you very much for your time and consideration.
Response to Reviewer aVhG [part 1]
We thank the reviewer for recognizing the value of geometry-conditioned world-action modeling and our controlled backbone comparison, and for highlighting concerns regarding future-view generation, computational overhead, and generalization. We provide point-by-point responses supported by additional ablations, inference profiling, and updated benchmark results.
Weakness 1: Contribution of Future-View Supervision to Action Learning
Although predicted visual latents are not decoded or fed back into the observation history, they remain active during joint denoising and interact bidirectionally with action tokens within each future block. To access the contribution of future-view supervision to action learning, we train an action-only variant using the same VGGT-Ω encoder, 3D Scene-to-Token Adapter, and 450-token geometry prefix. Unlike WNM-3D, it removes future visual tokens and the visual flow-matching objective, optimizing only the action flow-matching objective. Both variants use the same backbone initialization, Stage-I expert data, training updates, and optimization settings. We evaluate both Stage-I checkpoints on GN-Bench Unseen.
Compared with Action-Only, WNM-3D reduces NE by 0.8 and improves OS, SR, and SPL by 11.3, 8.3, and 7.2 percentage points, respectively. These results demonstrate that future-view supervision provides substantial benefits to Stage-I action learning beyond geometry-aware history conditioning. This Stage-I ablation assesses the contribution of future-view supervision during training; it does not separately establish the necessity of visual denoising at inference. We have added this ablation to the revised manuscript.
Weakness 2: Computational Overhead and Real-Time Feasibility
We profile WNM-2D and WNM-3D on a single NVIDIA H100 under matched hardware and precision settings. The table below reports component-wise latency, end-to-end inference time, peak GPU memory, and inference-limited replanning rate. To reduce overhead, we optimize VGGT-Ω and DiT inference using custom FlashAttention-3 and WGMMA kernels.
Runtime Overhead Breakdown. For WNM-3D, the 0.277-s runtime overhead comprises input transformation (0.098 s), text encoding (0.058 s), CLIP/image conditioning (0.064 s), target-video VAE encoding (0.049 s), and approximately 0.008 s for noise generation, scheduler setup, tensor transfers, and other wrapper operations. For WNM-2D, the 0.049-s VAE encoding cost is already included in the history-encoder row and is excluded from other runtime overhead, giving 0.228 s. Component times and totals are rounded; action decoding takes less than 1 ms.
These rates reflect model inference only; actual closed-loop frequency also depends on action execution and observation acquisition. Compared with WNM-2D, WNM-3D adds 0.435 s (32.7%) latency and 1.8 GB peak memory, while improving GN-Bench Unseen SR by 7.6 points (45.9% to 53.5%) and VLN-CE Val-Unseen SR by 15.9 points (52.3% to 68.2%). These results quantify the computational–performance trade-off.