acceptodds
Under review as a conference paper at ICLR 2027

WNM-3D: A World Navigation Model with 3D Scene Conditioning for Closed-Loop VLN

14 views

Abstract

Recent vision-language navigation (VLN) systems increasingly adapt pretrained vision-language models (VLMs) into vision-language-action (VLA) policies that map egocentric observations and language instructions directly to navigation actions. Although semantically capable, such action-centric training does not explicitly model how the agent's visual observations evolve under its predicted motion. Generative world-action models (WAMs) jointly predict future observations and actions, yet existing WAMs for continuous VLN do not condition joint future-view and action generation on geometry-aware representations inferred from the observation history. We present WNM-3D, a generative World Navigation Model with 3D scene conditioning for continuous VLN. A frozen feed-forward geometry encoder and trainable 3D Scene-to-Token Adapter convert monocular RGB history into a fixed-length geometry-aware prefix that persistently conditions joint future-view and action generation. We train WNM-3D with supervised world-action learning, DAgger-style adaptation on policy-visited states, and Counterfactual DanceGRPO refinement. Across 3DGS-based GN-Bench and VLN-CE, WNM-3D outperforms the backbone-matched WNM-2D counterpart and strong VLN baselines on GN-Bench. We further demonstrate closed-loop deployment on a physical robot, while component and stage-wise ablations support the Adapter design and the complementary roles of DAgger-SFT and DanceGRPO.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language Navigation, Embodied AI, World-Action Models, 3D Scene Understanding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_aVhGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes WNM-3D, a geometry-conditioned generative world-action model for continuous vision-language navigation. The method uses a frozen VGGT-Ω geometry encoder and a trainable 3D Scene-to-Token Adapter to extract geometry-aware representations from historical RGB observations. These representations serve as persistent conditioning tokens for joint future-view and navigation-action generation through a diffusion transformer. The framework adopts a three-stage training procedure: supervised world-action learning, DAgger-style adaptation, and Counterfactual DanceGRPO refinement. Experiments on GN-Bench and VLN-CE show improvements over a backbone-matched WNM-2D baseline.

Critical strengths
  1. A meaningful way to integrate 3D scene representations into generative navigation. The proposed framework brings history-derived geometric information into a joint world-action model, allowing future-view and action generation to share persistent spatial context.

  2. Strong empirical improvements with a controlled backbone comparison. The paper includes a backbone-matched WNM-2D baseline that shares the world-action architecture, training curriculum, and inference procedure.

Critical weaknesses
  1. The necessity of joint future-view generation is insufficiently validated. Although the model jointly generates future visual latents and navigation actions, the predicted visual latents are neither decoded nor fed back into the observation history during inference. Therefore, it remains unclear whether joint visual generation is actually necessary for effective navigation or mainly serves as auxiliary supervision during training. The paper lacks an action-only variant to isolate the contribution of future-view generation. This is a significant limitation because joint world-action modeling is a central motivation of the proposed framework.

  2. Computational overhead and real-time feasibility are not adequately evaluated. WNM-3D introduces a relatively expensive geometry encoder that processes 33 historical frames at 512 × 512 resolution at each replanning step, in addition to the large world-action diffusion backbone. However, the paper does not report end-to-end inference latency, geometry encoding time, GPU memory consumption, or replanning frequency. Without these measurements, it is difficult to determine whether the performance improvements justify the additional computational cost, particularly for real-time robotic navigation.

  3. Limited evidence of improved generalization and advantages over existing methods. Although WNM-3D consistently improves over WNM-2D, the gains are substantially larger on seen environments than on unseen environments. On GN-Bench, the success-rate improvement is 13.1 points on seen scenes but only 7.6 points on unseen scenes, and the seen–unseen performance gap remains substantial. Furthermore, WNM-3D does not outperform the strongest existing methods on VLN-CE, while comparisons with external baselines are not matched in terms of training data or model capacity. These results leave open questions about the generalization benefits and practical advantages of the proposed approach.

Questions for authors

Please refer to the weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer aVhG [part 1]

We thank the reviewer for recognizing the value of geometry-conditioned world-action modeling and our controlled backbone comparison, and for highlighting concerns regarding future-view generation, computational overhead, and generalization. We provide point-by-point responses supported by additional ablations, inference profiling, and updated benchmark results.


Weakness 1: Contribution of Future-View Supervision to Action Learning

Although predicted visual latents are not decoded or fed back into the observation history, they remain active during joint denoising and interact bidirectionally with action tokens within each future block. To access the contribution of future-view supervision to action learning, we train an action-only variant using the same VGGT-Ω encoder, 3D Scene-to-Token Adapter, and 450-token geometry prefix. Unlike WNM-3D, it removes future visual tokens and the visual flow-matching objective, optimizing only the action flow-matching objective. Both variants use the same backbone initialization, Stage-I expert data, training updates, and optimization settings. We evaluate both Stage-I checkpoints on GN-Bench Unseen.

VariantFuture-View ObjectiveNE ↓OS ↑SR ↑SPL ↑
WNM-3D (Stage-I)✓5.047.843.141.6
Action-Only (Stage-I)✗5.836.534.834.4

Compared with Action-Only, WNM-3D reduces NE by 0.8 and improves OS, SR, and SPL by 11.3, 8.3, and 7.2 percentage points, respectively. These results demonstrate that future-view supervision provides substantial benefits to Stage-I action learning beyond geometry-aware history conditioning. This Stage-I ablation assesses the contribution of future-view supervision during training; it does not separately establish the necessity of visual denoising at inference. We have added this ablation to the revised manuscript.


Weakness 2: Computational Overhead and Real-Time Feasibility

We profile WNM-2D and WNM-3D on a single NVIDIA H100 under matched hardware and precision settings. The table below reports component-wise latency, end-to-end inference time, peak GPU memory, and inference-limited replanning rate. To reduce overhead, we optimize VGGT-Ω and DiT inference using custom FlashAttention-3 and WGMMA kernels.

ComponentWNM-2D (s)WNM-3D (s)
History encoder (VAE / VGGT-Ω)0.0490.425
3D Adapter—0.010
DiT denoising (16 steps)1.0521.052
Action decoding<0.001<0.001
Other runtime overhead0.2280.277
End-to-end latency1.3291.764
Peak GPU memory (GB)31.032.8
Inference-limited rate (Hz)0.7520.567

Runtime Overhead Breakdown. For WNM-3D, the 0.277-s runtime overhead comprises input transformation (0.098 s), text encoding (0.058 s), CLIP/image conditioning (0.064 s), target-video VAE encoding (0.049 s), and approximately 0.008 s for noise generation, scheduler setup, tensor transfers, and other wrapper operations. For WNM-2D, the 0.049-s VAE encoding cost is already included in the history-encoder row and is excluded from other runtime overhead, giving 0.228 s. Component times and totals are rounded; action decoding takes less than 1 ms.

These rates reflect model inference only; actual closed-loop frequency also depends on action execution and observation acquisition. Compared with WNM-2D, WNM-3D adds 0.435 s (32.7%) latency and 1.8 GB peak memory, while improving GN-Bench Unseen SR by 7.6 points (45.9% to 53.5%) and VLN-CE Val-Unseen SR by 15.9 points (52.3% to 68.2%). These results quantify the computational–performance trade-off.

AuthorsOpenReview ↗

Response to Reviewer aVhG [part 2]

Weakness 3: Generalization and External Comparisons

Seen–Unseen Performance. WNM-3D improves SR by 13.1 percentage points (pp) on Seen and 7.6 pp on Unseen. Relative to WNM-2D, these gains are comparable: (88.7 − 75.6)/75.6 = 17.3% and (53.5 − 45.9)/45.9 = 16.6%, respectively. Although the Seen–Unseen gap increases from 29.7 to 35.2 pp, geometry conditioning improves absolute performance on both splits.

External Baseline Comparisons. External methods are compared on the same benchmark splits and evaluation metrics using their respective reported configurations. Differences in model capacity and training data are inherent to cross-method comparisons; these results therefore provide benchmark-level performance references rather than controlled architectural comparisons.

Updated VLN-CE Results. Following submission, we retrained both WNM variants. The retrained WNM-3D achieves 68.2% SR and 74.9% OS, slightly surpassing GN-BAE's reported SR (67.7%), although its OS remains slightly lower (75.6%). We have updated the results and revised our comparative claims accordingly.

Thank you very much for your constructive feedback. We have carefully addressed your comments in our response and updated the revised manuscript accordingly. We hope these changes have helped clarify the issues you raised and would greatly appreciate any further feedback when you have a chance. We look forward to hearing your thoughts and continuing the discussion.

AuthorsOpenReview ↗

Gentle Follow-up on Our Rebuttal

Dear Reviewer aVhG,

Thank you again for your valuable feedback. With only two days remaining in the discussion period, we would sincerely appreciate it if you could take another look at our responses and share your thoughts on whether your concerns have been resolved.

We remain happy to address any outstanding questions or provide further clarification.

Thank you very much for your time and consideration.

Reviewer_C3hNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

WNM-3D conditions joint future-view and action generation on geometry-aware features from monocular RGB history. A frozen VGGT-Ω encoder plus a trainable adapter builds a prefix for a DreamZero-style world-action diffusion model. Training uses offline supervision, DAgger-SFT, and counterfactual DanceGRPO. It beats WNM-2D on GN-Bench, is competitive on VLN-CE, and includes ablations plus a real-robot demo.

Critical strengths

The conditioning really helps navigation. With the same backbone and 450-token prefix, GN-Bench Unseen SR goes from 45.9% to 53.5%, and R2R Val-Unseen SPL from 45.3% to 57.9%. The GN-Bench gain is already there after Stage I (35.6% to 43.1%), before DAgger data, so it’s not just better collected data. The ablations are useful too: removing deformable resampling drops Stage-I Unseen SR from 43.1% to 32.0%; DAgger helps SR but hurts SPL, and DanceGRPO brings SPL back from 38.2% to 46.6%.

Critical weaknesses

The comparison doesn’t isolate geometry. WNM-3D adds VGGT-Ω at 512×512 and a 23.97M adapter, while WNM-2D uses VAE features at 160×320. Matching backbone and token count doesn’t control input resolution, pretrained representation, or adapter capacity. Table 3 keeps VGGT throughout, so attribution is still unclear. Also, the visual-action consistency claim isn’t directly shown. The tables evaluate navigation; the figures show rollouts, not generated futures checked against executed actions. Stage III compares against fixed reference observations, which may not match sampled off-trajectory actions. The flow reward only checks net XY direction for moving chunks and ignores yaw. I buy the navigation gains, but not yet the world-modeling claim.

Questions for authors
  1. Does the Stage-I gain hold against a pretrained appearance encoder matched on resolution, adapter size, prefix length, and training budget?
  2. How should generated views be scored against fixed reference video when sampled actions leave the recorded path? Can a held-out simulator comparison with observations from those sampled actions show better consistency than WNM-2D?
  3. Does counterfactual DanceGRPO beat ordinary DanceGRPO with the same rewards and matched sampling/update budgets?
5 comments
AuthorsOpenReview ↗

Response to Reviewer C3hN [part 1]

We thank the reviewer for recognizing the navigation improvements and the effectiveness of our adapter design, while raising concerns about geometry attribution, action-conditioned visual consistency, and counterfactual optimization. We address each point through clarifications and additional controlled experiments, as detailed below.


Weaknesses

Yaw and the flow reward. The optical-flow consistency term provides only translational feedback, with inferred flow yaw used for diagnostics only. Stage-III yaw supervision is provided by path-heading and yaw-rate consistency for moving actions, and ground-truth orientation supervision for pure-yaw controls. An additional flow-derived yaw term yielded no improvement and was therefore omitted.


Q1: Stage-I Gains under Matched Appearance Controls

To test whether adapter capacity or input resolution alone can explain the Stage-I gain, we introduce two complementary pretrained VAE-based appearance controls at 160×320 and 512×512. Both use a 23.97M-parameter adapter and produce 450 tokens; the latter directly matches VGGT-Ω's input resolution. All variants share the same DiT architecture, target-video resolution, Stage-I training data, number of updates, batch size, and optimization protocol. All variants are evaluated after Stage-I offline expert SFT only, without DAgger or RL.

ModelHistory ResolutionEncoderAdapter ParamsPrefix TokensNE ↓OS ↑SR ↑SPL ↑
WNM-2D160×320VAE04505.637.235.635.3
WNM-2D + Adapter160×320VAE23.97M4505.438.136.035.5
WNM-2D + Adapter512×512VAE23.97M45013.826.822.421.0
WNM-3D512×512VGGT-Ω23.97M4505.047.843.141.6

The results reveal three findings. First, adding a 23.97M-parameter adapter to native-resolution WNM-2D yields negligible SR gains (35.6% to 36.0%), showing little benefit from added adapter capacity in this control. Second, increasing VAE history resolution to 512×512 degrades SR to 22.4%, consistent with a mismatch to Wan's native 160×320 interface: the pre-adapter latent grid expands from 5×10 to 16×16 per frame (450 to 2,304 tokens), while altering the aspect ratio. Third, under matched 512×512 resolution, adapter capacity, and prefix length, WNM-3D achieves 43.1% SR, outperforming the 512×512 appearance control by 20.7 pp and the native-resolution control by 7.1 pp. Within these tested controls, increasing VAE adapter capacity or history resolution does not reproduce WNM-3D's Stage-I gain. Differences in pretrained encoder representations remain, so this experiment does not fully isolate the contribution of geometry.

AuthorsOpenReview ↗

Response to Reviewer C3hN [part 2]

Q2: Action-Conditioned Visual Consistency

Fixed-reference observations. We use fixed references as a computational trade-off. Each update samples 1,024 branches (64 prompts × 16 branches); generating action-matched targets would require online simulator execution and rendering for every branch at every update. We therefore use pre-recorded observations as scalable visual regularizers, not branch-specific ground-truth futures, and do not treat them as direct verification of full action-conditioned consistency.

Held-out simulator evaluation. To directly assess action-conditioned visual consistency, we evaluate WNM-2D and WNM-3D on all 5000 GN-Bench Unseen episodes. Given identical initial simulator states, instructions, and observation histories, each model generates 32 future frames and 32 actions in four aligned blocks. For this first-block diagnostic, we execute only the first 8-action block in the simulator and compare its eight predicted frames against the corresponding simulator-rendered observations. Each model yields eight valid frame pairs in every episode, totaling 40,000 pairs per model; no episode contributes fewer pairs. We report PSNR, SSIM, and LPIPS over the executed frames. The simulator is reset to the same initial state for each model, while each model is evaluated under its own sampled actions. Metrics are averaged over the eight aligned frame pairs within each episode and then across episodes.

ModelPSNR ↑SSIM ↑LPIPS ↓
WNM-2D14.9410.4970.450
WNM-3D14.2650.4890.456

Results. WNM-2D achieves slightly better scores on all three visual metrics (PSNR: 14.941 vs. 14.265; SSIM: 0.497 vs. 0.489; LPIPS: 0.450 vs. 0.456). Thus, this evaluation does not establish a visual-fidelity advantage for WNM-3D. It nevertheless directly evaluates each model's predicted first-block views against observations produced by its own sampled actions. These visual metrics measure prediction fidelity rather than navigation effectiveness, which is independently evaluated by SR and SPL.


Q3: Added value of counterfactual optimization

We compare standard and counterfactual DanceGRPO on GN-Bench Unseen, both initialized from the same Stage-II checkpoint, using identical visual, navigation, and stopping reward definitions and weights, sampling budgets, and update steps.

MethodNE ↓OS (%)SR (%)SPL (%)
Standard DanceGRPO4.754.144.639.3
Counterfactual DanceGRPO4.162.253.546.6

Counterfactual DanceGRPO improves NE, OS, SR, and SPL by 0.6, 8.1, 8.9, and 7.3 percentage points, respectively. Unlike standard full-trajectory replay, it shares non-intervened randomness across counterfactual branches and localizes gradient replay to the intervened denoising transition, while evaluating rewards on complete rollouts covering all four action chunks. This enables transition-level credit assignment without shortening the prediction horizon. These results demonstrate the added value of counterfactual optimization, although they do not isolate individual components.

Computational Implications. Mean actor-update time over two updates is 163.62 s for standard full replay and 33.43 s for counterfactual localized replay (4.9× speedup). Both methods were profiled under identical settings on 8 H100 GPUs, using 64 prompts, eight rollouts per prompt, and an actor micro-batch size of 16. Timing includes replay forward passes, backpropagation, and optimizer updates, and reflects actor-update rather than overall training acceleration.

We sincerely appreciate your valuable comments and constructive suggestions, which have helped us improve our work. We have made every effort to address your concerns in our response and revised manuscript. We would be grateful to hear your thoughts on whether our revisions have sufficiently addressed the points you raised. We look forward to continuing the discussion and welcome any further feedback.

Reviewer_18fuOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes WNM-3D, a generative world-action model (WAM) for continuous vision-language navigation (VLN).

It uses a frozen 3D geometry encoder (VGGT-Omega) and a trainable 3D Scene-to-Token Adapter to turn monocular RGB history into a fixed-length 3D token prefix. Conditioned on this prefix, a Diffusion Transformer (DiT) jointly predicts future video latents and navigation actions. The model is trained in three stages: offline SFT, closed-loop DAgger-SFT, and Counterfactual DanceGRPO.

On GN-Bench and VLN-CE, WNM-3D consistently beats its 2D baseline (WNM-2D) and shows working demos on a real robot.

Critical strengths
  1. Clean 2D vs. 3D controlled comparison. WNM-2D and WNM-3D share the exact same 5B DreamZero backbone, prefix length (450 tokens), attention mask, and training stages. WNM-3D improves SR on both GN-Bench Unseen and VLN-CE Val-Unseen, confirming that 3D VGGT-Omega history tokens are more effective than 2D VAE history tokens.
  2. Practical 3D Scene-to-Token Adapter. The adapter compresses dense 33-frame VGGT-Omega feature grids into 450 tokens using only 24M parameters via anchored 3D deformable resampling and factorized attention.
Critical weaknesses
  1. Incremental combination with overstated novelty. The paper reads like a system integration of very recent works: the joint video-action flow backbone and block-causal attention come from DreamZero (Ye et al., 2026); the 3D encoder is frozen VGGT-Omega (Wang et al., 2026a), which was already shown to help VLAs and used in VLN world-action modeling by FutureNav (Zhang et al., 2026b); and the three-stage curriculum (A* SFT -> closed-loop DAgger -> group-relative RL) directly mirrors GN0/GN-BAE (Li et al., 2026), just swapping DAPO for DanceGRPO (Xue et al., 2025) to fit a flow model. Claiming the three-stage curriculum as a core contribution overstates novelty, especially since GN0 already showed the same finding that DAgger is needed before RL.
  2. Unjustified video-generation overhead and no reward ablations. At test time, WNM-3D runs 16 denoising steps to generate 32 future video frames and then throws them away (in Section 3.5). Recent VLN/WAM works like FutureNav and GeoSem-WAM avoid test-time video generation to keep inference fast. The paper provides no ablation showing that joint video denoising beats an action-only policy with the same 3D adapter, nor does it ablate the heavily engineered Stage-III rewards (in Appendix A.4) to show whether the visual reward stream helps at all.
  3. Missing latency numbers and weaker VLN-CE performance. Running VGGT-Omega on 33 frames (512x512) plus a 5B video DiT at every replanning step is extremely heavy, yet no latency or FPS numbers are reported.
Questions for authors

Q1. Since GN0 (Li et al., 2026) already introduced the SFT -> DAgger -> GRPO/DAPO curriculum and A* oracle relabeling on GN-Bench, can you clarify what is fundamentally new in your training pipeline beyond adapting DanceGRPO with modality-routed credit?

Q2. Does generating future video frames actually help? What is the performance if you train an action-only flow head with the exact same VGGT-Omega + 3D Adapter prefix, or drop the visual reward (LvL_v) in Stage-III?

Q3. What is the wall-clock latency per replanning step (broken down by VGGT-Omega, Adapter, and DiT denoising) compared to GN-BAE and FutureNav?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 18fu [part 1]

We thank the reviewer for recognizing our controlled 2D–3D comparison and the 3D Scene-to-Token Adapter, and for raising concerns regarding methodological novelty, joint video-action generation, visual reward design, and computational efficiency. We address each comment point by point through clarifications and additional experiments.


Weakness 1:Incremental combination with overstated novelty

WNM-3D adopts DreamZero's joint video-action flow backbone and block-causal attention, and uses frozen VGGT-Ω as its geometry encoder; we do not claim these components as novel. Our contribution is the 3D Scene-to-Token Adapter, which maps multi-view geometric features into a fixed-length prefix that persistently conditions joint future-view and continuous-action generation. FutureNav is a VLM-based navigation framework that incorporates spatial features into an autoregressive policy and trains auxiliary inverse/forward dynamics and future spatial-state objectives. In contrast, WNM-3D uses a persistent history-derived geometry prefix to condition a shared DiT throughout joint denoising of future visual latents and continuous action chunks. Table 4 ablations isolate the contributions of anchored deformable resampling and factorized spatiotemporal refinement.


Weakness 3:Weaker VLN-CE performance

Following submission, we retrained both WNM variants while keeping the validation split and evaluation protocol unchanged. The retrained WNM-3D achieves 68.2% SR and 74.9% OS, attaining the highest SR among compared methods with publicly available code and checkpoints (GN-BAE: 67.7%).


Q1: Stage-III RL Novelty

We apologize for overstating the novelty of the three-stage curriculum and A* oracle relabeling; neither is claimed as a contribution. Stage-I adopts DreamZero's joint video-action flow backbone and offline expert SFT framework, whereas Stage-II and Stage-III training pipelines were independently implemented by us. GN0 describes an A* SFT–DAgger–DAPO training sequence, with GRPO-style advantage estimation. We implemented the closed-loop DAgger-SFT and Stage-III RL pipelines for WNM-3D's joint continuous action-chunk and visual-latent flow setting.

Beyond adapting DanceGRPO and modality-routed credit, our algorithmic contribution is denoising-stratum counterfactual credit assignment: matched branches share initial latents and all exogenous noise except at the intervention; rewards are evaluated on full rollouts, while gradients are replayed only through the intervened transition. Visual and navigation/stopping rewards use their respective likelihood ratios.

We have revised the contribution claims accordingly in the manuscript and provided the framework's source code through the supplementary material submitted on OpenReview.

AuthorsOpenReview ↗

Response to Reviewer 18fu [part 2]

Weakness 2 & Q2: Video Generation and Visual Reward Ablations

Role of Future-View Generation. Although predicted visual latents are neither decoded into RGB nor fed back into the observation history at inference, they are not simply discarded during generation. Visual and action tokens interact bidirectionally within each future block under our block-causal attention, allowing visual latents to participate in joint denoising. To assess the contribution of future-view supervision, we train a Stage-I action-only variant with the same VGGT-Ω encoder, 3D Adapter, and 450-token geometry prefix, but without future visual tokens or the visual flow-matching objective. Both Stage-I checkpoints are evaluated on GN-Bench Unseen.

VariantFuture-View ObjectiveNE ↓OS ↑SR ↑SPL ↑
WNM-3D (Stage-I)✓5.047.843.141.6
Action-Only (Stage-I)✗5.836.534.834.4

Future-view supervision reduces NE by 0.8 and improves OS, SR, and SPL by 11.3, 8.3, and 7.2 percentage points, respectively, demonstrating its contribution to Stage-I action learning beyond geometry-aware conditioning.

Visual Reward Ablation. We further remove the Stage-III visual reward (Lv), starting from the same Stage-II checkpoint while retaining joint video-action generation and identical navigation/stopping rewards, counterfactual sampling budgets, and update steps. Both variants are evaluated on GN-Bench Unseen.

VariantLvLvNE ↓OS ↑SR ↑SPL ↑
Full WNM-3D✓4.162.253.546.6
WNM-3D w/o LvLv✗4.362.650.241.5

Removing Lv reduces SR and SPL by 3.3 and 5.1 percentage points, respectively, and slightly worsens NE (4.1 vs. 4.3), while OS changes marginally (62.2 vs. 62.6). These results indicate that the Stage-III visual reward provides a complementary optimization signal, primarily benefiting task completion and path efficiency.

We have added these ablations and the corresponding discussion to the revised manuscript.


Q3: Inference Latency and Efficiency

We profile WNM-3D on a single NVIDIA H100 and optimize VGGT-Ω and DiT inference using custom FlashAttention-3 and WGMMA kernels. We measured GN-BAE and FutureNav on a single NVIDIA H100 with batch size 1, using their publicly released checkpoints. The tables below report WNM-3D's component-wise and end-to-end latency and peak GPU memory, together with configuration-specific per-invocation latency references for GN-BAE and FutureNav.

ComponentWNM-3D latency (s)
VGGT-Ω0.425
3D Adapter0.010
DiT denoising (16 steps)1.052
Action decoding<0.001
Other runtime overhead0.277
End-to-end1.764
Peak GPU memory (GB)32.8

Runtime Overhead Breakdown. The 0.277-s runtime overhead comprises input transformation (0.098 s), text encoding (0.058 s), CLIP/image conditioning (0.064 s), target-video VAE encoding (0.049 s), and approximately 0.008 s for noise generation, scheduler setup, tensor transfers, and other wrapper operations.

MethodLatency (s) ↓
GN-BAE0.572
FutureNav0.743
WNM-3D1.764

Latency Analysis. The reported per-invocation latencies are 1.764 s for WNM-3D, 0.572 s for GN-BAE, and 0.743 s for FutureNav; their prediction workloads differ. At each replanning step, WNM-3D encodes 33 history frames with VGGT-Ω and performs 16-step joint denoising to generate 32 future frames and 32 actions. In contrast, GN-BAE generates trajectory tokens, while FutureNav's default inference activates only its action-policy branch, leaving auxiliary world-modeling heads inactive. These per-invocation figures describe different native inference workloads and do not establish a controlled speed comparison. This reflects a trade-off between explicit future-view generation and inference efficiency.

Thank you again for your thoughtful and constructive feedback. We have carefully considered your suggestions and made corresponding revisions in both our response and the revised manuscript. We would greatly appreciate hearing whether these updates have adequately addressed your concerns or if any issues remain. We look forward to further discussion and would be happy to provide additional clarification if needed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.