acceptodds
Under review as a conference paper at ICLR 2027

FutureNav: Unified World-Action Modeling for Vision-and-Language Navigation

Abstract

Vision-and-language navigation (VLN) in continuous environments requires an agent to ground instructions in egocentric observations while maintaining spatial understanding across long action sequences. Recent navigation foundation models have shown strong progress by scaling vision-language models, but they often learn navigation primarily as direct action generation, without explicitly modeling world states or predicting their future evolution. We introduce FutureNav, a VLM-based unified world-action modeling framework for vision-and-language navigation. Specifically, FutureNav jointly encodes text, visual, and spatial features and feeds them into the LLM, and optimizes four objectives for simultaneous world and action modeling: an action policy objective for navigation action prediction, inverse and forward dynamics objectives for modeling state transitions, and a future generation objective for predicting future spatial states. This unified architecture strengthens action prediction while explicitly modeling the world, without sacrificing inference speed. Extensive experiments show that, with only a 4B-scale backbone, FutureNav achieves state-of-the-art performance on multiple VLN benchmarks and substantially outperforms prior VLN methods, paving the way toward future world-action models for VLN. We will release the code and models to support future research.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied Navigation, Embodied Intelligence

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_jxEaOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents FutureNav, a VLM-based framework for vision-and-language navigation (VLN) with auxiliary world-modeling objectives. It jointly trains four training objectives: action policy, inverse dynamics, forward dynamics, future generation within a VLM-based architecture, and leverages the auxiliary world modeling objectives to augment its VLN capability. At inference, the auxiliary prediction heads are removed and the model directly decodes navigation actions through the policy branch. Experiments on simulated benchmarks and real-world deployment demonstrate the effectiveness of the proposed approach.

Critical strengths

The implementation is simple and practical, allowing the model to benefit from additional supervision while retaining direct action decoding at inference. The ablation studies provide empirical evidence for the effectiveness of the world-action objectives and spatial latent representation.

Critical weaknesses

Overclaim on novelty. The four world-action training objectives -- policy learning, inverse dynamics, forward dynamics, future generation -- have substantial precedent in previous works. UWM (UWM; Zhu et al., RSS 2025) already shows that policy, forward dynamics, inverse dynamics and video generator can be unified in one model; LDA-1B (Lyu et al., RSS 2026) further utilizes this paradigm and performs world action modeling in visual latent space. Moreover, as for VLN tasks, SpaAct  (Li et al., 2026) explores using backward action reasoning and forward transition prediction for dynamic spatial awareness. The novelty of FutureNav relative to these prior methods should be clarified more carefully. Overclaim on performance. The paper claims that the performance of FutureNav surpasses all prior single-RGB SOTA methods (section 4.2). However, the paper doesn’t compare FutureNav with previous published works, such as CorrectNav (Yu et al., 2026), a prior monocular RGB-based VLA navigation method that reports 65.1/62.3 SR/SPL on R2R-CE and 69.3/63.3 on RxR-CE. It exceeds FutureNav's best RxR-CE result of 63.9/54.8. Other works like AwareVLN (Guo et al., 2026) also report stronger results under comparable settings. In this case, the claim of surpassing all prior single-RGB SOTA methods does not appear to be supported by the current comparison. The proposed formulation may be better characterized as auxiliary predictive representation learning rather than unified world-action modeling. The forward and future-generation objectives predict a one-step pooled latent representation of the next observation, and these predictions are used only as training supervision rather than for rollout or planning at inference. It is therefore unclear what additional world-modeling capability is demonstrated beyond the benefit of predictive auxiliary supervision for learning a better policy representation. References: Zhu et al., Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets, RSS 2025. Lyu et al., LDA-1B: Scaling Latent Dynamics Action Model via Universal Embodied Data Ingestion, RSS 2026. Li et al., SpaAct: Spatially-Activated Transition Learning with Curriculum Adaptation for Vision-Language Navigation, arXiv:2604.27620, 2026. Yu et al., CorrectNav: Self-Correction Flywheel Empowers Vision-Language-Action Navigation Model, AAAI 2026. Guo et al., AwareVLN: Reasoning with Self-awareness for Vision-Language Navigation, CVPR 2026.

Questions for authors

Novelty: Could the authors clarify the key methodological novelty of FutureNav relative to UWM and LDA-1B, which already jointly model policy, forward dynamics, inverse dynamics, and future visual states? It would also be helpful to discuss the relation to SpaAct, which explores backward action reasoning and forward transition prediction in VLN. Performance comparison: Why is CorrectNav (AAAI 2026) not included in Table 1? Since CorrectNav reports stronger results on RxR-CE under a monocular RGB setting, could the authors clarify whether the evaluation settings are directly comparable and revise the SOTA claim accordingly? World modeling vs. auxiliary supervision: What evidence indicates that the proposed forward/future prediction objectives learn a meaningful world model beyond serving as auxiliary predictive supervision for improving the policy representation? Could the authors further clarify what world-modeling capability is enabled by these objectives? Spatial-feature contribution: Does the policy-only baseline in Table 2 use the same VGGT spatial-feature fusion as the full FutureNav model? Appendix C.3 describes an ablation with alpha=0 to isolate spatial input fusion, but the corresponding quantitative results appear to be missing. Could the authors provide or clarify this ablation?

Reviewer_iVW4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces FutureNav, a vision-and-language navigation framework that augments a Qwen3-VL backbone with frozen VGGT spatial features. It jointly optimizes action prediction, forward dynamics, inverse dynamics, and future-state prediction, while removing the auxiliary prediction heads at inference. The method is evaluated on R2R-CE, RxR-CE, and additional simulated and real-world navigation settings.

Critical strengths
  1. The reported navigation results are competitive across R2R-CE and RxR-CE.
  2. The component ablation in Table 2 provides evidence that auxiliary supervision improves navigation performance within the proposed architecture.
  3. Removing the auxiliary prediction heads at inference is a practical design choice that avoids their runtime overhead.
Critical weaknesses
  1. The experiments do not directly validate the central world-modeling interpretation. Navigation improvements alone do not establish that the auxiliary heads learn action-sensitive transitions. The paper does not report held-out prediction quality, inverse-dynamics accuracy, or comparisons against current-state copying and action-shuffled conditioning. These controls would distinguish transition modeling from more generic representation regularization.

  2. The spatial target construction requires stronger justification. Equation 14 flattens the spatial feature tensor before adaptive pooling, rather than preserving feature channels while pooling patches. Appendix C.3 describes comparisons against channel-preserving pooling and variations of spatial input fusion, but provides no numerical outcomes. Appendix C.4 similarly describes data/budget controls without reporting their results.

  3. The incremental contribution relative to existing spatially augmented navigation and auxiliary dynamics learning is insufficiently established. JanusVLN already introduces geometric features, NaVIDA uses inverse-dynamics supervision, and Fast-WAM studies training-time world modeling without test-time future generation. A more controlled comparison is needed to establish the benefit of this particular combination.

  4. The efficiency claims lack direct measurements. Removing auxiliary heads avoids their inference cost, but the spatial encoder remains active. Figure 4 measures progress through training, not wall-clock time or compute.

Questions for authors

Does the inverse-dynamics input expose the previous action being predicted through historical action tokens? Can the authors provide the numerical results described in C.3–C.4 and measured latency including VGGT?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Dear ICLR 2027 AC, SAC, PC, and Reviewers

Dear ICLR 2027 AC, SAC, PC, and Reviewers,

Thank you and the reviewers for your time and constructive feedback. We summarize the recognized strengths, our responses, and the discussion outcomes below.

Strengths recognized across the reviewers

  • Competitive results on R2R-CE and RxR-CE (iVW4); with a 4B backbone, FutureNav exceeds several 7B/8B methods using similar or more external data (ddQL).
  • World-action objectives are training-only and the auxiliary heads are removed at inference, a simple and practical design (iVW4, jxEa).
  • Ablations show that the objectives improve navigation (SR 50.1 → 55.1), each one helps individually, and spatial latents outperform other representations (iVW4, jxEa, ddQL).

Reviewer jxEa (rating: 3, confidence: 4)

  • Strengths: A "simple and practical" implementation and supporting ablations.
  • Concerns: Novelty vs. UWM, LDA-1B and SpaAct; the SOTA claim given CorrectNav and AwareVLN; world modeling vs. auxiliary supervision; VGGT fusion in the policy-only baseline.
  • Our Response: Matched UWM-style (VAE latent) and LDA-1B-style (DINO latent) targets reach 51.0 / 51.8 vs. 55.1 SR. We will discuss CorrectNav and AwareVLN and qualify the SOTA wording. On held-out data, the forward head beats copy-current (NMSE 0.22 vs. 0.40) and degrades with shuffled actions (0.42); the inverse head reaches 0.86 macro-F1 (majority: 0.26). Controls without transition structure reach only 50.6–51.6 SR. All Table 2 rows share VGGT fusion, and the objectives help even without it (48.5 → 52.8 SR).
  • Reviewer's Feedback: jxEa replied: "I am pleased to see that the author has thoroughly addressed my questions and conducted extensive experiments, which have resolved most of my concerns." The reviewer then raised the rating from 2 to 3, with high confidence (4).

Reviewer ddQL (rating: 3, confidence: 4)

  • Strengths: "Strong benchmark results" and "a useful within-model ablation".
  • Concerns: Positioning vs. JanusVLN and SpaAct; missing controls (full-data policy-only, C.3/C.4, SpaAct-style); action leakage; seed variance; cost and latency; real-world uncertainty.
  • Our Response: Under identical settings, the objectives improve the full-data policy-only model by +6.1 SR / +4.8 SPL (R2R-CE) and +4.1 / +4.0 (RxR-CE); a matched SpaAct-style variant reaches 50.6 vs. 55.1 SR. We reported all C.3/C.4 results, the input template (no action tokens), three-seed std ≤ 0.7, and 255 ms latency including VGGT. Our follow-up reconciles the full-data baseline with per-seed results, commits to documenting the SpaAct-style setup and qualifying the SOTA claims, and adds a paired real-world bootstrap.
  • Reviewer's Feedback: ddQL wrote that the new controls "address important gaps in my original review" and that "the additional evidence strengthens the empirical case for the auxiliary objectives", leaving points on reporting consistency, reproducibility and claim scope, which our follow-up addresses. ddQL maintained the positive rating (3) with high confidence (4).

Reviewer iVW4 (rating: 2, confidence: 3)

  • Strengths: "Competitive" results, supporting ablations, and "a practical design choice" of removing the heads at inference.
  • Concerns: Direct evidence of world modeling; the spatial target (Eq. 14) and missing C.3/C.4 results; novelty vs. JanusVLN, NaVIDA and Fast-WAM; action leakage; efficiency.
  • Our Response: Provided the evidence above (probing, controls, C.3/C.4, leakage, latency) and matched JanusVLN-, NaVIDA- and Fast-WAM-style variants (50.1 / 50.8 / 53.0 vs. 55.1 SR).
  • Reviewer's Feedback: iVW4 acknowledged that "the additional experiments provide useful evidence for action-sensitive prediction, and the clarifications address my concerns about action-label leakage and missing measurements". The remaining concern was the incremental contribution of the joint formulation, as the individual objectives and the spatial-target design were not isolated. Our second response addresses both points: (i) in the ablation of all eight objective combinations, removing any single objective from the full model lowers SR (Inv. −0.9, Fwd. −2.7, Gen. −1.1), so each contributes within the joint formulation; (ii) among matched-dimension targets, layout-free ones reach 52.6–52.9 SR and layout-preserving ones 54.6–55.1 regardless of the pooling operator, so the gain comes from preserving spatial layout. No reply yet.

In summary, all three reviewers acknowledged that the additional experiments strengthen the empirical evidence; after jxEa raised the rating, both high-confidence reviewers (jxEa, ddQL) support weak acceptance. On the shared novelty concern, FutureNav outperforms matched re-implementations of six prior paradigms (50.1–53.0 vs. 55.1 SR).

We hope this summary helps your assessment of the paper in light of the additional evidence and the reviewers' feedback.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.