acceptodds
Withdrawn from ICLR 2027

WorldWeave: Jointly Learning World Dynamics Across Embodied Navigation Tasks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Mobile embodied agents continuously alter their observations through their own motion, while external dynamics further shape future interactions. Different mobile tasks therefore capture complementary patterns of state evolution, yet jointly learning world dynamics across such interactions remains less explored. We investigate this problem through Vision-and-Language Navigation (VLN) and Embodied Visual Tracking (EVT), which respectively involve navigation toward a stationary goal and interaction with a moving target. We propose WorldWeave, a cross-task world model that couples shared predictive dynamics with task-specific adaptation. Instead of conditioning state transitions on heterogeneous physical actions, WorldWeave learns latent actions from observed state changes via inverse–forward dynamics, providing a common transition representation across heterogeneous action spaces. The resulting future-state predictions are adaptively integrated into each task's representation for downstream decision making. Experiments across multiple VLN and EVT benchmarks demonstrate consistent improvements over specialized baselines.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied Navigation, World Models, Vision-and-Language Navigation, Embodied Visual Tracking, Latent Dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Khu4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents WorldWeave, a world-modeling framework that jointly learns predictive dynamics across vision-and-language navigation (VLN) and embodied visual tracking (EVT). The proposed approach learns task-agnostic latent actions from unlabeled videos through inverse and forward dynamics modeling, enabling dynamics sharing across heterogeneous action spaces. The learned future-state predictions are then incorporated into downstream policies through task-specific adaptation modules. Experiments on multiple simulated benchmarks and real-world robot deployments demonstrate improvements over task-specific baselines.

Critical strengths

Meaningful research motivation. Learning shared predictive dynamics across different embodied tasks is an interesting research direction. VLN and EVT provide complementary settings involving ego-motion and target motion, making them reasonable tasks for studying cross-task dynamics learning. Practical framework design. Learning latent transitions without requiring unified action spaces is a reasonable approach for various embodied tasks. The combination of shared dynamics and task-specific adaptation offers a practical way to mitigate negative transfer during multi-task learning. Extensive experimental evaluation. The paper evaluates WorldWeave across multiple VLN and EVT benchmarks, including R2R-CE, RxR-CE, EVTBench, UnrealZoo, and Gym-UnrealCV. The consistent improvements over matched baselines are encouraging.

Critical weaknesses

Limited methodological novelty. Most technical components are closely related to existing methods. Learning latent actions from video transitions has been extensively explored in prior work, including Genie [1] and LAPA [2]. Predictive world modeling has also been investigated separately for VLN (NavMorph [3]) and EVT (USS [4]). Furthermore, sharing world dynamics while introducing task-specific components has been explored in multi-task world modeling, such as Mixture-of-World Models [5]. Although applying these ideas jointly to VLN and EVT provides a different task setting, the proposed framework appears largely to combine established techniques. Insufficient evidence for the benefit of cross-task dynamics sharing. The main contribution of the paper is the claim that jointly learned world dynamics can benefit multiple embodied tasks. However, there lacks a comparison against independently pretrained world models for VLN and EVT under matched data and computational budgets. Therefore, it remains unclear whether sharing the dynamics model provides positive knowledge transfer compared with simply training separate dynamics models using the same data. Lack of direct validation of learned world dynamics. The paper primarily evaluates downstream task performance rather than the quality of the learned dynamics. There is no direct quantitative evaluation of future-state prediction accuracy, latent-action consistency, or cross-task dynamics transfer. Moreover, sharing the parameters of a dynamics model does not necessarily imply that the model learns a unified or transferable transition representation. Since the latent actions are inferred from observation changes rather than grounded in physical actions, they may also encode task-specific visual changes without capturing reusable motion dynamics. Additional analyses are needed to demonstrate what information is shared and whether the learned transition representations genuinely transfer across tasks. References: [1] Bruce et al., Genie: Generative Interactive Environments, ICML 2024. [2] Ye et al., Latent Action Pretraining from Videos, ICLR 2025. [3] Yao et al., NavMorph: A Self-Evolving World Model for Vision-and-Language Navigation in Continuous Environments, ICCV 2025. [4] Xie et al., USS: Unified Spatial-Semantic Prompts for Embodied Visual Tracking with Latent Dynamics Learning, 2026. [5] Zhang et al., Mixture-of-World Models: Scaling Multi-Task Reinforcement Learning with Modular Latent Dynamics, ICLR 2026.

Questions for authors

Could the authors clarify the key novelty of WorldWeave compared with existing latent-action learning and multi-task world-modeling approaches? In particular, what is the fundamental contribution beyond combining established techniques in a unified VLN-EVT framework? What is the direct benefit of sharing world dynamics across VLN and EVT? Could the authors compare the proposed shared world model against two independently pretrained task-specific world models, using matched pretraining data, model capacity, and computational budgets? This comparison seems essential to validate the central hypothesis. What evidence supports the transferability of the learned latent actions and dynamics representations? Could the authors provide direct evaluations of future-state prediction accuracy or cross-task transfer experiments? It would be particularly interesting to investigate whether latent actions corresponding to similar ego-motion patterns are consistent across VLN and EVT. How are the task-specific state representations kept compatible with the frozen dynamics model? Since Stage II introduces separate task-specific LoRA adapters while freezing the pretrained dynamics modules, it would be helpful to clarify whether the resulting representation shifts affect prediction accuracy.

Reviewer_Nq2EOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates sharing predictive world dynamics between vision-and-language navigation (VLN) and embodied visual tracking (EVT). It learns latent transitions from videos through inverse–forward dynamics, together with a causal latent-action predictor that enables future-state prediction without access to future observations. Shared predictions are incorporated through task-specific adaptation modules, allowing the two tasks to retain different representations and action spaces. The framework uses video-based dynamics pretraining followed by task-specific adaptation. Evaluation covers multiple navigation and tracking benchmarks, component ablations, and real-robot experiments. The reported results show improvements over the corresponding single-task baselines.

Critical strengths
  1. This paper addresses a meaningful cross-task learning problem with a coherent design. VLN and EVT involve different goals, action spaces, and sources of environmental change. Sharing predictive dynamics while retaining task-specific adaptation is a reasonable approach to exploiting their common temporal structure.

  2. The empirical evaluation is broad and demonstrates practical benefits. The paper evaluates both tasks across several benchmarks and includes real-robot deployment. The additional benchmarks and adaptation ablations strengthen the evidence that the overall framework is useful beyond a single evaluation setting.

Critical weaknesses
  1. The reported results contain an inconsistency between SR and OSR. Under the definition, OSR should be at least as high as SR when computed over the same episodes with the same distance threshold. However, Table 2 reports OSR/SR of 50.1/52.7 for WorldWeave and 41.2/45.8 for the baseline. This limits confidence in the affected results and their interpretation.

  2. The improvement in ambiguity tracking comes with a substantial collision-rate increase. On EVT-Bench (AT), WorldWeave improves SR while CR also increases substantially from 8.70 to 18.62. Collision avoidance is consequential for embodied tracking, and the results therefore provide mixed evidence about the overall quality of closed-loop behavior in this setting.

  3. The benefit of cross-task dynamics sharing is not clear. There is no comparison with a task-specific dynamics model. For this reason, there is no clear evidence to support the gain from sharing dynamics across VLN and EVT, compared with learning predictive dynamics.

  4. The experiments do not sufficiently isolate the proposed predictive mechanism. The paper claims latent actions as a way to share dynamics without aligning heterogeneous physical action spaces. However, a shared predictor that directly maps observation history to future features could also operate without physical action annotations. The result doesn't isolate the contribution of inverse dynamics with this alternative.

Questions for authors
  1. How to interpret the SR, OSR inconsistencies in Tables 2? Were both metrics computed on the same episodes using the same success threshold?

  2. Table 2 and Table 3 provide single-task comparisons, and Table 4 evaluates several sharing strategies. Is there also a comparison against task-specific dynamics model using the same architecture, adaptation components, and comparable training resources? Such evidence would help distinguish the benefit of cross-task dynamics sharing from the benefit of adding predictive modeling to each task.

  3. Has the proposed inverse-forward dynamics been compared with direct future prediction? In other words, is there other evidence that isolates the contribution of the inverse dynamics model?

  4. In Stage II, C,FC,F is frozen while task-specific VLM adapter is updated. If the input representations of C,FC,F change during LoRA adapation, will the next-state prediction accuracy been affected? Is there any evaluation of this potential problem?

Reviewer_hMC3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether VLN and EVT, typically studied separately, can benefit from shared predictive world dynamics. To this end, the authors propose WorldWeave, which learns latent dynamics from videos without action annotations and integrates predicted future states into task-specific decision processes. Experiments on multiple VLN and EVT benchmarks show consistent improvements over task-specific baselines, further supported by ablation studies and real-world robot evaluations. The results suggest that sharing predictive dynamics with task-specific adaptation can benefit both tasks.

Critical strengths
  1. This paper explores whether predictive world dynamics can be shared across heterogeneous embodied tasks, rather than directly unifying their action spaces or task representations. This is a relevant research direction that provides useful insights into cross-task knowledge sharing in embodied learning.

  2. This paper evaluates WorldWeave across multiple VLN and EVT benchmarks, showing consistent improvements over task-specific baselines. The ablation studies provide evidence for the proposed design choices, while real-world robot experiments further demonstrate the practical potential of the framework.

Critical weaknesses
  1. The paper tackles a meaningful problem, but its core components largely build on existing techniques. Latent-action learning has already been explored in LAPA and DreamDojo, and multi-task dynamics sharing has been studied in Mixture-of-World Models. WorldWeave mainly combines these ideas with task-specific adaptation, and it is unclear what new mechanism it introduces for sharing dynamics across tasks. As a result, the methodological novelty feels limited.

  2. While the ablation studies show improvements from incorporating shared predictive dynamics, the paper lacks a direct comparison with independently trained task-specific world models under comparable settings. Consequently, it remains unclear whether the gains primarily result from cross-task dynamics sharing or from predictive modeling itself. This weakens the empirical support for the paper's central claim.

Questions for authors

Could the authors clarify what evidence supports the benefit of jointly learning world dynamics across VLN and EVT, compared with learning task-specific dynamics independently? While Table 4 examines different sharing strategies, it remains unclear whether the improvements primarily come from cross-task knowledge transfer or predictive modeling itself. Clarifying this distinction would help assess the paper's central contribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.