acceptodds
Withdrawn from ICLR 2027

World-4D: 4D Embodied World Models for Robotic Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Robotic manipulation in the open world requires not only recognizing what a scene looks like, but also anticipating how its 3D structure moves under interaction. We argue that synchronized RGB, depth, and optical flow (RGB-DF) provide a geometry-aware projective representation that makes scene structure and pixel-level motion explicit. Compared to 2D pixel videos, this multi-modal synergy aligns visual appearance with geometric structure and temporal motion, creating a representation space significantly closer to low-level end-effector actions demanded by robotic systems, narrowing the gap between world prediction and policy learning. Building on this insight, we introduce World-4D, a generative model that co-produces future RGB frames, depth maps, and optical flow from a single RGB-D image and a language instruction within one unified diffusion process. This 4D world model features a tri-branch architecture that integrates cross-modal attention with frame-wise 3D RoPE, ensuring that appearance, geometry, and motion evolve consistently. To supply training data at scale, we curate World-4D-200M, a large-scale dataset containing approximately 254 million frames across egocentric human and robotic manipulation videos with pseudo-labels for depth and optical flow. We further propose World-4D-Policy, an inverse dynamics head that consumes the internal 4D representations of World-4D in a single forward pass, bypassing expensive multi-step denoising, to output robot actions in a closed-loop manner. Experiments show that World-4D produces temporally and spatially coherent 4D predictions, and that World-4D-Policy achieves strong performance on real-world bimanual manipulation tasks, outperforming the evaluated baselines on most tasks, with notable gains on tasks demanding spatial precision and temporal coordination.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot manipulation, Robot foundation models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bzmcOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces World-4D, a language-conditioned model that jointly predicts RGB, depth, and optical-flow videos from a single RGB-D observation. Building on Wan 2.2-TI2V-5B, the method uses three modality-specific branches connected through frame-wise cross-modal attention, together with a staged training procedure. The authors curate World-4D-200M, containing approximately 254.4 million frames augmented with depth, flow, and language pseudo-annotations. For downstream control, World-4D-Policy extracts intermediate features from the frozen backbone in a single forward pass, aggregates them through a Flow Former, and uses a flow-matching action head to produce ten-step action chunks. Validation includes a 50-sequence world-model evaluation, architectural and modality ablations, an action-regression probe across four datasets, and six real-world bimanual manipulation tasks with 35 trials per task. The paper reports improved visual and geometric prediction metrics and best or tied-best policy success on five of the six manipulation tasks.

Critical strengths

1.Jointly modeling depth and optical flow provides a plausible way to expose complementary geometric and motion cues while retaining a representation compatible with pretrained video models. The modality-specific branches and cross-modal attention provide a concrete implementation of this idea. The ablations in Table 1 offer encouraging support: compared with independent branches, joint modeling improves reported depth AbsRel from 0.737 to 0.310 and RGB SSIM from 0.683 to 0.754. Additional ablations examine modality adaptation, positional encoding, and parameter sharing.

  1. The experiments connect the generative model to a 54-dimensional dual-arm, dexterous-hand control system and evaluate six tasks involving coordination, contact, and precise placement. The reported mean success rate of 74.76%, with best or tied-best performance on five tasks, provides meaningful evidence of downstream utility. The common-decoder comparisons with adapted world models and the modality ablations are also useful experimental choices.
Critical weaknesses
  1. The paper does not sufficiently establish that improved 4D prediction leads to better downstream action quality. The world-model evaluation mainly reports visual, depth, and optical-flow metrics, while the downstream evaluation is limited to success rates on six real-robot tasks. Improvements in depth reconstruction or flow AEPE do not necessarily demonstrate more accurate actions or stronger closed-loop robustness. The LARY probing result is also an indirect regression proxy. A controlled simulation benchmark, or analysis connecting prediction quality to action error and task success under perturbations, would better support the claimed downstream benefit.
  2. The specific mechanism and regime in which optical flow helps are under-analyzed. Although Table 2 includes modality ablations (RGB, RGB+Depth, RGB+Optical Flow, and RGB-DF), the results do not explain why flow improves success. It would be informative to evaluate whether flow helps especially under motion, occlusion, contact disturbances, or other OOD perturbations, and to visualize successful and failed cases with predicted flow, 3D scene flow, and actions. In addition, Mimic-Video is a relatively old baseline for this setting. The comparison would be more compelling if it included a recent 2026 world-action-model baseline, such as FastWAM, in simulation or on the real robot.
Questions for authors

Regarding the weaknesses described above, I would appreciate further explanation.

Reviewer_h9AeOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

World-4D jointly predicts RGB, depth, and optical flow using a tri-branch video diffusion model pretrained on approximately 254M pseudo-annotated frames. A downstream policy uses intermediate features from the frozen model to generate action chunks. The paper reports improvements in video/depth prediction and six real-world manipulation tasks.

Critical strengths
  • Well-motivated representation. Joint RGB, depth, and flow prediction combines appearance, geometry, and temporal correspondence in a format compatible with pretrained video models.
  • Substantial implementation effort. The work integrates 254M-frame pseudo-annotation, staged multimodal training, and a complete real-robot control pipeline.
  • Encouraging empirical results. Evaluation covers generation quality and six real-world manipulation tasks, with useful architecture/modality ablations and comparisons using a shared action decoder.
Critical weaknesses
  1. The benefit of the full tri-branch architecture over simpler expert-based alternatives is insufficiently established. The reported depth and flow annotations are produced by Depth Anything 3 and DPFlow, respectively, while the downstream policy consumes intermediate features rather than decoded predictions. This raises a natural question: why are full Wan branches needed to leverage these experts’ knowledge? Please compare against directly encoding estimated depth/flow or expert features, and, more critically, a shared video backbone with lightweight heads predicting future depth/flow or expert features under the same supervision. Report policy success and computational cost, and clarify whether modality ablations retrain reduced-branch models or only omit features. These experiments would distinguish the benefit of geometry/motion supervision from that of the proposed architecture. Convincing evidence of the latter would make me willing to revise my assessment to Weak Accept.

  2. The depth, flow, and 3D visualizations are not sufficiently clear or convincing. Larger, higher-resolution comparisons with ground truth, motion boundaries, and failure cases would better demonstrate geometric and temporal consistency.

Questions for authors

Please see the clarification requests and experimental comparisons in the Weaknesses section; addressing these could materially change my assessment.

Reviewer_yzDiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a world model that jointly generates RGB, depth, and optical-flow videos to learn representations for robotic manipulation. The method extends a pretrained video diffusion model into a three-branch architecture with cross-modal attention and trains it on approximately 254 million pseudo-annotated frames. A downstream policy extracts intermediate representations through a single backbone forward pass and generates action chunks. Evaluation covers video generation, geometric prediction, and six real-world manipulation tasks.

Critical strengths

Jointly modeling appearance, geometry, and motion is well motivated. Explicit depth and motion information could reduce the difficulty of learning action-relevant representations. The work involves substantial dataset construction and real-world evaluation. The world-model experiments include ablations of cross-modal interaction, modality adaptation, and positional encoding. Using the same action decoder to compare representations from different world models is a useful experimental design. The proposed policy achieves the best or tied-best success rate on five of the six evaluated tasks.

Critical weaknesses

The optical-flow metrics and interpretation of metric 3D motion require further clarification. Appendix K.1 normalizes flow magnitudes before color encoding, but does not specify the actual normalization constant, whether it varies across frames or clips, or how it is recovered for generated samples. Consequently, it is unclear whether the reported AEPE of 0.170 has a consistent pixel-displacement interpretation. Appendix F also directly subtracts back-projected points from consecutive frames. For sequences with camera motion, these points must first be expressed in a common coordinate frame before their difference can represent physical scene motion. Visually plausible trajectories alone do not establish geometric or physical consistency.

The experiments do not sufficiently isolate the contribution of optical flow. Sharing the action decoder controls decoding differences, but does not control backbone capacity, pretraining data, or training budget. In the more direct internal comparison, RGB-D achieves approximately 71.91% average success, compared with 74.76% for RGB-DF. On Hand-over, which is used to motivate motion reasoning, both variants achieve 28.57%. With only 35 trials per task, several improvements correspond to one to three additional successes. Stronger evidence is needed to attribute the policy gains specifically to optical-flow representations.

The high-frequency closed-loop control claim conflates action execution with visual replanning. Appendices A, H, and J.3 report approximately 1.106 seconds per inference, followed by execution of ten actions. The visual replanning rate is therefore approximately 0.9 Hz, while 9 Hz describes action execution. The statement in Appendix H that the “next 9 Hz update” replans the trajectory is inconsistent with this implementation. This distinction matters for the claimed responsiveness and disturbance-recovery capabilities.

Depth preprocessing may affect the validity of the baseline comparison. Appendix D applies sequence-wise min–max normalization to 4DNeX depth, whereas Appendix B describes only multiplicative median scaling during evaluation. The additive offset introduced by min–max normalization generally cannot be removed by multiplicative scaling. The authors should clarify whether the original depth representation is restored before evaluation.

Questions for authors

What exact flow-color encoding, magnitude normalization, and inverse mapping are used? What are the AEPE units and the encoding–decoding reconstruction error? Under matched data and training budgets, does a paired evaluation establish a reliable advantage of RGB-DF over RGB-D? Can the authors distinguish visual replanning, action execution, and low-level control frequencies, and provide disturbance-recovery evidence consistent with the actual sensing-to-action latency? Is the 4DNeX min–max transformation fully inverted before computing depth metrics?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.