acceptodds
Under review as a conference paper at ICLR 2027

WorldFlow: Joint Video and 3D Motion Prediction via Large-Scale Pretraining

96 views

Abstract

Predicting future physical states stands as a central challenge in embodied intelligence. Existing video world models leave 3D dynamics implicit, action-based formulations remain tied to specific robot configurations, and 3D motion prediction models use flow-only supervision, providing a limited learning signal for physical dynamics. To circumvent these limitations, we introduce WorldFlow, a generative world model that jointly predicts future video and query-based 3D trajectories. By parameterizing 3D motion as continuous displacement tokens aligned with the video grid, a unified Mixture-of-Transformers (MoT) backbone jointly denoises both modalities under a rectified-flow objective. This formulation couples explicit geometric dynamics with visual evolution, effectively unlocking the physical commonsense embedded within pretrained video models. To empirically scale and validate this unified architecture, we train WorldFlow on a heterogeneous mixture of 1.44 million object-level samples spanning simulated and real-world domains. We further introduce WorldFlowBench, a benchmark comprising three evaluation sets, including SpatialCog for assessing spatial reasoning. Across all three evaluation sets, WorldFlow achieves the lowest ADE and FDE among the evaluated learning-based methods. Crucially, on SpatialCog, our joint model reduces Average Displacement Error (ADE) by 55.9% and Final Displacement Error (FDE) by 68.5% relative to a flow-only ablation, demonstrating that generative visual priors fundamentally regularize complex geometric forecasting. Our code, pretrained models, and benchmarks will be made publicly available.

Primary area:
generative models
Keywords:
World Models, 3D Motion Forecasting, Joint Video–Flow Generation, Embodied AI

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Ck3uOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

WorldFlow jointly predicts video and query-conditioned metric 3D point trajectories from an observed image, text, and initial 3D queries. It extends a pretrained Cosmos-Nano Mixture-of-Transformers generator with continuous displacement tokens, query-geometry embeddings, and video-aligned positional coordinates. Training combines joint rectified-flow losses with auxiliary motion objectives on approximately 1.44 million object-level samples. Evaluation covers Sim7, Real3, and SpatialCog, plus a modular robot manipulation pipeline. The strongest evidence is the matched flow-only ablation and the comparison between direct flow prediction and tracking the same generated videos.

Critical strengths
  1. The paper makes video prediction more useful for downstream interaction by generating explicit, query-conditioned 3D motion alongside the visual future. Representing motion as metric point displacements provides an interface that a controller can use without first recovering trajectories from generated pixels. The architecture connects these outputs through shared attention, initial point geometry, and positional coordinates aligned with the video tokens. This is a coherent design for combining the rich supervision available in video with the geometric precision needed for manipulation.

  2. The experiments provide direct support for the value of joint prediction. The flow-only comparison uses the same backbone, training data, and optimization recipe, while the post-hoc tracking comparison uses videos from the same WorldFlow checkpoint. Joint prediction reduces ADE from 0.099 to 0.051 m on Sim7 and from 0.136 to 0.060 m on SpatialCog, and directly generated trajectories outperform both trackers on all three subsets. The manipulation experiment adds practical evidence: with grasping and control held fixed, replacing the flow predictor improves success from 4/36 to 24/36. Together, these results support the usefulness of the predicted motion beyond qualitative visual examples.

Critical weaknesses
  1. The evaluation establishes better motion prediction more clearly than coherent joint video–motion prediction. The quantitative results focus on trajectory accuracy, while Fig. 2 provides only qualitative overlays of generated trajectories and video. Comparing direct flow with tracks extracted from generated videos measures each output against the reference future; it does not measure whether the two generated outputs agree with each other. Consequently, the results support video generation as a useful source of supervision, but leave a central property of the proposed joint world model unverified. Quantitative cross-modal consistency and an assessment of whether video quality is preserved are needed to substantiate the joint-generation claim.

  2. The Real3 results leave the method's benefit for real-world motion forecasting insufficiently characterized. On this subset, predicting no motion gives lower ADE and FDE than WorldFlow (0.062 versus 0.081 m and 0.086 versus 0.113 m) and higher PWT (0.464 versus 0.311). WorldFlow improves over the flow-only model in ADE/FDE, but slightly reduces PWT from 0.320 to 0.311. The paper correctly limits its ranking to learned methods; nevertheless, the aggregate scores do not reveal whether the model helps on genuinely moving points or whether gains are concentrated in particular data sources or motion regimes. This matters because Real3 is the main benchmark evidence for broad real-world forecasting, distinct from the small, task-specific robot experiment.

Questions for authors
  1. Do the jointly generated video and 3D trajectories describe the same motion? Can you evaluate their agreement on the existing test outputs—for example, by comparing projected trajectories with independently tracked image motion, accounting for camera motion, visibility, and tracker error—and report whether joint training preserves video quality relative to the video backbone?

  2. How do WorldFlow, its flow-only variant, and Static compare on Real3 when results are separated by motion magnitude and constituent data source? In particular, does WorldFlow outperform Static on moving points, and where does the decrease in fine-scale PWT arise?

5 comments
AuthorsOpenReview ↗

Response to W1/Q1: Video-Motion Consistency and Video Quality

We thank the reviewer for their insightful comments.

W1/Q1. Video-Motion Consistency and Video Quality

We thank the reviewer for their valuable suggestions regarding consistency and video quality in joint generation. We have added two evaluations to examine cross-modal consistency and video quality relative to the original backbone.

Video-motion consistency. We have added a Video-Motion Consistency section to the supplementary material, comparing projected 3D trajectories with independently tracked point positions in the jointly generated videos, while accounting for estimated camera motion, visibility, and tracking confidence. The table summarizes pixel discrepancies across 1,678 test videos at 480 × 288 resolution.

Tracking scoreFrame 1Frame 16Frame 32Frame 48All 48 frames
>0.53.109.9520.0026.8712.30
>0.73.088.3517.9425.4710.26
>0.83.057.4616.5023.208.97
>0.92.976.1213.2321.037.31

Mean pointwise discrepancy in pixels (lower is better). Thresholds apply to the tracker’s visibility-confidence product.

At a tracking-confidence threshold of 0.9, the mean discrepancy over all 48 future frames is 7.31 pixels, supporting spatial correspondence between the two generated modalities on high-confidence observations. Increasing the threshold from 0.5 to 0.9 reduces the mean discrepancy from 12.30 to 7.31 pixels, indicating that WorldFlow’s projected 3D trajectories align more closely with pixel trajectories assigned higher confidence by the tracker. Generated outputs and camera projections remain fixed; only the retained observations change. Detailed settings, confidence-sensitivity analysis, and visualizations are provided in the supplementary section.

Video quality and dynamic degree. We compare videos generated by WorldFlow and the original Cosmos-Nano under matched generation settings on all 1,678 test inputs, using MUSIQ [1] to measure image quality and VBench Dynamic Degree [2] to quantify the proportion of videos containing substantial motion.

DomainCosmos-Nano MUSIQ ↑WorldFlow MUSIQ ↑Cosmos-Nano Dynamic DegreeWorldFlow Dynamic Degree
Sim772.3572.2522.73%31.06%
Real363.4261.5879.71%90.36%
SpatialCog68.9066.3441.75%92.50%
Overall66.8465.2357.21%76.88%

Overall, WorldFlow’s MUSIQ decreases by 2.40% relative to Cosmos-Nano, while its dynamic degree increases by 19.67 percentage points. This indicates a slight decrease in image quality, while generated videos more frequently contain substantial motion. Together with our motion-prediction results, these findings characterize an empirical trade-off between stronger flow prediction and a slight reduction in image quality.

References

[1] Ke et al. MUSIQ: Multi-Scale Image Quality Transformer. ICCV, 2021.

[2] Huang et al. VBench: Comprehensive Benchmark Suite for Video Generative Models. CVPR, 2024.

AuthorsOpenReview ↗

Response to W2/Q2 Part 1: Why PWT Slightly Decreases Relative to Flow-only on Real3

Real3: Explaining the PWT Difference Between WorldFlow and Flow-only

We thank the reviewer for highlighting the different trends in ADE/FDE and PWT on Real3. The slight PWT decrease primarily comes from fewer predictions within 1 cm, while WorldFlow improves the cumulative hit rate within 5 cm. Further analysis shows that Flow-only’s 1 cm hits are concentrated in the first eight frames. Even on clips where Flow-only has a higher early-frame hit rate, WorldFlow achieves lower full-trajectory ADE/FDE.

1. The PWT decrease primarily arises in the ≤1 cm interval.

PWT evaluates prediction errors at the point–frame level and averages the cumulative hit rates at 1, 2, and 5 cm. The mutually exclusive error intervals on Real3 are:

Prediction errorFlow-onlyWorldFlow
≤1 cm17.71%14.79%
>1–2 cm12.12%12.62%
>2–5 cm18.77%23.97%
>5 cm51.40%48.61%

WorldFlow increases the proportion of predictions in the >1–5 cm interval from 30.89% to 36.60%, while reducing the proportion above 5 cm from 51.40% to 48.61%. Consequently, the cumulative 5 cm hit rate improves from 48.60% to 51.39%.

However, the reduction in the ≤1 cm interval exceeds the increase in the >1–2 cm interval, lowering both the 1 cm and 2 cm cumulative hit rates. Averaging across the three thresholds therefore yields a slight PWT decrease from 0.3205 to 0.3120.

2. Flow-only’s early-frame hit-rate advantage does not translate into better full-trajectory performance.

Of all Flow-only predictions with errors within 1 cm, 47.80% occur in the first eight frames. Thus, just 16.67% of the prediction horizon accounts for nearly half of its 1 cm hits, contributing 8.47 percentage points to its overall 17.71% hit rate.

To connect this early performance with complete trajectories, we select the 683 clips where Flow-only has a higher 1 cm hit rate over the first eight frames. We then evaluate full 48-frame ADE and final-frame FDE on the same clips and query points:

MethodFull 48-frame ADE ↓Final-frame FDE ↓
Flow-only10.59 cm15.87 cm
WorldFlow9.11 cm12.36 cm
Error reduction1.48 cm3.51 cm

Even on clips that favor Flow-only under the early-frame strict hit criterion, WorldFlow achieves lower full-trajectory and endpoint errors. ADE/FDE also derive from point-position errors, but summarize the full prediction horizon and its endpoint, providing a trajectory-level assessment. PWT counts threshold hits and does not distinguish error magnitudes beyond its largest threshold.

For complete future-motion forecasting, full-trajectory and endpoint accuracy are more directly relevant to the prediction objective than strict hits concentrated in the initial few frames. On this subset, Flow-only’s early fine-scale hits do not translate into more accurate complete motion predictions, whereas WorldFlow reduces both full-trajectory ADE and endpoint FDE.

AuthorsOpenReview ↗

Response to W2/Q2 Part 2: Static Baseline Analysis on Real3

Analysis of the Static baseline on Real3

We thank the reviewer for suggesting further analysis by motion magnitude and data source. The stratified results show that WorldFlow’s ADE/FDE improvements over flow-only extend across all data sources and motion groups; cases where WorldFlow predicts more accurately than Static occur in both larger- and smaller-motion regimes. Our qualitative analysis further suggests that interpreting the Real3 results requires considering the difference between multiple plausible futures allowed by the instructions and single-reference evaluation, rather than attributing the results solely to motion magnitude. Below, we present the three-method comparison, quantify the distribution of cases with more accurate predictions, and discuss instruction ambiguity.

1. WorldFlow’s ADE/FDE improvements over flow-only extend across data sources and motion magnitudes.

We categorize each query point by its mean ground-truth displacement between consecutive frames over the 48 future steps, comparing all three methods on the same queries within each group.

Source / Mean Step DisplacementStaticFlow-onlyWorldFlow
AgiBot8.29/12.06/.344311.43/17.77/.31179.63/13.76/.2572
Something-Something V25.05/7.19/.53928.93/13.29/.34066.25/8.80/.4174
EPIC6.21/8.20/.38919.32/13.34/.30598.30/10.50/.2535
0–0.5 cm/frame2.68/3.57/.60326.32/9.81/.43604.81/6.55/.4344
>0.5–1 cm/frame8.34/12.27/.241911.38/17.41/.21109.53/13.60/.1970
>1–2 cm/frame14.82/21.60/.141317.75/26.39/.139114.71/20.60/.1184
>2 cm/frame22.15/29.73/.077725.84/35.42/.082021.75/28.74/.0603

Entries report ADE/FDE (cm)/PWT. Bold marks the best value for each metric within a row. “Frame” refers to a sampled evaluation frame.

WorldFlow achieves lower ADE/FDE than flow-only across all three sources and all four motion groups, indicating that its improvements are not confined to a particular subset. Relative to Static, WorldFlow achieves lower ADE/FDE in both groups above 1 cm/frame, while Static remains stronger in the smaller-motion groups.

2. Cases where WorldFlow predicts more accurately than Static span all motion ranges, including small motions.

We identify clips where WorldFlow has lower clip-level ADE than Static. Here, we average the mean step displacement across all queries within each clip and group clips using the same magnitude intervals. Among the 882 Real3 clips, WorldFlow achieves lower ADE on 195 clips, including 156 clips where both ADE and FDE are lower.

Clip Mean Step DisplacementTotal ClipsClips with Lower WorldFlow ADEWithin-Group Percentage
0–0.5 cm/frame4295412.59%
>0.5–1 cm/frame3068126.47%
>1–2 cm/frame1345440.30%
>2 cm/frame13646.15%

The proportion of clips where WorldFlow achieves lower ADE than Static increases with motion magnitude, but these cases are not limited to larger motions. Within the 0–0.5 cm/frame group, WorldFlow achieves lower ADE on 54 clips, including 39 clips with lower ADE and FDE. Small motion therefore does not inherently prevent WorldFlow from making more accurate predictions.

3. Instruction ambiguity and single-reference evaluation help interpret the Real3 results.

ADE and FDE measure the distance between predicted trajectories and a single reference trajectory, whereas the same initial observation and text instruction may allow multiple plausible future motions. When the predicted motion direction or path differs from the actual recording, even a prediction consistent with the input conditions can incur a large trajectory error.

The Results – Real3 Static Baseline Analysis section of our supplementary website illustrates this phenomenon: some predictions are visually plausible and consistent with the instructions, but their motion directions differ from the reference trajectories, resulting in higher ADE/FDE than Static. Real3 consists of clips from everyday real-world scenes, where descriptions typically specify the task but do not fully constrain the motion direction, path, magnitude, or endpoint. Consequently, the same initial observation and instruction may correspond to multiple plausible future motions. When the recorded motion remains close to the initial position, Static may achieve lower error than a model predicting another plausible trajectory.

Together, the stratified results and qualitative examples suggest that motion magnitude alone does not fully explain performance on Real3; interpreting the results requires considering the difference between multiple plausible futures and a single recorded reference.

Reviewer_va1pOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WorldFlow builds on the pretrained Cosmos-Nano Mixture-of-Transformers architecture to jointly generate future video and 3D point trajectories from an initial RGB image, a language instruction, and 3D query points. It represents inter-frame 3D displacements as flow tokens, which interact with video tokens through joint attention and rectified-flow denoising. Auxiliary objectives supervise cumulative motion direction and trajectory endpoints. The authors assemble approximately 1.44 million object-level samples for large-scale training and introduce WorldFlowBench for evaluation.

Critical strengths
  1. The proposed joint video and 3D motion generation framework draws on pretrained video priors to explicitly model scene dynamics.
  2. The paper introduces WorldFlowBench, where WorldFlow achieves lower ADE and FDE than the evaluated baselines based on (1) video generation followed by 3D tracking or (2)direct flow generation.
Critical weaknesses
  1. The main evaluation is limited to the proposed WorldFlowBench. Although the test samples are excluded from training, they come from the same underlying data sources as the training corpus. These results therefore do not establish generalization to unseen data domains.
  2. The construction of WorldFlowBench raises concerns about the reliability of its evaluation data. SpatialCog was collected, annotated, and split by the authors and is not yet publicly available. The reference trajectories for Real3 and Sim7 are pseudo-labels generated by Track4World, whose reliability as ground truth for quantitative evaluation requires further validation.
Questions for authors
  1. In Table 1, every evaluated learned method performs worse than the Static baseline on Real3, why?
  2. How does WorldFlow compare with competing methods in computational efficiency? parameter counts, peak GPU memory usage during inference, latency, and FLOPs, etc.
  3. Are there plans to publicly release the training dataset?
3 comments
AuthorsOpenReview ↗

Response to W1–W2: Generalization and Annotation Reliability

W1. Generalization to Unseen Environments

We thank the reviewer for their valuable comments on evaluating generalization to unseen data domains. We have added OOD simulation experiments on RoboLab [1] to both the main paper’s experimental section and the supplementary material. Its simulation environments lie outside WorldFlow’s training data domains. The evaluation covers all 64 tasks in the public Simple suite under three instruction variants, with 576 trials per method under each execution setting. We directly use the pretrained model without RoboLab-specific post-training or fine-tuning and assess success using the official automated task-completion criterion.

The table reports success rates under three instruction variants, with published policies shown in their original configurations for reference. Orch evaluates full manipulation; Attach uses simulator-assisted object retention. Within each setting, only the motion predictor changes.

MethodVagueDefaultSpecificOverall
Cosmos3-Nano-Policy [2]23.28%40.63%42.03%35.31%
π₀.₅ [3]16.25%29.69%29.84%25.26%
DreamZero [4]15.00%26.09%25.63%22.24%
GR00T N1.6 [5]7.19%8.75%7.50%7.81%
μ₀-Orch [6]3.13%3.65%4.69%3.82%
WorldFlow-Orch28.13%44.79%42.71%38.54%
μ₀-Attach [6]8.33%8.33%10.94%9.20%
WorldFlow-Attach32.29%52.60%56.77%47.22%

With all other execution modules shared and only the motion predictor replaced, WorldFlow-Orch achieves 38.54% end-to-end success, compared with 3.82% for μ₀-Orch. Under simulator-assisted attachment, WorldFlow-Attach achieves 47.22%, compared with 9.20% for μ₀-Attach. For system-level context, WorldFlow-Orch also exceeds the published RoboLab success rates of Cosmos3-Nano-Policy (35.31%) and π₀.₅ (25.26%). These results provide generalization evidence beyond WorldFlowBench, demonstrating that WorldFlow’s motion predictions transfer to unseen simulation environments and effectively guide downstream robotic manipulation.

W2. Annotation Reliability

We thank the reviewer for their valuable comments on the reliability of the evaluation annotations. Sim7’s ground-truth 3D point trajectories come directly from the simulator and are not pseudo-labels. For Real3, obtaining accurate 3D trajectories at scale from existing real-world videos remains a major challenge across the field, so we use Track4World to generate pseudo-labels. Estimating 3D motion using vision models is also adopted in related work, such as MolmoMotion [7], Dream2Flow [8], and μ₀ [6].

We likewise recognize that the quality and reliability of trajectory labels are crucial when they serve as references for quantitative evaluation. The Quality Control subsection of the supplementary material describes how we filter unreliable trajectories to improve pseudo-label quality. Starting from 2.30 million raw candidate samples, we obtain 1.44 million object-level training samples after quality control and other data processing. We check geometric validity, temporal continuity, observation completeness, and motion plausibility to filter samples with invalid geometry, trajectory discontinuities, missing observations, or implausible motion. We provide the detailed filtering criteria and processing pipeline in the supplementary material and hope these details help clarify our efforts to improve annotation quality.

References

[1] Yang et al. RoboLab. RSS 2026.

[2] NVIDIA. Cosmos 3. 2026.

[3] Black et al. π₀.₅. CoRL 2025.

[4] Ye et al. DreamZero. 2026.

[5] NVIDIA. GR00T N1.6. 2025.

[6] Lee et al. μ₀. 2026.

[7] Zhang et al. MolmoMotion. 2026.

[8] Dharmarajan et al. Dream2Flow. ICRA 2026.

AuthorsOpenReview ↗

Response to Q1–Q3: Static Baseline, Computational Efficiency, and Data Release

Q1. The Static Baseline and Motion Magnitude on Real3

We thank the reviewer for this question about the Static baseline.

Static serves as a statistic for quantifying motion magnitude across evaluation domains. It keeps each query point at its initial 3D position, so its ADE equals the mean displacement of the reference trajectories from their starting positions. ADE and FDE measure the distance between a predicted trajectory and a single reference trajectory, whereas the same initial observation and text instruction may allow multiple plausible future motions. When the predicted direction or path differs from the recorded motion, even a prediction consistent with the inputs can incur a large trajectory error.

The Results – Real3 Static Baseline Analysis section of our supplementary website illustrates this phenomenon: some predictions are visually plausible and consistent with the instructions, but their motion directions differ from the reference trajectories, resulting in higher ADE/FDE than Static. Real3 consists of clips from everyday real-world scenes, where the correspondence between textual descriptions and specific motions involves considerable uncertainty: descriptions typically specify the task but do not fully constrain the motion direction, path, magnitude, or endpoint. Consequently, the same initial observation and instruction may correspond to multiple plausible future motions. Under these circumstances, Static may achieve a lower error by keeping points at their initial positions, while a model predicting another plausible trajectory may incur a larger error penalty. This helps interpret why learned methods fail to outperform Static on Real3.

Q2. Computational Efficiency

We thank the reviewer for requesting a comparison of computational efficiency. We report prediction accuracy alongside computational cost to assess the resources required to achieve the reported forecasting performance.

The table below combines the ADE/FDE/PWT results from Table 1 with four efficiency metrics: parameter count (B), peak GPU memory during inference (GiB), inference latency (s), and computational cost per prediction (TFLOPs). Methods are ordered by ascending mean ADE across Sim7, Real3, and SpatialCog.

MethodSim7Real3SpatialCogParams (B)↓Mem (GiB)↓Time (s)↓TFLOPs↓
WorldFlow.051/.094/.564.081/.113/.311.060/.078/.4491631.2013.906647.49
Nano+4RC.096/.196/.531.085/.122/.332.141/.263/.40117.532.07/14.858.152293.81
LingBot+4RC.104/.207/.492.092/.134/.295.131/.232/.37631.568.08/14.8540.425436.12
Nano+T4W.101/.202/.451.088/.133/.310.141/.257/.39817.732.07/12.188.192026.92
LingBot+T4W.105/.207/.425.097/.145/.267.134/.239/.36431.768.08/12.1840.475169.23
NovaFlow.196/.318/.289.095/.129/.293.165/.295/.32015.675.70/13.1578.4411337.63
μ₀.186/.307/.069.116/.156/.148.213/.270/.0552.596.89.3968.74
EgoWAM.198/.336/.020.130/.190/.056.249/.383/.080.21.893.152.0662
Molmo-AR-H1.176/.323/.280.185/.335/.222.282/.533/.1734.469.70183.9146.27

Accuracy: ADE/FDE (m)/PWT. Nano = Cosmos3-Nano; LingBot = LingBot-Video-MoE; Molmo-AR-H1 = MolmoMotion-AR-H1. Two-stage parameters, latency, and FLOPs are totals; peak memory is reported as generation/tracking.

With 40 sampling steps, WorldFlow jointly generates a 49-frame video at 480 × 288 resolution and 64 query-point trajectories in 13.90 s, including RGB decoding. Among the video-generation-based approaches, it achieves lower ADE/FDE on every subset while running faster than NovaFlow and both LingBot pipelines. Compared with the faster Cosmos3-Nano pipelines, WorldFlow improves trajectory accuracy at additional computational cost. Among direct motion predictors, μ₀ and EgoWAM offer lower latency and resource usage but substantially higher trajectory errors, while MolmoMotion-AR-H1 requires fewer FLOPs yet incurs much higher latency. Overall, WorldFlow achieves the lowest ADE and FDE among the evaluated learning-based methods on all three subsets while jointly producing video and 3D trajectories, providing stronger forecasting accuracy at a lower latency than several video-based alternatives.

Q3. Public Release of the Training Dataset

We thank the reviewer for their interest in the public release of the training data.

Yes. We will release all data used in this work, including the training dataset.

We thank the authors for detailed reply. The explanation for static baseline is useful, but it raises my concern about the reliability of the metrics ADE/FDE/PWT. As WorldFlow doesn't have clear advantage on efficiency, reliable metrics to evaluate its trajectory accuracy is necessary. However, I agree with the workload of data engineering and the joint modeling exploration. Therefore, I will raise my score to weak accept.

Reviewer_xmqiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces WorldFlow, a generative world model that jointly predicts future video and query-conditioned 3D object trajectories. Given an observed image, a language instruction, and sparse 3D query points, WorldFlow represents 3D motion as continuous frame-to-frame displacement tokens aligned with the video token grid. A unified Mixture-of-Transformers backbone jointly denoises video latents and flow tokens under a rectified-flow objective, with auxiliary motion losses on direction, displacement magnitude, and endpoints. The authors train on a heterogeneous corpus of approximately 1.44 million object-level samples spanning simulation and real-world domains, and introduce WorldFlowBench with Sim7, Real3, and SpatialCog evaluation sets. They report the lowest ADE/FDE among evaluated learning-based methods on all three sets, show that joint video-flow generation substantially outperforms a flow-only variant.

Critical strengths
  1. The core formulation is well motivated. Treating 3D flow as an explicit generative modality alongside video, with shared spatiotemporal positional alignment and a joint rectified-flow objective, is a sensible way to combine dense visual priors with metric motion supervision.

  2. The scale and diversity of training data, together with the direct-vs-tracking comparison, make the empirical study more than a single-benchmark result.

Critical weaknesses
  1. The benchmark comparison is difficult to interpret because the strongest reference is excluded, and the reported margins are inconsistent across subsets. In Table 1, the “Static” zero-flow reference is excluded from ranking, yet it outperforms WorldFlow on Real3 ADE (0.062 vs. 0.081) and on Real3 PWT (0.464 vs. 0.311), and it is competitive on Sim7.

  2. The paper claims the lowest ADE/FDE among learning-based methods, but it does not analyze what fraction of query points are nearly stationary, whether the metric is dominated by static regions, or why the learned model fails to beat a trivial baseline on Real3.

  3. The real-world manipulation evidence is too small and too controlled to support the downstream claims. The robot evaluation uses 12 tasks with only three trials each (36 executions total per method), a single reproduced baseline, and matched initial layouts, identical grasp results, and fixed control settings. Success is judged by stable placement, but no confidence intervals or per-task breakdowns are reported.

  4. The pipeline depends on external grasp and local patch-fitting modules, so it is unclear how much of the gain comes from WorldFlow’s flow predictions versus the surrounding system.

Questions for authors

Please refer to paper weakness

2 comments
AuthorsOpenReview ↗

Response to W1–W2: Static Baseline and Stationary Query Analysis

We thank the reviewer for their insightful comments.

W1. The Static in Table 1

We thank the reviewer for their valuable comments on the Static baseline and the interpretation of Table 1.

Static is a statistic for quantifying motion magnitude across evaluation domains. It keeps each query point at its initial 3D position, so its ADE exactly equals the mean displacement of the reference trajectories from their starting positions. ADE and FDE measure the distance between predicted trajectories and a single reference trajectory, whereas the same initial observation and text instruction may allow multiple plausible future motions. When the predicted motion direction or path differs from the actual recording, even a prediction consistent with the input conditions can incur a large trajectory error.

The Results – Real3 Static Baseline Analysis section of our supplementary website illustrates this phenomenon: some predictions are visually plausible and consistent with the instructions, but their motion directions differ from the reference trajectories, resulting in higher ADE/FDE than Static. Real3 consists of clips from everyday real-world scenes, where the correspondence between textual descriptions and specific motions involves considerable uncertainty: descriptions typically specify the task but do not fully constrain the motion direction, path, magnitude, or endpoint. Consequently, the same initial observation and instruction may correspond to multiple plausible future motions. Under these circumstances, Static may achieve a lower error by keeping points at their initial positions, while a model predicting another plausible trajectory may incur a larger error penalty. This helps interpret why learned methods fail to outperform Static on Real3.

W2. Stationary Point Analysis and Their Impact on Evaluation

We thank the reviewer for highlighting the need to analyze nearly stationary queries and their impact on evaluation metrics. We define a query as nearly stationary if its ground-truth position remains within 1 cm of its initial position throughout the 48 future frames. These queries account for 10.77%, 10.74%, and 0.32% of Sim7, Real3, and SpatialCog, respectively, and 8.27% of the full benchmark. WorldFlow’s ADE/FDE (cm) on all, nearly stationary, and non-stationary queries are reported below:

DomainAll queriesNearly stationary queriesNon-stationary queries
Sim75.19 / 9.422.35 / 5.025.60 / 9.98
Real38.17 / 11.343.32 / 4.378.50 / 11.76
SpatialCog6.04 / 7.842.48 / 2.336.04 / 7.84
All6.96 / 10.053.11 / 4.507.24 / 10.41

Nearly stationary queries involve small motions and yield lower ADE/FDE. However, excluding these queries increases the benchmark-wide ADE/FDE by only 4.02% and 3.52%, respectively. Measured by the relative change after exclusion, the impact of nearly stationary queries on both metrics is below 4.1%, indicating that the overall evaluation results are not primarily driven by static regions.

AuthorsOpenReview ↗

Response to W3–W4: OOD Evaluation and Motion Prediction Contributions

W3. OOD Evaluation on RoboLab and Real-Robot Results

We thank the reviewer for highlighting the limited scale and diversity of the downstream manipulation evaluation. We have added OOD simulation experiments on RoboLab [1] to both the main paper’s experimental section and the supplementary material, covering all 64 tasks in the public Simple suite under three instruction variants, with 576 trials per method per execution setting, without RoboLab-specific fine-tuning. Success is assessed using the official automated task-completion criterion.

The table reports success rates under three instruction variants, with published policies included for reference. Orch evaluates full manipulation; Attach uses simulator-assisted object retention. With other modules shared, WorldFlow outperforms μ₀ across all instruction variants in both settings.

MethodVagueDefaultSpecificOverall
Cosmos3-Nano-Policy [4]23.28%40.63%42.03%35.31%
π₀.₅ [2]16.25%29.69%29.84%25.26%
DreamZero [5]15.00%26.09%25.63%22.24%
GR00T N1.6 [3]7.19%8.75%7.50%7.81%
μ₀-Orch [6]3.13%3.65%4.69%3.82%
WorldFlow-Orch28.13%44.79%42.71%38.54%
μ₀-Attach [6]8.33%8.33%10.94%9.20%
WorldFlow-Attach32.29%52.60%56.77%47.22%

With all other execution modules shared and only the motion predictor replaced, WorldFlow-Orch achieves 38.54% end-to-end success, substantially outperforming μ₀-Orch at 3.82%. For system-level context, WorldFlow-Orch also exceeds the published RoboLab success rates of Cosmos3-Nano-Policy (35.31%) and π₀.₅ (25.26%). These results further support the downstream utility of WorldFlow’s predicted motion in unseen robotic environments. Visual comparisons of the simulation experiments are provided in the Simulation section of the supplementary website.

Regarding the reporting of real-robot success rates and per-task outcomes. The real-robot experiments use matched initial layouts and fixed execution modules to control other factors and assess the contribution of motion prediction. We have added per-task results for all 12 tasks and visualizations of individual trials to the Real-Robot Experiments section of the supplementary website, separately presenting flow-prediction and robot-execution outcomes for inspection.

References

[1] Yang et al. RoboLab. RSS 2026.

[2] Black et al. π₀.₅. CoRL 2025.

[3] NVIDIA. GR00T N1.6. 2025.

[4] NVIDIA. Cosmos 3. 2026.

[5] Ye et al. DreamZero. 2026.

[6] Lee et al. μ₀. 2026.

W4. Disentangling Motion Prediction Quality from Downstream Execution Success

We thank the reviewer for raising the important question of distinguishing the contribution of motion prediction from that of the surrounding modules. We have added flow-prediction evaluations on the real-robot tasks to the main-paper table and supplementary material, reporting motion-prediction quality separately from final execution success. We manually inspect 3D visualizations to assess whether the predicted motion is plausible and consistent with the task instruction.

MethodFlow-Prediction SuccessReal-Robot Execution Success
μ₀25.0% (9/36)11.1% (4/36)
WorldFlow97.2% (35/36)66.7% (24/36)

With identical grasp results, local fitting, and control settings, WorldFlow’s advantage is already evident in motion-prediction quality before execution and translates into higher robot execution success within the shared pipeline. Per-case results and visualizations are available in the Real-Robot Experiments section of the supplementary website.

The newly added RoboLab simulation experiments distinguish these two aspects through the Attach and Orch configurations. Attach measures flow-prediction success through task completion with grasp failures and slippage removed, while Orch measures the execution success of the full grasping, transport, and placement pipeline. Within each configuration, both methods share all other modules, with only the motion predictor replaced.

MethodAttach Task SuccessOrch Task Success
μ₀9.20% (53/576)3.82% (22/576)
WorldFlow47.22% (272/576)38.54% (222/576)

WorldFlow achieves higher task success both with reduced grasp-related interference and under full simulation execution, further supporting the contribution of its motion predictions to downstream task performance.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Updates to the Paper and Supplementary Materials

We thank all reviewers for their careful evaluations and valuable feedback. We have updated the paper PDF and supplementary materials with additional experiments and clarifications, and invite the reviewers to examine the revised materials. Revised and newly added text is highlighted in blue to make the changes easier to follow. The main updates are:

  • Section 4.1, DATASETS, and Supplementary Section A.1, DATA PROCESSING DETAILS: Minor wording revisions to clarify data sources and processing procedures.
  • Section 4.4, FLOW-GUIDED ROBOTIC MANIPULATION: New experiments on the public OOD benchmark RoboLab, evaluating WorldFlow’s flow-guided task completion and end-to-end robotic execution under separate settings.
  • Section 4.4.2, REAL-ROBOT MANIPULATION: An updated results table that separately reports WorldFlow’s trajectory prediction accuracy, distinguishing prediction quality from end-to-end execution performance.
  • Supplementary Section D, VIDEO-MOTION CONSISTENCY: Additional experiments and analysis assessing consistency between the jointly generated video and motion predictions.

We also include an anonymous project website in the supplementary materials. After extracting the archive, reviewers can open index.html to browse extensive visualizations, per-task results, and additional experimental findings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.