- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
WorldFlow jointly predicts video and query-conditioned metric 3D point trajectories from an observed image, text, and initial 3D queries. It extends a pretrained Cosmos-Nano Mixture-of-Transformers generator with continuous displacement tokens, query-geometry embeddings, and video-aligned positional coordinates. Training combines joint rectified-flow losses with auxiliary motion objectives on approximately 1.44 million object-level samples. Evaluation covers Sim7, Real3, and SpatialCog, plus a modular robot manipulation pipeline. The strongest evidence is the matched flow-only ablation and the comparison between direct flow prediction and tracking the same generated videos.
Critical strengths
-
The paper makes video prediction more useful for downstream interaction by generating explicit, query-conditioned 3D motion alongside the visual future. Representing motion as metric point displacements provides an interface that a controller can use without first recovering trajectories from generated pixels. The architecture connects these outputs through shared attention, initial point geometry, and positional coordinates aligned with the video tokens. This is a coherent design for combining the rich supervision available in video with the geometric precision needed for manipulation.
-
The experiments provide direct support for the value of joint prediction. The flow-only comparison uses the same backbone, training data, and optimization recipe, while the post-hoc tracking comparison uses videos from the same WorldFlow checkpoint. Joint prediction reduces ADE from 0.099 to 0.051 m on Sim7 and from 0.136 to 0.060 m on SpatialCog, and directly generated trajectories outperform both trackers on all three subsets. The manipulation experiment adds practical evidence: with grasping and control held fixed, replacing the flow predictor improves success from 4/36 to 24/36. Together, these results support the usefulness of the predicted motion beyond qualitative visual examples.
Critical weaknesses
-
The evaluation establishes better motion prediction more clearly than coherent joint video–motion prediction. The quantitative results focus on trajectory accuracy, while Fig. 2 provides only qualitative overlays of generated trajectories and video. Comparing direct flow with tracks extracted from generated videos measures each output against the reference future; it does not measure whether the two generated outputs agree with each other. Consequently, the results support video generation as a useful source of supervision, but leave a central property of the proposed joint world model unverified. Quantitative cross-modal consistency and an assessment of whether video quality is preserved are needed to substantiate the joint-generation claim.
-
The Real3 results leave the method's benefit for real-world motion forecasting insufficiently characterized. On this subset, predicting no motion gives lower ADE and FDE than WorldFlow (0.062 versus 0.081 m and 0.086 versus 0.113 m) and higher PWT (0.464 versus 0.311). WorldFlow improves over the flow-only model in ADE/FDE, but slightly reduces PWT from 0.320 to 0.311. The paper correctly limits its ranking to learned methods; nevertheless, the aggregate scores do not reveal whether the model helps on genuinely moving points or whether gains are concentrated in particular data sources or motion regimes. This matters because Real3 is the main benchmark evidence for broad real-world forecasting, distinct from the small, task-specific robot experiment.
Questions for authors
-
Do the jointly generated video and 3D trajectories describe the same motion? Can you evaluate their agreement on the existing test outputs—for example, by comparing projected trajectories with independently tracked image motion, accounting for camera motion, visibility, and tracker error—and report whether joint training preserves video quality relative to the video backbone?
-
How do WorldFlow, its flow-only variant, and Static compare on Real3 when results are separated by motion magnitude and constituent data source? In particular, does WorldFlow outperform Static on moving points, and where does the decrease in fine-scale PWT arise?
5 comments
Response to W2/Q2 Part 1: Why PWT Slightly Decreases Relative to Flow-only on Real3
Real3: Explaining the PWT Difference Between WorldFlow and Flow-only
We thank the reviewer for highlighting the different trends in ADE/FDE and PWT on Real3. The slight PWT decrease primarily comes from fewer predictions within 1 cm, while WorldFlow improves the cumulative hit rate within 5 cm. Further analysis shows that Flow-only’s 1 cm hits are concentrated in the first eight frames. Even on clips where Flow-only has a higher early-frame hit rate, WorldFlow achieves lower full-trajectory ADE/FDE.
1. The PWT decrease primarily arises in the ≤1 cm interval.
PWT evaluates prediction errors at the point–frame level and averages the cumulative hit rates at 1, 2, and 5 cm. The mutually exclusive error intervals on Real3 are:
| Prediction error | Flow-only | WorldFlow |
|---|---|---|
| ≤1 cm | 17.71% | 14.79% |
| >1–2 cm | 12.12% | 12.62% |
| >2–5 cm | 18.77% | 23.97% |
| >5 cm | 51.40% | 48.61% |
WorldFlow increases the proportion of predictions in the >1–5 cm interval from 30.89% to 36.60%, while reducing the proportion above 5 cm from 51.40% to 48.61%. Consequently, the cumulative 5 cm hit rate improves from 48.60% to 51.39%.
However, the reduction in the ≤1 cm interval exceeds the increase in the >1–2 cm interval, lowering both the 1 cm and 2 cm cumulative hit rates. Averaging across the three thresholds therefore yields a slight PWT decrease from 0.3205 to 0.3120.
2. Flow-only’s early-frame hit-rate advantage does not translate into better full-trajectory performance.
Of all Flow-only predictions with errors within 1 cm, 47.80% occur in the first eight frames. Thus, just 16.67% of the prediction horizon accounts for nearly half of its 1 cm hits, contributing 8.47 percentage points to its overall 17.71% hit rate.
To connect this early performance with complete trajectories, we select the 683 clips where Flow-only has a higher 1 cm hit rate over the first eight frames. We then evaluate full 48-frame ADE and final-frame FDE on the same clips and query points:
| Method | Full 48-frame ADE ↓ | Final-frame FDE ↓ |
|---|---|---|
| Flow-only | 10.59 cm | 15.87 cm |
| WorldFlow | 9.11 cm | 12.36 cm |
| Error reduction | 1.48 cm | 3.51 cm |
Even on clips that favor Flow-only under the early-frame strict hit criterion, WorldFlow achieves lower full-trajectory and endpoint errors. ADE/FDE also derive from point-position errors, but summarize the full prediction horizon and its endpoint, providing a trajectory-level assessment. PWT counts threshold hits and does not distinguish error magnitudes beyond its largest threshold.
For complete future-motion forecasting, full-trajectory and endpoint accuracy are more directly relevant to the prediction objective than strict hits concentrated in the initial few frames. On this subset, Flow-only’s early fine-scale hits do not translate into more accurate complete motion predictions, whereas WorldFlow reduces both full-trajectory ADE and endpoint FDE.
Response to W2/Q2 Part 2: Static Baseline Analysis on Real3
Analysis of the Static baseline on Real3
We thank the reviewer for suggesting further analysis by motion magnitude and data source. The stratified results show that WorldFlow’s ADE/FDE improvements over flow-only extend across all data sources and motion groups; cases where WorldFlow predicts more accurately than Static occur in both larger- and smaller-motion regimes. Our qualitative analysis further suggests that interpreting the Real3 results requires considering the difference between multiple plausible futures allowed by the instructions and single-reference evaluation, rather than attributing the results solely to motion magnitude. Below, we present the three-method comparison, quantify the distribution of cases with more accurate predictions, and discuss instruction ambiguity.
1. WorldFlow’s ADE/FDE improvements over flow-only extend across data sources and motion magnitudes.
We categorize each query point by its mean ground-truth displacement between consecutive frames over the 48 future steps, comparing all three methods on the same queries within each group.
| Source / Mean Step Displacement | Static | Flow-only | WorldFlow |
|---|---|---|---|
| AgiBot | 8.29/12.06/.3443 | 11.43/17.77/.3117 | 9.63/13.76/.2572 |
| Something-Something V2 | 5.05/7.19/.5392 | 8.93/13.29/.3406 | 6.25/8.80/.4174 |
| EPIC | 6.21/8.20/.3891 | 9.32/13.34/.3059 | 8.30/10.50/.2535 |
| 0–0.5 cm/frame | 2.68/3.57/.6032 | 6.32/9.81/.4360 | 4.81/6.55/.4344 |
| >0.5–1 cm/frame | 8.34/12.27/.2419 | 11.38/17.41/.2110 | 9.53/13.60/.1970 |
| >1–2 cm/frame | 14.82/21.60/.1413 | 17.75/26.39/.1391 | 14.71/20.60/.1184 |
| >2 cm/frame | 22.15/29.73/.0777 | 25.84/35.42/.0820 | 21.75/28.74/.0603 |
Entries report ADE/FDE (cm)/PWT. Bold marks the best value for each metric within a row. “Frame” refers to a sampled evaluation frame.
WorldFlow achieves lower ADE/FDE than flow-only across all three sources and all four motion groups, indicating that its improvements are not confined to a particular subset. Relative to Static, WorldFlow achieves lower ADE/FDE in both groups above 1 cm/frame, while Static remains stronger in the smaller-motion groups.
2. Cases where WorldFlow predicts more accurately than Static span all motion ranges, including small motions.
We identify clips where WorldFlow has lower clip-level ADE than Static. Here, we average the mean step displacement across all queries within each clip and group clips using the same magnitude intervals. Among the 882 Real3 clips, WorldFlow achieves lower ADE on 195 clips, including 156 clips where both ADE and FDE are lower.
| Clip Mean Step Displacement | Total Clips | Clips with Lower WorldFlow ADE | Within-Group Percentage |
|---|---|---|---|
| 0–0.5 cm/frame | 429 | 54 | 12.59% |
| >0.5–1 cm/frame | 306 | 81 | 26.47% |
| >1–2 cm/frame | 134 | 54 | 40.30% |
| >2 cm/frame | 13 | 6 | 46.15% |
The proportion of clips where WorldFlow achieves lower ADE than Static increases with motion magnitude, but these cases are not limited to larger motions. Within the 0–0.5 cm/frame group, WorldFlow achieves lower ADE on 54 clips, including 39 clips with lower ADE and FDE. Small motion therefore does not inherently prevent WorldFlow from making more accurate predictions.
3. Instruction ambiguity and single-reference evaluation help interpret the Real3 results.
ADE and FDE measure the distance between predicted trajectories and a single reference trajectory, whereas the same initial observation and text instruction may allow multiple plausible future motions. When the predicted motion direction or path differs from the actual recording, even a prediction consistent with the input conditions can incur a large trajectory error.
The Results – Real3 Static Baseline Analysis section of our supplementary website illustrates this phenomenon: some predictions are visually plausible and consistent with the instructions, but their motion directions differ from the reference trajectories, resulting in higher ADE/FDE than Static. Real3 consists of clips from everyday real-world scenes, where descriptions typically specify the task but do not fully constrain the motion direction, path, magnitude, or endpoint. Consequently, the same initial observation and instruction may correspond to multiple plausible future motions. When the recorded motion remains close to the initial position, Static may achieve lower error than a model predicting another plausible trajectory.
Together, the stratified results and qualitative examples suggest that motion magnitude alone does not fully explain performance on Real3; interpreting the results requires considering the difference between multiple plausible futures and a single recorded reference.
Response to W1/Q1: Video-Motion Consistency and Video Quality
We thank the reviewer for their insightful comments.
W1/Q1. Video-Motion Consistency and Video Quality
We thank the reviewer for their valuable suggestions regarding consistency and video quality in joint generation. We have added two evaluations to examine cross-modal consistency and video quality relative to the original backbone.
Video-motion consistency. We have added a Video-Motion Consistency section to the supplementary material, comparing projected 3D trajectories with independently tracked point positions in the jointly generated videos, while accounting for estimated camera motion, visibility, and tracking confidence. The table summarizes pixel discrepancies across 1,678 test videos at 480 × 288 resolution.
Mean pointwise discrepancy in pixels (lower is better). Thresholds apply to the tracker’s visibility-confidence product.
At a tracking-confidence threshold of 0.9, the mean discrepancy over all 48 future frames is 7.31 pixels, supporting spatial correspondence between the two generated modalities on high-confidence observations. Increasing the threshold from 0.5 to 0.9 reduces the mean discrepancy from 12.30 to 7.31 pixels, indicating that WorldFlow’s projected 3D trajectories align more closely with pixel trajectories assigned higher confidence by the tracker. Generated outputs and camera projections remain fixed; only the retained observations change. Detailed settings, confidence-sensitivity analysis, and visualizations are provided in the supplementary section.
Video quality and dynamic degree. We compare videos generated by WorldFlow and the original Cosmos-Nano under matched generation settings on all 1,678 test inputs, using MUSIQ [1] to measure image quality and VBench Dynamic Degree [2] to quantify the proportion of videos containing substantial motion.
Overall, WorldFlow’s MUSIQ decreases by 2.40% relative to Cosmos-Nano, while its dynamic degree increases by 19.67 percentage points. This indicates a slight decrease in image quality, while generated videos more frequently contain substantial motion. Together with our motion-prediction results, these findings characterize an empirical trade-off between stronger flow prediction and a slight reduction in image quality.
References
[1] Ke et al. MUSIQ: Multi-Scale Image Quality Transformer. ICCV, 2021.
[2] Huang et al. VBench: Comprehensive Benchmark Suite for Video Generative Models. CVPR, 2024.