- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a new benchmark for video world models. The authors introduced Reasoning-Structured Videos, with 11,37 videos and built on Unreal Engine 5. The videos are exploration and relation-structured and have action and camera-pose annotations. The evaluation can be have 3 trajectory categories, including inverse, loop, and equivalence. The evaluation separates self-consistency from reference fidelity and enhances image-based metrics with layout and geometry readouts. The evaluation has been conducted across 9 world-model configurations.
Critical strengths
-
The proposed benchmark contains a large-scale of videos with detailed structured designs. This could be useful for the video world model designs.
-
The trajectory categories of inverse, loop, and equivalence are reasonable. And exactly what the video world model evaluation needed.
Critical weaknesses
-
The benchmark is named with "Reasoning". However, I cannot understand what is the relationship between this work with reasoning. "Reasoning" in the large-model era has been deeply studied for years, for very different domains. I do not support Inverse, Loop, and Equivalence could be rephrased as reasoning.
-
The videos in the benchmarks are simulated from Unreal Engine. Are there any real-world videos could be included? For example, the autonomous driving domain has some real traffic videos with actions. They could make the benchmark more useful.
-
The evaluation metrics are LPIPS, PSNR, and FID, which are too old-school. Could the authors designs some more meaningful metrics. For example, including the wall-clock time that each world model spends to finish a loop trajectory, to measure the efficiency of the world model. And combine it with the accuracy-level metrics. This will highlights the practicality of video world models.
-
Some of the tested world models are trained using game videos, which includes videos from Unreal Engine. It might be in-domain and unfair for evaluation.
Questions for authors
- Based on the evaluation results from the benchmarks, are there any tips the authors can summarize to benefit the future video world model designs or training?
4 comments
W2/W3/W4. Real-world recordings and wall-clock time.
The videos in the benchmarks are simulated from Unreal Engine. Are there any real-world videos could be included? Could the authors designs some more meaningful metrics. For example, including the wall-clock time that each world model spends to finish a loop trajectory, to measure the efficiency of the world model. And combine it with the accuracy-level metrics.
Following this suggestion, we collected data from public real-world datasets—ARKitScenes, ETH3D, TUM RGB-D, 7-Scenes, and Oxford Spires—and used their GT camera trajectories to select samples satisfying our three trajectory relations. Even across these datasets, only a small number of samples met the criteria. This highlights the value of our synthetic dataset in scaling up data with specific trajectory structures. We evaluated each model on 100 graphs covering the three relations: 33 Inverse, 32 Loop, and 35 Equivalence.
Each Real rollout is conditioned on a single real image and its associated camera trajectory. We compare the generated endpoint with the corresponding GT endpoint, averaging the two branch-to-GT scores for Equivalence. For wall-clock timing, each model generates a 5-second video at 360p on a single NVIDIA A100 GPU using the same input. We report the mean time over five runs. Quality scores below are LPIPS ↓ / PSNR ↑.
| Model | Inverse UE | Inverse Real | Loop UE | Loop Real | Equivalence UE | Equivalence Real | Time ↓ (s) |
|---|---|---|---|---|---|---|---|
| minWM | 0.571 / 12.93 | 0.781 / 9.67 | 0.620 / 12.67 | 0.785 / 7.94 | 0.582 / 13.59 | 0.812 / 8.80 | 14.80 |
| LingBot | 0.461 / 15.97 | 0.784 / 10.56 | 0.550 / 14.07 | 0.759 / 10.24 | 0.544 / 14.27 | 0.759 / 10.79 | 18.09 |
| MagicWorld-Fast | 0.517 / 14.49 | 0.682 / 13.00 | 0.572 / 14.54 | 0.703 / 11.22 | 0.551 / 14.57 | 0.797 / 10.26 | 14.58 |
| SANA-WM | 0.589 / 11.82 | 0.760 / 10.06 | 0.667 / 10.47 | 0.716 / 9.40 | 0.654 / 11.21 | 0.795 / 9.23 | 18.13 |
MagicWorld ranks higher and shows a greater advantage on real-world data than on UE videos, consistent with its training primarily on real videos. This highlights the difficulty of comparing models trained on different datasets and the value of real-world data for a more comprehensive evaluation.
We provide sample IDs for examples satisfying the three trajectory relations on this link: Real-domain Data. The release contains only sample identifiers, without redistributing the original videos. We have also checked that our use of the source datasets for research purposes is permitted under their respective licenses.
Thanks for your detailed reply. Most of my concerns have been resolved. I improved the rate to weak acceptance. Thanks.
Thank you for your positive assessment of our work and your constructive feedback. We will incorporate the additional experiments on real-world datasets into Section 5.2 to facilitate comparison and joint analysis with the original Unreal Engine dataset. We will also add a new subsection to Section 5 to discuss the practical implications of our experimental findings for training world models. We sincerely appreciate your valuable suggestions and the time you have devoted to reviewing our work.
We thank the reviewer for recognizing the value of our work for designing and evaluating video world models and the soundness of our three trajectory designs. We also appreciate the valuable suggestions to incorporate real-world data for fairer evaluation and provide deeper analysis. Following these suggestions, we have added experiments on real-world data and further analysis of the implications for model design and training, which have strengthened our paper.
Q1. Insights into training and designing consistent video world models
Controlled comparisons are difficult because models differ in training data, parameter counts, and training procedures. Our core contribution is a synthetic dataset for fine-grained evaluation of world consistency. Its results inform model training and design but do not isolate the causal effects of individual choices.
(1) Strong short-range consistency does not necessarily translate into strong long-range consistency, suggesting that training on longer video sequences is important for long-range consistency. Table 2 in paper evaluates Inverse self-consistency at control-step gaps of 10–40 on the same 50 pose-certified graphs. Among the roughly 1.3B-scale Wan-based models, Chunk-AR achieves lower LPIPS at a gap of 10 than LingBot Light and MagicWorld-Fast (0.322 versus 0.435 and 0.456, respectively). At a gap of 40, however, these differences narrow substantially (0.631 versus 0.641 and 0.654). Training only on short video sequences may therefore yield an advantage that does not persist during long-term inference, motivating the inclusion of longer sequences in training.
(2) Broad scene coverage and training strategies that address error accumulation are promising directions for video world models. We use a user study and average benchmark scores to compare all models. The table below lists the top four models in each evaluation, excluding Chunk-AR, our in-domain reference model. In the user study, each of the 34 participants rated 18 examples on a five-point scale for object preservation, spatial layout, appearance/style consistency, and visual quality. The table reports mean scores. We provides example images of the samples used in this experiment (in the user study, both images and videoes are provided) at this link: User study image link.
Across the quantitative results and human ratings above, HY-WorldPlay, LingBot, and Matrix-Game 2.0 stand out overall. They combine real and synthetic videos to broaden scene coverage and expand the training data. They also address error accumulation through Context Forcing, distribution matching distillation over long self-rollouts, and Self-Forcing, respectively. However, minWM also employs advanced forcing techniques without showing a substantial advantage on our benchmark, suggesting that training data scale and model size may play a more important role.