acceptodds
Under review as a conference paper at ICLR 2027

Reasoning-Structured Videos: A Stratified Diagnostic Suite for Compositional Consistency in World Models

Abstract

Existing evaluations of video world models measure visual fidelity, control adherence, and revisit consistency. We extend this perspective from within-rollout revisitation to agreement across distinct histories reaching the same reference state. We introduce Reasoning-Structured Videos, an Unreal Engine 5 benchmark for evaluating these relationships in static scenes. The dataset comprises 11,377 exploration and relation-structured videos with action and camera-pose annotations. Its relation-structured trajectories form rooted graphs with shared initial states and annotated endpoint relationships. Inverse tests recovery after retracing a path, Loop tests return after an exploratory detour, and Equivalence tests agreement between distinct paths reaching the same state. Our evaluation separates self-consistency from reference fidelity and complements image-based metrics with layout and geometry readouts. Across nine world-model configurations, return-consistency rankings broadly align with distributional quality, whereas cross-path agreement yields different rankings. Strong layout agreement can coexist with poor geometric fidelity, and revisit consistency deteriorates over longer intervals. Together, these findings show why cross-history agreement, reference fidelity, and long-range retention should be evaluated separately.

Primary area:
datasets and benchmarks
Keywords:
World Model, Video Generation, Dataset and Benchmark

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_cyEvOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a new benchmark for video world models. The authors introduced Reasoning-Structured Videos, with 11,37 videos and built on Unreal Engine 5. The videos are exploration and relation-structured and have action and camera-pose annotations. The evaluation can be have 3 trajectory categories, including inverse, loop, and equivalence. The evaluation separates self-consistency from reference fidelity and enhances image-based metrics with layout and geometry readouts. The evaluation has been conducted across 9 world-model configurations.

Critical strengths
  1. The proposed benchmark contains a large-scale of videos with detailed structured designs. This could be useful for the video world model designs.

  2. The trajectory categories of inverse, loop, and equivalence are reasonable. And exactly what the video world model evaluation needed.

Critical weaknesses
  1. The benchmark is named with "Reasoning". However, I cannot understand what is the relationship between this work with reasoning. "Reasoning" in the large-model era has been deeply studied for years, for very different domains. I do not support Inverse, Loop, and Equivalence could be rephrased as reasoning.

  2. The videos in the benchmarks are simulated from Unreal Engine. Are there any real-world videos could be included? For example, the autonomous driving domain has some real traffic videos with actions. They could make the benchmark more useful.

  3. The evaluation metrics are LPIPS, PSNR, and FID, which are too old-school. Could the authors designs some more meaningful metrics. For example, including the wall-clock time that each world model spends to finish a loop trajectory, to measure the efficiency of the world model. And combine it with the accuracy-level metrics. This will highlights the practicality of video world models.

  4. Some of the tested world models are trained using game videos, which includes videos from Unreal Engine. It might be in-domain and unfair for evaluation.

Questions for authors
  1. Based on the evaluation results from the benchmarks, are there any tips the authors can summarize to benefit the future video world model designs or training?
4 comments

We thank the reviewer for recognizing the value of our work for designing and evaluating video world models and the soundness of our three trajectory designs. We also appreciate the valuable suggestions to incorporate real-world data for fairer evaluation and provide deeper analysis. Following these suggestions, we have added experiments on real-world data and further analysis of the implications for model design and training, which have strengthened our paper.

Q1. Insights into training and designing consistent video world models

Based on the evaluation results from the benchmarks, are there any tips the authors can summarize to benefit the future video world model designs or training?

Controlled comparisons are difficult because models differ in training data, parameter counts, and training procedures. Our core contribution is a synthetic dataset for fine-grained evaluation of world consistency. Its results inform model training and design but do not isolate the causal effects of individual choices.

(1) Strong short-range consistency does not necessarily translate into strong long-range consistency, suggesting that training on longer video sequences is important for long-range consistency. Table 2 in paper evaluates Inverse self-consistency at control-step gaps of 10–40 on the same 50 pose-certified graphs. Among the roughly 1.3B-scale Wan-based models, Chunk-AR achieves lower LPIPS at a gap of 10 than LingBot Light and MagicWorld-Fast (0.322 versus 0.435 and 0.456, respectively). At a gap of 40, however, these differences narrow substantially (0.631 versus 0.641 and 0.654). Training only on short video sequences may therefore yield an advantage that does not persist during long-term inference, motivating the inclusion of longer sequences in training.

(2) Broad scene coverage and training strategies that address error accumulation are promising directions for video world models. We use a user study and average benchmark scores to compare all models. The table below lists the top four models in each evaluation, excluding Chunk-AR, our in-domain reference model. In the user study, each of the 34 participants rated 18 examples on a five-point scale for object preservation, spatial layout, appearance/style consistency, and visual quality. The table reports mean scores. We provides example images of the samples used in this experiment (in the user study, both images and videoes are provided) at this link: User study image link.

RankHuman evaluationMean score ↑Our benchmark evaluationMean LPIPS ↓
1HY-WorldPlay3.817 / 5HY-WorldPlay0.493
2LingBot Light3.373 / 5LingBot Light0.559
3Matrix-Game 2.03.138 / 5MagicWorld-Fast0.571
4MagicWorld-Fast3.088 / 5Matrix-Game 2.00.609

Across the quantitative results and human ratings above, HY-WorldPlay, LingBot, and Matrix-Game 2.0 stand out overall. They combine real and synthetic videos to broaden scene coverage and expand the training data. They also address error accumulation through Context Forcing, distribution matching distillation over long self-rollouts, and Self-Forcing, respectively. However, minWM also employs advanced forcing techniques without showing a substantial advantage on our benchmark, suggesting that training data scale and model size may play a more important role.

W2/W3/W4. Real-world recordings and wall-clock time.

The videos in the benchmarks are simulated from Unreal Engine. Are there any real-world videos could be included? Could the authors designs some more meaningful metrics. For example, including the wall-clock time that each world model spends to finish a loop trajectory, to measure the efficiency of the world model. And combine it with the accuracy-level metrics.

Following this suggestion, we collected data from public real-world datasets—ARKitScenes, ETH3D, TUM RGB-D, 7-Scenes, and Oxford Spires—and used their GT camera trajectories to select samples satisfying our three trajectory relations. Even across these datasets, only a small number of samples met the criteria. This highlights the value of our synthetic dataset in scaling up data with specific trajectory structures. We evaluated each model on 100 graphs covering the three relations: 33 Inverse, 32 Loop, and 35 Equivalence.

Each Real rollout is conditioned on a single real image and its associated camera trajectory. We compare the generated endpoint with the corresponding GT endpoint, averaging the two branch-to-GT scores for Equivalence. For wall-clock timing, each model generates a 5-second video at 360p on a single NVIDIA A100 GPU using the same input. We report the mean time over five runs. Quality scores below are LPIPS ↓ / PSNR ↑.

ModelInverse UEInverse RealLoop UELoop RealEquivalence UEEquivalence RealTime ↓ (s)
minWM0.571 / 12.930.781 / 9.670.620 / 12.670.785 / 7.940.582 / 13.590.812 / 8.8014.80
LingBot0.461 / 15.970.784 / 10.560.550 / 14.070.759 / 10.240.544 / 14.270.759 / 10.7918.09
MagicWorld-Fast0.517 / 14.490.682 / 13.000.572 / 14.540.703 / 11.220.551 / 14.570.797 / 10.2614.58
SANA-WM0.589 / 11.820.760 / 10.060.667 / 10.470.716 / 9.400.654 / 11.210.795 / 9.2318.13

MagicWorld ranks higher and shows a greater advantage on real-world data than on UE videos, consistent with its training primarily on real videos. This highlights the difficulty of comparing models trained on different datasets and the value of real-world data for a more comprehensive evaluation.

We provide sample IDs for examples satisfying the three trajectory relations on this link: Real-domain Data. The release contains only sample identifiers, without redistributing the original videos. We have also checked that our use of the source datasets for research purposes is permitted under their respective licenses.

Thanks for your detailed reply. Most of my concerns have been resolved. I improved the rate to weak acceptance. Thanks.

Thank you for your positive assessment of our work and your constructive feedback. We will incorporate the additional experiments on real-world datasets into Section 5.2 to facilitate comparison and joint analysis with the original Unreal Engine dataset. We will also add a new subsection to Section 5 to discuss the practical implications of our experimental findings for training world models. We sincerely appreciate your valuable suggestions and the time you have devoted to reviewing our work.

Reviewer_dRidOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces a benchmark based on UE5 to test whether video world models keep scenes consistent across movement paths. It tests retracing a path, returning after a detour, and taking different routes to the same endpoint. Its main contribution is to compare the final views reached through different paths and separately evaluate how well generated frames match the ground truth. Experiments on nine model configurations show that good visual quality does not guarantee consistency, consistent predictions can still be wrong, and consistency generally declines over longer revisit intervals.

Critical strengths
  1. This paper evaluates whether different paths to the same state produce consistent final views. This adds a useful dimension beyond standard revisits does.

  2. The experiments show an important conclusion in video world model: visual quality, endpoint agreement, and geometric fidelity may not be consistent. This helps identify failures that a single quality score would miss.

Critical weaknesses
  1. The comparisons with NMR and FVD are useful, but different model rankings alone do not show what Equivalence adds beyond existing revisit tests. It would help to show cases where revisit tests suggest good consistency but different paths lead to conflicting scenes, and report how often this happens.

  2. The SAM/depth metrics and pose-error calibration are useful checks, but it remains unclear how well the scores capture scene consistency. Appearance changes may affect the scores even when the scene is preserved, while meaningful object or layout changes may be missed. Validation against human judgments or ground-truth scene annotations would make the evaluation more convincing.

Questions for authors
  1. What are the actual start–end pose errors in the Loop evaluation set, and do the main findings hold on trajectories with exact loop closure?

  2. Beyond the SAM/depth comparisons and pose-error calibration, is there evidence that the scores agree with human judgments or ground-truth object and layout consistency?

  3. Do the existing comparisons identify clear scene conflicts that Equivalence detects but revisit metrics miss? If so, how frequently does this occur?

5 comments

We thank the reviewer for recognizing that our work evaluates models from complementary perspectives, yields useful insights, and highlights the importance of cross-path consistency through Equivalence. We also appreciate the constructive suggestions on aligning metrics with human judgments, especially for difficult aspects such as layout and spatial consistency, and on testing whether Equivalence reveals errors missed by revisit evaluation.

In response, we conducted metric–human correlation analyses and a human study comparing Equivalence and revisit across models. We also added hundreds of analysis examples, which have substantially strengthened the paper. We sincerely thank the reviewer for these insightful comments.

Q1. How closely do Loop trajectories return to their starting pose?

What are the actual start–end pose errors in the Loop evaluation set, and do the main findings hold on trajectories with exact loop closure?

Closure error is small and far below model error, so the observed revisit errors primarily reflect model inconsistency rather than imperfect closure. We also release 1,000 new numerically closed Loop videos with camera/action metadata: Hugging Face link.

Because Loop trajectories were generated stochastically, small closure errors could occur. Closure error over all Loop trajectories (all Inverse and Equivalence endpoints hold to numerical precision):

TranslationRotationAvg. path lengthAvg. angular travelRelative translationRelative rotation
12.71 cm0.393°13.04 m213.64°0.97%0.18%

The mean LPIPS is 0.103 for GT and 0.713 for model outputs, even exact-loop GT endpoints yield 0.042 due to lighting and slight leaf motion. Thus, the resulting systematic bias is limited to Loop-SC and is small relative to the model error, which is typically much larger due to long-horizon drift and memory limitations. GT-anchor remains valid for imperfectly closed Loops because it compares each output with the GT image at the actual endpoint. Our core findings remain valid because they are derived jointly across all three trajectory relations and both evaluation tiers, SC and GT-anchor.

Q2 / W2. Do the scores agree with human judgments of scene consistency?

Beyond the SAM/depth comparisons and pose-error calibration, is there evidence that the scores agree with human judgments or ground-truth object and layout consistency?

We use human study to show that SAM and Depth correlate with human judgments of scene consistency.

In our human study, 34 participants (gamers and artists) each rated 18 anonymized examples on a 5-point scale for layout and object consistency. Across the study, we evaluated 162 unique outputs, corresponding to 54 model–graph examples for each relation. The table reports Pearson r with human ratings.

ScoreInv. LayoutInv. ObjectLoop LayoutLoop ObjectEquiv. LayoutEquiv. ObjectMeanMin
PSNR0.2600.3270.5420.4020.3950.3010.3710.260
−LPIPS0.4890.5950.4380.4510.2960.1550.4040.155
Depth0.5800.6610.5110.5120.3770.4850.5210.377
SAM0.4660.5110.4930.5030.4890.5300.4990.466

We report −LPIPS so that larger values consistently indicate better agreement.

These results show that the SAM/Depth scores agree with human judgments of object and layout consistency. We provides example images of the samples used in this experiment (in the user study, both images and videoes are provided) at this link: User study image link.

Q3 / W1. Can Equivalence reveal inconsistencies that Loop misses?

Do the existing comparisons identify clear scene conflicts that Equivalence detects but revisit metrics miss? If so, how frequently does this occur?

In 31.1% of cases, Equivalence reveals more inconsistency than revisit

To address this question within the existing evaluation, we identify a subset of path pairs from the current Relation Evaluation Set that satisfies the following setup, no new trajectories or model runs are introduced.

Setup. We use path pairs (A, B) that occupy the same GT pose at times i and j after following different routes. All four frames correspond to the same ground-truth pose: the paths meet at time i, diverge, and then return to that pose at time j. Loop compares frames within each path (Aᵢ–Aⱼ, Bᵢ–Bⱼ), whereas Equivalence compares frames across paths at the same generation times (Aᵢ–Bᵢ, Aⱼ–Bⱼ). Both comparisons therefore use the same four generated frames. This controls for changes in generation quality with rollout length. The set contains 25 time pairs from 16 graphs, evaluated on 9 configurations, giving 225 comparisons. Gaps range from 1.1 to 9 s (median 2.25 s).

Human study. Seven participants judged all 225 comparisons blind, and each comparison received its majority label. They were asked whether scene inconsistencies were greater under revisits or Equivalence.

OutcomeCount (%)95% CI
Equiv. more revealing70 (31.1%)22.7–40.4%
Difficult to compare146 (64.9%)57.7–71.3%
Revisit more revealing9 (4.0%)1.1–7.0%

Confidence intervals are obtained from 10,000 graph-level bootstrap resamples.

The remaining 64.9% were judged difficult to compare, meaning that the two relations exposed inconsistencies of similar severity. This is expected: Equivalence is not intended to replace revisit, but to probe a distinct failure mode. Revisit tests whether a model remembers a state along a single trajectory, whereas Equivalence tests whether different trajectories agree on the same state. The two are complementary, and we therefore retain both in our benchmark.

We provide all comparisons and annotations for this experiment at: Hugging Face link. These results show that good revisit performance does not guarantee consistency across equivalent paths: a model may retain sufficient memory to revisit a previously observed state, yet still generate inconsistent scenes when following alternative paths to the same state.

Dear Reviewer, we appreciate the time and care you have put into reviewing our paper. Your questions prompted us to analyze Loop closure and conduct human studies assessing both metric agreement with human judgments and the additional inconsistencies revealed by Equivalence. If your schedule permits, we would be grateful to hear your perspective on this new evidence. Please let us know if any aspect would benefit from further explanation. Many thanks for your thoughtful suggestions and continued consideration.

Reviewer_XnL2OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces a diagnostic benchmark for video world models to evaluate compositional consistency in static 3D environments. Rather than relying solely on perceptual fidelity (FVD) or within-trajectory revisitation, it frames consistency around three algebraic relations: Inverse, Loop, and Equivalence (different paths reaching the same state). It decouples evaluation into a GT-anchored tier (ground-truth reference correctness) and a Self-Consistency (SC) tier (internal model agreement), supplemented by SAM layout and depth readouts.

Critical strengths
  1. Algebraic Formulation Beyond Revisit: Extends evaluation from simple reversal to cross-history path equivalence (TA=TBT_A = T_B). This isolates route dependence from generation quality.

  2. Decoupled Two-Tier Evaluation: Separating GT-anchored fidelity from internal self-consistency exposes critical failure modes, such as models achieving high internal consensus on collapsed or shared-wrong states (e.g., MiniWorld).

  3. Insightful Empirical Findings: The paper clearly demonstrates that revisit consistency metrics correlate heavily with FVD unless decoupled, and confirms that short-range consistency does not predict long-range retention.

Critical weaknesses
  1. Static-Only Scope: Evaluates only camera navigation in static scenes. Real-world world modeling often involves irreversible physics, dynamic agents, and multi-agent interactions, which fall outside this framework.

  2. Action Space & Embodiment Discrepancy: The action space primarily relies on a fixed set of 9 discrete actions and continuous trajectory interpolation, which remains oversimplified compared to realistic continuous motion control or non-rigid character control (e.g., complex third-person animations)..

Questions for authors
  1. Fot the use of metric, how do you distinguish between stochastic variation in texture/details and genuine geometric failure when comparing two rollouts reaching the same terminal pose?

  2. Context length bias: To what extent does the large discrepancy in conditioning history (e.g., minWM receiving 180 frames vs. HY-WorldPlay receiving 1 frame) explain the relative retention gaps?

  3. Extension to Irreversible Transitions: Can the compositional consistency framework be extended to environments with non-invertible physical dynamics?

6 comments

We thank the reviewer for recognizing our trajectory relations and decoupled two-tier evaluation as sound and valuable designs for fine-grained model assessment, and for highlighting our insightful empirical findings. We also appreciate the suggestions for further analysis of conditioning history length and the evaluation of texture and spatial layout. Following these suggestions, we conducted additional analyses and experiments, yielding interesting findings and strengthening our paper.

Q1. Texture/details and genuine geometric failure

Fot the use of metric, how do you distinguish between stochastic variation in texture/details and genuine geometric failure when comparing two rollouts reaching the same terminal pose?

When comparing two rollouts reaching the same terminal pose, we combine texture metrics (e.g., PSNR) and spatial layout metrics (e.g., Depth and SAM) to evaluate Equivalence. We use SAM features and DepthAnything-derived depth maps to assess semantic and spatial consistency beyond pixel-level similarity. SAM's deep features capture higher-level scene content and spatial organization, reducing sensitivity to variations in texture and fine details, while DepthAnything provides a complementary comparison of relative depth structure. Together, these measures help distinguish appearance variation from changes in scene layout and geometry.

We further designed a human study in which 34 participants (gamers and artists) each rated 18 anonymized examples on a five-point scale for layout and object consistency. The study covered 162 unique outputs, with 54 model–graph examples per relation. The table reports Pearson's r between metric scores and human ratings.

ScoreInv. LayoutInv. ObjectLoop LayoutLoop ObjectEquiv. LayoutEquiv. ObjectMeanMin
PSNR0.2600.3270.5420.4020.3950.3010.3710.260
−LPIPS0.4890.5950.4380.4510.2960.1550.4040.155
Depth0.5800.6610.5110.5120.3770.4850.5210.377
SAM0.4660.5110.4930.5030.4890.5300.4990.466

We report −LPIPS so that larger values consistently indicate better agreement.

SAM and Depth both show positive correlations with human judgments across all three relations and both rating dimensions, with mean correlations of 0.521 for Depth and 0.499 for SAM. This supports their use as complementary indicators of object and spatial layout consistency. Example images are available on this link: User Study link, participants were shown both images and videos.

Q2. Effect of conditioning history length

To what extent does the large discrepancy in conditioning history explain the relative retention gaps?

We compared single-frame and 180-frame GT conditioning for three models on 150 graphs, with 50 per trajectory relation. The table reports LPIPS ↓ / PSNR ↑, with better values within each model shown in bold.

ModelConditioning historyInverseLoopEquivalence
MagicWorld-Fast1-frame0.5170 / 14.490.5720 / 14.540.5510 / 14.57
MagicWorld-Fast180-frame0.6085 / 14.830.6063 / 15.240.6232 / 14.17
SANA-WM1-frame0.5890 / 11.820.6670 / 10.470.6540 / 11.21
SANA-WM180-frame0.5168 / 14.850.5901 / 14.220.6097 / 13.70
Matrix-Game 2.01-frame0.4870 / 13.820.5920 / 13.210.5690 / 12.46
Matrix-Game 2.0180-frame0.4969 / 14.060.5850 / 13.280.5515 / 13.16

The two settings use frame 180 and frames 1–180 as context, respectively, and evaluate predictions for frames 181–360.

This comparison further probes whether models can effectively use longer observed histories to generate future frames. Matrix-Game 2.0 shows relatively small changes. MagicWorld-Fast shows modest PSNR changes but worse LPIPS, with no consistent benefit from the longer history. In contrast, SANA-WM improves across all three relations: its mean LPIPS decreases from 0.6367 to 0.5722, while mean PSNR rises from 11.17 to 14.26 dB. This improvement may reflect its long-context design: frame-wise Gated DeltaNet maintains a recurrent memory with gated updates and forgetting, combined with softmax attention and training on minute-long sequences (SANA-WM, Sections 3.1–3.2). These results suggest that the benefit of additional conditioning depends on how effectively each model uses that history, rather than its length alone.

We will standardize conditioning history lengths across models and report results under both single-frame and multi-frame settings to provide a more comprehensive assessment of their capabilities.

Q3 / W1 / W2. Scenes, action space and irreversible physical changes

Extension to Irreversible Transitions: Can the compositional consistency framework be extended to environments with non-invertible physical dynamics?

Our framework remains useful when scene changes are small and localized, as it captures larger consistency failures while allowing some tolerance for minor variations.

We would like to clarify that our scenes include some dynamic elements: they include changing shadows, swaying leaves, rotating fans, and flowing water, reflecting some of the changes found in real-world environments. However, they do not include substantial motion from vehicles or people, which we consider an interesting direction for future work.

We believe that irreversible physical changes require more fine-grained evaluation and analysis. When such changes are localized—for example, a cup falling and breaking—they can be considered local scene variations, similar in spatial extent to the shadow changes already present in our scenes. Our metrics, including PSNR, LPIPS, FID, SAM, and Depth, allow some tolerance for these changes in detail. In contrast, the failures of current video world models on revisits and equivalent paths more often involve forgetting due to ineffective memory mechanisms, producing plausible or visually appealing but inconsistent content, or long-term drift that changes the visual style or deforms objects. These failures currently have a greater impact than minor variations in detail.

In these cases, our framework remains applicable because the metrics can capture larger inconsistencies while allowing some tolerance for minor variations. Their agreement with human judgments is further supported by the human study reported in Q1.

Regarding Action Space & Embodiment Discrepancy, our actions are discrete, but each translation or rotation uses a small increment, mimicking discrete keyboard controls in games (e.g., WASD for movement). A fully continuous action space would make constructing Loop, Inverse, and Equivalence trajectories more challenging, but remains an interesting and valuable direction for future work.

Dear Reviewer, thank you again for your thoughtful feedback. We have provided detailed responses to your comments, including a human study assessing metric alignment with layout and object consistency judgments, and experiments comparing single-frame and 180-frame conditioning. We would appreciate it if you could take a look when you have a chance and let us know whether our responses address your concerns. We would be happy to provide any further clarification. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.