acceptodds
Under review as a conference paper at ICLR 2027

A Plausible Future Is Not a Causal Future: Can Video Models Generate How the World Unfolds?

Abstract

Large-scale video generation models are increasingly viewed as a promising path toward world models, yet their ability to capture causally coherent physical dynamics remains poorly understood. Existing evaluations largely emphasize visual quality, prompt adherence, or broad physical plausibility, leaving open whether a model can faithfully realize a complete causal cascade, in which an initiating event propagates through causally dependent interactions to a terminal outcome. We introduce **CausalCascade**, a simulator-grounded benchmark centered on 50 event-rich Blender scenes with structured event graphs, graph-grounded diagnostic VQA, three levels of future-event specification, and 2,832 controlled counterfactual variants, and use it to evaluate nine image-to-video models with complementary motion-, event-, and counterfactual-level diagnostics. Across the evaluated models, generated motion is more reliably localized spatially than realized with correct spatiotemporal evolution, while richer future-event specification improves event recovery but leaves physical realization limited. Counterfactual evaluation further reveals weak selective responsiveness to physical interventions, while most audited failures are foundational, arising at scene consistency or event occurrence. Together, these results position causal cascade generation as a process-level test of world modeling and highlight the need to jointly recover the intended future, faithfully realize its physical evolution, selectively respond to physical interventions, and maintain a coherent physical world over time.

Primary area:
datasets and benchmarks
Keywords:
Video Generation, World Models, Causal Dynamics, Physical Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_rCS3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents CausalCascade, a causal cascade generation benchmark to evaluate whether I2V models can generate physically plausible events with causal dependencies sequentially. The benchmark includes 50 Blender scenes, structured event graphs, three-level future event prompts, and 2832 counterfactual variants. The authors evaluated nine I2V models on CausalCascade. The results show that more detailed prompts can improve event recovery, but the physical process, temporal coherence, and intervention response remain unreliable.

Critical strengths
  1. Future-event specification at three levels and Temporal alignment separately examine the impact of insufficient event information and temporal misalignment, which is more helpful in explaining the causes of failure than simply reporting the quality of the generated results.
  2. The authors consider four types of physical interventions and independently re-simulate each modified scene. The evaluation subset is selected without reference to model performance, and factual-counterfactual pairs use matched prompts. These choices make the pairwise comparisons more reliable.
Critical weaknesses
  1. CF Spatial Effect IoU and CF Preserve Spatial IoU seem to be unable to effectively differentiate between the model's fundamental generation errors and true causal response errors. Suppose a model already struggles to generate accurate factual rollouts; even if the model can respond to interventions, it may still fail to obtain the correct motion mask due to insufficient base generation quality.
  2. The benchmark seems to conflate fidelity to a specific simulator-generated rollout with causal correctness. It compares each generated video against a single reference rollout, potentially assigning low scores to alternative physically plausible futures that preserve the correct causal dependencies but differ in motion details or timing.
Questions for authors

Please see the weakness part.

6 comments
AuthorsOpenReview ↗

Response to Reviewer rCS3

We sincerely thank the reviewer for recognizing the diagnostic value of our three-level future-event specification and temporal-alignment analyses, as well as for the thoughtful assessment of our factual-counterfactual evaluation design.

W1: Decomposing Base Generation Quality and Conditional Intervention Response.

We thank the reviewer for this insightful observation regarding the interpretation of our motion metrics. It motivated us to further disentangle base-motion realization from conditional intervention response.

The original CF Spatial Effect IoU and CF Preserve Spatial IoU provide useful end-to-end measures of agreement with the expected intervention-related motion. We therefore retain these metrics while introducing a unified two-stage decomposition of the recall-side coverage of both Preserve and Effect into Base Realization (CC) and Conditional Response (RR).

For Preserve, let P∗P^\ast denote the ground-truth shared-motion map as defined in Appendix F. Base Realization (CP=∣P∗∩M^f∣∣P∗∣C_P=\frac{|P^\ast\cap\hat M^f|}{|P^\ast|}) measures how much of the ground-truth shared motion is realized in the factual generation; Conditional Response (RP=∣P∗∩M^f∩M^cf∣∣P∗∩M^f∣R_P=\frac{|P^\ast\cap\hat M^f\cap\hat M^{cf}|}{|P^\ast\cap\hat M^f|}) measures how much of this realized motion is retained in the counterfactual generation.

For Effect, following Appendix F, let S∗S^\ast and A∗A^\ast denote the ground-truth motion maps to be suppressed and induced, respectively, and S^\hat S and A^\hat A their corresponding model generated motion maps.

  • Suppression: CS=∣S∗∩M^f∣∣S∗∣C_S=\frac{|S^\ast\cap\hat M^f|}{|S^\ast|} measures the realization of target motion in the factual generation; RS=∣S∗∩S^∣∣S∗∩M^f∣R_S=\frac{|S^\ast\cap\hat S|}{|S^\ast\cap\hat M^f|} measures how much of this motion is correctly suppressed in the counterfactual generation.
  • Induction: CA=∣A∗∩M^cf∣∣A∗∣C_A=\frac{|A^\ast\cap\hat M^{cf}|}{|A^\ast|} measures the realization of target motion in the counterfactual generation; RA=∣A∗∩A^∣∣A∗∩M^cf∣R_A=\frac{|A^\ast\cap\hat A|}{|A^\ast\cap\hat M^{cf}|} measures how much of this motion is genuinely absent from the factual generation.

In Table R3-1, all entries report Spatial metrics, following the original scene-macro aggregation and equal weighting across the four edit types and nine models. Complete definitions and corresponding weighted results have been provided in the Appendix H.3 of the revised PDF.

Table R3-1. Decomposition of Base Motion Realization and Conditional Intervention Response.

TaskBase realizationConditional response
Preserve57.50%81.73%
Suppression43.47%25.66%
Induction45.51%30.85%

Notably, the results reveal an asymmetry between preserving existing dynamics and selectively modifying them. Together, these results highlight two complementary bottlenecks: incomplete base-motion realization and limited selective intervention responsiveness, with the latter persisting even after conditioning on realized target motion.

W2: Interpreting Reference-based Evaluation.

We thank the reviewer for raising this concern. CausalCascade evaluates physical process realization conditioned on the initial scene states and relevant physical properties provided in the L1/L2 prompts, rather than open-ended future prediction. Under these specified conditions, Blender provides a reproducible reference rollout with fixed simulation settings.

Our motion evaluation also allows for variations in motion timing. Both Spatial IoU (which aggregates motion masks over time) and Weighted Spatial IoU (which accounts for each pixel's temporal activation frequency) relax the requirement for frame-wise temporal correspondence. For Strict ST-IoU, which is sensitive to temporal alignment, our original evaluation (Finding 3) further compares strict temporal matching with global affine alignment and constrained DTW, accounting for global and local timing variations.

Complementarily, graph-grounded VQA evaluates event occurrence, causal pathways, temporal ordering, and outcomes without requiring the generated video to reproduce the full motion details and timing of the reference rollout.

We thank the reviewer again for the time and thoughtful consideration. If our responses have adequately addressed your concerns, we would sincerely appreciate your reconsideration of the score.

Sincerely,
The Authors

AuthorsOpenReview ↗

A Gentle Follow-up on Our Rebuttal

Dear Reviewer rCS3,

Thank you again for your time and valuable feedback. We are writing to ask whether our rebuttal has addressed your concerns, and we would be happy to discuss any remaining issues if needed.

AuthorsOpenReview ↗

A Second Follow-up on Our Rebuttal

Dear Reviewer rCS3,

We would like to briefly follow up on our previous message. We have put considerable effort into addressing the concerns raised in your review, including conducting additional experiments and incorporating the corresponding analyses and clarifications into the revised manuscript.

If you have had a chance to review our responses, we would greatly appreciate any feedback on whether the additional evidence has addressed your concerns, or if there are still any points that require further clarification. We would be glad to address them promptly.

We sincerely appreciate your time and consideration.

Best regards,
The Authors

Reviewer_sbmVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CausalCascade, a benchmark for evaluating multi-stage physical generation and intervention responses in image-to-video models. It comprises 50 Blender rigid-body scenes, reference simulations, human-reviewed graphs containing 267 events and 226 direct causal edges, and 872 diagnostic questions. Three prompt levels provide the physical state, add the trigger, or replace the structured prompt with a full event-chain description. The authors retain 2,832 delete, move, scale, and mass variants. Across nine open-source models, they evaluate all 50 base scenes using motion-overlap metrics and hybrid VQA combining a tool-assisted VLM with deterministic scene-consistency rules, and assess 225 factual-counterfactual pairs using Effect/Preserve scores. The VQA evaluator agrees with one human annotator on 89 of 100 audited questions. Results show low motion and counterfactual agreement; the authors interpret richer event descriptions as benefiting event recovery more than physical realization. A manual audit of 90 rollouts from four base scenes assigns 84.4% of earliest failures to scene consistency or event occurrence.

Critical strengths
  1. The benchmark makes multi-stage physical generation concrete and inspectable. The combination of simulator references, human-reviewed event graphs, diagnostic questions, and re-simulated interventions supports investigation of specific events and dependencies. This is a useful foundation for studying how failures propagate through a physical process. The documented prompts, thresholds, seeds, and inference configurations also support reproducibility. These features make the benchmark a potentially useful resource for evaluation and diagnosis.

  2. The complementary analyses expose useful failure modes. The temporal-alignment analysis distinguishes strict timing agreement from agreement after limited alignment: model-average L3 Strict ST-IoU increases from 0.067 to 0.086 with affine alignment and to 0.104 with constrained DTW. The VQA breakdown and qualitative examples identify issues such as spontaneous motion, state replay, and causal shortcuts. These analyses offer more specific guidance than an aggregate plausibility or final-outcome score, although the prevalence estimates need to be interpreted within the audited sample.

Critical weaknesses
  1. Aggregate scores do not isolate intervention selectivity or explain Finding 2's relational gap. If a model copies its factual generation as its counterfactual, Preserve Spatial IoU becomes IoU(P∗,M^f)\mathrm{IoU}(P^*,\hat M^f) (Appendix F). It can be low from factual errors, without any intervention-induced disruption. Low Preserve alone cannot establish Finding 5's claim that responses disturb unaffected dynamics; this requires conditioning on factual realization and an actual response. Its correlation with factual Spatial IoU is 0.94–0.99, remaining 0.86–0.98 without the strongest model, although scene sets differ (Tables 7, 10–13). Temporal-ordering and causal-pathway scores also conflate absent endpoints with incorrect relations. Without the coverage and conditional accuracy defined but unreported in Appendix B.3, Finding 2's relational comparison cannot distinguish missing events from incorrect relationships between realized events.

  2. Limited scope and effective scale constrain the benchmark contribution. The 2,832 variants derive from 50 synthetic rigid-body Blender scenes built around recurring interaction motifs; only 225 pairs are evaluated. Increasing edits within these scenes does not establish broad coverage of cascade structures or physical mechanisms. RQ2 further selects interventions with clear downstream consequences, excluding edits that leave the observable process unchanged (Appendix D.5). This leaves an important regime of selective intervention response untested. The quantitative failure audit covers 90 rollouts from only four base scenes, and the model study is restricted to nine open-source systems. These choices limit both the resource's coverage and the reach of its findings. The submission establishes a focused testbed but provides insufficient evidence for the broader significance and representativeness of its benchmark contribution.

Questions for authors
  1. Do the scores isolate the claimed causal failures? For RQ2, please report a factual-copy baseline to quantify the Preserve attainable without responding to an intervention. On matched scene sets, does Finding 5 hold among pairs with adequate factual event realization and verified intervention responses? Please report retained counts and coverage if restricting the analysis, and use intervention-specific event checks or paired human judgments to distinguish incorrect responses from unrelated generation errors. For RQ1, please report the endpoint coverage and conditional relation accuracy defined in Appendix B.3: how much of the low pathway and ordering scores comes from missing events versus incorrect relationships between realized events?

  2. How sensitive are the motion results to frame rate and evaluation horizon? Appendix E.2 extracts masks before resampling and uses a fixed per-frame background update, giving time constants of 4.10 seconds at 8 fps and 1.37 seconds at 24 fps. Could you test the same reference videos at different frame rates, or recompute masks using a common physical-time scale, to quantify score sensitivity? Please also clarify how often relevant reference events extend beyond each model's output duration and whether a supplementary shared-horizon analysis changes the main findings.

  3. Does the prompt-level finding survive a controlled and calibrated comparison? In the L1–L2 comparison, which keeps the state description fixed, model-average Strict ST-IoU rises by 23.3% (0.043 to 0.053) and VQA Overall by 25.4% (0.193 to 0.242). From L1 to L3, the increases are 55.8% and 68.4%. Neither absolute nor relative gains are directly comparable measures of the underlying capabilities; what calibration supports the claim that event recovery benefits more? Could you retain the complete L1 state and constraints and append only the event chain, accounting for text-encoder truncation? Does the event-recovery benefit persist, and does scene consistency still decline for seven of nine models?

  4. What establishes the breadth and representativeness of the benchmark? Please report how the evaluated pairs cover distinct base scenes, cascade structures, intervened objects, and mass multipliers. Does the benchmark assess physically valid edits whose correct response is no observable change, in addition to the consequential edits selected for RQ2? How consistent are the main findings across scene families, and what supports extending the four-scene audit's failure pattern beyond those cases? Do the main conclusions remain stable across generation seeds? Please report paired score differences for the principal comparisons, with confidence intervals computed at the base-scene level.

  5. How is the validity of the reference targets and evaluator established? Does the initial image plus L1 text specify all necessary initial velocities, external forces, and trigger conditions? Figure 1 illustrates a push as a Level 2 trigger; please clarify how such triggers are represented in the actual benchmark inputs. Where residual input uncertainty exists, could you assess physically valid alternative rollouts under that uncertainty? Pipeline controls using reference videos and videos with known violations of specific dependencies would help assess sensitivity to causal errors. For the 100-question human audit, please report the observed answer distribution and agreement by answer category, so that the 89% agreement can be interpreted beyond its aggregate value.

9 comments
AuthorsOpenReview ↗

Response to Reviewer sbmV (1/5)

We sincerely thank the reviewer for recognizing the value of CausalCascade's inspectable and reproducible benchmark design, as well as the diagnostic value of our complementary analyses.

Q1 (1), W1: Report factual-copy baseline; The Preserve metric is not enough.

(a) Factual-Copy Baseline.

We construct a factual-copy baseline by directly reusing each model's factual generation as its counterfactual generation. We recompute Actual and Factual-Copy Preserve scores on the same original 2,025 model-edit pairs, following the original scene-macro aggregation.

Table R2-1. Factual-Copy Baseline for RQ2 Preserve Metrics.

MetricActualFactual-Copy
Binary Preserve IoU16.32%15.81%
Weighted Preserve IoU9.18%8.02%

(b) Decomposing Preserve into Base Realization and Conditional Response.

To disentangle insufficient factual-motion realization from the subsequent loss of shared motion in counterfactual generations, we decompose the recall-side coverage of Preserve into two stages. Let P∗=min⁡(Mf∗,Mcf∗)P^\ast=\min(M^{f\ast},M^{cf\ast}) denote the ground-truth shared-motion map as defined in Appendix F.

Base Realization (CP=∣P∗∩M^f∣∣P∗∣C_P=\frac{|P^\ast\cap\hat M^f|}{|P^\ast|}) measures how much of the ground-truth shared motion is realized in the factual generation; Conditional Response (RP=∣P∗∩M^f∩M^cf∣∣P∗∩M^f∣R_P=\frac{|P^\ast\cap\hat M^f\cap\hat M^{cf}|}{|P^\ast\cap\hat M^f|}) measures how much of this realized shared motion is retained in the counterfactual generation.

Table R2-2. Decomposition of Spatial Preservation into Base Realization and Conditional Response.

Motion metricBase Realization CPC_PConditional Response RPR_P
Spatial57.50%81.73%
Weighted51.21%81.14%

All results follow the original scene-macro aggregation, followed by equal weighting across the nine models and four intervention types.

Factual generations realize 57.5% of the reference-shared motion, of which counterfactual generations retain 81.7%. This decomposition distinguishes missing factual motion from the subsequent non-retention of already-realized shared motion. We have revised Finding 5 to state more precisely: Shared-motion preservation exhibits a substantial factual-realization gap.

Q1 (2), W1: Report the endpoint coverage and conditional relation accuracy defined in Appendix B.3.

We report the Endpoint Coverage and Conditional Relation Accuracy defined in Appendix B.3, separately for Causal Pathway and Temporal Ordering. All results follow the original scene-macro aggregation, followed by equal weighting across the nine models.

Table R2-3. Decomposition of Causal Pathway and Temporal Ordering scores into missing event endpoints and incorrect relations.

LevelDiagnostic DimensionEndpoint CoverageConditional Relation AccuracyOriginal Score (Table 9)Score Deficit: Missing EndpointsScore Deficit: Incorrect Relations
L1Causal Pathway8.74%89.11%7.78%98.97%1.03%
L1Temporal Ordering9.66%86.35%8.34%98.56%1.44%
L2Causal Pathway14.05%89.10%12.52%98.25%1.75%
L2Temporal Ordering15.77%88.87%14.01%97.96%2.04%
L3Causal Pathway30.08%75.98%22.85%90.63%9.37%
L3Temporal Ordering31.76%83.29%26.45%92.79%7.21%

These results clarify that the low Causal Pathway and Temporal Ordering scores primarily reflect incomplete event realization, rather than incorrect relations among already-realized events. Endpoint Coverage increases substantially from L1 to L3, yet missing endpoints still account for 90.6–92.8% of the score deficit under L3.

AuthorsOpenReview ↗

Response to Reviewer sbmV (2/5)

Q2: Frame-rate Sensitivity, GT Event-chain Duration Coverage and Shared-horizon Robustness.

(a) Physical-time Normalization and Metric Robustness

To assess the sensitivity of our motion metrics to the per-frame background update, we additionally adopt a common physical-time constant τ=1.368s\tau=1.368 s, corresponding to the original 24-FPS setting, and adjust the background update coefficient according to α(f)=1−(1−0.03)24/f\alpha(f)=1-(1-0.03)^{24/f}. We then recompute all RQ1 and RQ2 motion metrics using this physical-time normalization.

As shown in Table R2-4, the absolute changes in nine-model average scores are marginal, while model rankings remain highly consistent, with minimum Spearman correlations of 0.9333 and 0.9833 respectively.

Table R2-4. Robustness of RQ1 and RQ2 Motion Metrics to Physical-time Normalization.

ExperimentMaximum Absolute Score ChangeMinimum Spearman's ρ\rho
RQ10.53 pp0.9333
RQ20.27 pp0.9833

Importantly, the principal L3-over-L1 improvements remain positive under the physical-time normalization policy, with all paired 95% CIs excluding zero: Spatial IoU (+6.40 pp, [4.62, 8.24]), Strict ST-IoU (+2.39 pp, [1.65, 3.27]), and Weighted Spatial IoU (+5.18 pp, [3.75, 6.65]).

Together, these results demonstrate that our principal findings are robust to the choice of background-update policy and the associated physical-time normalization.

(b) Event-chain Duration Coverage and Shared-horizon Robustness.

We annotate the completion time of the last key event in each of the 50 GT scenes to assess whether model outputs cover the complete event chain. The last key event falls within the output duration in 50/50 scenes for CogVideoX-Fun-V1.5; 42/50 (84%) for Cosmos3-Nano, LTX-Video-13B, Wan2.1-I2V-14B, and Wan2.2-I2V-A14B; and 17/50 (34%) for HunyuanVideo-I2V, MAGI-1-4.5B, SkyReels-V2-I2V, and Wan2.2-TI2V-5B.

We then conduct a shared-horizon analysis, restricting each scene to the shorter of its GT duration and 5.00 seconds (the shortest decoded duration among all models). All scores follow the original scene-macro aggregation, followed by equal weighting across the nine models. Paired 95% confidence intervals are computed using 10,000 base-scene-level bootstrap resamples.

Table R2-5. Motion Scores and Paired Improvements under the Shared Horizon.

MetricL1L2L3L2−L1 (95% CI)L3−L2 (95% CI)
Spatial IoU0.186480.214820.24434+0.02833 [0.01902, 0.03804]+0.02953 [0.01173, 0.04745]
Strict ST-IoU0.054290.069270.08190+0.01498 [0.01027, 0.01982]+0.01264 [0.00445, 0.02134]
Weighted Spatial IoU0.113030.143360.16646+0.03033 [0.02253, 0.03801]+0.02309 [0.00828, 0.03828]

All three motion metrics exhibit consistent improvements from L1 to L2 and from L2 to L3, with all six paired 95% confidence intervals excluding zero. Spatial IoU also remains substantially higher than Strict ST-IoU at every prompt level. Model rankings remain highly consistent with full-duration evaluations (Spearman's ρ=0.9333–1.0000\rho=0.9333–1.0000).

For RQ2, the shared-horizon scores remain comparable to full-duration results: Effect Spatial IoU changes from 0.03311 to 0.03011, Weighted Effect from 0.04106 to 0.04015, Preserve Spatial IoU from 0.16331 to 0.15791, and Weighted Preserve from 0.09406 to 0.10691. Thus, Effect remains low and Preserve remains limited under the shared horizon.

Overall, the shared-horizon analysis preserves the principal motion-based findings of both RQ1 and RQ2.

AuthorsOpenReview ↗

Response to Reviewer sbmV (3/5)

Q3 (1): What calibration supports the claim that event recovery benefits more?

To clarify, our finding 2 was intended to distinguishes event recovery from faithful physical realization; it does not claim that their gains are directly comparable. We have clarified this distinction in the revised PDF.

As stated in Finding 2, the VQA breakdown in Table 9 further supports that Event Occurrence exhibits the largest absolute score increase from L1 to L3 (+0.290), compared with Causal Pathway (+0.151), Temporal Ordering (+0.182), and Outcome/Termination (+0.163).

The additional Endpoint Coverage and Conditional Relation Accuracy in Table R2-3 further characterize this pattern. Endpoint Coverage increases consistently from L1 to L3 for both Causal Pathway (8.74% to 30.08%) and Temporal Ordering (9.66% to 31.76%), while Conditional Relation Accuracy does not exhibit the same trend. These results indicate that improvements in Causal Pathway and Temporal Ordering scores primarily arise from realizing more required event endpoints, rather than from improved relations among realized events.

Q3 (2): Could you retain the complete L1 state and constraints and append only the event chain, accounting for text-encoder truncation?

We additionally evaluate an L1+EC condition that retains the complete L1 prompt while appending a compressed event-chain description. To accommodate text-encoder context limits, the appended event chain is a shortened and compressed version of the L3 event-chain description.

As shown in Table R2-6, the event-recovery benefit persists: mean Event Occurrence increases from 0.230 to 0.413, with improvements across all nine models. VQA Overall also increases from 0.193 to 0.257. Meanwhile, mean Scene Consistency decreases from 0.491 to 0.364, with declines in eight of nine models.

Table R2-6. Controlled Comparison of L1 and L1+EC.

ConditionVQA OverallEvent OccurrenceScene Consistency
L10.1930.2300.491
L1+EC0.2570.4130.364
Change+0.064+0.183−0.127

We emphasize that the original L1, L2, and L3 represent increasingly explicit future-event specifications, rather than cumulative textual prefixes. L1+EC is an additional controlled variant, not another hierarchical prompt level.

Q4 (1): Report how the evaluated pairs cover distinct base scenes, cascade structures, intervened objects, and mass multipliers.

As shown in Table R2-7, these statistics characterize the structural and intervention coverage of CausalCascade within its controlled rigid-body setting.

Table R2-7. Coverage of the 225 Evaluated Counterfactual Pairs.

Coverage dimensionStatistics
Base scenes48/50 scenes; 1–11 pairs per scene (median 5)
Cascade structures19 distinct event-graph topologies; max depth 6
Intervened objects153 distinct targets
Delete50 pairs, 39 scenes, 50 targets
Move50 pairs, 43 scenes, 50 targets
Scale50 pairs, 31 scenes, 47 targets
Mass75 pairs, 40 scenes, 69 targets; 75 distinct multipliers (0.01096–74.93636, median 2.882)

Together, these statistics characterize the benchmark's coverage across scene instances, causal structures, intervention targets, and intervention types.

Q4 (2), W2: Assess edits whose correct response is no observable change.

To additionally assess cases where physically valid interventions produce no substantial downstream event change, we test 80 supplementary pairs (20 per edit type), allowing minor motion-detail or timing variations. The GT factual-counterfactual videos achieve a Spatial IoU of 89.82%, confirming that their underlying spatial motion patterns remain highly consistent.

Table R2-8. Evaluation of Physically Valid Interventions with No Substantial Downstream Event Change (nine-model equal-weighted average).

MetricSpatialWeighted
Preserve IoU18.73%10.42%
Base Realization CPC_P50.33%41.80%
Conditional Response RPR_P78.87%78.46%

Despite the high GT factual-counterfactual consistency, models achieve limited Preserve IoU on these cases. The decomposition further shows that both base-motion realization and conditional retention remain imperfect, even when the correct response involves no substantial downstream event change. We have included the full per-model results in the revised PDF.

Reviewer_oKruOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes an evaluation benchmark, CausalCascade, to test whether the Tusheng video model can generate a complete and coherent physical causal chain from the initial picture. The authors constructed 50 simulation scenarios with 2,832 physical intervention variants, and evaluated nine models using motion matching, event question answering, and counterfactual comparisons. It was found that the model can roughly generate expected events, but often cannot accurately represent the process of collision, transmission and termination; even if the complete event chain is clearly informed, the physical process is still not accurate enough. After changing the scene conditions, it is difficult for the model to "change what should be changed and what should not be changed remain unchanged". Based on this, the paper points out that continuous and stable object and scene states are the key to the video model's reliable causal prediction ability.

Critical strengths
  1. This approach extends evaluation to encompass the entire causal process. While benchmarks like Physics-IQ and WorldBench primarily assess physical phenomena or concepts, and CausalVQA focuses on causal question-answering within videos, this work goes further: it examines whether events in model-generated videos are triggered by the correct antecedents, unfold in the proper sequence, and conclude in a stable manner. By integrating motion-based and question-answering metrics, the event graph framework can distinguish between simply "generating the target event" and "correctly generating the entire event process," while also pinpointing exactly where a causal chain breaks down. This provides more specific feedback for model improvement.
  2. Refine the counterfactual evaluation into "change what should be changed and keep what should not be changed." What-If World has used paired scenes to test the results after physical changes, and CRONOS tests the physical consistency of the model under changes in perspective, appearance, etc.; this article makes controllable intervention in factors such as object position, size, quality, and measures the changes in affected motion and the retention of irrelevant motion respectively. In particular, mass changes that are not directly visible from the first frame allow it to test whether the model uses physical conditions to deduce subsequent effects. This provides an operational test for measuring selective causal responses of world models.
Critical weaknesses
  1. Without a thorough comparison between single-step and multi-step events, it is difficult to determine whether a model’s failure stems from a lack of fundamental generative capability or an inability to handle sequential causal relationships.
  2. The focus is primarily on simple rigid-body motion from a fixed perspective; it lacks validation in complex, real-world scenarios, limiting the generalizability of the conclusions.
Questions for authors
  1. Is the model’s failure primarily due to "key events failing to occur," or is it that the events did occur but in the wrong sequence or with incorrect causal relationships? Could you provide a simple categorical breakdown or some illustrative examples, drawing on existing evaluation results?
  2. In which types of scenarios does the model perform best, and in which does it perform worst?
5 comments
AuthorsOpenReview ↗

Response to Reviewer oKru (1/2)

We thank the reviewer for the thoughtful assessment and constructive questions. We particularly appreciate the recognition of our efforts to evaluate the full causal event process and selective counterfactual responses beyond isolated event generation or visual plausibility.

Q1, W1: Is the model’s failure primarily due to key events failing to occur, or is it that the events did occur but in the wrong sequence or with incorrect causal relationships?

Our results indicate that incomplete event realization is the primary source of low relational scores, rather than incorrect relations among already-realized events.

We quantify this distinction using Endpoint Coverage and Conditional Relation Accuracy defined in Appendix B.3. As shown in Table R1-1, even at L3, missing event endpoints account for 90.63% and 92.79% of the score deficits in Causal Pathway and Temporal Ordering, respectively. Incorrect relations among realized endpoints account for only the remaining 9.37% and 7.21%. Thus, although relation errors do occur, incomplete event realization is the dominant contributor to the low relational scores.

Table R1-1. Attribution of Relational-Score Deficits to Missing Events and Incorrect Relations.

LevelDiagnostic DimensionEndpoint CoverageConditional Relation AccuracyOriginal Score (Table 9)Missing Endpoints / Incorrect Relations
L1Causal Pathway8.74%89.11%7.78%98.97% / 1.03%
L1Temporal Ordering9.66%86.35%8.34%98.56% / 1.44%
L2Causal Pathway14.05%89.10%12.52%98.25% / 1.75%
L2Temporal Ordering15.77%88.87%14.01%97.96% / 2.04%
L3Causal Pathway30.08%75.98%22.85%90.63% / 9.37%
L3Temporal Ordering31.76%83.29%26.45%92.79% / 7.21%

Q2: In which types of scenarios does the model perform best, and in which does it perform worst?

We summarize six scene families and their performance in Table R1-2.

Table R1-2. Scenario Characteristics and Model Performance across Six Scene Families.

Scene FamilyPhysical Scenario CharacteristicsRQ1: Spatial / Strict ST / VQA (L3)RQ2: Effect / Preserve
DominoDomino chains; sequential and branching toppling cascades0.367 / 0.082 / 0.2070.026 / 0.208
GaltonPegboard interactions; repeated collisions and trajectory branching toward different targets0.067 / 0.031 / 0.3340.021 / 0.044
InstabilitySupport loss, stack and platform collapse0.306 / 0.097 / 0.3120.042 / 0.137
SeesawSeesaw interactions; balancing, tilting and support0.281 / 0.073 / 0.3720.040 / 0.235
RailRail-guided motion and track-constrained interactions0.232 / 0.041 / 0.4730.046 / 0.163
Two-BallCoupled two-ball motion and interactions0.208 / 0.031 / 0.4270.021 / 0.148

Across the six scene families:

Galton is consistently the most challenging family for physical realization, achieving the lowest Spatial IoU and Strict ST-IoU across all three prompt levels. Its repeated collisions and branching trajectories pose challenges for accurately modeling sequential interactions.

Other families also exhibit distinct strengths. Rail achieves the highest VQA Overall across all three prompt levels, while Instability achieves the highest Strict ST-IoU under L3. These results further illustrate how different interaction structures challenge different aspects of model capabilities.

Counterfactual preservation also varies across families. Galton has the lowest Preserve Spatial IoU (0.044), whereas Seesaw has the highest (0.235). Effect Spatial IoU remains low across all six families.

AuthorsOpenReview ↗

Response to Reviewer oKru (2/2)

W1: Fundamental Event Generation and Sequential Causal Reasoning.

We thank the reviewer for highlighting the need to distinguish fundamental event-generation failures from difficulties in handling sequential causal relationships.

We stratify the 50 base scenes by maximum causal depth into short (0–3 edges), medium (4 edges), and long (5–6 edges), and analyze 1,350 rollouts across nine models and three prompt levels. Event Occurrence decreases from 0.427 on short chains to 0.285 on long chains, with a paired difference of −0.142 (95% CI: [−0.253, −0.032]), and this trend holds for all nine models.

To further distinguish missing events from incorrect relations, we conduct a relation-level analysis on the 48 scenes with relation questions, excluding 0-edge scenes. Accordingly, short chains comprise 2–3 edges (15 scenes), while medium and long chains comprise 4 edges (21 scenes) and 5–6 edges (12 scenes), respectively. Endpoint Coverage decreases by 5.4 pp, but its confidence interval includes zero. In contrast, Conditional Relation Accuracy, evaluated only when both queried events are present, drops significantly from 88.9% to 71.0% (−17.9 pp, 95% CI: [−31.9, −4.3]).

Table R1-3. Causal-chain Length Analysis.

MetricShortMediumLongLong − Short (95% CI)
Event Occurrence0.4270.3700.285−0.142 [−0.253, −0.032]
Causal Pathway0.1690.1510.100−0.069 [−0.140, −0.002]
Temporal Ordering0.1830.1820.104−0.079 [−0.161, −0.006]
Endpoint Coverage19.8%19.6%14.4%−5.4 pp [−13.6, +2.5]
Conditional Relation Accuracy88.9%85.0%71.0%−17.9 pp [−31.9, −4.3]

Together, these results indicate that incomplete event realization is a major bottleneck, while longer causal chains also pose additional challenges to establishing correct causal and temporal relations among realized events.

W2: The focus is primarily on simple rigid-body motion from a fixed perspective; it lacks validation in complex, real-world scenarios.

We appreciate the reviewer's thoughtful comments on the scope and generalizability of our benchmark. Our focus on rigid-body simulation is a design choice to ensure controlled and reliable evaluation. Rigid-body simulation allows us to explicitly specify physical properties and initial conditions, precisely isolate interventions, and obtain reliable physical trajectories and event-level ground truth.

More importantly, simulation enables strictly matched factual-counterfactual pairs by changing only the target intervention while keeping other conditions fixed. Such controlled comparisons are difficult to establish in real-world settings, where the counterfactual outcome under the same initial conditions is inherently unobservable and physical factors may be difficult to measure or control. Meanwhile, our benchmark goes beyond isolated rigid-body motions, covering diverse event-graph structures with causal chains of up to six levels.

We agree that deformable objects, viewpoint variations, and complex real-world interactions remain important directions for future work.

We thank the reviewer again for the time and thoughtful consideration.

Sincerely, The Authors

AuthorsOpenReview ↗

A Gentle Follow-up on Our Rebuttal

Dear Reviewer oKru,

Thank you again for your time and valuable feedback. We are writing to ask whether our rebuttal has addressed your concerns, and we would be happy to discuss any remaining issues if needed.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank oKru for recognizing the value of evaluating complete causal processes and selective responses to physical interventions, sbmV for highlighting the benchmark's inspectability, reproducibility, and diagnostic analyses, and rCS3 for appreciating the controlled intervention design and complementary analyses of event specification and temporal alignment. The reviewers' suggestions have prompted us to conduct additional experiments and strengthen our work.

The revised PDF is now available. New in the rebuttal:

  1. A more diagnostic counterfactual evaluation. We decompose recall-side Effect and Preserve into Base Realization and Conditional Response, and refine Finding 5 to highlight a substantial factual-realization gap in recall-side shared-motion preservation. We further evaluate 80 physically valid interventions with no substantial downstream event change: although reference factual-counterfactual pairs achieve 89.82% Spatial IoU, generated pairs achieve only 18.73% Preserve Spatial IoU. Tables 22–24 / Appendix H.3.

  2. A refined analysis of event recovery and physical realization. We revise Finding 2 to clarify that future-event specification mainly improves event recovery through better endpoint coverage, without consistent gains in relation accuracy. Physical realization remains limited, with Strict ST-IoU increasing only from 0.043 to 0.067. A controlled L1+EC condition further supports the event-recovery benefit, increasing event occurrence from 0.230 to 0.413 and VQA Overall from 0.193 to 0.257. Tables 14–17 / Appendix H.2.

  3. Robustness to evaluation and initial-condition specifications. All six paired RQ1 improvements remain positive under a shared evaluation horizon. Physical-time normalization changes RQ1 and RQ2 scores by at most 0.53 pp and 0.27 pp, respectively. We also verify the Level 1 initial conditions: explicitly specifying objects at rest, gravity, and the absence of external impulses leaves motion-based results broadly comparable to the original L1 setting. Tables 8–9 / Appendices E and H.2.

  4. An expanded cross-scene earliest-failure audit. We extend the audit to 138 rollouts from the 46 remaining scenes, finding that the earliest primary failure occurs at a foundational stage in 121 cases (87.7%), consistent across all six scene families (95% CI: [81.9%, 92.8%]). Table 30 / Appendix J.5.

  5. Stronger validation of the VQA evaluator. We further report answer-category-wise agreement for the existing 100-question human audit. On controlled tests, the evaluator achieves 91.7% accuracy on balanced four-way questions and 90.0% exact relation accuracy on 60 matched Blender-rendered intervention pairs. Tables 5–6 / Appendix B.

  6. Broader benchmark coverage and scene-family robustness. We characterize the 225 evaluated pairs across 48 base scenes, 19 event-graph structures, and 153 intervention targets. All 18 family-level L3-over-L1 improvements remain positive, including under leave-one-family-out evaluation. Tables 7 and 25 / Appendices D.5 and H.4.

Together, these additions provide stronger validation, more nuanced interpretations, and broader robustness evidence for our central findings. We thank the reviewers for helping us strengthen CausalCascade as a benchmark that evaluates complete causal processes, probes selective responses to physical interventions, and offers a more systematic testbed for measuring progress in causal video generation.

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.