- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces YoCausal, a benchmark for probing temporal and causal sensitivity in video diffusion models using forward and temporally reversed real-world videos. It measures forward-video preference through the Reverse Surprise Index (RSI), based on paired denoising losses, and introduces the Causality Cognition Index (CCI), which contrasts RSI between videos with and without salient causal interactions. Experiments on 13 open-source models reveal different rankings under the two metrics and gaps relative to human performance. The paper further examines agreement with human judgments of generated videos, sensitivity to the VLM used for dataset partitioning, and the effect of text conditioning.
Critical strengths
-
The use of paired forward and reversed real-world videos provides a practical, extensible evaluation protocol while controlling for visual content within each pair. The two-level design addresses the important distinction between sensitivity to temporal direction and sensitivity to salient causal interactions.
-
The paper supports its proposed metrics with complementary validation, including human evaluations of generated videos, alternative VLM partitioners, and a null-prompt ablation. The correlation with human judgments provides useful evidence that the benchmark captures behavior relevant to the causal plausibility of generated videos.
Critical weaknesses
-
CCI’s interpretation as causal sensitivity is not sufficiently established, since similar optical-flow magnitudes do not rule out differences in event content or reversal difficulty between the two subsets. CCI can also approach zero when both subsets are recognized reliably, making low scores ambiguous despite the proposed rank aggregation.
-
The stability of the reported scores and rankings under denoising-loss estimation remains unclear. Evaluation uses only ten sampled timesteps and one noise sample per timestep, without reporting sensitivity to random seeds or larger sampling budgets.
-
YoCausal requires access to internal denoising predictions and therefore cannot directly evaluate closed-source models. This restriction reduces its generality and limits the applicability of its conclusions beyond the evaluated open-source models.
Questions for authors
-
Do the CCI differences and model rankings persist when causal and non-causal videos are matched for event content, motion characteristics, and reversal difficulty? How should low CCI be interpreted when RSI is high on both subsets, and does combining CCI with RSI improve agreement with human judgments over RSI alone?
-
How stable are RSI, CCI, and the aggregate rankings across random seeds and larger timestep/noise sampling budgets? Does the reported bootstrap account for variability from timestep and noise sampling, or only from resampling videos?
-
Could the authors evaluate MiniMax-H3 to examine whether the RSI–CCI differences and human–model gap persist?
3 comments
I thank the authors for their careful rebuttal, which has addressed most of my concerns. I therefore maintain my positive rating.
We sincerely thank reviewer uJxQ for the time and thoughtful feedback, and are glad our response addressed the concerns.
Response to Reviewer uJxQ
We thank the reviewer for recognizing that paired forward/reversed clips control visual content, that the two-level design separates temporal direction from causal interactions, and that our validation (human evaluation, alternative VLMs, null-prompt ablation) shows the benchmark captures causal plausibility.
W1 / Q1 — Interpreting CCI.
Reversal difficulty. We agree the subsets differ in event content and causal videos can be easier to recognize as reversed. That extra detectability is the causal signal itself, not a confound: reversing a causal video creates a causal anomaly (e.g., shards reassembling) that only a model expecting cause before effect can exploit. Non-causal cues are matched or nearly matched:
Model behaviour agrees: if the causal subset were simply easier, high-RSI models would also score high on CCI, yet RSI and CCI rank the 13 models with only τ = 0.13.
Saturation. We agree CCI could approach zero if both subsets were recognized reliably, but current models are still far from saturation, and CCI is designed to be read jointly with RSI rather than as an absolute score (Sec. 3.4).
Agreement with humans. In rank correlation, the aggregate ranking matches RSI alone (Kendall's τ = 0.667 for both), but this only reflects pairwise ordering. To test whether CCI adds information, we regress each model's human preference score (Table 7) on its metrics:
Adding CCI raises R2 (variance in the 13 models' human preference scores explained) from 0.56 to 0.79 and adjusted R2 from 0.52 to 0.75; a nested F-test confirms the gain is significant (p = 0.007): CCI carries information about human-perceived causality that RSI misses.
W2 / Q2 — Seed and timestep stability.
RSI averages each pair's denoising loss over K = 10 timesteps, with one Gaussian noise sample per timestep shared by both clips; CCI is causal-subset RSI minus non-causal-subset RSI. Directly: the bootstrap resamples videos only and excludes timestep and noise variability. We ran two new experiments on four models covering all three prediction targets: AnimateDiff-SD1.5 (ϵ), CogVideoX-2B (v), Wan2.1-1.3B and LTXV-2B (flow), otherwise following the paper.
The seed std is 0.30–1.23 points for RSI and 0.33–1.52 for CCI, i.e., moderate. From K = 10 to 15, scores change by at most 1.4 points except LTXV-2B's RSI, so K = 10 is sufficient. As Monte-Carlo estimates of the expected denoising loss, RSI and CCI would become more robust with more seeds given sufficient compute. Each extra seed or budget needs a full benchmark pass, so the larger models (5B–14B) would not finish within the rebuttal period on our 24 GB RTX 4090; we will add them in the revision once larger GPUs are available.
W3 — Closed-source models.
Loss access limits which models can be tested today, not the method: YoCausal needs only the denoising loss at a given timestep and noise seed, not the weights. Providers could expose a minimal loss-query interface without revealing internals, or run YoCausal internally to select checkpoints. Open-weight models are also quickly closing the gap to the state of the art, widening YoCausal's reach.
Q3 — MiniMax-H3.
We agree a stronger model would further test our findings. YoCausal evaluates every model in its official setting; H3's full-precision weights (~66 GB DiT plus a 51.5 GB text encoder) exceed our 24 GB RTX 4090. Even with sufficient memory, a 33B model would not finish the full benchmark within the rebuttal period. We will include H3 in the revision once larger GPUs are available.
We thank the reviewer again. If these responses address your concerns, we would be grateful if you would consider adjusting your score.
Sincerely,
The Authors