acceptodds
Withdrawn from ICLR 2027

CRONOS: Benchmarking Counterfactual Physical Consistency in Video Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video prediction is increasingly viewed as a path toward generalizable world models, yet it remains unclear whether these systems learn underlying causal structure or merely exploit superficial visual correlations for future prediction. We introduce CRONOS, an intervention-based benchmark to evaluate counterfactual physical consistency: whether a model's predictions of physical events respond appropriately to controlled changes in the visual input, such as variations of camera viewpoint, scene, object category, and object appearance. Built in a photorealistic Unreal Engine environment, CRONOS enables controlled, high-fidelity generation of videos across diverse scenes and dynamics, including sampling multiple physically plausible outcomes of the same physical event. In contrast to previous benchmarks, that evaluate video prediction on scenes CRONOS systematically intervenes on four factors—camera viewpoint, scene, object category, and object appearance—while keeping the underlying physical event type, such as a fall, collision, or occlusion, fixed. Our evaluation of recent open-source video generators reveals substantial failures in counterfactual physical consistency: prediction quality for the same physical event type is affected by appearance, environment, and particularly viewpoint changes. CRONOS provides a controlled and reproducible testbed for diagnosing how generated video quality changes across interventions, establishing a concrete target for developing models that perform consistently across changes of multiple conditions. The dataset and evaluation code will be available upon acceptance.

Primary area:
datasets and benchmarks
Keywords:
Video models, benchmarking

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KbiKOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose CRONOS, an intervention-based benchmark with evaluation metrics for video generation. This benchmark considers three basic physical events and applies four controlled counterfactual visual interventions to them, and also presents several findings.

Critical strengths
  1. The authors build a well-controlled benchmark with systematic interventions on camera viewpoints, scenes, object categories, and appearances for video generation models;

  2. The authors evaluate various video generation models under I2V and V2V and investigate the impact of model size on physical consistency;

  3. The paper provides human user study and finding, like shows visual changes can affect the physical consistency of generated videos.

Critical weaknesses
  1. The writing is a bit verbose, like the long background in the Introduction and please add clear contribution summarization in introduction. The related works could be more focus on video generation benchmark and evaluation metrics.

  2. This paper lacks comparisons with highly relevant works like PhyWorldBench[1] and Impossible Videos[2]. I not see any novel innovation because the concept "counterfactual physical consistency" very similar to the "Anti-Physics". This already presented in previous paper[1]. It is better to claim your novelty.

  3. Only evaluating three simple physical events. While the authors argue these are fundamental interactions, a comprehensive standard benchmark needs much broader coverage.

  4. The evaluation metrics (like motion similarity and appearance stability) simply adopt existing designs like VBench. The authors need to claim your metric design is uniquely suited for evaluating counterfactual physical consistency. And I have doubts about whether using DINOv2 [CLS] token can measure true appearance stability.

  5. The experiments seem just translates the numbers from the tables into text. It lacks deeper insights or specific observations based on generated video results from different models. This makes the takeaways basic. And I recommend the author to expend your benchmark to include proprietary models.

[1] PhyWorldBench: A Comprehensive Evaluation of Physical Realism in Text-to-Video Models. ICLR2026.
[2] Impossible videos. ICML 2025.
[3] VBench. CVPR2024

Questions for authors

see weakness.

1 comment

We thank the reviewers for their insightful feedback. After carefully considering their concerns, we have substantially revised the design of our evaluation metrics, resulting in updated results and findings. We believe these changes go beyond the scope of a rebuttal and have therefore decided to withdraw our submission. We thank the reviewers again for their time and valuable contributions to improving our work.

Reviewer_4935OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CRONOS, a benchmark for evaluating video generation models under controlled visual interventions. It constructs photorealistic videos of falling, collision, and occlusion events, while varying camera viewpoint, scene, object category, and appearance using UE. It evaluates generated videos using several visual and physical quality metrics, and introduces intervention sensitivity to measure how generation quality changes across these variations. Experiments on multiple I2V and V2V models show low success rates on basic physical events and sensitivity to some interventions.

Critical strengths
  1. Well motivated. The controlled intervention setup is a useful complement to evaluations of isolated physical events. Photorealistic simulations allow the authors to systematically vary visual factors while controlling the underlying physical setup.
  2. The evaluation covers multiple video generation models an quality dimensions with human study. The results offer useful insights into current models' performance and robustness on basic physical events.
Critical weaknesses
  1. The connection between intervention sensitivity and counterfactual physical consistency needs stronger validation. The proposed metric compares generation-quality scores across interventions rather than directly testing whether predicted outcomes satisfy the expected physical relationships. Different, physically inconsistent trajectories can receive similar quality scores and thus yield low sensitivity. Some underlying quality metrics may themselves vary with viewpoint or appearance. Consequently, the results directly establish robustness of generation quality, but it is less clear to what extent they measure the claimed counterfactual physical consistency.
Questions for authors
  1. Could the authors further justify why using score consistency as a proxy for Counterfactual Physical Consistency? It would also be helpful to clarify how intervention sensitivity relates to human judgments of consistency between paired outputs.
  2. Could the authors clarify whether intervention sensitivity is affected by viewpoint or appearance changes on ground-truth simulated videos? Such effects may arise from the evaluation metrics even when the underlying physics is correct.
2 comments

We thank the reviewers for their insightful feedback. After carefully considering their concerns, we have substantially revised the design of our evaluation metrics, resulting in updated results and findings. We believe these changes go beyond the scope of a rebuttal and have therefore decided to withdraw our submission. We thank the reviewers again for their time and valuable contributions to improving our work.

Thank you for the comment. I think systematically evaluating video generation models under controlled variations in visual conditions is an interesting and meaningful direction, and I believe CRONOS has the potential to make a valuable contribution to the community. Best of luck!

Reviewer_T47AOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

CRONOS evaluates video models on simulated falling, collision, and occlusion events. It varies scene, viewpoint, appearance, and object identity while controlling the other factors. Generated continuations are scored for motion, object stability, and physical plausibility. The paper measures how these scores change under interventions and interprets lower sensitivity as greater counterfactual consistency. It also includes human evaluation and a transfer study using real inputs.

Critical strengths
  1. I like the controlled setup. Varying scene, appearance, viewpoint, and object identity separately is a useful way to study robustness. The distinction between changes that preserve physics and changes that affect it is also important.
  2. Evaluation is thorough. It includes repeated generations, human ratings, calibrated thresholds, and threshold-stability checks. Allowing several valid futures also acknowledges that an initial image does not determine every physical parameter.
Critical weaknesses
  1. The sensitivity score compares quality scores, not physical outcomes across interventions. A systematic change in collision behavior after recoloring can go unnoticed if both outcomes score equally well. Reporting success separately helps, but does not resolve this gap.
  2. I did not find an oracle check showing that valid simulated videos score consistently across viewpoints and appearances. Segmentation, shape reconstruction, or motion features could contribute to the measured differences. This weakens the claim about pixel-based world representations.
  3. The human validation does not test the main diagnostic. The reported correlations are across model-level averages. I need evidence that the metrics track human judgments of those differences, not just which model is generally better.
  4. The Cosmos Super receives essentially perfect physical-plausibility scores, but the disappearance detector flags many of its videos. The VLM questions explicitly cover disappearance and reappearance. I would like a manual audit to explain this disagreement and its effect on the rankings.
  5. The prompt templates change with viewpoint, appearance, object, and scene, and already describe the expected event. These are valid joint-input tests, but they do not isolate visual grounding. Fixed-text and text-only controls would help support the interpretation.
Questions for authors

See Weaknesses.

1 comment

We thank the reviewers for their insightful feedback. After carefully considering their concerns, we have substantially revised the design of our evaluation metrics, resulting in updated results and findings. We believe these changes go beyond the scope of a rebuttal and have therefore decided to withdraw our submission. We thank the reviewers again for their time and valuable contributions to improving our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.