- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces What-If World, a benchmark for evaluating whether video world models respond correctly to controlled physical interventions. Instead of judging each generated video independently, the benchmark compares paired generations that differ in one physical variable. The benchmark contains 319 prompt pairs across autonomous driving and robotic manipulation, covering six physical primitives. The authors also introduce the APEO evaluation framework and evaluate nine recent video generation models. A key finding is that videos can look physically plausible individually while still failing to respond correctly to the intervention.
Critical strengths
-
The benchmark targets an important evaluation gap that is not captured by conventional single-video metrics. The key idea: evaluating whether generated futures change appropriately when one controlled variable changes—is well motivated and consequential for the use of video models as world simulators. A model that generates two individually realistic videos but ignores the intervention is unsuitable for planning or counterfactual simulation, even though conventional video-quality or physics metrics may rate both outputs highly. The paper demonstrates that this is not merely a hypothetical issue: the substantial gap between single-video and paired physics performance appears consistently across all nine evaluated models.
-
The empirical study is unusually thorough for a benchmark paper. The paper evaluates a diverse set of open and closed source models, repeats generation three times, reports uncertainty through scene-cluster bootstrapping, provides detailed per-primitive and per-subcategory analysis, and performs human validation of the VLM judge. The extensive failure-mode analysis is also valuable. This makes the benchmark substantially more diagnostic than a single scalar metric.
Critical weaknesses
-
For variables such as friction, mass, or force magnitude, the exact outcome can depend on physical properties that are not directly observable from a single frame and text prompt. This makes it somewhat difficult to separate model failure from ambiguity in the benchmark specification.
-
The evaluation still relies heavily on the VLM judge. The human agreement results are reasonably good, but not perfect, and some of the tested phenomena are quite subtle. I would like to see stronger evidence that the main conclusions are stable under human evaluation or alternative judges.
Questions for authors
-
For categories such as friction, mass, and material properties, how do the authors ensure that the expected counterfactual outcome is sufficiently unambiguous from the information given to the model?
-
Do the main conclusions, especially the single-vs-paired physics gap, remain similar when evaluated directly by humans?
-
For the environment interventions that use edited initial frames, how much does the image editing itself affect the paired evaluation?
5 comments
W2 + Q2: Human and second-judge validation
The main conclusions, including the single–paired Physics gap, hold under direct human scoring and under a second judge. We keep the VLM as the main judge because it covers every test case and metric and gives consistent scores at a scale where human annotation is impractical, as in VideoPhy [4], PhyGenBench [5] and WorldModelBench [7]. Three checks based on evidence other than Gemini's judgments confirm its conclusions: (1) human labels alone show a large gap (Q2), (2) a second judge, GPT-5, reaches the same conclusion (W2), and (3) on subtle phenomena, the judge agrees with humans as often as annotators do and follows motion measured by object tracking (W2).
(1) Human labels also show a large gap (Q2)
Under direct human scoring of 341 records, the gap (single-video minus paired Physics pass rate) is 29.3 pp (95% CI 21.8–36.7): without any Gemini judgment, annotators rate 61.6% of single videos as physically plausible but only 32.3% of pairs as showing the correct physical relation (Table R1). With the stricter baseline that requires both videos to pass, the gap is still 15.2 pp (95% CI 6.8–23.4).
Table R1. Human labels also show a large single–paired Physics gap
| Single-video baseline | Single-video pass rate (%) | Paired pass rate (%) | Gap, pp [95% CI] |
|---|---|---|---|
| Mean of the two videos | 61.6 | 32.3 | 29.3 [21.8, 36.7] |
| Both videos pass | 47.5 | 32.3 | 15.2 [6.8, 23.4] |
(2) A second judge, GPT-5, reaches the same conclusion (W2)
GPT-5, scoring the same run-1 videos, also finds a large gap (52.3 pp, against 37.9 pp for Gemini on these videos), and, as reported in Appendix E.7, the two judges rank the models similarly (Spearman = 0.87, with the same three leading models).
(3) The judge stays reliable on subtle phenomena (W2)
For the subtle phenomena, we checked the judge in two ways.
First, the judge agrees with humans as often as annotators agree with each other, both overall (82.5%, = 0.65, against 84.1%, = 0.67, Section 3.3) and on the primitives that may be hard to see: Surface Friction 84.0% against 84.0%, Material/Medium 83.0% against 84.0%, and Obstacle Configuration 80.0% against 77.0% (Appendix E.7, Table 18). Its Avg also tracks the human Avg across (model, sub-category) groups (Pearson = 0.79).
Second, where motion can be measured, the judge's scores follow measurements that use no judge at all. On 88 scenes whose expected outcome is a single motion quantity, we tracked objects with Grounding DINO [8] and SAM 2 [9] and counted an effect as clear only if it exceeded , the variation between two generations of the same prompt. The judge passes none of the 28 pairs whose motion goes clearly in the wrong direction, and 68% of pairs with a clear effect in the expected direction (Table R2). A judge that read only the prompt or guessed would not show this pattern. We will add these checks to the appendix.
Table R2. Judge Outcome scores by measured motion effect (judge passes / measurable pairs)
| Measured effect | First generation run (nine models) | Second generation run (four models) |
|---|---|---|
| Wrong direction, beyond () | 0 / 28 (0%) | 0 / 16 (0%) |
| Within , not in the expected direction () | 47 / 262 (18%) | 15 / 99 (15%) |
| Within , in the expected direction () | 93 / 350 (27%) | 55 / 146 (38%) |
| Expected direction, beyond () | 39 / 57 (68%) | 39 / 50 (78%) |
| AUC within sub-category and model [95% CI] | 0.68 [0.61, 0.75] | 0.71 [0.61, 0.79] |
Note. = measured A/B difference in the expected direction divided by .
Q3: Effect of edited initial frames
Image editing has only a small effect on the main result, the single–paired Physics gap, because each edit changes only the intervened condition and every edited frame was checked manually. Each edit changes only the environmental condition that the intervention requires, such as snow on the road, and keeps the scene layout, objects, agents and camera viewpoint unchanged (Section 3.2, Appendix D.3). Each edited frame was manually checked against its original and the edit instruction (Appendix D.5). Any unintended difference between the two scenes would also fail the paired Environment check (), which requires the shared scene to be preserved, so it would lower a model's score rather than raise it (Section 3.3). The exclusion test below quantifies the effect on the main result.
The edited pairs change the gap only slightly and do not inflate it. As an additional controlled experiment, we excluded all 72 edited-frame pairs and found that the observed differences remained. On the remaining 247 pairs, whose two branches start from byte-identical images, the single–paired Physics gap is 39.2 pp (95% CI 35.0–43.2), against 37.6 pp on the full benchmark (nine models and three runs, Table R3).
Table R3. Single–paired Physics gap with and without the 72 edited-frame pairs
| Pairs included | Pairs | Gap vs. mean single-video Physics, pp [95% CI] | Gap vs. both-videos-pass baseline, pp [95% CI] |
|---|---|---|---|
| All pairs | 319 | 37.6 [34.1, 41.0] | 21.5 [17.8, 25.2] |
| Edited pairs removed | 247 | 39.2 [35.0, 43.2] | 26.4 [22.3, 30.1] |
Removing the edited pairs changes the gap by only +1.6 pp (95% CI of the difference 0.4–2.8), about 4% of the gap, and toward a larger gap, not a smaller one. The stricter gap, which counts a pair only if both videos pass single-video Physics, rises the same way (from 21.5 to 26.4 pp, +4.8 pp, 95% CI 3.4–6.4). We will add this exclusion analysis to the appendix.
References
[1] Gupta et al., 2024. The Essential Role of Causality in Foundation World Models for Embodied AI. https://arxiv.org/html/2402.06665v2
[2] Sun et al., 2026. When Numbers Speak: Aligning Textual Numerals and Visual Instances in Text-to-Video Diffusion Models. https://arxiv.org/html/2604.08546v1
[3] Upadhyay et al., 2026. WorldBench: Benchmarking Physical Understanding of World Models by Isolating Physics Concepts. https://arxiv.org/abs/2601.21282
[4] Bansal et al., 2024. VideoPhy: Evaluating Physical Commonsense for Video Generation. https://arxiv.org/html/2406.03520v2
[5] Meng et al., 2025. Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation. https://proceedings.mlr.press/v267/meng25c.html
[6] Motamed et al., 2026. Do Generative Video Models Understand Physical Principles? https://arxiv.org/html/2501.09038
[7] Li et al., 2025. WorldModelBench: Judging Video Generation Models As World Models. https://arxiv.org/abs/2502.20694
[8] Liu et al., 2024. Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection. https://arxiv.org/abs/2303.05499
[9] Ravi et al., 2024. SAM 2: Segment Anything in Images and Videos. https://arxiv.org/abs/2408.00714
We thank the reviewer for the positive assessment and the careful questions. We address the weaknesses and questions below and will incorporate the results of this discussion into the revised paper.
W1 + Q1: Unobservable properties and the expected direction
Unobservable properties do not make the expected outcome ambiguous, because they do not change the direction of the difference between the two videos, and we score only this direction. Scoring the direction follows prior evaluations of physical reasoning. Our contribution is to make the evaluation contrastive, and to our knowledge What-If World is the first benchmark to do so for general world models. Below, (1) explains why unobservable properties do not change the direction we score and why scoring the direction is a meaningful test (W1), and (2) shows how construction removes scenes in which the direction could still be unclear (Q1).
(1) Unobservable properties do not change the direction, which is what we score (W1)
In the scenes we keep, unobservable properties mainly affect how large the difference between the two videos is, not its direction, and we only score the direction.
First, properties other than the intervened variable are the same in both videos, because the two branches share the same frame and text except for the intervention (Section 3.2). Such a property changes the size of the effect but not its direction.
Second, the exact value of the intervened variable is also unobservable, but the inputs state how it differs between the two branches, and this alone fixes the direction of the outcome.
Third, we score only this direction, not exact values: Outcome checks whether the two videos differ in the direction the intervention predicts (Table 1). This is a meaningful test of physical reasoning: any correct numerical prediction must also match the direction, and a planner comparing candidate actions needs this direction [1]. Exact values would instead require exact parameters in the prompt, but general world models have difficulty grounding even exact object counts from text [2], so a failure could then reflect an ignored number rather than wrong physics. Scoring the direction also follows prior work, which evaluates physical understanding separately from parameter estimation [3] and assesses physical capability through generated videos [4][5][6]. We will state in Section 3.2 that each expected outcome specifies the direction of change.
(2) Construction removes scenes in which the direction could still be unclear (Q1)
The direction could still be unclear if the scene itself were ambiguous or contained another element that also responds to the change. Two construction steps remove such scenes, and a control checks the result.
First, two researchers independently screened every candidate frame, and only frames that both kept were admitted (Appendix D.3). A frame was rejected if its geometry was ambiguous, if either branch was not a plausible future of it, if the effect would not be visible, or if another element would also respond to the target variable. For example, friction scenes exclude a lead vehicle whose braking would also change with the surface, and mass scenes need a push strong enough for mass to matter (Table 16).
Second, two researchers independently drafted each prompt pair, a third cross-checked it, and only pairs that all three approved were kept (Appendix D.4). About 37% of drafts were rejected, mainly because the wording did not match the frame or left the intervention ambiguous.
Third, the control shows that the expected direction can be produced from the given inputs: with the intervention, the leading models produce the expected difference far more often than without it. In a no-intervention control (see our response to Reviewer yy49), pairs of two generations from the same prompt pass Outcome in 14.5% of cases, against 25.9% for real intervention pairs, and the leading models exceed their own floors by 18.5 to 32.4 pp.