acceptodds
Under review as a conference paper at ICLR 2027

What-If World: A Contrastive Reasoning Benchmark for Video World Models in Embodied Scenarios

Abstract

Video generation models are increasingly used as world simulators for driving and robotic manipulation, where their value lies in answering what-if questions: if the car brakes harder, or the object is heavier, what happens next? Answering such questions requires a model to change its output correctly when its input changes. Many existing benchmarks score each video in isolation rather than test this contrast: a model can generate two videos that each look plausible yet are wrong relative to each other. We introduce What-If World, a contrastive reasoning benchmark of 319 prompt pairs grounded in real frames from nuScenes and DROID. The two prompts in each pair describe the same scene with a single physical variable altered, drawn from a taxonomy of six variables shared across driving and manipulation. The prompts differ only slightly in wording, but the correct videos should diverge substantially. We score each pair with APEO, a rubric with four checks: whether each video follows its prompt (Adherence), whether its motion is physically consistent (Physics), whether the shared scene is preserved (Environment), and whether the two videos differ in the expected outcome (Outcome). Across nine state-of-the-art models, contrastive reasoning remains challenging: none exceeds 53% on the paired score, and the four open-source models average 28.6%. This failure is easy to miss when each video looks plausible on its own. Every model scores 28 to 49 points lower on paired physics than on single-video physics, a gap we call the contrastive bottleneck: a common failure is to produce similar trajectories under both prompts. Beyond being hard to see, the failure falls on the what-if questions that matter for embodied agents; for example, models are often far better at making things move than at making them stop. Realism therefore does not imply contrastive reasoning, which What-If World measures one intervention at a time, a useful test before a video generation model serves as a world simulator.

Primary area:
datasets and benchmarks
Keywords:
Video World Models, Video Generation, Embodied AI, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_s59POpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces What-If World, a benchmark for evaluating whether video world models respond correctly to controlled physical interventions. Instead of judging each generated video independently, the benchmark compares paired generations that differ in one physical variable. The benchmark contains 319 prompt pairs across autonomous driving and robotic manipulation, covering six physical primitives. The authors also introduce the APEO evaluation framework and evaluate nine recent video generation models. A key finding is that videos can look physically plausible individually while still failing to respond correctly to the intervention.

Critical strengths
  1. The benchmark targets an important evaluation gap that is not captured by conventional single-video metrics. The key idea: evaluating whether generated futures change appropriately when one controlled variable changes—is well motivated and consequential for the use of video models as world simulators. A model that generates two individually realistic videos but ignores the intervention is unsuitable for planning or counterfactual simulation, even though conventional video-quality or physics metrics may rate both outputs highly. The paper demonstrates that this is not merely a hypothetical issue: the substantial gap between single-video and paired physics performance appears consistently across all nine evaluated models.

  2. The empirical study is unusually thorough for a benchmark paper. The paper evaluates a diverse set of open and closed source models, repeats generation three times, reports uncertainty through scene-cluster bootstrapping, provides detailed per-primitive and per-subcategory analysis, and performs human validation of the VLM judge. The extensive failure-mode analysis is also valuable. This makes the benchmark substantially more diagnostic than a single scalar metric.

Critical weaknesses
  1. For variables such as friction, mass, or force magnitude, the exact outcome can depend on physical properties that are not directly observable from a single frame and text prompt. This makes it somewhat difficult to separate model failure from ambiguity in the benchmark specification.

  2. The evaluation still relies heavily on the VLM judge. The human agreement results are reasonably good, but not perfect, and some of the tested phenomena are quite subtle. I would like to see stronger evidence that the main conclusions are stable under human evaluation or alternative judges.

Questions for authors
  1. For categories such as friction, mass, and material properties, how do the authors ensure that the expected counterfactual outcome is sufficiently unambiguous from the information given to the model?

  2. Do the main conclusions, especially the single-vs-paired physics gap, remain similar when evaluated directly by humans?

  3. For the environment interventions that use edited initial frames, how much does the image editing itself affect the paired evaluation?

5 comments

We thank the reviewer for the positive assessment and the careful questions. We address the weaknesses and questions below and will incorporate the results of this discussion into the revised paper.

W1 + Q1: Unobservable properties and the expected direction

Unobservable properties do not make the expected outcome ambiguous, because they do not change the direction of the difference between the two videos, and we score only this direction. Scoring the direction follows prior evaluations of physical reasoning. Our contribution is to make the evaluation contrastive, and to our knowledge What-If World is the first benchmark to do so for general world models. Below, (1) explains why unobservable properties do not change the direction we score and why scoring the direction is a meaningful test (W1), and (2) shows how construction removes scenes in which the direction could still be unclear (Q1).

(1) Unobservable properties do not change the direction, which is what we score (W1)

In the scenes we keep, unobservable properties mainly affect how large the difference between the two videos is, not its direction, and we only score the direction.

First, properties other than the intervened variable are the same in both videos, because the two branches share the same frame and text except for the intervention (Section 3.2). Such a property changes the size of the effect but not its direction.

Second, the exact value of the intervened variable is also unobservable, but the inputs state how it differs between the two branches, and this alone fixes the direction of the outcome.

Third, we score only this direction, not exact values: Outcome checks whether the two videos differ in the direction the intervention predicts (Table 1). This is a meaningful test of physical reasoning: any correct numerical prediction must also match the direction, and a planner comparing candidate actions needs this direction [1]. Exact values would instead require exact parameters in the prompt, but general world models have difficulty grounding even exact object counts from text [2], so a failure could then reflect an ignored number rather than wrong physics. Scoring the direction also follows prior work, which evaluates physical understanding separately from parameter estimation [3] and assesses physical capability through generated videos [4][5][6]. We will state in Section 3.2 that each expected outcome specifies the direction of change.

(2) Construction removes scenes in which the direction could still be unclear (Q1)

The direction could still be unclear if the scene itself were ambiguous or contained another element that also responds to the change. Two construction steps remove such scenes, and a control checks the result.

First, two researchers independently screened every candidate frame, and only frames that both kept were admitted (Appendix D.3). A frame was rejected if its geometry was ambiguous, if either branch was not a plausible future of it, if the effect would not be visible, or if another element would also respond to the target variable. For example, friction scenes exclude a lead vehicle whose braking would also change with the surface, and mass scenes need a push strong enough for mass to matter (Table 16).

Second, two researchers independently drafted each prompt pair, a third cross-checked it, and only pairs that all three approved were kept (Appendix D.4). About 37% of drafts were rejected, mainly because the wording did not match the frame or left the intervention ambiguous.

Third, the control shows that the expected direction can be produced from the given inputs: with the intervention, the leading models produce the expected difference far more often than without it. In a no-intervention control (see our response to Reviewer yy49), pairs of two generations from the same prompt pass Outcome in 14.5% of cases, against 25.9% for real intervention pairs, and the leading models exceed their own floors by 18.5 to 32.4 pp.

W2 + Q2: Human and second-judge validation

The main conclusions, including the single–paired Physics gap, hold under direct human scoring and under a second judge. We keep the VLM as the main judge because it covers every test case and metric and gives consistent scores at a scale where human annotation is impractical, as in VideoPhy [4], PhyGenBench [5] and WorldModelBench [7]. Three checks based on evidence other than Gemini's judgments confirm its conclusions: (1) human labels alone show a large gap (Q2), (2) a second judge, GPT-5, reaches the same conclusion (W2), and (3) on subtle phenomena, the judge agrees with humans as often as annotators do and follows motion measured by object tracking (W2).

(1) Human labels also show a large gap (Q2)

Under direct human scoring of 341 records, the gap (single-video minus paired Physics pass rate) is 29.3 pp (95% CI 21.8–36.7): without any Gemini judgment, annotators rate 61.6% of single videos as physically plausible but only 32.3% of pairs as showing the correct physical relation (Table R1). With the stricter baseline that requires both videos to pass, the gap is still 15.2 pp (95% CI 6.8–23.4).

Table R1. Human labels also show a large single–paired Physics gap

Single-video baselineSingle-video pass rate (%)Paired pass rate (%)Gap, pp [95% CI]
Mean of the two videos61.632.329.3 [21.8, 36.7]
Both videos pass47.532.315.2 [6.8, 23.4]

(2) A second judge, GPT-5, reaches the same conclusion (W2)

GPT-5, scoring the same run-1 videos, also finds a large gap (52.3 pp, against 37.9 pp for Gemini on these videos), and, as reported in Appendix E.7, the two judges rank the models similarly (Spearman ρ\rho = 0.87, with the same three leading models).

(3) The judge stays reliable on subtle phenomena (W2)

For the subtle phenomena, we checked the judge in two ways.

First, the judge agrees with humans as often as annotators agree with each other, both overall (82.5%, κ\kappa = 0.65, against 84.1%, κ\kappa = 0.67, Section 3.3) and on the primitives that may be hard to see: Surface Friction 84.0% against 84.0%, Material/Medium 83.0% against 84.0%, and Obstacle Configuration 80.0% against 77.0% (Appendix E.7, Table 18). Its ppAvg also tracks the human ppAvg across (model, sub-category) groups (Pearson rr = 0.79).

Second, where motion can be measured, the judge's scores follow measurements that use no judge at all. On 88 scenes whose expected outcome is a single motion quantity, we tracked objects with Grounding DINO [8] and SAM 2 [9] and counted an effect as clear only if it exceeded τ\tau, the variation between two generations of the same prompt. The judge passes none of the 28 pairs whose motion goes clearly in the wrong direction, and 68% of pairs with a clear effect in the expected direction (Table R2). A judge that read only the prompt or guessed would not show this pattern. We will add these checks to the appendix.

Table R2. Judge Outcome scores by measured motion effect (judge passes / measurable pairs)

Measured effectFirst generation run (nine models)Second generation run (four models)
Wrong direction, beyond τ\tau (z<−1z < -1)0 / 28 (0%)0 / 16 (0%)
Within τ\tau, not in the expected direction (−1≤z≤0-1 \le z \le 0)47 / 262 (18%)15 / 99 (15%)
Within τ\tau, in the expected direction (0<z≤10 < z \le 1)93 / 350 (27%)55 / 146 (38%)
Expected direction, beyond τ\tau (z>1z > 1)39 / 57 (68%)39 / 50 (78%)
AUC within sub-category and model [95% CI]0.68 [0.61, 0.75]0.71 [0.61, 0.79]

Note. zz = measured A/B difference in the expected direction divided by τ\tau.

Q3: Effect of edited initial frames

Image editing has only a small effect on the main result, the single–paired Physics gap, because each edit changes only the intervened condition and every edited frame was checked manually. Each edit changes only the environmental condition that the intervention requires, such as snow on the road, and keeps the scene layout, objects, agents and camera viewpoint unchanged (Section 3.2, Appendix D.3). Each edited frame was manually checked against its original and the edit instruction (Appendix D.5). Any unintended difference between the two scenes would also fail the paired Environment check (EpE_p), which requires the shared scene to be preserved, so it would lower a model's score rather than raise it (Section 3.3). The exclusion test below quantifies the effect on the main result.

The edited pairs change the gap only slightly and do not inflate it. As an additional controlled experiment, we excluded all 72 edited-frame pairs and found that the observed differences remained. On the remaining 247 pairs, whose two branches start from byte-identical images, the single–paired Physics gap is 39.2 pp (95% CI 35.0–43.2), against 37.6 pp on the full benchmark (nine models and three runs, Table R3).

Table R3. Single–paired Physics gap with and without the 72 edited-frame pairs

Pairs includedPairsGap vs. mean single-video Physics, pp [95% CI]Gap vs. both-videos-pass baseline, pp [95% CI]
All pairs31937.6 [34.1, 41.0]21.5 [17.8, 25.2]
Edited pairs removed24739.2 [35.0, 43.2]26.4 [22.3, 30.1]

Removing the edited pairs changes the gap by only +1.6 pp (95% CI of the difference 0.4–2.8), about 4% of the gap, and toward a larger gap, not a smaller one. The stricter gap, which counts a pair only if both videos pass single-video Physics, rises the same way (from 21.5 to 26.4 pp, +4.8 pp, 95% CI 3.4–6.4). We will add this exclusion analysis to the appendix.

References

[1] Gupta et al., 2024. The Essential Role of Causality in Foundation World Models for Embodied AI. https://arxiv.org/html/2402.06665v2

[2] Sun et al., 2026. When Numbers Speak: Aligning Textual Numerals and Visual Instances in Text-to-Video Diffusion Models. https://arxiv.org/html/2604.08546v1

[3] Upadhyay et al., 2026. WorldBench: Benchmarking Physical Understanding of World Models by Isolating Physics Concepts. https://arxiv.org/abs/2601.21282

[4] Bansal et al., 2024. VideoPhy: Evaluating Physical Commonsense for Video Generation. https://arxiv.org/html/2406.03520v2

[5] Meng et al., 2025. Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation. https://proceedings.mlr.press/v267/meng25c.html

[6] Motamed et al., 2026. Do Generative Video Models Understand Physical Principles? https://arxiv.org/html/2501.09038

[7] Li et al., 2025. WorldModelBench: Judging Video Generation Models As World Models. https://arxiv.org/abs/2502.20694

[8] Liu et al., 2024. Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection. https://arxiv.org/abs/2303.05499

[9] Ravi et al., 2024. SAM 2: Segment Anything in Images and Videos. https://arxiv.org/abs/2408.00714

Reviewer_yy49OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces What-If World, a contrastive benchmark that tests whether video generation models change their output correctly when one physical variable in the input is changed. It contains 319 prompt pairs built on real frames from nuScenes and DROID and organized by a taxonomy of six physical primitives shared across driving and manipulation. Each pair is scored with APEO, which checks adherence, physics, and environment consistency for single videos and pairs, and the expected outcome difference for pairs, using a VLM judge validated against human labels.

Critical strengths
  1. Evaluating pairs rather than single videos is a simple and well-motivated idea for using video models as world simulators, and it reveals failures that per-video scores miss, such as two plausible videos that follow the same trajectory. This is the main reason I lean towards acceptance.

  2. The evaluation is careful: three generation runs, cluster bootstrap intervals, judge validation with agreement close to inter-annotator agreement, a second judge, and a full release of videos and judge outputs.

Critical weaknesses
  1. The noise floor of the Outcome check is not well established. Control pairs that should fail, such as two samples of the same prompt, still pass Op in 13% of cases, which is close to the Op scores of all open-source models, and this control uses only 15 pairs from one model.

  2. The model selection limits the conclusions. The open-source models are the smallest available variants, so the closed/open gap mixes access type with model size, and no action-conditioned or domain-specific world models for driving or manipulation are included, although these are the models most likely to serve as simulators.

  3. The contrastive bottleneck is partly built into the metrics. Paired physics requires the two videos to diverge as predicted, which largely overlaps with the Outcome check, while single-video physics only checks plausibility, so a large gap between them is expected by design.

Questions for authors
  1. Clarification: Could the authors run the false-pass control on more pairs and for every model, and report Op relative to this floor?

  2. Clarification: How do larger open-source variants, such as Wan2.2-A14B or Cosmos-Predict2-14B, perform on the paired score?

  3. Clarification: Clips are only 4 to 5 seconds long. Are the timing interventions in driving, which score lowest, fully observable within this length?

6 comments

We thank the reviewer for the positive assessment and the concrete questions. We address the weaknesses and questions below and will incorporate the results of this discussion into the revised paper.

W1 + Q1: Results of the expanded false-pass control

Our expanded control shows that the Outcome scores reflect responses to the intervention rather than generation noise: pooled over the nine models, real intervention pairs pass Outcome in 25.9% of cases, 11.4 pp more often than same-prompt pairs without any intervention (14.5%, 95% CI of the difference 8.4–14.6), and every model scores above its own no-intervention floor (Table R1).

The control now covers all nine models with 108 pairs each

Each model is tested on six pairs from each of the 18 sub-categories, which extends the 15-pair, single-model control in the paper. The expanded floor (14.5% pooled, 11.1% to 18.5% per model) is close to the 13% of that control. Same-prompt pairs (AA and BB) contain no intervention, so any pass is a false pass. Real and same-prompt pairs both take their two videos from two independent generations, so they differ only in whether the intervention changes, and all pairs are judged with the original prompts, expected difference and rubric.

Table R1. Outcome pass rate: real intervention versus no-intervention floor (108 pairs per model)

ModelAccessReal (%)Floor (%)Difference, pp95% CIAdjusted CI
Veo 3.1Closed50.017.632.4[24.6, 40.4][21.5, 43.4]
Grok ImagineClosed40.718.522.2[13.3, 31.5][10.0, 35.2]
Seedance 2.0Closed31.913.418.5[11.5, 25.5][8.7, 28.4]
Seedance 1.5Closed22.711.111.6[5.6, 18.1][3.2, 21.1]
Kling 3.0Closed24.113.410.6[3.2, 18.7][0.4, 22.0]
Cosmos-Predict2-2BOpen16.713.92.8[−3.2, 8.6][−5.9, 10.9]
CogVideoX1.5-5BOpen15.313.02.3[−3.3, 8.0][−5.8, 10.8]
HunyuanVideo1.5-8.3BOpen16.214.41.9[−3.8, 7.7][−5.9, 10.4]
Wan2.2-5BOpen15.715.30.5[−3.8, 4.9][−5.6, 6.7]
Pooled, nine models25.914.511.4[8.4, 14.6]

Note. Floor: two generations of the same prompt, judged against the same expected difference. Differences computed before rounding. Adjusted CI: Bonferroni correction for nine models.

Veo 3.1, Grok Imagine and Seedance 2.0 exceed their floors by 32.4, 22.2 and 18.5 pp, with intervals above zero after Bonferroni correction for nine models, so their advantage is not explained by random generation differences. We will add this control to the appendix and show each model's floor next to its OpO_p in Table 2.

W2: Model size and domain-specific world models

We respectfully note that the model selection does not limit the paper's central finding, because the single–paired gap is measured within each model. (1) On model size, the larger Wan2.2-A14B improves ppAvg but remains clearly behind the leading closed-source models, and its within-model gap is nearly the same as that of Wan2.2-5B. (2) On domain-specific models, we deliberately evaluate general world models, which apply to more tasks, whereas domain-specific models take specialized inputs for a narrower task.

(1) Scale explains part of the closed/open difference, but does not close the gap to the top models or change the within-model gap

Scale does bring an improvement: Wan2.2-A14B gains 7.1 pp ppAvg (the mean of the four paired checks) over Wan2.2-5B, but it still stays 6.2 to 16.1 pp below the three leading closed-source models (Tables R2 and R3, in our answer to Q2), so the larger variant does not close the gap to the leading models. Closed-source model sizes are not disclosed, so in Section 4.1 we will describe this difference as a comparison between the evaluated variants rather than between access types.

The central finding does not depend on this comparison: within each model, individually plausible futures can still fail to show the correct physical relation under a changed intervention. This within-model gap stays nearly the same with the larger model (38.1 pp for Wan2.2-A14B, 37.9 pp for Wan2.2-5B), so it is not an artifact of using small open-source variants. We will name each model's exact variant in Table 2 and add the Wan2.2-A14B results.

(2) We evaluate general world models because we want to provide a general insight

We evaluate general world models because they are more widely used than domain-specific ones and apply to a much wider range of tasks, so our benchmark design generalizes to more use cases, with driving and manipulation as two examples (Sections 3.1–3.2). Our question is whether one general video model can predict how the consequences change under a physical intervention: the image and text specify the scene and the changed action or property, but not the expected consequence.

Specialized world models instead target control and planning within one domain and take domain-specific inputs: steering, speed or ego trajectories in the driving models Vista [1] and Drive-WM [2], and robot actions or end-effector trajectories in manipulation. These inputs already specify the motion that our probes ask the model to predict, for example how braking changes on an icy road. Evaluating them would therefore require separate, domain-specific handling of the inputs, and such a setup would not transfer to other applications.

Our results are still relevant to specialized models, because these are often built on general video models: Vista starts from pretrained Stable Video Diffusion before driving-specific training [1], and Cosmos evaluates the physical alignment of pretrained models separately from domain post-training [3]. Evaluating specialized models directly is a clear next step after our cross-domain evaluation, and we will state this scope in Section 3 and list it as future work in Appendix A.

W3: Single and paired Physics

The gap reveals a failure that matters in practice and that single-video evaluation cannot detect. This contrastive bottleneck appears when a model generates two individually plausible videos that do not differ as the intervention predicts, whereas a world model used to compare actions or conditions must respond to the intervention [4]. Many existing benchmarks score each video in isolation, so this ability has rarely been measured or targeted. To our knowledge, What-If World is the first benchmark to reveal this gap (Section 4.2). Below, we show that the gap reflects the models rather than the metric design: (1) its size differs significantly across models, and the failures are visible in the videos and found by human annotators, so neither the metric definition nor the judge creates them, and (2) Outcome does not enter the gap, so its overlap with paired Physics cannot inflate it.

(1) The gap reveals model failures of different degrees

The degree of failure differs significantly across models. The gap ranges from 27.7 pp (Veo 3.1, 95% CI 23.6–31.5) to 49.3 pp (HunyuanVideo1.5, 95% CI 44.4–53.8), with non-overlapping intervals, and the two leading models, Veo 3.1 and Grok Imagine, have the smallest gaps (27.7 and 29.5 pp, Appendix Table 27). The gap is therefore model-specific, so it tells us something about each model.

These failures are real: they are created neither by the metric definition nor by the VLM judge. First, the failure is visible in the videos themselves, so the metric definition does not create it: in Figure 3, a maximum-effort launch and a gentle launch cover nearly the same distance, although each video looks plausible on its own. Second, human annotators find the same failure, so the judge does not create it: with human labels alone, 77 of the 162 pairs whose two videos annotators rate as physically plausible, nearly half, still fail paired Physics.

(2) PpP_p and OpO_p check different things, and their overlap does not drive the results

The overlap with Outcome is partial. PpP_p checks the trajectory relation: matching motion before the intervention and physically appropriate divergence after it. OpO_p checks only whether the expected state difference appears at some point in the clip (Table 1). Outcome also does not enter the gap, which is computed from Physics pass rates alone, so the overlap cannot inflate it. Every weighting from 0.1 to 0.4 per check, including counting P and O as one group, also keeps the same three leading models (see our response to Reviewer mxJs, Q2). We will clarify in Section 4.2 what the gap measures and how PpP_p differs from OpO_p.

Q2: Larger open-source variants

Wan2.2-A14B reaches 35.6% ppAvg, 7.1 pp above Wan2.2-5B (95% CI 2.9–11.4), but it remains 6.2 to 16.1 pp below the three leading closed-source models (Tables R2 and R3). Its single–paired Physics gap (38.1 pp) is nearly the same as that of Wan2.2-5B (37.9 pp).

Table R2. Wan2.2-A14B versus Wan2.2-5B on the same 319 pairs

ScoreWan2.2-5B (%)Wan2.2-A14B (%)Difference, pp [95% CI]
ppAvg28.435.6+7.1 [2.9, 11.4]
Paired Physics (PpP_p)14.122.3+8.2 [2.8, 13.6]
Single-video Physics (PsP_s)52.060.3+8.3 [3.3, 13.5]
Single–paired Physics gap (Ps−PpP_s - P_p)37.938.1+0.2 [−5.5, 5.9]

Note. Differences computed before rounding.

Scale raises both Physics scores but barely changes the gap

Both single-video and paired Physics rise by about 8 pp: the larger model produces more plausible videos and more correct pairs, but the gap between the two barely changes (+0.2 pp, 95% CI −5.5 to 5.9).

Table R3. ppAvg of all ten models

ModelAccessppAvg(%)
Grok ImagineClosed51.7
Veo 3.1Closed50.9
Seedance 2.0Closed41.8
Kling 3.0Closed38.6
Wan2.2-A14B (new)Open35.6
Cosmos-Predict2-2BOpen32.9
Seedance 1.5Closed32.4
Wan2.2-5BOpen28.4
HunyuanVideo1.5-8.3BOpen27.0
CogVideoX1.5-5BOpen22.7

Note. Original nine models as in Appendix Table 23.

Reviewer_mxJsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces What-If World, a contrastive benchmark designed to evaluate whether video generation models can produce physically consistent outcomes under different interventions. The benchmark contains 319 prompt pairs covering autonomous driving and robotic manipulation, with six physical intervention categories. The authors further propose the APEO evaluation framework, which measures Adherence, Physics, Environment, and Outcome using a VLM-based evaluator. Experiments across nine video generation models reveal significant performance gaps between single-video physical plausibility and paired contrastive consistency.

Critical strengths
  1. The paper addresses an important problem in video world model evaluation: assessing whether generated dynamics respond consistently to changes in physical conditions rather than merely producing visually plausible videos.

  2. The contrastive evaluation paradigm is intuitive, and the benchmark covers two representative embodied domains.

  3. The authors provide extensive experimental results across nine video generation models, together with human evaluation, repeated generation experiments, and failure-mode analysis.

Critical weaknesses
  1. Lack of explicit physical parameterization. The benchmark evaluates physical interventions primarily through semantic changes in prompts or edited images, rather than explicitly controlling physical parameters. For example, surface friction is modified by describing dry versus icy roads without specifying friction coefficients. Consequently, the benchmark mainly evaluates semantic responsiveness rather than quantitative physical simulation, weakening its claims about physical reasoning.

  2. Physical reasoning is conflated with generation controllability. The benchmark does not clearly distinguish physical reasoning failures from video generation limitations. A model may understand the expected physical consequences but fail to generate the corresponding motion due to insufficient controllability. Conversely, generating the expected outcome does not necessarily demonstrate genuine physical reasoning. This ambiguity limits the interpretation of the experimental findings.

  3. The single-variable intervention assumption is not fully satisfied. For 72 benchmark pairs, the authors use image editing to modify environmental conditions. However, image editing may introduce unintended changes in appearance, texture, or geometry, potentially affecting video generation beyond the intended intervention. Additional controlled experiments are needed to verify that the observed differences arise from the targeted physical variable.

  4. Overreliance on VLM-based evaluation. The benchmark relies primarily on Gemini 3.1 Pro for binary judgments of physical consistency, without quantitative measurements of trajectories, velocities, or object displacement. Although human validation is provided, the reported agreement (82.5%, κ = 0.65) indicates remaining uncertainty. Quantitative physical measurements would provide stronger validation of the benchmark's reliability.

Questions for authors
  1. Why are physical interventions such as friction, mass, and force specified only through qualitative semantic descriptions? Without explicitly controlling physical parameters, how can the benchmark distinguish genuine physical simulation from semantic prompt following?

  2. What justifies assigning equal weights to the four APEO dimensions, considering their conceptual overlap? How robust are the conclusions to alternative weighting schemes?

6 comments

We thank the reviewer for the detailed review. We address the weaknesses and questions below and will incorporate the results of this discussion into the revised paper.

W1 + Q1: Physical parameters and physical reasoning

We would like to clarify that predicting how a physical change alters motion is itself physical reasoning [1][2]. Such reasoning can be tested without exact parameters, as prior work evaluates intuitive physical understanding separately from parameter estimation [1]. We score the direction of the change, which any correct numerical prediction must also match and which a planner needs when comparing candidate actions [2] (see (1)). Fine-grained control of exact physical parameters is an important research topic on its own, but it is outside the scope of this work: precise numerical grounding is still an open problem for general world models, so exact parameters in the prompt would confound the test (see (2)). To our knowledge, What-If World is the first benchmark to test contrastive reasoning in general world models, which are widely used across many downstream applications. We follow prior work in how physical reasoning is evaluated. Our contribution is to evaluate it contrastively, by checking whether a model's prediction changes correctly when a single physical variable changes, and our four checks separate this from semantic prompt following (see (3)).

(1) Testing the direction of the effect does not weaken the physical-reasoning claim (W1)

A correct numerical prediction must match both the direction and the magnitude of an effect, and a model that predicts the wrong direction has made a physical error regardless of precision. The direction is therefore a necessary part of physical prediction, not a weaker substitute for it. WorldBench also treats intuitive physical understanding and parameter estimation as separate evaluation targets [1], and a planner that compares candidate actions by their consequences requires this direction to be correct [2].

(2) Precise numerical grounding of general world models is still an open problem (Q1)

Current general world models, which take text as input, have difficulty in grounding even exact object counts from text [3], and models that control physical parameters explicitly need dedicated parameter inputs [4], which general world models lack. With numbers in the prompt, a failure could reflect an ignored number rather than wrong physics, which is not what we aim to test.

(3) Our four checks jointly identify correct physical reasoning (Q1)

Our four checks jointly cover what physical reasoning about an intervention requires, so passing them reflects physical reasoning rather than semantic prompt following (Table 1). The intervention must be applied (Adherence), everything else in the scene must stay fixed (Environment), the motion must remain physically valid and diverge only after the intervention (Physics), and the predicted consequence must be correct (Outcome). Each way of passing without physical reasoning fails at least one check. A model that only follows the text can render an icy road and pass Adherence, but it fails Outcome unless the car also travels farther before stopping than on the dry road. A model that creates a difference by changing the scene rather than through the intervention fails Environment, and one that reaches the right end state through implausible motion fails Physics. The checks also separate in practice: among Surface Friction pairs in which both videos show the requested surface, only 28.7% show the expected difference in motion (Appendix C.6, Table 9). VideoPhy [5] and WorldModelBench [6] likewise score prompt following separately from physical correctness.

W2: Physical capability and generation

We would like to clarify that our main contribution is evaluating the contrastive physical reasoning of general world models. We evaluate it through the predictions the models produce, the generated videos, which is common practice: VideoPhy [5], PhyGenBench [7] and Physics-IQ [8] also evaluate physical capability through generated videos, because the predicted video is what a world model offers for downstream use. Whether a model internally understands the underlying physical process remains an open problem [9] and applies equally to these benchmarks. This open question does not weaken our conclusions about the physical ability of general world models, because downstream tasks use the predicted videos.

Case 1: Why understanding right, generation wrong will not limit our finding

In practice, the generated future is the model's prediction of what happens next, and the video is what downstream tasks use. If a model understands the consequence but cannot generate it, its prediction is still wrong and cannot be used downstream, so the benchmark is right to score it as a failure. For downstream use, the key question is where the video goes wrong, and our metrics record this separately (Table 1 and Appendix C.6, Table 9): Adherence shows whether the intervention is rendered, single-video Physics whether each motion is plausible, and the paired checks whether the pair responds correctly to the intervention.

Case 2: Why understanding wrong, generation right will not limit our finding

The success rates in our benchmark are clearly separated from coincidence, so a model that scores well does not do so by chance: real interventions pass Outcome well above a no-intervention false-pass floor. In a new control (see our response to Reviewer yy49), pairs of two generations from the same prompt, which contain no intervention but are judged against the same expected A/B difference, pass Outcome in 14.5% of cases, compared with 25.9% for real intervention pairs (+11.4 pp, 95% CI 8.4–14.6). Model scores and rankings are also stable across three independent generation runs (Appendix F.2).

W3: Effect of edited initial frames

The edits change only the intervened condition by design, and every edited frame was checked manually, so unintended changes do not determine our results. Each edit changes only the environmental condition that the intervention requires, such as snow on the road, and keeps the scene layout, objects and their positions, agents and camera viewpoint unchanged (Section 3.2, Appendix D.3). Each edited frame was manually checked against its original and the edit instruction (Appendix D.5). Any unintended difference between the two scenes would also fail the paired Environment check (EpE_p), which requires the shared scene to be preserved (Section 3.3), so it would lower a model's score rather than raise it. The controlled experiment below confirms that the edited frames change the main result only slightly and do not inflate it.

The edited pairs change the gap only slightly and do not inflate it

As an additional controlled experiment, we excluded all 72 edited-frame pairs and found that the observed differences remained. On the remaining 247 pairs, whose two branches start from byte-identical images, the single–paired Physics gap (single-video minus paired Physics pass rate) is 39.2 pp (95% CI 35.0–43.2), compared with 37.6 pp on the full benchmark (nine models and three runs, Table R1).

Table R1. Single–paired Physics gap with and without the 72 edited-frame pairs

PairsGap, pp [95% CI]Stricter gap, pp [95% CI]
All 319 pairs37.6 [34.1, 41.0]21.5 [17.8, 25.2]
247 pairs without edited frames39.2 [35.0, 43.2]26.4 [22.3, 30.1]
Difference+1.6 [0.4, 2.8]+4.8 [3.4, 6.4]

Removing the edited pairs changes the gap by only +1.6 pp (95% CI of the difference 0.4–2.8), about 4% of the gap, and in the direction of a larger gap, not a smaller one. The stricter version of the gap in Table R1, which counts a pair only if both videos pass single-video Physics, rises the same way (from 21.5 to 26.4 pp, a difference of +4.8 pp, 95% CI 3.4–6.4). The finding therefore holds without branch-specific edits and changes little when they are included. We will add the exclusion analysis to the appendix.

W4: Judge reliability and motion measurements

Quantitative measurements can evaluate the VLM-as-judge but cannot replace it, and three new checks confirm that the VLM judge is reliable. Tracking only works where the outcome is a single motion quantity, while other sub-categories require judging events, such as a successful grasp or a deforming material, so such measurements cannot cover all our test cases and metrics. We therefore keep the VLM as the main judge: it covers every test case and metric and gives consistent scores at a scale where human annotation is impractical, and VideoPhy [5], PhyGenBench [7] and WorldModelBench [6] also use VLM judges for such evaluations.

Three checks based on evidence other than Gemini's judgments further support the judge: (1) motion measured by object tracking, as the reviewer suggested, agrees with the judge's Outcome scores (Table R2), (2) human labels alone show the paper's main finding (Table R3), and (3) a second judge, GPT-5 (Appendix E.7), finds it too.

(1) Measured motion agrees with the judge

We measured how the objects actually move in the A and B videos and checked whether the judge's Outcome scores follow these measurements. We used the 88 scenes whose expected outcome is a single motion quantity, such as the distance to the lead vehicle or how far an object is pushed or lifted. We tracked the objects with Grounding DINO [10] and SAM 2 [11] and measured the A/B difference in that quantity (for speed changes, the difference in distance covered), without using judge or human labels. To separate a real effect from generation noise, we compared each difference with τ\tau, the 95th percentile of the difference between two generations of the same prompt.

A judge that passed pairs by reading the prompt or by chance would pass them at the same rate no matter how the objects moved. Instead, the judge's scores track what happens in the videos. (a) The judge passes none of the 28 pairs whose measured motion goes clearly in the wrong direction (beyond τ\tau). (b) Its pass rate rises with the measured effect, from 18% (within τ\tau, not in the expected direction) and 27% (within τ\tau, in the expected direction) to 68% (beyond τ\tau, in the expected direction). (c) Within the same sub-category and model, when a pair the judge passes is compared with one it fails, the passed pair shows the larger measured effect 68% of the time (AUC 0.68, 95% CI 0.61–0.75).

Most passes fall within τ\tau, a strict threshold, but even within it the judge passes pairs in the expected direction more often (27% against 18%), as (b) shows. A second generation run of four models shows the same pattern (0 of 16, 78% and AUC 0.71, 95% CI 0.61–0.79). The pass rate on clear effects stays below 100% partly because one motion quantity does not capture everything Outcome checks. For example, the distance to the lead vehicle does not show which vehicle changed speed.

Table R2. Judge Outcome scores by measured motion effect (judge passes / measurable pairs)

Measured effectFirst generation run (nine models)Second generation run (four models)
Wrong direction, beyond τ\tau (z<−1z < -1)0 / 28 (0%)0 / 16 (0%)
Within τ\tau, not in the expected direction (−1≤z≤0-1 \le z \le 0)47 / 262 (18%)15 / 99 (15%)
Within τ\tau, in the expected direction (0<z≤10 < z \le 1)93 / 350 (27%)55 / 146 (38%)
Expected direction, beyond τ\tau (z>1z > 1)39 / 57 (68%)39 / 50 (78%)
AUC within sub-category and model [95% CI]0.68 [0.61, 0.75]0.71 [0.61, 0.79]

Note. zz = measured A/B difference in the expected direction divided by τ\tau.

(2) Human scoring and (3) a second judge confirm the main findings

On the reported agreement, the judge's 82.5% (κ\kappa = 0.65) is on par with the agreement between annotators themselves (84.1%, κ\kappa = 0.67, Section 3.3), so the remaining uncertainty is at the level of human disagreement. With human labels alone and no Gemini judgment, the paper's central pattern also appears: most single videos are judged physically plausible, but many pairs miss the correct physical relation. On 341 existing human-labeled records, annotators rate 61.6% of single videos but only 32.3% of pairs as passing Physics, a 29.3 pp gap (95% CI 21.8–36.7, Table R3).

Table R3. Human labels also show a large single–paired Physics gap

Single-video baselineSingle-video pass rate (%)Paired pass rate (%)Gap, pp [95% CI]
Mean of the two videos61.632.329.3 [21.8, 36.7]
Both videos pass47.532.315.2 [6.8, 23.4]

GPT-5, scoring the same run-1 videos, also finds the gap (52.3 pp, against 37.9 pp for Gemini on these videos), and, as in Appendix E.7, it ranks the models similarly to Gemini (Spearman ρ\rho = 0.87, with the same three leading models). We will add these analyses to the appendix.

Comments on the paper

1 comment

Dear Area Chair 7yA7 and Reviewers,

We sincerely thank the Area Chair for organizing and overseeing the rebuttal process, and the reviewers for the time and care they put into reading our paper and for their thoughtful questions and suggestions. Their comments have been very valuable for improving our work.

We have uploaded a revised version that incorporates the results and clarifications from the discussion, with all changes marked in blue. If you have any further comments, please let us know, and we will be glad to revise the paper further.

Thank you again for the time and effort you have all given to helping us improve this work.

Sincerely,

The Authors of Paper 37439

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.