acceptodds
Under review as a conference paper at ICLR 2027

PhysicsLENS: Diagnosing Physical Property Blindness in Video Generation Models

Abstract

Reliable video world models could provide scalable predictive environments for robot learning, planning, and evaluation. However, robot videos can violate physical principles and complete tasks through physically implausible behavior, limiting their reliability for robot learning and planning. Current video-generation benchmarks exclude physics that are inherently hidden by visuals (eg: weight, viscosity, friction). Due to this, video models are evaluated on the fidelity of physics- not the underlying accuracy of physics. We introduce PhysicsLENS, a dataset and benchmark for evaluating plausibility of physical properties grounded in robotics. PhysicsLENS uses matched scenario pairs that hold the conditioning frame, task, and scene description. Scenarios are curated from public robot video sources and annotated across seven physical domains: collision, gravity, momentum, friction, deformation, fluid, and causality. We evaluate across four video generation models, producing over 400 human-annotated labels. Results show a consistent accuracy gap between observable and unobservable conditions, indicating that models rely on visual appearance rather than grounded text-specified properties.

Primary area:
datasets and benchmarks
Keywords:
video generation models, physics evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_fRTeOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PhysicsLENS, a benchmark that tests whether image-to-video models follow physical properties that are stated in the prompt but cannot be inferred from the initial frame, such as viscosity, surface condition, elasticity, and mass in robot manipulation scenes. It builds 80 observable prompts from public robot datasets and 30 matched unobservable variants that keep the frame and task fixed while changing only the stated property, generates 439 videos with four open-source models, and collects human ratings of physical plausibility, task completion, and property adherence. The authors find that most videos rated plausible under the unobservable condition do not follow the stated property, and propose an automated evaluator that combines motion signals with targeted VLM questions.

Critical strengths
  1. The matched-pair design is a useful idea: keeping the frame and task fixed and changing only the stated property separates physical plausibility from property grounding, which existing physics benchmarks do not distinguish. This is the main reason I see value in the paper.

  2. The observation that plausible-looking videos often ignore the stated property is relevant to using video models as data sources for robot learning.

Critical weaknesses
  1. The model coverage is too narrow. Only four open-source models are evaluated, without commercial models such as Veo, Kling, or Seedance, or robot-specific world models such as DreamGen or DreamDojo, so it is unclear whether property blindness generalizes beyond the tested models.

  2. The annotation and generation are thin. Each clip has a single annotation without any agreement measure, and each prompt is generated only once, so annotation and generation noise cannot be separated from the measured effects.

  3. The central claim is not statistically supported. The plausibility gap between observable and unobservable conditions, presented as consistent in the abstract, is not significant for any model or in the pooled test, partly because there are only 30 unobservable scenarios.

Questions for authors
  1. Clarification: Could the authors report inter-annotator agreement on a subset rated by several annotators, and results over multiple seeds per prompt, to show that the observable/unobservable difference exceeds annotation and generation noise?

  2. Clarification: A low adherence score cannot distinguish a contradicted property from a missing interaction, and many plausible but ungrounded videos come from a generator that barely moves. How many of these videos actually show the interaction that would reveal the property?

If the authors adequately address my concerns, I will consider raising my rating.

3 comments

We thank the reviewer for recognizing the value of our matched-pair setup in separating physical plausibility from property grounding and its relevance to robot learning. We also appreciate your constructive suggestions regarding model coverage, sample size, and annotation reliability. We have now addressed all of these concerns by adding 3 commercial models, expanding from 30 to 80 unobservable scenarios with 3 seeds per prompt, and collecting multi-annotator ratings across the entire benchmark. Please see our detailed responses and updated statistical results below.

W1. Does property blindness generalize beyond the four open-source models tested?

In our study, we started with 4 open-source models to establish the matched-prompt setup and ensure clarity. Following the reviewer's suggestion, we have conducted additional experiments with 3 of the latest closed-source commercial models (Gemini Omni Flash, Grok Imagine Video, and Grok Imagine Video 1.5 Lite) alongside our original 4 models, giving 7 models in total (240 unobservable videos per model, or 1,680 in total).

We observe that property blindness persists across these commercial models as well. Among the generated videos, the 3 closed-source models ignore the stated property in 56-65% of cases, which is comparable to the open-source models (49-71%, excluding MAGI, which has only 1 realistic and testable video). Even Gemini Omni Flash, which produces the most realistic videos, ignores the property in 56% of such cases, indicating that higher visual quality alone does not resolve this issue. Please refer to the table in Q2 for the per-model breakdown. We also plan to release our benchmark and evaluation code publicly so that any future commercial or robot-specific world models can be tested directly.

W2, W3 & Q1. Is the observable–unobservable plausibility gap significant and larger than annotation and generation noise?

We agree that a single annotation and generation per prompt can make it difficult to discern noise from actual physical property grounding. To mitigate this, we expanded hidden-property scenarios from 30 to 80 and now generate 3 seeds per prompt, totaling 3,360 videos. Each of the four main categories (mass, surface condition, viscosity, elasticity) now has at least 15 scenarios, or at least 315 annotated videos. Every video now has independent ratings from at least 2 annotators (41% receive 3), giving 8,095 annotations in total. Agreement is moderate across all four labels:

Label (yes/no)Percent agreement (↑)Cohen’s κ (↑) (average over annotator pairs)
Realistic (P ≥ 3)78%0.56
Task completed80%0.60
Property testable84%0.62
Follows hidden property (H ≥ 3)77%0.52

With this expanded evaluation across 80 matched scenarios, the performance gap between observable and unobservable properties is statistically significant. Videos are rated 0.31 ± 0.03 points less plausible (mean ± SE; 95% CI [0.25, 0.37]; two-sided paired t-test, p<10−10p < 10^{-10}) when the hidden property is specified, with 88% of scenarios exhibiting a drop (scenario SD = 0.28).

This drop is consistent across generation seeds and holds across each annotator's ratings evaluated independently (mean drops between 0.16 and 0.37 points, all paired t-test p<0.05p < 0.05), demonstrating that the observed effect exceeds both generation and annotation noise. The plausibility penalty remains robust in a linear mixed-effects model after controlling for generator, annotator, and seed simultaneously (β=−0.29,p<10−32\beta = -0.29, p < 10^{-32}), and is consistent across all three seeds. Plausibility scores drop under unobservable properties for all seven generators, reaching individual statistical significance (p<0.05p < 0.05) in five of them.

We agree with the reviewer that specifying this drop as "consistent" in the abstract was too strong for the original 30 scenarios, and we will revise the abstract to reflect these new findings.

SeedAvg. plausibility drop (observable → unobservable)Scenarios where unobservable is less plausible
Seed 00.2971% (57/80)
Seed 10.3176% (61/80)
Seed 20.3478% (62/80)
All seeds0.31 ± 0.2888% (70/80)

Q2. Separating missing interaction from contradicted properties

We recognize that a low adherence score may not fully distinguish a contradicted property from a missing interaction. Following this suggestion, we introduced a binary "Property Testable" label into our annotation protocol: before scoring adherence, annotators judge whether the video shows enough of the relevant interaction to test the stated property. Across all unobservable videos, 63% (1,056 of 1,680) are testable by all annotators.

After removing untestable videos, 59% (246 of 417) of realistic videos (physical plausibility score P ≥ 3 on our 1 to 4 scale) still ignore the stated property, and 57% (210 of 367) when the task must also be completed. Our main conclusion, that realistic videos often ignore the stated property, also holds under strict annotator consensus: among the 336 unobservable videos where every annotator agreed on both realism and testability, 57% still ignored the property.

TypeVideo generation modelHidden property is testableRealistic (P ≥ 3) & testable videosIgnores hidden property (among realistic & testable)
Proprietary models (new)Gemini Omni Flash85% (204/240)13556% (75/135)
Grok Imagine Video93% (224/240)9758% (56/97)
Grok Imagine Video 1.5 Lite88% (210/240)10965% (71/109)
Open-weight modelsCosmos3-Nano56% (135/240)3171% (22/31)
HunyuanVideo 1.569% (166/240)3549% (17/35)
Wan 2.244% (105/240)956% (5/9)
MAGI 4.5B5% (12/240)10% (0/1)
All63% (1,056/1,680)41759% (246/417)

We would like to thank the reviewer again for their thoughts and constructive suggestions, which have helped us improve our evaluation. These new findings will be incorporated into the camera-ready version. We hope the additional results address your concerns, and we would be grateful if you would consider increasing your score given these new results.

All of my concerns have been addressed, so I’ve raised my rating to Weak Accept.

Reviewer_WjsAOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents PhysicsLENS, a benchmark for testing whether generated robot videos follow physical properties specified in text but not visible in the initial image. It contains 110 prompts built from 80 robotic scenarios, including 30 matched hidden-property variants, and 439 human-annotated videos from four generators. The authors also develop an automated evaluator that combines motion measurements with VLM judgments to assess physical plausibility, task completion, and hidden-property adherence.

Critical strengths
  1. The paper makes a useful distinction between a video looking physically plausible and actually following the physical properties in its prompt. The matched prompts are a sensible way to study this, since they keep the initial image and task fixed. Among the hidden-property videos, 34 of the 47 rated as physically plausible did not demonstrate the specified property. This supports evaluating plausibility and property adherence separately.
  2. The public dataset and evaluation code give other researchers a concrete starting point for testing this problem. The comparison of standard VLM questions, targeted physics-error questions, and motion signals is also informative. The full evaluator reaches an AUC of 0.72 for overall physical plausibility and 0.81 for task completion (Table 4), although its performance on hidden-property adherence is less convincing.
Critical weaknesses
  1. The 34/47 result needs a clearer breakdown. Section 4.1 already shows that the pattern persists without MAGI: 13 of the remaining 26 plausible videos still fail to follow the property. However, removing MAGI does not separate videos that contradict the property from those where the relevant interaction never occurs. I would like to see the failure rate among videos with enough interaction to judge the property, regardless of whether the task is completed. Without this distinction, it is difficult to tell how much of the reported failure rate reflects incorrect physical behavior rather than a lack of evidence.

  2. The added value of the VLM ensemble for hidden-property scoring is less clear. The full method reaches an AUC of 0.71, compared with 0.69 for motion signals alone and 0.68 for generator identity. The paper reports within-generator results for plausibility and task completion, but not for property adherence. That missing comparison matters here: it would show whether the evaluator can distinguish adherence among videos from the same generator, rather than mainly reflecting quality differences between generators.

Questions for authors
  1. Among the plausible videos, how many contain enough interaction to judge the specified property? What is the adherence failure rate within this subset, and how does it vary across generators?

  2. Could the authors report hidden-property adherence AUC within each generator? Does the improvement over motion-only signals hold in this setting, or when evaluating on a held-out generator?

3 comments

Thank you for your supportive comments and for recognizing the value of separating visual plausibility from physical property adherence, as well as our public dataset and VLM evaluation. We appreciate your constructive suggestions regarding separating missing interaction from property violation, per-generator failure rates, and within-generator evaluation of hidden-property adherence. We supplement this rebuttal with additional experiments and analyses to address each of your questions below.

W1 & Q1. Separating static/incomplete videos from active property failures.

MAGI is a clear outlier and a clearer metric is needed to strengthen our argument and distinguish failure through inaction versus violating the hidden property. We address this suggestion by implementing a binary label “Property Testable” for our annotation evaluation to explicitly filter whether the relevant physical interaction occurs and allows the property to be visibly judged. Of the 593 realistic (physical plausibility score P ≥ 3 on our 1 to 4 scale) unobservable videos, 417 (70%) contain enough interaction to test the property according to every annotator. Within this subset, 59% still ignore the hidden property. Filtering to require task completion gives nearly the same result (57%).

TypeVideo Generation ModelHidden Property is TestableRealistic (P ≥ 3) & Testable VideosIgnores Hidden Property (among Realistic & Testable)
Proprietary models (new)Gemini Omni Flash85% (204/240)13556% (75/135)
Grok Imagine Video93% (224/240)9758% (56/97)
Grok Imagine Video 1.5 Lite88% (210/240)10965% (71/109)
Open-weight modelsCosmos3-Nano56% (135/240)3171% (22/31)
HunyuanVideo 1.569% (166/240)3549% (17/35)
Wan 2.244% (105/240)956% (5/9)
MAGI 4.5B5% (12/240)10% (0/1)
All63% (1,056/1,680)41759% (246/417)

The failure rate is similar across generators with enough videos (49–71%), so it reflects incorrect physical behavior rather than missing evidence. We will replace the 34/47 summary in Section 4.1 with this breakdown.

W2 & Q2: Within-generator and held-out adherence AUC

We now report hidden-property adherence AUC within each generator, where generator identity carries no information, and with each generator held out (leave-one-generator-out).

Evaluation SettingMotion-only (↑)Motion + DINO (↑)VLM-only Ensemble (↑)Full Ensemble (↑)
Cosmos3-Nano0.602 ± 0.0000.602 ± 0.0000.758 ± 0.0170.805 ± 0.015
HunyuanVideo 1.50.667 ± 0.0000.667 ± 0.0000.842 ± 0.0050.850 ± 0.001
MAGI 4.5B-distill*0.571 ± 0.0000.571 ± 0.0000.429 ± 0.0000.429 ± 0.000
Wan 2.20.550 ± 0.0000.550 ± 0.0000.690 ± 0.0100.689 ± 0.012
Gemini Omni Flash0.429 ± 0.0000.429 ± 0.0000.768 ± 0.0040.753 ± 0.009
Grok Imagine Video0.573 ± 0.0000.573 ± 0.0000.770 ± 0.0120.766 ± 0.015
Grok Imagine Video 1.5 Lite0.618 ± 0.0000.618 ± 0.0000.760 ± 0.0130.760 ± 0.008
Mean Within-generator AUC0.573 ± 0.0000.573 ± 0.0000.717 ± 0.0020.722 ± 0.002
Held-out-generator Mean AUC0.613 ± 0.0000.610 ± 0.0000.716 ± 0.0020.723 ± 0.005

The improvement over motion-only signals holds in both settings: 0.722 vs. 0.573 within generators (+0.149 ± 0.002) and 0.723 vs. 0.613 on held-out generators (+0.110 ± 0.005). The full ensemble outperforms motion-only for 6 of the 7 generators; the exception is MAGI, which has very few eligible videos (16). This gain comes almost entirely from the VLM judges: the VLM ensemble alone reaches 0.717, and adding motion and DINO signals raises it only to 0.722. Performance is also unchanged on held-out generators (0.716 VLM-only, 0.723 full), so the evaluator does not rely on recognizing which generator produced the video. This shows that the VLMs can tell whether a video follows the hidden property, rather than simply recognizing a good generator from a bad one.

Individual VLMs vary widely, from 0.461 to 0.710 (table below). The ensemble is only slightly better than the best single VLM (Gemma 4-31B, 0.710) but is more stable across runs. We will include these findings in the camera-ready version.

EvaluatorWithin-generator AUC (↑)Held-out-generator AUC (↑)
Original evaluators
Qwen3-VL-32B0.681 ± 0.0070.681 ± 0.007
InternVL3-14B0.680 ± 0.0130.680 ± 0.013
Qwen2.5-VL-32B0.668 ± 0.0080.668 ± 0.008
Gemma3-12B0.659 ± 0.0210.659 ± 0.021
Mistral-Small-3.1-24B0.635 ± 0.0040.635 ± 0.004
LLaVA-OV-7B0.587 ± 0.0070.587 ± 0.007
Qwen3-VL-8B0.583 ± 0.0020.583 ± 0.002
Qwen2.5-VL-7B0.574 ± 0.0080.574 ± 0.008
InternVL3-8B0.525 ± 0.0120.525 ± 0.012
Llama-4-Scout-17B0.461 ± 0.0420.461 ± 0.042
Open-weight SOTA (Newly added evaluators)
Gemma 4-31B0.710 ± 0.0110.710 ± 0.011
Qwen3.8-27B0.697 ± 0.0220.697 ± 0.022
Ensembles using all 12 judges
VLM-only ensemble: 12 judges0.725 ± 0.0030.726 ± 0.004
Full ensemble: VLM + motion + DINO0.741 ± 0.0130.739 ± 0.013

We again thank Reviewer WjsA for the valuable suggestions. We hope these new analyses address your concerns, and we would be grateful if you would consider raising your score in light of these results.

Thanks for your response. I am inclined to accept this paper. But it is not strong enough for a score of 4; I will keep the score.

Reviewer_ExHPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PhysicsLENS, a benchmark for evaluating whether video generation models follow physical properties that are specified in text but not visible in the input image. Its key observation is that a generated video can look physically plausible while still ignoring properties such as weight, friction, elasticity, or viscosity. The benchmark uses matched prompts that keep the initial scene and task unchanged while varying the hidden property. The authors evaluate several video generators with human annotations and also explore an automatic evaluator that combines VLM judgments with motion-based measurements. Results suggest that visible realism and correct physical-property grounding should be evaluated separately.

Critical strengths
  1. The distinction between physical plausibility and property grounding is important and well motivated. Existing benchmarks may accept a realistic-looking video even when it shows the wrong behavior for the specified mass, friction, elasticity, or viscosity. The matched-prompt setup provides a useful way to expose this problem while keeping the initial image and task fixed.

  2. The evaluation separates task completion, visible physics, and hidden-property adherence rather than combining them into one score. This makes several otherwise hidden failure modes easy to see. The paper is also transparent about the limitations of VLM judges and evaluates multiple VLMs, measured video signals, and their combinations against human annotations.

Critical weaknesses
  1. The hidden-property evaluation is still fairly small, especially once the examples are divided among viscosity, friction, elasticity, and mass. Each model also generates only one video for each prompt. This makes it difficult to separate a consistent grounding failure from the normal randomness of video generation.

  2. I found the meaning of a low property-adherence score somewhat unclear. A model may actively produce behavior that contradicts the stated property, or it may simply generate a static or incomplete video in which the property is never tested. These are different failures, but they are counted together in the main analysis.

  3. Each video receives only one final human annotation. Judging whether a liquid is sufficiently viscous or an object behaves sufficiently heavy can be subjective, so some estimate of agreement between annotators would make the conclusions more convincing.

Questions for authors
  1. If the analysis is restricted to videos that complete the task or at least perform the relevant interaction, how many plausible videos still fail to follow the stated property?

  2. Did the authors consider adding an “insufficient evidence” option for videos in which the interaction is missing or too limited to reveal the property?

  3. Since only one video is generated for each prompt and model, do the authors have any evidence that the observable–unobservable difference remains stable across multiple generation seeds?

  4. Could a subset of the videos be rated independently by several annotators? I would especially like to see agreement numbers for hidden-property adherence.

2 comments

We sincerely thank Reviewer ExHP for their vote of confidence and for recognizing the importance of disentangling physical property grounding from visual realism through our matched-prompt setup. We appreciate your constructive suggestions regarding multi-seed stability, task-completion filtering, and inter-annotator agreement. We have now supplemented this rebuttal with additional analyses and experiments to address each of your questions below.

W1 & Q3. Benchmark scale and stability across multiple seeds.

We initially started with 4 open-source models to establish the matched-prompt setup. Following the reviewer's recommendation, we have raised our hidden-property scenarios from 30 to 80 and now evaluate 3 seeds per prompt to account for generation randomness. We also expanded our evaluation to include 3 closed-source models alongside the 4 open-source models from the main paper (7 models in total, yielding 240 unobservable videos per model and 1,680 unobservable-property videos overall). Each category (mass, viscosity, friction, and elasticity) now has at least 15 scenarios, or at least 315 videos per category across all models that get annotated. With this expanded multi-seed evaluation, we find that the gap between observable and unobservable properties still persists. Specifically, unobservable property scenarios decrease physical plausibility by 0.31 ± 0.03 points on average (95% CI [0.25, 0.37]) across all seven models (mean ± standard error over the 80 matched scenarios; two-sided paired t-test, p<10−10p < 10^{-10}). These results are further corroborated by multi-annotator ratings across the entire dataset (detailed in W3 & Q4 below).

W2, Q1, & Q2. Separating static/incomplete videos from active property failures.

The reviewer makes a good point that static or incomplete videos where the property is never tested should be distinguished from active grounding contradictions. Following this suggestion, we introduced a binary “Property Testable” criterion into our annotation protocol to explicitly filter whether the relevant physical interaction occurs and allows the property to be visibly judged. The hidden property is able to be visibly tested in 63% of unobservable videos. Filtering these for realism (physical plausibility score P ≥ 3 on our 1 to 4 scale), 59% still ignore the hidden property. If we also require the task to be completed, the result is close to the same: 57% ignore the hidden property. Results can be seen below.

TypeVideo Generation ModelHidden Property is TestableRealistic (P ≥ 3) & Testable VideosIgnores Hidden Property (among Realistic & Testable)
Proprietary models (new)Gemini Omni Flash85% (204/240)13556% (75/135)
Grok Imagine Video93% (224/240)9758% (56/97)
Grok Imagine Video 1.5 Lite88% (210/240)10965% (71/109)
Open-weight modelsCosmos3-Nano56% (135/240)3171% (22/31)
HunyuanVideo 1.569% (166/240)3549% (17/35)
Wan 2.244% (105/240)956% (5/9)
MAGI 4.5B5% (12/240)10% (0/1)
All63% (1,056/1,680)41759% (246/417)

W3 & Q4. Inter-annotator agreement across multiple raters.

We agree with the reviewer’s suggestion that judging physical properties such as viscosity or weight can be subjective. In light of that, we now have independent ratings from at least 2 annotators per video (41% receiving 3 ratings) across the entire updated benchmark, including the newly added observable and unobservable videos across all 3 seeds. Annotators agree on whether the hidden property was followed 77% of the time (Cohen's κ=0.52\kappa = 0.52), and 78% for physical plausibility (κ=0.56\kappa = 0.56), showing consistent agreement across all four rating criteria:

Label (yes/no)Videos rated by ≥ 2 annotatorsPercent agreement (↑)Cohen's κ (avg over annotator pairs) (↑)
Realistic (P ≥ 3)3,36078%0.56
Task completed3,36080%0.60
Property testable1,68084%0.62
Follows hidden property (H ≥ 3)1,17477%0.52

Furthermore, our main conclusion, that realistic-looking videos often ignore the stated property, holds even under strict annotator consensus. When looking only at the unobservable videos where every annotator agreed on both physical realism and property testability, 57% (190/336) still ignored the stated property.

We thank the reviewer again for the insightful suggestions, which have helped us strengthen our evaluation further. We will include these new findings in the camera-ready version. We hope these additional results clear up all of your questions, and we would be grateful if you would consider increasing your score in light of these new results.

Thank you for the additional experiments. The larger multi-seed evaluation addresses my concern about dataset size and generation randomness. My other concerns are also reflacted and solved in the rebuttal. I decide to keep my original positive rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.