- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces PhysicsLENS, a benchmark that tests whether image-to-video models follow physical properties that are stated in the prompt but cannot be inferred from the initial frame, such as viscosity, surface condition, elasticity, and mass in robot manipulation scenes. It builds 80 observable prompts from public robot datasets and 30 matched unobservable variants that keep the frame and task fixed while changing only the stated property, generates 439 videos with four open-source models, and collects human ratings of physical plausibility, task completion, and property adherence. The authors find that most videos rated plausible under the unobservable condition do not follow the stated property, and propose an automated evaluator that combines motion signals with targeted VLM questions.
Critical strengths
-
The matched-pair design is a useful idea: keeping the frame and task fixed and changing only the stated property separates physical plausibility from property grounding, which existing physics benchmarks do not distinguish. This is the main reason I see value in the paper.
-
The observation that plausible-looking videos often ignore the stated property is relevant to using video models as data sources for robot learning.
Critical weaknesses
-
The model coverage is too narrow. Only four open-source models are evaluated, without commercial models such as Veo, Kling, or Seedance, or robot-specific world models such as DreamGen or DreamDojo, so it is unclear whether property blindness generalizes beyond the tested models.
-
The annotation and generation are thin. Each clip has a single annotation without any agreement measure, and each prompt is generated only once, so annotation and generation noise cannot be separated from the measured effects.
-
The central claim is not statistically supported. The plausibility gap between observable and unobservable conditions, presented as consistent in the abstract, is not significant for any model or in the pooled test, partly because there are only 30 unobservable scenarios.
Questions for authors
-
Clarification: Could the authors report inter-annotator agreement on a subset rated by several annotators, and results over multiple seeds per prompt, to show that the observable/unobservable difference exceeds annotation and generation noise?
-
Clarification: A low adherence score cannot distinguish a contradicted property from a missing interaction, and many plausible but ungrounded videos come from a generator that barely moves. How many of these videos actually show the interaction that would reveal the property?
If the authors adequately address my concerns, I will consider raising my rating.
3 comments
Q2. Separating missing interaction from contradicted properties
We recognize that a low adherence score may not fully distinguish a contradicted property from a missing interaction. Following this suggestion, we introduced a binary "Property Testable" label into our annotation protocol: before scoring adherence, annotators judge whether the video shows enough of the relevant interaction to test the stated property. Across all unobservable videos, 63% (1,056 of 1,680) are testable by all annotators.
After removing untestable videos, 59% (246 of 417) of realistic videos (physical plausibility score P ≥ 3 on our 1 to 4 scale) still ignore the stated property, and 57% (210 of 367) when the task must also be completed. Our main conclusion, that realistic videos often ignore the stated property, also holds under strict annotator consensus: among the 336 unobservable videos where every annotator agreed on both realism and testability, 57% still ignored the property.
| Type | Video generation model | Hidden property is testable | Realistic (P ≥ 3) & testable videos | Ignores hidden property (among realistic & testable) |
|---|---|---|---|---|
| Proprietary models (new) | Gemini Omni Flash | 85% (204/240) | 135 | 56% (75/135) |
| Grok Imagine Video | 93% (224/240) | 97 | 58% (56/97) | |
| Grok Imagine Video 1.5 Lite | 88% (210/240) | 109 | 65% (71/109) | |
| Open-weight models | Cosmos3-Nano | 56% (135/240) | 31 | 71% (22/31) |
| HunyuanVideo 1.5 | 69% (166/240) | 35 | 49% (17/35) | |
| Wan 2.2 | 44% (105/240) | 9 | 56% (5/9) | |
| MAGI 4.5B | 5% (12/240) | 1 | 0% (0/1) | |
| All | 63% (1,056/1,680) | 417 | 59% (246/417) |
We would like to thank the reviewer again for their thoughts and constructive suggestions, which have helped us improve our evaluation. These new findings will be incorporated into the camera-ready version. We hope the additional results address your concerns, and we would be grateful if you would consider increasing your score given these new results.
All of my concerns have been addressed, so I’ve raised my rating to Weak Accept.
We thank the reviewer for recognizing the value of our matched-pair setup in separating physical plausibility from property grounding and its relevance to robot learning. We also appreciate your constructive suggestions regarding model coverage, sample size, and annotation reliability. We have now addressed all of these concerns by adding 3 commercial models, expanding from 30 to 80 unobservable scenarios with 3 seeds per prompt, and collecting multi-annotator ratings across the entire benchmark. Please see our detailed responses and updated statistical results below.
W1. Does property blindness generalize beyond the four open-source models tested?
In our study, we started with 4 open-source models to establish the matched-prompt setup and ensure clarity. Following the reviewer's suggestion, we have conducted additional experiments with 3 of the latest closed-source commercial models (Gemini Omni Flash, Grok Imagine Video, and Grok Imagine Video 1.5 Lite) alongside our original 4 models, giving 7 models in total (240 unobservable videos per model, or 1,680 in total).
We observe that property blindness persists across these commercial models as well. Among the generated videos, the 3 closed-source models ignore the stated property in 56-65% of cases, which is comparable to the open-source models (49-71%, excluding MAGI, which has only 1 realistic and testable video). Even Gemini Omni Flash, which produces the most realistic videos, ignores the property in 56% of such cases, indicating that higher visual quality alone does not resolve this issue. Please refer to the table in Q2 for the per-model breakdown. We also plan to release our benchmark and evaluation code publicly so that any future commercial or robot-specific world models can be tested directly.
W2, W3 & Q1. Is the observable–unobservable plausibility gap significant and larger than annotation and generation noise?
We agree that a single annotation and generation per prompt can make it difficult to discern noise from actual physical property grounding. To mitigate this, we expanded hidden-property scenarios from 30 to 80 and now generate 3 seeds per prompt, totaling 3,360 videos. Each of the four main categories (mass, surface condition, viscosity, elasticity) now has at least 15 scenarios, or at least 315 annotated videos. Every video now has independent ratings from at least 2 annotators (41% receive 3), giving 8,095 annotations in total. Agreement is moderate across all four labels:
With this expanded evaluation across 80 matched scenarios, the performance gap between observable and unobservable properties is statistically significant. Videos are rated 0.31 ± 0.03 points less plausible (mean ± SE; 95% CI [0.25, 0.37]; two-sided paired t-test, p<10−10) when the hidden property is specified, with 88% of scenarios exhibiting a drop (scenario SD = 0.28).
This drop is consistent across generation seeds and holds across each annotator's ratings evaluated independently (mean drops between 0.16 and 0.37 points, all paired t-test p<0.05), demonstrating that the observed effect exceeds both generation and annotation noise. The plausibility penalty remains robust in a linear mixed-effects model after controlling for generator, annotator, and seed simultaneously (β=−0.29,p<10−32), and is consistent across all three seeds. Plausibility scores drop under unobservable properties for all seven generators, reaching individual statistical significance (p<0.05) in five of them.
We agree with the reviewer that specifying this drop as "consistent" in the abstract was too strong for the original 30 scenarios, and we will revise the abstract to reflect these new findings.