- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces a benchmark that aims to understand whether VLMs answer physical reasoning questions based on their general knowledge of physics or what they observe in videos. It generates 3,382 simulation videos by changing six physical factors, including both normal Earth settings and settings with altered physical parameters. It also uses Cosmos to create paired translated versions of the simulation videos to reduce realism gap between simulated vs real data. The benchmark includes 15,229 questions across five tasks: Law identification, Parameter estimation, Temporal reasoning, Analytic counterfactual comparison, and Law induction. Models are evaluated under five input settings: text only, ordered and shuffled simulation frames, and ordered and shuffled Cosmos-translated frames. By comparing these settings while keeping the questions and answers fixed, the paper studies how VLMs use visual information for physical reasoning.
Critical strengths
- The number of simulation videos and questions is sufficiently large to support a broad evaluation.
- The different tasks and input settings make sense for studying different parts of physical understanding. Keeping the questions fixed across input settings is useful for measuring how visual information and frame order affect model performance.
- The number and variety of evaluated models are sufficient to support general observations about the tested models. The findings in Section 4 are informative, especially the differences between normal and altered scenes and the distinction between consistent answers and correct answers.
Critical weaknesses
- Although the paper mentions simulation checks, frame sampling checks, and human review of question usability, it does not report a clear, separate evaluation of whether Cosmos preserves the motion and physical information needed to answer the questions. The translated videos use the same reference answers as the simulation videos. If Cosmos changes the motion or removes important details, these answers may no longer be supported by the translated videos. Therefore, the score differences may partly reflect changes introduced by Cosmos.
- The evaluation is limited to synthetic scenes, including the Cosmos-translated videos. It is unclear whether the same findings would hold for real-world videos, where object appearance, motion, and interactions can be more varied.
- My main concern is whether the provided visual information is sufficient to determine a unique answer for some questions, especially those asking which physical factor differs from normal Earth physics. Friction depends on the interacting surfaces, and bounce behavior depends on the properties of both the object and the surface. There is no single normal friction or restitution value for all real-world scenes. A difference from the benchmark's reference value may therefore reflect an ordinary material difference rather than a violation of physical laws.
- The simulator provides a fixed reference answer, but it is not clear whether the model can always infer this answer from the provided frames. This concern may also affect some parameter estimation questions, although directly estimating an observable quantity, such as a bounce-height ratio, does not necessarily require material information. The paper should explain how it handles cases with multiple possible physical explanations and whether additional scene information is needed for the affected questions.
Questions for authors
- Was a separate quality check performed to verify that the Cosmos-translated videos preserve the motion and the validity of the original answers? If so, could you provide the procedure and results?
- Could different material or surface properties lead to multiple valid physical explanations for the same visual observations? How do you ensure that the reference answer is uniquely supported by the information given to the model, particularly for law identification and parameter estimation?
- Could you discuss the similarities and differences between your benchmark and LikePhys [1]? LikePhys also uses physics-based simulations to construct paired videos with physically valid and invalid behavior, including temporal reordering to create invalid sequences. These choices seem related to your use of normal and altered physics and your comparison of ordered and shuffled frames . [1] Yuan et al. LikePhys: Evaluating Intuitive Physics Understanding in Video Diffusion Models via Likelihood Preference. ICLR 2026. https://arxiv.org/abs/2510.11512