acceptodds
Withdrawn from ICLR 2027

InfiniteWorld: Separating Visual Evidence from Familiar Physics

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Physical reasoning requires adapting familiar expectations to the dynamics of the observed world. Existing benchmarks establish broad physical capabilities, yet an overall score leaves unclear whether answers follow visual evidence when familiar rules no longer apply. We introduce InfiniteWorld, a benchmark that pairs physical-reasoning questions with simulator videos whose physical parameters are systematically altered. Its generation pipeline connects simulation, scene screening, paired video translation, question construction, and input checks. Five tasks assess law identification, parameter estimation, temporal reasoning, analytic counterfactual comparison, and law induction. The evaluation defines five matched input settings for each question, comprising text-only evaluation and ordered or shuffled observations from simulator and translated videos. We evaluate a diverse set of video-language models under these matched settings. The results expose a persistent preference for familiar physical rules. Frames improve performance for most models, but gains differ across tasks. Strong accuracy on Earth-normal questions coexists with poor recognition of altered dynamics, and Earth-normal answers remain common when the observed physics changes. Task scores and semantic agreement further distinguish matching physical factors from matching their amount, and correct answers from repeated errors. InfiniteWorld makes these behaviors measurable through paired changes in the evidence supplied for the same question, separating visual evidence from familiar physics.

Primary area:
datasets and benchmarks
Keywords:
video understanding, physical reasoning, multimodal evaluation, world models, counterfactual reasoning, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DHMVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a benchmark that aims to understand whether VLMs answer physical reasoning questions based on their general knowledge of physics or what they observe in videos. It generates 3,382 simulation videos by changing six physical factors, including both normal Earth settings and settings with altered physical parameters. It also uses Cosmos to create paired translated versions of the simulation videos to reduce realism gap between simulated vs real data. The benchmark includes 15,229 questions across five tasks: Law identification, Parameter estimation, Temporal reasoning, Analytic counterfactual comparison, and Law induction. Models are evaluated under five input settings: text only, ordered and shuffled simulation frames, and ordered and shuffled Cosmos-translated frames. By comparing these settings while keeping the questions and answers fixed, the paper studies how VLMs use visual information for physical reasoning.

Critical strengths
  1. The number of simulation videos and questions is sufficiently large to support a broad evaluation.
  2. The different tasks and input settings make sense for studying different parts of physical understanding. Keeping the questions fixed across input settings is useful for measuring how visual information and frame order affect model performance.
  3. The number and variety of evaluated models are sufficient to support general observations about the tested models. The findings in Section 4 are informative, especially the differences between normal and altered scenes and the distinction between consistent answers and correct answers.
Critical weaknesses
  1. Although the paper mentions simulation checks, frame sampling checks, and human review of question usability, it does not report a clear, separate evaluation of whether Cosmos preserves the motion and physical information needed to answer the questions. The translated videos use the same reference answers as the simulation videos. If Cosmos changes the motion or removes important details, these answers may no longer be supported by the translated videos. Therefore, the score differences may partly reflect changes introduced by Cosmos.
  2. The evaluation is limited to synthetic scenes, including the Cosmos-translated videos. It is unclear whether the same findings would hold for real-world videos, where object appearance, motion, and interactions can be more varied.
  3. My main concern is whether the provided visual information is sufficient to determine a unique answer for some questions, especially those asking which physical factor differs from normal Earth physics. Friction depends on the interacting surfaces, and bounce behavior depends on the properties of both the object and the surface. There is no single normal friction or restitution value for all real-world scenes. A difference from the benchmark's reference value may therefore reflect an ordinary material difference rather than a violation of physical laws.
  4. The simulator provides a fixed reference answer, but it is not clear whether the model can always infer this answer from the provided frames. This concern may also affect some parameter estimation questions, although directly estimating an observable quantity, such as a bounce-height ratio, does not necessarily require material information. The paper should explain how it handles cases with multiple possible physical explanations and whether additional scene information is needed for the affected questions.
Questions for authors
  1. Was a separate quality check performed to verify that the Cosmos-translated videos preserve the motion and the validity of the original answers? If so, could you provide the procedure and results?
  2. Could different material or surface properties lead to multiple valid physical explanations for the same visual observations? How do you ensure that the reference answer is uniquely supported by the information given to the model, particularly for law identification and parameter estimation?
  3. Could you discuss the similarities and differences between your benchmark and LikePhys [1]? LikePhys also uses physics-based simulations to construct paired videos with physically valid and invalid behavior, including temporal reordering to create invalid sequences. These choices seem related to your use of normal and altered physics and your comparison of ordered and shuffled frames . [1] Yuan et al. LikePhys: Evaluating Intuitive Physics Understanding in Video Diffusion Models via Likelihood Preference. ICLR 2026. https://arxiv.org/abs/2510.11512
Reviewer_oBJROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces InfiniteWorld, a benchmark that tests whether video-language models follow visual evidence when physical conditions differ from familiar expectations. It contains 15,229 questions across five tasks, built from 3,382 simulated videos. Each question is tested under five input settings that vary whether frames are provided, their order, and their visual appearance. The results show that videos help most models, but the gains vary across tasks, and some models still favor normal-physics answers when the observed physics changes.

Critical strengths
  1. The matched input settings help measure the effects of visual input, frame order, and rendering on the same questions.
  2. The benchmark covers several physical factors and tests both what changes in the physics and how large the change is.
  3. The analysis reveals problems that overall scores can hide, such as repeated wrong answers and difficulty telling different levels of physical change apart.
Critical weaknesses
  1. The frames lack timestamps, even though some questions ask about specific times, so errors may partly reflect missing information.
  2. Simple rules chosen after checking the labels achieve high scores on some tasks without using the videos, suggesting strong answer patterns.
  3. The translated videos may change important motion details, but the original reference answers are still used.
  4. Scene layouts are linked to physical factors, so models may use layout cues without fully understanding the motion.
Questions for authors
  1. How should models match the times in the questions to frames without timestamps or video duration? Can humans reliably answer these questions using the same inputs?
  2. How is “the same amount of change” defined in the law induction task when the actual parameter values can vary across scene templates within the same category?
  3. Have you checked whether the counterfactual answers from simplified formulas agree with full simulations of the same scenes under normal physics, especially when collisions occur?
Reviewer_1GVMOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper focuses on investigating whether current VLMs can overcome familiar physics biases when they are contradicted by actual physical observations. The authors propose different evaluation settings, including text-only vs. text + video, as well as different frame orders and renderings, to examine how VLMs rely on visual evidence and familiar physical priors. Experiments on the proposed benchmark reveal that VLMs still exhibit strong biases toward familiar physics despite contradictory visual observations.

Critical strengths
  1. The paper addresses an important problem in physical reasoning: whether VLMs can rely on visual observations when they conflict with familiar physical priors.

  2. The proposed benchmark covers multiple physical factors and reasoning tasks. The matched-input evaluation settings provide a systematic way to investigate the effects of visual evidence, temporal order, and rendering.

  3. The authors evaluate many VLMs and provide detailed analyses of their performance across different tasks and evaluation settings, revealing persistent biases toward familiar physics.

Critical weaknesses
  1. The key motivation of this paper substantially overlaps with InPhyRe (2026), which has already investigated the reliance of VLMs on familiar physical priors when reasoning about unfamiliar physical dynamics. While InfiniteWorld extends the evaluation to more physical factors and introduces matched-input settings to examine the effects of visual evidence, these contributions appear largely incremental. The main empirical finding, that models continue to favor familiar physical expectations despite contradictory visual observations, is also consistent with prior findings. Furthermore, the paper does not introduce a method to address this limitation or provide substantially new insights into its underlying causes.

  2. The key contribution of this paper is to separate the effects of visual evidence from familiar physical priors. However, the proposed evaluation primarily measures the accuracy difference between text-only and video-input settings. I am concerned that such performance differences do not necessarily reveal the extent to which a model relies on visual evidence versus its learned physical priors. For example, improved accuracy with video input does not necessarily indicate that the model can override its physical priors when they conflict with visual observations. A more direct evaluation could use paired videos with identical initial conditions but different physical dynamics, examining whether the model changes its predictions accordingly.

Questions for authors

Please see the two major weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.