acceptodds
Withdrawn from ICLR 2027

CognPhys: A Benchmark for Inductive Physical Reasoning in Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Existing benchmarks for physical reasoning in Vision-Language Models (VLMs) primarily assess rule following—predicting outcomes governed by known Newtonian laws—yet a hallmark of human physical cognition is rule discovery: observing unfamiliar phenomena and inducing the underlying principles from scratch. This capability is essential for embodied agents adapting to novel environments, robot control, and automated scientific discovery. We introduce CognPhys, a diagnostic benchmark comprising multiple physics worlds, each governed by distinct rules that systematically deviates from real-world physics. We evaluate a broad suite of open-source and closed-source VLMs using a structured scoring protocol. Our extensive evaluations reveal a profound deficiency in state-of-the-art VLMs: they are severely bottlenecked at rule induction. Even the most capable models consistently fail to synthesize coherent rules from observed motion, constrained by an over-reliance on memorized physical priors. CognPhys uncovers a fundamental blind spot in current VLM physical reasoning and establishes a challenging testbed for scientific and embodied AI.

Primary area:
datasets and benchmarks
Keywords:
physical reasoning, benchmark, rule discovery

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_SEKBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

CognPhys is a benchmark for testing visual physical-rule induction in procedurally generated, non-Newtonian worlds. It contains 284 MuJoCo worlds: Track A has 150 single-law worlds for parameter estimation, and Track B has 134 worlds composed from a closed vocabulary of eleven laws for rule identification. A third task evaluates future trajectory prediction. The design uses parameter-twin worlds, a same-pixel classical estimator as an identifiability check/reference, and a second camera view for Track A. Across 18 closed-source API models and five local open-weight models, the best exact-set rule-determination score is 20.1% (versus 1.6% for random subsets), the best parameter accuracy within 10% is 15.3% (versus 90.7% for a classical estimator), and the best future-prediction hit rate within 0.15 ball radii is 33.3% (versus 87% for replay with the true rules). The results provide a careful diagnosis within a controlled synthetic benchmark, with more limited support for open-ended or real-world physical law discovery.

Critical strengths
  • CognPhys combines video input, multiple counterfactual physics worlds, and explicit rule/parameter tasks in a coherent benchmark. Track A and Track B separate quantitative parameter estimation from closed-vocabulary rule identification.
  • The identifiability gate is a useful design choice: worlds are retained only when a classical method can recover the parameter from the same pixels. Twin worlds that differ in one parameter and the second camera view further help diagnose whether a model is responding to visual evidence.
Critical weaknesses
  • The related-work coverage is incomplete at the point most relevant to novelty. The manuscript surveys fixed-physics visual benchmarks, numerical law-discovery tasks, and general physical-reasoning benchmarks, but does not cite or compare InPhyRe, a 2025 preprint accepted to TMLR in 2026 that tests inductive reasoning under unseen physics using synthetic collision videos; QuantiPhy, a CVPR 2026 video benchmark for quantitative size, velocity, and acceleration inference; or PhysBench, which evaluates physical-world understanding interleaved video-image-text entries in four domains, including physics-based dynamics. These are not identical tasks: InPhyRe focuses on predicting collision outcomes after demonstrations, QuantiPhy uses known physical relations and priors, and PhysBench covers broader physical-world understanding rather than CognPhys’s controlled non-Newtonian rule and parameter induction. They are nevertheless close enough that Section 2 and Table 1 should explain the distinctions. Their omission leaves the “first benchmark to unify” positioning and the claim that video benchmarks do not test induction insufficiently substantiated.
  • The headline reference comparison is easy to misread and hides stronger simple baselines. Table 3 places 87% future-prediction accuracy under “same-pixel ceiling,” but this value comes from replaying the simulator with the ground-truth rules and parameters; it is an oracle, not a pixel-based predictor available to a model. Appendix Table 6 reports a 43% no-video last-velocity predictor, which exceeds the best VLM’s 33.3%, and a 24% no-video family-median parameter predictor, which exceeds the best VLM’s 15.3%. These baselines should appear in the main table and be clearly separated from oracle ceilings. For Track B, the 1.6% uniformly random rule-subset baseline is weak; a majority-set or per-rule base-rate predictor is needed before “well above chance” can be interpreted as evidence of useful rule induction.
  • Model coverage and reported denominators are not comparable enough for the displayed rankings. Figure 7 reports 150 Track A worlds for closed-source models but only 40 for open-weight models, omitting two rule families; the main table does not give per-model denominators or a same-world comparison. Future prediction is based on only 30 worlds. In addition, Appendix A.4 reports 2,818 model-world observations for Track A’s log-error decomposition, whereas the stated coverage implies 18×150 + 5×40 = 2,900 possible observations. Since unparsable outputs are counted incorrect in the main evaluation, the paper should state whether and why 82 observations are absent from this analysis. Figure 6 reports binomial intervals without making the independent sampling unit clear; Track A twin worlds share seeds and initial conditions, and each world is evaluated by multiple models. If these observations are treated as independent, the uncertainty is understated. Please report paired/cluster-aware intervals and API decoding or repeat-call variability.
  • The visual evidence and the interpretation of failure need stronger validation. Clips are 12–20 seconds in Track A and 14 seconds in Track B, while models receive frames described as 2 fps with a maximum of 16. Section 3.2 says frames are evenly spaced, but the paper does not make clear how that rule interacts with the cap for each clip length or show sensitivity to the resulting temporal sampling. More importantly, the Track A identifiability gate validates parameter recovery by a hand-written estimator, not all visual signatures in Track B. The 97.5% forced choice between counterfactual twins is a different, easier discrimination task and does not establish that perception is solved for all eleven composed rules. The conclusion that the bottleneck is induction rather than perception is therefore stronger than the diagnostic evidence supports.
  • Implementation provenance and reproducibility are not documented at sufficient detail. The paper describes a procedural rule engine, MuJoCo simulation, rule templates, and state logging, but does not state how the simulator programs were authored (handwritten versus code-generation or AI coding assistance), how each rule is implemented in the engine, or what independent unit/regression tests validate the event conditions and ground-truth labels. These implementation choices define the physical phenomena being benchmarked, so authorship workflow, implementation details, and validation artifacts are important for assessing validity and reproducibility.
Questions for authors
  1. Can the main results table include the no-video family-median and last-velocity baselines, while labeling ground-truth simulator replay explicitly as an oracle ceiling? For Track B, can you add majority-set and per-rule prevalence baselines?
  2. Please report per-task and per-model sample counts, the exact 40-world open-weight subset and omitted families, and a same-world comparison against the closed-source models. How do the 2,818 observations in Appendix A.4 relate to the 2,900 expected Track A model-world evaluations?
  3. How are the 2 fps and 16-frame cap applied to 12–20 second clips: are frames selected across the full clip or truncated? Can you provide a sensitivity analysis showing that the evidence-bearing events remain visible under this sampling policy?
  4. Can the uncertainty analysis account for paired twin worlds and shared worlds across models, and report API decoding/repeat-call stability? What additional visual-evidence diagnostic supports the claim that rule induction, rather than perception, is the main bottleneck on Track B?
  5. How were the world-generator and simulator programs authored: handwritten by the authors, generated in part with coding assistants/LLMs (including vibe-coding-style workflows), or by another workflow? Please disclose any code-generation assistance, explain how each rule maps to MuJoCo (including timestep, force, and event handling), and provide implementation/configuration details plus unit or validation tests that reproduce the 284 worlds and their ground-truth labels.
Reviewer_qNTEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces CognPhys, a benchmark designed to evaluate whether VLMs can infer physical rules from visual observations rather than simply apply familiar physical knowledge. The benchmark contains 284 procedurally generated physics worlds, organized into two tracks: one focuses on estimating physical parameters, while the other requires identifying multiple governing rules. The authors evaluate 23 open-source and closed-source VLMs on three tasks: rule determination, parameter estimation, and future trajectory prediction. The results show that current VLMs struggle across all three tasks, particularly when precise numerical estimates are required. The authors argue that these failures reflect a limitation in the models' ability to infer physical rules from observations, rather than a lack of basic visual perception.

Critical strengths

I think the distinction between applying known physical rules and inferring rules from observations is important. Most existing physical reasoning benchmarks focus on understanding or predicting events under familiar physical assumptions. CognPhys attempts to evaluate a different capability, which is relevant to scientific reasoning and embodied intelligence. Combining visual observations with multiple physical environments is a useful contribution. Also, the procedural generation framework allows the authors to systematically manipulate physical parameters while controlling initial conditions. I particularly appreciate the use of paired worlds and the identifiability check, which helps establish that the parameters can be recovered from visual observations. The evaluation also includes several useful analyses, such as comparisons across physical rule families, camera viewpoints, and prediction tolerances. These make the benchmark more informative than a single aggregate accuracy score.

Critical weaknesses
  • My main concern is whether the proposed tasks actually measure the ability to discover unfamiliar physical laws. In Track A, the physical law is largely specified by the question, and the model primarily needs to estimate an unknown numerical parameter. In Track B, the model selects governing laws from a predefined vocabulary of eleven rules, with their definitions provided in the prompt. These are meaningful tasks, but they are closer to parameter estimation and multi-label rule identification than open-ended physical law discovery. Moreover, many of the underlying laws, such as constant acceleration, velocity-dependent drag, and friction, are familiar physical mechanisms with different parameters or configurations. I therefore find the paper's distinction between Newtonian rule following and novel rule induction somewhat overstated.

  • The authors conclude that the primary limitation is rule induction rather than visual perception. However, I do not think the current evidence fully supports this interpretation. For example, the high accuracy on paired forced-choice comparisons does not necessarily establish that models can accurately extract trajectories, velocities, or accelerations from individual videos. These tasks have different perceptual and numerical requirements. The additional coordinate analysis in Appendix A.5 is helpful, but it is limited to one model and does not clearly separate perception errors from subsequent prediction errors across the benchmark. I would like to see a more controlled evaluation where models receive ground-truth trajectories or object coordinates instead of images.

  • The paper reports very low accuracy, but the selected thresholds are demanding, especially the 10% relative error criterion for parameter estimation and the 0.15-ball-radius tolerance for future prediction. More importantly, Appendix A.2 shows that a simple no-video baseline achieves 24.0% on parameter estimation and 43% on future prediction, exceeding the best VLM's 15.3% and 33.3%, respectively. This is an important finding that deserves more discussion in the main paper.

    The classical references also require careful interpretation. In particular, the future prediction reference uses the world's actual physical rules and ground-truth parameters, making it closer to an oracle reference than a directly comparable prediction method. Additionally, future prediction is evaluated on only 30 worlds, making the reported percentages relatively sensitive to individual examples. These issues do not invalidate the benchmark, but they weaken the interpretation that low scores specifically demonstrate a failure of physical law induction.

Questions for authors
  • What exactly constitutes novel physical law discovery in CognPhys? Since Track A primarily evaluates parameter estimation and Track B provides a fixed vocabulary of candidate laws, could the authors clarify how these tasks distinguish inductive rule discovery from conventional parameter estimation or rule classification?
  • Can the authors separate perceptual errors from physical reasoning errors more directly? For example, how would VLM performance change if models received ground-truth object positions or trajectories instead of sampled video frames?
  • How should the classical and no-video references be interpreted? In particular, does the classical parameter estimator operate on exactly the same sampled frames and resolution provided to VLMs? For future prediction, could the authors distinguish an oracle simulation using ground-truth rules from a baseline that must infer the dynamics from observations?
Reviewer_B5tjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposed Cognphys, a benchmark to test VLM’s capability of physics rule discovery. The benchmark is built within Mujoco and generated with 150 Track A worlds for parameter estimation, and 134 Track B worlds for identifying rule combinations. Both closed sourced VLMs and open-source VLMs are tested and it has shown that Opus is strongest with 20.1% exact rule-set accuracy, 15.3% parameter-estimation accuracy, and 33.3% future-position accuracy. While the idea is interesting, I think the concern of under-specification of the physics environment remains.

Critical strengths
  1. Recovering physical rules from observations is an important evaluation target for VLMs. The counterfactual pairs, which share initial conditions while changing a governing parameter, provide a useful control for testing whether models respond to observed dynamics. Separating rule identification, parameter estimation, and prediction also provides a more informative assessment than a single aggregate score.

  2. The evaluation covers a broad set of 23 models, including both closed-source and open-source VLMs. The classical estimator, no-video baseline, tolerance sweeps, and family-level analyses provide useful reference points for interpreting their failures.

Critical weaknesses
  1. The comparison with the classical estimator needs clearer documentation before the performance gap can be attributed primarily to rule induction. The paper explicitly states that the estimator operates on the same pixels and reports 90.7% parameter accuracy; this is meaningful evidence that the retained tasks are solvable under its setup. However, I am not sure if "the same pixels" mean that the exact frame selection, temporal sampling, image resolution, timestamps, metric calibration, and prior knowledge of the law family are given VLMs too. The evaluation also does not establish whether a VLM allowed to generate and execute tracking and fitting code could reproduce the estimator. Such a comparison would help distinguish limitations in selecting an inference procedure from limitations of the direct-answer protocol. The future-prediction reference additionally uses ground-truth rules and parameters, so it should be distinguished from a method that infers dynamics from the clip.

  2. The scene distribution is narrow relative to the broader motivation of physical cognition and embodied adaptation. The benchmark indeed includes acceleration, drag, friction, localized forces, fields, and collisions, but these laws are mainly in simple ball-based scenes. I think the conclusions should be restricted to this controlled setting; and the claim of "general rule extraction" would require more diverse objects and interactions.

Questions for authors
  1. Beyond the statement that the classical estimator uses the same pixels, can you provide an exact input comparison? In particular, does it receive the same selected frames, temporal sampling, resolution, timestamps, and metric calibration as the VLMs? What prior knowledge of the governing law family does each method receive?

  2. Are VLMs allowed to generate and execute programs for tracking objects, fitting parameters, and simulating motion? If the reported evaluation only permits direct answers, would a code-enabled condition using the same observations and metadata change the conclusions about rule-induction ability?

  3. How should the scope of the findings be interpreted given the concentration on controlled ball-based scenes? What extensions in object geometry or contact and interaction types would be needed to assess broader physical rule discovery?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.