- Rating
- 4(+1)
- Confidence
- 3(+1)
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper introduces NewtPhys, a benchmark curated by augmenting real-world 3DGS scenes with objects and simulating their interactions using a simulator. It provide simulated sequences with high-level event annotations and dense pixel-aligned labels for physical quantities, kinematics, semantics, and geometry. The dataset also supports VQA across six categories of physical understanding.
The authors evaluate VLMs on the VQA benchmark and VFMs using lightweight decoders trained to predict pixel-level quantities such as collisions, gravity, and scene flow. Further analyses examine language-prior reliance, counterfactual reasoning, the effects of spatial and physical cues, and ood gravity prediction. The pipeline is carefully designed to obtain physical annotations that are difficult to collect directly, and it incorporates object properties estimated from visual cues and metadata. The suppl also reports checks of these estimates against vendor information for a subset of objects.
Critical strengths
A useful benchmark with rich annotations. NewtPhys combines realistic 3DGS scenes with controlled simulation and provides both VQA data and dense pixel-level labels. This enables evaluation of high-level physical reasoning as well as probing whether representations contain localized physical signals, and this quite different from existing VLM physics benchmarks.
A thoughtfully designed data-generation pipeline. Recovering physical quantities and dense labels from real-world interactions is challenging. The use of simulation makes these annotations available at scale, while accounting for object properties such as mass and material parameters. The supplementary checks of estimated object properties provide some support for the quality of these inputs.
Broad evaluation and useful diagnostic analyses. The paper evaluates multiple VLMs and VFMs and studies language priors, counterfactual reasoning, spatial and physical cues, and OOD gravity prediction. These analyses help characterize where current models succeed or fail beyond aggregate benchmark scores and provide insights on further building more physically capable VLM/VFMs.
Critical weaknesses
The physical validity of the generated dynamics and labels remains difficult to assess. The paper reports visual-fidelity and physical-plausibility metrics, checks 3DGS reconstruction dimensions for a subset of objects, and evaluates estimated object properties against vendor information. However, it does not compare simulated dynamics or pixel-level physical labels against measurements from real-world interactions. As a result, it still remains unclear how simulator assumptions and estimated physical properties affect the generated dynamics and labels. Even a small subset of this experiments would be fine.
The effect of language priors on VQA scores could be analyzed more deeply. The authors deserve credit for explicitly testing whether models rely on language priors and question construction, including ROI-masking and counterfactual evaluations. However, roughly one third of the tested models improve when the queried visual region is masked. This is a notable result, but the paper does not establish what drives the increase: for example, whether masking removes distracting visual evidence, changes the model's answer strategy, or exposes biases in the question and answer choices.
Questions for authors
-
The appendix shows that basic property labels align with the simulator state, but this does not establish physical accuracy. Could the authors validate the pipeline on videos with known states and compare selected dynamics with an independent simulator or controlled real-world recordings, while quantifying the effect of errors in estimated ones?
-
Roughly one third of the tested models improve after the queried region is masked. Could the authors investigate whether this comes from distracting visual content, a changed answer strategy, or biases in the questions and answer choices?
-
The OOD test varies gravity magnitude, but how well do the pixel-level probes transfer to held-out scenes, object instances, and material combinations? Cross-object or cross-scene splits could test whether the representations capture physical information rather than superficial appearance or simulator cues.
-
Could you discuss how future work can establish human reference performance and test whether models trained with NewtPhys transfer to real interactions? It would also be useful to clarify how NewtPhys complements Beyond Static Vision: Scene Dynamic Field Unlocks Intuitive Physics Understanding in Multi-modal Large Language Models and PhysicsMind: Sim and Real Mechanics Benchmarking for Physical Reasoning and Prediction in Foundational VLMs and World Models.
7 comments
Rebuttal (2)
Investigation of effect of language priors on VQA
We first detail our masked-ROI experiment (Sec. 4.2, Fig. 7). The latter is conducted on a 2K subset of object-related VQAs and reports before (baseline) and after masking the queried object(s) with a black-filled minimum enclosing circle, concealing their contours. Questions, background, and other objects remain perfectly unchanged as we aim to isolate the effect of removing visual evidence about the queried objects.
Magnitude of improvements. While we mention a third of the 31 VLMs models improved, there are confined to only 5 of 19 open-VLM families studied, with most positive variations being marginal. In fact, only four models (InternVL2-1B/40B/76B and Mantis-8B) improve by more than relative accuracy. We hypothesize that this bias may partly stem from training objectives, consistent with our VFM findings (Tab. 2), where self-supervised models generalize better to physics understanding.
Question-level bias analysis. To investigate whether these changes arise from specific questions among our 2K set, we measure the overlap of questions whose answers change after masking across models. Tab. B (below) reports variations among these sets using three Jaccard metrics per model group: average pairwise models agreement, majority agreement, and general agreement. Agreement is generally low: average pairwise Jaccard is 0.089 across all models and 0.101 for improving models. The generalized Jaccard is close to 0, showing that not all models changed their answers on the same set of questions. Relaxing the constraint, we look at majority agreement across improved models which is 0.04205 (ie, 46 out of the 1094 changed questions, from our 2K set). Furthermore, only two of these questions changed from wrong to right on at least three of the improved models. This shows that shared answer changes were uncommon, and shared gains were rarer still.
Tab B Agreement among models which changed answers
| Model group | Models | Union size | Pairwise Jaccard | Majority Jaccard | Generalized Jaccard |
|---|---|---|---|---|---|
| All models | 31 | 1,843 | 0.08852 | 0.00054 | 0.00000 |
| Improved () | 4 | 1,094 | 0.10052 | 0.04205 | 0.00091 |
| Degraded () | 14 | 1,669 | 0.10591 | 0.00779 | 0.00000 |
Our results are consistent with prior work suggesting that VLMs underuse visual cues and rely on language priors (Chen et al., NeurIPS 2024). Our contribution is to investigate this behavior specifically in physical reasoning tasks, a regime where LLMs appear to struggle to compensate visual deficiencies. We will clarify that our experiment measures reliance on queried-object visual evidence and provides evidence consistent with language-prior effects, rather than establishing their causal origin. We will detail the protocol and additional analysis in the supplementary.
Rebuttal (3)
OOD variations
We appreciate the suggested experiments, which we are currently running. We will report results as soon as they are ready.
Discussion on future works (human evaluation, transferability)
Thanks for opening the discussion on future works. NewtPhys opens questions about how models infer physical properties from visual evidence, how these inferences compare with human perception, and how they might support real-world interaction.
1. Human evaluation.
NewtPhys reveals gaps in vision-language models’ physical understanding, leaving open how humans would perform on the same tasks. Studies in cognitive science show that humans can reliably judge relative differences in physical quantities such as mass, depth, and elasticity, even when their estimates are systematically biased. For example, verbal estimates of distance track physical distance (Philbeck and Loomis, 1997), and observers can estimate mass under some viewing conditions (Shim et al., 2009). More directly relevant to NewtPhys, Paulun et al. (2026) found that human ratings strongly track the elasticity of simulated bouncing cubes ().
These findings motivate future work at the intersection of cognitive science and computer vision. NewtPhys could help test whether humans and models differ in their ability to judge relative differences and estimate absolute values. Human performance provides an empirical reference, and some vision models have already achieved lower errors than human participants in specific settings for absolute depth estimation (Kubota et al., 2025) and multi-view mass estimation (Standley et al., 2017).
As a first effort in this direction, we are conducting a user study to establish an initial human reference for NewtPhys and aim to report the results by the rebuttal deadline.
2. Transferability
Whether training on NewtPhys improves performance on real-world tasks, and whether NewtPhys scores predict that performance, are key directions to investigate. Prior work shows that training on simulated videos can improve the physical consistency of generated motion (PisaBench, ICML 2025; KineMask, ECCV 2026; PhyCo, CVPR 2026), and preliminary results from our follow-up work suggest that fine-tuning on NewtPhys could yield stronger benefits. In robotics, training on offline synthetic data proved to boost manipulation task like grasping (Tobin et al., IROS 2017), which could be investigated with NewtPhys richer data. A separate question is whether NewtPhys scores predict downstream performance on tasks such as grasping or throwing, and could therefore guide model selection? For transfer to real-world physical VQA, obtaining comparable physical ground truth in real scenes is a central challenge and was a key motivation for NewtPhys. Reconstruction and simulation provide physically grounded annotations at scale for quantities that would otherwise require specialized instrumentation and controlled experiments. A first step would be to evaluate transfer in controlled laboratory settings, using independently measured properties such as mass or Young’s modulus.
Rebuttal (1)
We appreciate the feedback and constructive questions. We respond to your comments below and will update our reply as we complete our experiments. We welcome suggestions for additional experiments that we can run within the rebuttal period.
Physical validity
We thank the reviewer for these suggestions and agree that physical validity deserves careful evaluation.
Existing validation. NewtPhys already assesses (i) visual and physical realism against existing physics benchmarks (Tab. 1b), (ii) reconstruction accuracy (App. B.1, L935–946), and (iii) physical properties (App. B.2, L960–970, Fig. 13). For dynamics, we employ NVIDIA Simplicits (Modi et al., TOG 2024), a well-established mesh-free elastic simulator whose accuracy has been quantitatively investigated against finite-element references, including in Xiang et al. (CVPR 2026). Together, these evaluations support the physical plausibility of NewtPhys, although they cannot guarantee exact real-world dynamics.
Additional evaluation. To further assess physical plausibility, we evaluated all datasets using the recent GeoPhys metric (Internò et al., NeurIPS 2026), which measures plausibility from video-feature trajectories. NewtPhys achieves again the best score (0.4918, versus 0.5530 for the closest baseline), reinforcing our results across complementary realism metrics (Tab. A).
Importantly, prior works relying on substantially simpler simulated interactions (PisaBench, ICML 2025; KineMask, ECCV 2026; PhyCo, CVPR 2026) have demonstrated transfer or generalization to real-world scenarios. While this does not establish physical accuracy, it supports the value of simulation-based physical supervision beyond synthetic settings.
Regarding the suggested experiments:
While both suggestions are valuable in principes, establishing meaningful comparisons across our diverse scenes would require extensive calibration and constitute a substantial study, even for a small subset, which is beyond the present scope for rebuttal. We will clarify these limitations and explicitly distinguish physical plausibility, extensively assessed in NewtPhys, from real-world physical accuracy, which remains an open direction.
We remain open to suggestion of feasible experiments within the short rebuttal period.
Tab A. NewtPhys realism metrics