acceptodds
Under review as a conference paper at ICLR 2027

NewtPhys: Dissecting Newtonian Understanding In Foundation Models

Abstract

Previous work has evaluated physics reasoning in foundation models using synthetic or semi-synthetic scenes and visual question-answering tasks. However, these benchmarks emphasize high-level events and lack the visual fidelity needed to assess low-level Newtonian understanding. We introduce **NewtPhys**, a 4D physically annotated dataset built from multiview images of real-world scenes with physics-grounded simulations. The dataset provides dense, fine-grained annotations across timesteps — including 3D forces and amodal per-pixel quantities covering physics, tracking, semantics and geometry — bridging the gap between simplistic synthetic setups and realistic visual complexity. Using NewtPhys, we systematically evaluate 57 VLMs, including 54 open models and 3 frontier closed models, and 10 VFMs and reveal limitations in low-level physics reasoning. Beyond benchmarking, our study enables future research in physics-grounded vision and next-generation physics-aware evaluation. Code and datasets will be public.

Primary area:
datasets and benchmarks
Keywords:
computer vision, vision foundation models, physics, benchmark, 3DGS, visual question answering, multimodal large language models

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_hQJZOpenReview ↗
Rating
4(+1)
Confidence
3(+1)
Informativeness
3
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper introduces NewtPhys, a benchmark curated by augmenting real-world 3DGS scenes with objects and simulating their interactions using a simulator. It provide simulated sequences with high-level event annotations and dense pixel-aligned labels for physical quantities, kinematics, semantics, and geometry. The dataset also supports VQA across six categories of physical understanding.

The authors evaluate VLMs on the VQA benchmark and VFMs using lightweight decoders trained to predict pixel-level quantities such as collisions, gravity, and scene flow. Further analyses examine language-prior reliance, counterfactual reasoning, the effects of spatial and physical cues, and ood gravity prediction. The pipeline is carefully designed to obtain physical annotations that are difficult to collect directly, and it incorporates object properties estimated from visual cues and metadata. The suppl also reports checks of these estimates against vendor information for a subset of objects.

Critical strengths

A useful benchmark with rich annotations. NewtPhys combines realistic 3DGS scenes with controlled simulation and provides both VQA data and dense pixel-level labels. This enables evaluation of high-level physical reasoning as well as probing whether representations contain localized physical signals, and this quite different from existing VLM physics benchmarks.

A thoughtfully designed data-generation pipeline. Recovering physical quantities and dense labels from real-world interactions is challenging. The use of simulation makes these annotations available at scale, while accounting for object properties such as mass and material parameters. The supplementary checks of estimated object properties provide some support for the quality of these inputs.

Broad evaluation and useful diagnostic analyses. The paper evaluates multiple VLMs and VFMs and studies language priors, counterfactual reasoning, spatial and physical cues, and OOD gravity prediction. These analyses help characterize where current models succeed or fail beyond aggregate benchmark scores and provide insights on further building more physically capable VLM/VFMs.

Critical weaknesses

The physical validity of the generated dynamics and labels remains difficult to assess. The paper reports visual-fidelity and physical-plausibility metrics, checks 3DGS reconstruction dimensions for a subset of objects, and evaluates estimated object properties against vendor information. However, it does not compare simulated dynamics or pixel-level physical labels against measurements from real-world interactions. As a result, it still remains unclear how simulator assumptions and estimated physical properties affect the generated dynamics and labels. Even a small subset of this experiments would be fine.

The effect of language priors on VQA scores could be analyzed more deeply. The authors deserve credit for explicitly testing whether models rely on language priors and question construction, including ROI-masking and counterfactual evaluations. However, roughly one third of the tested models improve when the queried visual region is masked. This is a notable result, but the paper does not establish what drives the increase: for example, whether masking removes distracting visual evidence, changes the model's answer strategy, or exposes biases in the question and answer choices.

Questions for authors
  1. The appendix shows that basic property labels align with the simulator state, but this does not establish physical accuracy. Could the authors validate the pipeline on videos with known states and compare selected dynamics with an independent simulator or controlled real-world recordings, while quantifying the effect of errors in estimated ones?

  2. Roughly one third of the tested models improve after the queried region is masked. Could the authors investigate whether this comes from distracting visual content, a changed answer strategy, or biases in the questions and answer choices?

  3. The OOD test varies gravity magnitude, but how well do the pixel-level probes transfer to held-out scenes, object instances, and material combinations? Cross-object or cross-scene splits could test whether the representations capture physical information rather than superficial appearance or simulator cues.

  4. Could you discuss how future work can establish human reference performance and test whether models trained with NewtPhys transfer to real interactions? It would also be useful to clarify how NewtPhys complements Beyond Static Vision: Scene Dynamic Field Unlocks Intuitive Physics Understanding in Multi-modal Large Language Models and PhysicsMind: Sim and Real Mechanics Benchmarking for Physical Reasoning and Prediction in Foundational VLMs and World Models.

7 comments
AuthorsOpenReview ↗

Rebuttal (1)

We appreciate the feedback and constructive questions. We respond to your comments below and will update our reply as we complete our experiments. We welcome suggestions for additional experiments that we can run within the rebuttal period.


Physical validity

We thank the reviewer for these suggestions and agree that physical validity deserves careful evaluation.

Existing validation. NewtPhys already assesses (i) visual and physical realism against existing physics benchmarks (Tab. 1b), (ii) reconstruction accuracy (App. B.1, L935–946), and (iii) physical properties (App. B.2, L960–970, Fig. 13). For dynamics, we employ NVIDIA Simplicits (Modi et al., TOG 2024), a well-established mesh-free elastic simulator whose accuracy has been quantitatively investigated against finite-element references, including in Xiang et al. (CVPR 2026). Together, these evaluations support the physical plausibility of NewtPhys, although they cannot guarantee exact real-world dynamics.

Additional evaluation. To further assess physical plausibility, we evaluated all datasets using the recent GeoPhys metric (Internò et al., NeurIPS 2026), which measures plausibility from video-feature trajectories. NewtPhys achieves again the best score (0.4918, versus 0.5530 for the closest baseline), reinforcing our results across complementary realism metrics (Tab. A).

Importantly, prior works relying on substantially simpler simulated interactions (PisaBench, ICML 2025; KineMask, ECCV 2026; PhyCo, CVPR 2026) have demonstrated transfer or generalization to real-world scenarios. While this does not establish physical accuracy, it supports the value of simulation-based physical supervision beyond synthetic settings.

Regarding the suggested experiments:

  • Independent simulator. We agree this would be informative, but a meaningful comparison requires matching geometry, material parameters, contact models, and initial conditions. Beyond toy experiments (eg, primitive interactions), cross-simulator discrepancies will be confounded by differences in their representation (mesh-free in Simplicits vs mesh-based in Isaac Sim/Blender), material parameterization (eg, YMS and Poisson for Simplicits/Isaac Sim vs spring-based model in Blender), and calibration. Consequently, even if physical parameters are matched, differences in numerical representations and calibration may yield divergent trajectories and these discrepancies would be difficult to attribute to physical inaccuracies rather than modeling differences. Moreover, to our knowledge, there is not any established ground-truth simulator for such complex interactions.
  • Controlled real-world recordings. We explored this approach during our research but could not identify a sufficiently representative and reliable protocol. Quantitative comparisons require accurately measured geometry (potentially requiring specialized 3D acquisition), with similar initial conditions (positioning, release conditions, etc.), and material parameters (difficult to measure) to produce comparable interactions. GAUGE (Wang et al., 2026) illustrates the difficulty of such controlled validation, highlighting in Sec. 6 that material variability and calibration limitations persist even under controlled settings, complicating the attribution of discrepancies to simulation inaccuracies.

While both suggestions are valuable in principes, establishing meaningful comparisons across our diverse scenes would require extensive calibration and constitute a substantial study, even for a small subset, which is beyond the present scope for rebuttal. We will clarify these limitations and explicitly distinguish physical plausibility, extensively assessed in NewtPhys, from real-world physical accuracy, which remains an open direction.
We remain open to suggestion of feasible experiments within the short rebuttal period.

Tab A. NewtPhys realism metrics

DatasetFID ↓KID ↓VBench ↑VJEPA (max) ↓VJEPA (top-3) ↓GeoPhys ↓\textcolor{blue}{\text{GeoPhys ↓}}
ContPhy272.590.16650.6913‾\underline{0.6913}0.46160.44690.5530‾\textcolor{blue}{\underline{0.5530}}
IntPhys251.600.15560.65840.43840.42770.7761\textcolor{blue}{0.7761}
IntPhys2215.110.0951‾\underline{0.0951}0.68770.49350.47670.6818\textcolor{blue}{0.6818}
PhysBench216.170.10750.67990.46290.44910.5688\textcolor{blue}{0.5688}
Physion++193.82‾\underline{193.82}0.11850.64420.4370‾\underline{0.4370}0.4254‾\underline{0.4254}0.6790\textcolor{blue}{0.6790}
NewtPhys189.680.07830.69700.42870.42160.4918\mathbf{\textcolor{blue}{0.4918}}
AuthorsOpenReview ↗

Rebuttal (2)

Investigation of effect of language priors on VQA

We first detail our masked-ROI experiment (Sec. 4.2, Fig. 7). The latter is conducted on a 2K subset of object-related VQAs and reports before (baseline) and after masking the queried object(s) with a black-filled minimum enclosing circle, concealing their contours. Questions, background, and other objects remain perfectly unchanged as we aim to isolate the effect of removing visual evidence about the queried objects.

Magnitude of improvements. While we mention a third of the 31 VLMs models improved, there are confined to only 5 of 19 open-VLM families studied, with most positive variations being marginal. In fact, only four models (InternVL2-1B/40B/76B and Mantis-8B) improve by more than 22\\% relative accuracy. We hypothesize that this bias may partly stem from training objectives, consistent with our VFM findings (Tab. 2), where self-supervised models generalize better to physics understanding.

Question-level bias analysis. To investigate whether these changes arise from specific questions among our 2K set, we measure the overlap of questions whose answers change after masking across models. Tab. B (below) reports variations among these sets using three Jaccard metrics per model group: average pairwise models agreement, majority agreement, and general agreement. Agreement is generally low: average pairwise Jaccard is 0.089 across all models and 0.101 for improving models. The generalized Jaccard is close to 0, showing that not all models changed their answers on the same set of questions. Relaxing the constraint, we look at majority agreement across improved models which is 0.04205 (ie, 46 out of the 1094 changed questions, from our 2K set). Furthermore, only two of these questions changed from wrong to right on at least three of the improved models. This shows that shared answer changes were uncommon, and shared gains were rarer still.

Tab B Agreement among models which changed answers [new]\textcolor{blue}{\text{[new]}}

Model groupModelsUnion sizePairwise JaccardMajority JaccardGeneralized Jaccard
All models311,8430.088520.000540.00000
Improved (>+2>+2\\%)41,0940.100520.042050.00091
Degraded (<−2<−2\\%)141,6690.105910.007790.00000

Our results are consistent with prior work suggesting that VLMs underuse visual cues and rely on language priors (Chen et al., NeurIPS 2024). Our contribution is to investigate this behavior specifically in physical reasoning tasks, a regime where LLMs appear to struggle to compensate visual deficiencies. We will clarify that our experiment measures reliance on queried-object visual evidence and provides evidence consistent with language-prior effects, rather than establishing their causal origin. We will detail the protocol and additional analysis in the supplementary.

AuthorsOpenReview ↗

Rebuttal (3)

OOD variations

We appreciate the suggested experiments, which we are currently running. We will report results as soon as they are ready.


Discussion on future works (human evaluation, transferability)

Thanks for opening the discussion on future works. NewtPhys opens questions about how models infer physical properties from visual evidence, how these inferences compare with human perception, and how they might support real-world interaction.

1. Human evaluation.

NewtPhys reveals gaps in vision-language models’ physical understanding, leaving open how humans would perform on the same tasks. Studies in cognitive science show that humans can reliably judge relative differences in physical quantities such as mass, depth, and elasticity, even when their estimates are systematically biased. For example, verbal estimates of distance track physical distance (Philbeck and Loomis, 1997), and observers can estimate mass under some viewing conditions (Shim et al., 2009). More directly relevant to NewtPhys, Paulun et al. (2026) found that human ratings strongly track the elasticity of simulated bouncing cubes (R2=0.84R^2=0.84).

These findings motivate future work at the intersection of cognitive science and computer vision. NewtPhys could help test whether humans and models differ in their ability to judge relative differences and estimate absolute values. Human performance provides an empirical reference, and some vision models have already achieved lower errors than human participants in specific settings for absolute depth estimation (Kubota et al., 2025) and multi-view mass estimation (Standley et al., 2017).

As a first effort in this direction, we are conducting a user study to establish an initial human reference for NewtPhys and aim to report the results by the rebuttal deadline.

2. Transferability

Whether training on NewtPhys improves performance on real-world tasks, and whether NewtPhys scores predict that performance, are key directions to investigate. Prior work shows that training on simulated videos can improve the physical consistency of generated motion (PisaBench, ICML 2025; KineMask, ECCV 2026; PhyCo, CVPR 2026), and preliminary results from our follow-up work suggest that fine-tuning on NewtPhys could yield stronger benefits. In robotics, training on offline synthetic data proved to boost manipulation task like grasping (Tobin et al., IROS 2017), which could be investigated with NewtPhys richer data. A separate question is whether NewtPhys scores predict downstream performance on tasks such as grasping or throwing, and could therefore guide model selection? For transfer to real-world physical VQA, obtaining comparable physical ground truth in real scenes is a central challenge and was a key motivation for NewtPhys. Reconstruction and simulation provide physically grounded annotations at scale for quantities that would otherwise require specialized instrumentation and controlled experiments. A first step would be to evaluate transfer in controlled laboratory settings, using independently measured properties such as mass or Young’s modulus.

Reviewer_b6hNOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a new 4D physically annotated dataset to evaluate and reveal limitations of VLMs and VFMs. This dataset includes 3D forces and covers physics, tracking, semantic and geometry annotations.This dataset was built from real world scenes in which the authors added diverse synthetic objects using 3D Gaussian Splatting and a Newtonian simulator to generate the sequence.

The authors show that VLMs struggle at predicting granular physical properties, which highlight that those models still did not learn a correct model of the physical world and leave room for improvement. This analysis also extends to VFMs. The contributions are in three fold 1) The pipeline to generate reliable physical scene using 3DGS 2) The dataset generated using this pipeline 3) The analysis of the limitations of current models and challenging around physical word modeling.

The datasets leveraged 53 DLV3D scenes and 109 different GSO objects that give around 11K sequences (up to 10s at 25fps). The metadata contains ground truth maps of physic events as well as kinetics, semantic and depth maps. Then, the authors built a set of questions/answers using LLMs around different axis leveraging the ground truth labels. They obtain 141 VQA (84 Videos and 57 images).

Using this data, they evaluated 67 models, from open-source to close frontier models. The best models are GPT-6 Astra and Gemini 3.1 across the 6 evaluations axis (Material understanding, mechanics, Spatial reasoning, Viewpoint, Permanence and temporal reasoning). The authors also added counter factual QA to assess whether the model truly generalizes. The authors highlight that large models might be prone to leveraging language bias and not looking at the visual data to answer the questions.

Critical strengths

Overall, the strengths are a very solid paper, which is well presented and introduces a much welcome benchmark. The pipeline is well documented and the ablations in the appendix are very informative.

Critical weaknesses

A limitation of the work acknowledged by the author is the sim to real gap that is not really studied in depth. There is also a lack of human baselines. There is also some references that are missing, For example Table 1, b) show dataset like ContPhy or IntPhys 2 but I don’t find the corresponding citations.

Questions for authors
  • Any plan to have some human baseline on at least a subset of the data if it's too costly?
3 comments

Thank you for the constructive feedback and questions. We address your comments below and would be happy to run additional experiments feasible within the rebuttal period.


Sim-to-real Gap

We agree that the sim-to-real gap warrants a close look. To assess this, we evaluated both visual and physical realism, reporting an improvement in all metrics of Tab. 1b. To our knowledge, this is the best possible effort to measure realism (ie, sim-to-real gap) of our data. In addition to details of Appendix C.2, we highlight that distributional metrics of Tab. 1b (ie, FID/KID) use as reference the real set of PisaBench, which captures gravity-driven falling objects in real various real-world conditions and is the closest to our setting. For physical plausibility, which more complex to evaluate, in Tab. 1b we report the well established VJEPA surprise measuring the latent discrepancy from a predictor model trained on real-world robotics videos showing a broad range of physics dynamics. Not only metrics on VJEPA surprise is better on NewtPhys (0.4287/0.4216 for max/top-3) but is also close to the real Pisa Bench set (0.4017/0.3931). To further demonstrate physical realism, we report the recent GeoPhys (Internò et al., NeurIPS 2026), which scores videos using their feature trajectories. As shown in Tab. A below, NewtPhys also outperforms all other baselines on this metric (0.4918 compared to 0.5530 for the second best).

Tab A. NewtPhys realism metrics

DatasetFID ↓KID ↓VBench ↑VJEPA (max) ↓VJEPA (top-3) ↓GeoPhys ↓\textcolor{blue}{\text{GeoPhys ↓}}
ContPhy272.590.16650.6913‾\underline{0.6913}0.46160.44690.5530‾\textcolor{blue}{\underline{0.5530}}
IntPhys251.600.15560.65840.43840.42770.7761\textcolor{blue}{0.7761}
IntPhys2215.110.0951‾\underline{0.0951}0.68770.49350.47670.6818\textcolor{blue}{0.6818}
PhysBench216.170.10750.67990.46290.44910.5688\textcolor{blue}{0.5688}
Physion++193.82‾\underline{193.82}0.11850.64420.4370‾\underline{0.4370}0.4254‾\underline{0.4254}0.6790\textcolor{blue}{0.6790}
NewtPhys189.680.07830.69700.42870.42160.4918\mathbf{\textcolor{blue}{0.4918}}

Further, we stress that there are concrete evidences in the literature that even simpler and less realistic settings than NewtPhys transfer to real-world robotics (Tobin et al., IROS 2017) or video generation (PisaBench, ICML 2025; KineMask, ECCV 2026; PhyCo, CVPR 2026).

We are happy to investigate other measurements of the sim-to-real gap, should the reviewer suggest other alternatives, feasible during the rebuttal period.


References missing

Thanks for noticing the omission, this will be fixed. ContPhy is (Zheng et al., ICML 2024) and IntPhys2 is (Bordes et al., NeurIPS 2025).


Human evaluation

We agree human evaluation may help positioning the performance of existing models, although human performance should not been seen as a gold standard as models are currently outperforming in a large variety of general (Li et al., ICML 2025) or specialist (Saha et al., The lancet 2024) computer vision tasks.

Given the short rebuttal period, we are constructing a human evaluation and will deploy best effort to obtain statistically significant results and include it before rebuttal period ends.


Bordes et al. Intphys 2: Benchmarking intuitive physics understanding in complex synthetic environments. NeurIPS 2025
Internò, Christian, et al. GEOPHYS: The Geometry of Physical Plausibility. NeurIPS 2026.  
Li, Fanfei, et al. Laion-c: An out-of-distribution benchmark for web-scale vision models. In ICML 2025.  
Saha, Anindo, et al. Artificial intelligence and radiologists in prostate cancer detection on MRI. The Lancet Oncology 2024  
Zheng, Zhicheng, et al. Contphy: Continuum physical concept learning and reasoning from videos. In ICML 2024.   
Reviewer_b6hNOpenReview ↗

Response to Authors

Thanks you for your response, really appreciate Table A. I will keep my acceptance recommendation.

AuthorsOpenReview ↗

Rebuttal (2)

We appreciate the reviewer's feedbacks. In addition, we report below the human evaluation study.
We highlight also that all the new results will be included in the revised manuscript version, along with the extended discussions and suggested clarifications.

Human evaluation

We conducted an online user study focused on physics understanding VQAs. Our study includes 796 responses from 44 participants aged 22–60 years, approximately 2424\\% of whom were women. In the preamble, we emphasize that the study provides a human reference, not an upper bound on performance, as machine models are known to outperform humans on several estimation tasks.

Methodology

Each participant was presented with up to 20 VQAs, including both image- and video-based questions, all of which were sampled randomly from NewtPhys, within 'Material Understanding' and 'Mechanics', and with sampling weights used to ensure sufficient representation of each question type across users.
A short instruction was displayed before the study, though without any example or trial questions. For each questions, users were presented an image/video and 4 possible answers. To address potential language barriers, when a technical term was mentioned (Young's Modulus, Poisson ratio, etc.) a helper card described the term and its scale in few words (typically, less than 25 words). The card provided no specific question guidance. Besides, to measure 'answerability', users could abstain on a question by clicking a button "I can't tell". Importantly, in all analyses below the abstentions are replaced by random answers.

We enforced no time limit and a user could also quit at any time. For analysis, we keep results of users having completed at least 10 questions.

Results and interpretation

Overall, human accuracy was 51.851.8\\% which seats between open VLM models (32.232.2\\%, Fig. 5) and the best frontier VLM model tested (ChatGPT 6 Astra; 56.856.8\\%). As shown in Tab. C below, humans performed better on Mechanics (59.459.4\\%) than Material understanding (44.244.2\\%), consistent with the trend observed for models. These rather low scores align with prior studies (Standley et al., CoRL 2017; Paulun et al., Biological Sciences 2026) that show quantity estimation is normally easier for computer vision models than humans -- who perform better instead at relative measurements (Philbeck and Loomis, 1997).

In contrast, humans excel at high level understanding like collision VQA (eg, "Which object collide with ..?") and we measured twice better performance for humans than open models (68.568.5\\% vs 32.932.9\\% from Fig. 6), even better than ChatGPT 6 Astra (65.765.7\\%). These results highlight a substantial gap between humans reasoning and VLMs. Among humans of our study, we also note a large variance in accuracy (σmaterial=14.0\sigma_\text{material}=14.0, σmechanics=22.6\sigma_\text{mechanics}=22.6).

Human performance exceeded the four-choice chance level of (2525\\%) which supports the answerability of NewtPhys and strengthen our analysis since most open models seat just above random chance (Fig. 5). Further, we note that users abstained overall on ≈9≈9\\% of the questions, with a slight tendency for higher-performing participants to abstain more frequently.

Tab C. Performance on physics across humans and models from Fig. 5 [new]\textcolor{blue}{\text{[new]}}

Material understandingMechanicsMean
Humans (44 users)\textcolor{blue}{\text{Humans (44 users)}}\textcolor{blue}{44.2\\%}\textcolor{blue}{59.4\\%}\textcolor{blue}{51.8\\%}
Open-models (54 models)31.231.2\\%33.133.1\\%32.232.2\\%
Frontier-models (3 models)48.048.0\\%53.653.6\\%50.850.8\\%
Best model (ChatGPT 6 Astra)50.550.5\\%63.163.1\\%56.856.8\\%

We will include details of the user study in the revised version of the manuscript along with: adding "Humans reference" in plot of Fig. 5, better positioning of NewtPhys and reporting detailed user performance with user distribution performance in the appendix.

Reviewer_UGopOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces NewtPhys, a benchmark combining reconstructed 3D Gaussian Splatting scenes and scanned objects with a customized Simplicits simulator. The pipeline generates approximately 11k sequences with dense physical, geometric, and semantic annotations, together with 141k visual questions. The authors evaluate 57 VLMs through VQA and 10 VFMs through pixel-level physics probing, and investigate counterfactual questions, additional input cues, and alternative question formulations. The central empirical claim is that current foundation models have limited low-level Newtonian understanding. The combination of visually complex scenes, controlled simulation, and dense annotations is potentially useful. My main concern is whether the evaluation sufficiently separates limitations in physical understanding from insufficient visual evidence, ambiguous task definitions, and sensitivity to the supplied information.

Critical strengths
  1. Useful integration of reconstruction, simulation, and annotation. The pipeline supports access to physical quantities and controlled interventions that are difficult to obtain from ordinary recorded footage. Its potential extends beyond VQA to dense prediction and representation analysis.

  2. Broad model evaluation. Evaluating both VLMs and VFMs provides complementary perspectives. The physics-probing experiments in Section 4.5 demonstrate a concrete use of the dense annotations, while the cue and formulation experiments investigate possible sources of failure.

  3. Substantial documentation and some validation. The appendix provides question templates, examples, simulation details, dataset statistics, and ambiguity-filtering procedures.

Critical weaknesses
  1. The answerability of the questions is insufficiently validated, weakening the interpretation of model failures. Some questions require quantities that may not be recoverable from the supplied observations, such as a lunch bag’s absolute mass or a container’s numerical Poisson ratio. Informative interactions could constrain these quantities, but the paper does not establish that the relevant questions consistently provide such evidence. Appendix D.5 describes ambiguity filtering, yet a unique simulator label does not guarantee that the visual input distinguishes the correct answer from the alternatives. Without a human reference or equivalent validation, low accuracy cannot reliably distinguish limited physical understanding from insufficient information. This is consequential because the paper uses these failures to diagnose deficiencies in current models’ Newtonian understanding.
  2. The diversity of physical scenarios is not established by the dataset’s scale. The described generation process centers on spawning objects and simulating gravity-driven motion and subsequent contact. The 11k sequences, varied assets, and approximately 15 distinct collisions per sequence demonstrate scale and interaction frequency, but do not establish coverage of diverse causal scenarios. The demonstrations also leave uncertainty about how frequently informative deformation occurs. This matters beyond dataset breadth: the available interactions determine which physical properties can be inferred and which abilities the benchmark can meaningfully evaluate. Event-type and initial-condition coverage are therefore important to assessing the dataset’s contribution and the scope of its conclusions.
  3. The incremental contribution of both the dataset and the diagnostic study needs stronger justification. The integration of reconstructed scenes, simulation, and dense annotations is potentially useful, but the paper should more clearly establish what additional evaluation capabilities it provides over existing alternatives. Visual realism is an important part of this positioning, yet Table 1(b)’s distributional and learned proxies provide only indirect support for that advantage. The diagnostic study also does not clearly articulate what new understanding emerges beyond the broad observations that models struggle with physical quantities and that their performance depends on the task and supplied cues. Extensive benchmarking is useful, but the paper’s stronger claim to dissect model limitations requires a clearer account of what the analyses newly explain. This leaves uncertainty about both the resource’s additional value and the distinct empirical insight obtained from it.
Questions for authors
  1. Answerability — clarification and validation: How does the ambiguity filter establish that the supplied observations distinguish the correct physical-property answer from the alternatives? Please clarify what timing and scale information is available to the models, and what validation—including any human evaluation—supports answerability under the actual input protocol.

  2. Physical-event coverage — clarification: What physical event types and initial conditions are represented beyond gravity-driven release and subsequent contact? How frequently do sequences contain visible deformation or interactions that provide evidence about the queried material properties?

  3. Additional value of the dataset — clarification and supporting evidence: Which evaluation capabilities specifically benefit from reconstruction and simulation, rather than annotation procedures transferable to existing real imagery? Which results demonstrate this additional value, and how do the reported realism metrics support the claimed advantage?

  4. New insight from the diagnostic study — clarification: Which findings most clearly extend existing knowledge about VLMs’ physical understanding? In particular, what do the analyses establish beyond performance measurements on the new benchmark, and what evidence supports the proposed explanations of model limitations?

6 comments
AuthorsOpenReview ↗

Rebuttal (1)

We appreciate the effort put for a careful review, and recognize the value of the questions raised. We believe NewtPhys is unique not only by its scale, but also by how it enables indepth study of vision foundation models, unlocks new computer vision tasks, and lead to new findings. We hope to address below your remarks, and will complement our answer as our experiments completes.


Q1. Answerability of the question

We agree that answerability depends on whether the supplied frames contain enough evidence to distinguish the correct option from the alternatives. Mechanics questions concern directly observable events, such as visible collisions, yet model accuracy in this category is relatively low. For questions about latent properties such as mass or Poisson’s ratio, the relevant visual cues vary across sequences, yet prior studies suggest that such properties can be estimated from visual data. For example, both human observers and models can estimate mass from single image (Standley et al., CoRL 2017), while videos enables estimation of physical elasticity (Paulun et al., 2026). On NewtPhys, the best frontier model achieves 50.550.5\\% on Material understanding (mostly latent physical properties) and 63.163.1\\% on Mechanics (mostly directly observable events), compared with mean accuracies of 31.231.2\\% and 33.133.1\\%, respectively, across the open models we evaluated. This advocates that at least some of these questions are tractable for stronger models and that open models have room to improve.

1. Answerability in NewtPhys.

We designed NewtPhys to provide visual cues through object interactions, use physically plausible material properties, and reduce ambiguity in question generation.

(a) Object interactions. Visible collisions are key to answerability since they reveal changes in motion or deformation that provide cues to physical properties. To foster interactions, objects are randomly spawned in selected regions that trigger rich interactions, such as above a statue or on a chair. In 7676\\% of physics VQAs, the frames supplied with the question show the queried object colliding with at least one other element in the scene. Compared with VQA not showing collisions, model accuracy on VQA showing the queried object colliding is higher by 0.4 percentage points for Material Understanding and by 5.5 percentage points higher for Mechanics. This supports that interactions help disambiguation. Appendix A.1 (Fig. 11a) also reports higher accuracy for softer objects (+2.32 points on Material), whose deformation are easier to observe.

(b) Physical realism. Answerability is also supported by the realism of NewtPhys (assessed in Sec. 3.2 and below), and especially Appendix B.2 showing that our objects' physical properties are close to reference values. We argue that this enables models to also reason from their prior knowledge.

(c) Filtering of ambiguous VQAs. The filtering (Appendix D.5) is intended to automatically discard VQAs with: multiple correct answers, visual ambiguities such as when the answer appears at multiple occurences (eg, twice the same objects), or near-tie answers (eg, asking which object is heaviest when the two heaviest objects have nearly equal masses). Practically, we require the correct answer has a minimum separation with other objects of 1010\\% for mass (relative), 0.02 for Poisson’s ratio, and 0.3 in log⁡10\log_{10} Young’s modulus (roughly a factor of two). For numerical questions, we apply the same thresholds to each pair of distractors.

2. Models inputs

By default, models receive the question, answer choices, and one to eight frames, with no explicit timing or metric scale reference (Appendices D.1, D.2, and D.5).

We tested the duration of sequence duration, in Fig. 10b, yielding modest accuracy gains for 9 of 31 models (“Name + Duration”). Adding exact or approximate object mass improves more strongly the accuracy for up to 28 of 31 models (“Name + XXX Mass”). These cues may help models interpret the supplied frames or directly favor certain answer choices (Appendix A.3).

3. Human evaluation

We follow your suggestion and are currently conducting a user study to establish human reference for physics understanding in NewtPhys. Importantly, we highlight that human observers should not be seen as an upper bound but rather as a reference, since quantities estimation by humans have been largely overpassed by computer vision models.
We aim to report the user study by the rebuttal deadline.

AuthorsOpenReview ↗

Rebuttal (2)

Q2. Clarification on physical-event coverage

NewtPhys focuses primarily on gravity-driven Newtonian dynamics. Within this scope, we conduct an in-depth analysis: varying object geometry, material properties, and drop conditions to produce a range of object–object and object–scene interactions. Following the reviewer's suggestion we extract diverse statistic below, to better assess our coverage.

1. Diversity of physical properties

Expanding on Fig. 14, NewtPhys uses 109 objects spanning 11 object types and 9 material types. Our properties cover a wide spectrum: object masses range from 0.04 to 44 kg, densities from 30 to 2,100 kg/m³ (equivalent of varying density from a light packing foam to a dense brick), and volumes from 0.27 to 280 dm³ (roughly varying the volume from a mug to a refrigerator). Drop heights range from 0.64 to 6.21 m above the ground. Observed object speeds reach 24.25 m/s, and camera speeds reach 10.52 m/s.

2. Interaction events

Overall, the datasets shows 61k individual objects, and captures 7.24M objects instances, and 7.87.8M frame-level collisions; among which 7474\\% are 2-way collisions (e.g., object-object or object-scene), 2222\\% 3-way, 33\\% 4-way, 0.40.4\\% 5-way, <0.05<0.05\\% 6-way. We also extract temporal events showing: 14k bounces of at least 5 cm after impact, 32k objects disappearing, and 19k objects reappearing after disappearance.

3. Deformations

Measuring objects deformation is not trivial, so we rely on material stress as a proxy. Overall, 2424\\% of the falling objects exhibit a significant material stress (504504k visible occurrences) among which 4444\\% experience a maximum material stress (9393k visible occurrences), with a 2-to-1 predominance of contraction versus expansion.

AuthorsOpenReview ↗

Rebuttal (3)

Q3. Additional value of the dataset

Which evaluation capabilities specifically benefit from reconstruction and simulation, rather than annotation procedures transferable to existing real imagery?

Reconstruction provides the scene geometry, and simulation yields dense physical annotations aligned with the rendered frames. In NewtPhys, we enabes pixel-, frame-, and event-level labels with 2D and 3D locations, and allows evaluation of per-pixel forces, contact timing and location, deformation, material properties, and kinematics. Many of these labels would be extremely difficult to recover from existing RGB footage, even when considered individually, especially in complex scenes and with camera motion. For example, recovering contact forces would need instrumented contact measurements, while Young’s modulus or Poisson’s ratio typically requires controlled loading and measurements of the resulting deformation. Our pipeline further enables controlled comparisons. We vary gravity or object size (Fig. 8) or Young’s modulus (Figs. 11a and 13a) while holding other scene settings fixed. This lets us test whether model predictions respond to changes in a specific physical parameter.

We stress that progress in computer graphics and simulation dynamics have reduced the sim-to-real gap, with concrete evidences that even simpler and less realistic settings than NewtPhys transfer to real-world robotics (Tobin et al., IROS 2017) or video generation (PisaBench, ICML 2025; KineMask, ECCV 2026; PhyCo, CVPR 2026). On this, NewtPhys proves not only more diverse (Tab. 1a) but better in visual and physical realistic (Tab. 1b and Tab. A below).

Which results demonstrate this additional value, and how do the reported realism metrics support the claimed advantage?

Realism is key to assess the validity of our dataset, both visually and physically. On all five metrics of Tab. 1b NewtPhys already outperforms the existing physics-understanding datasets, which we complement with the recent GeoPhys metric (Interno et al., NeurIPS 2026) further supports this in Tab. A below. Beyond metrics, we refer to visuals and videos of our dataset (Figs. 1,4 and supplementary website) and comparisons with others (Fig. 2 `Low-level understanding'); highlighting a striking visual difference which comes in addition of the richer features offered by NewtPhys (Tab. 1a).

This richer and more realistic data, allows to measure models capacity with unprecedented breadth and granularity. Among others, probing physical properties of materials, per-force information (gravity, collision, etc.) or even fine-grained kinematics (speed, acceleration) and spatial reasoning (position, volume, etc.), are simply not feasible without NewtPhys. This unveils novel insights which we describe below.

Tab A. NewtPhys realism metrics

DatasetFID ↓KID ↓VBench ↑VJEPA (max) ↓VJEPA (top-3) ↓GeoPhys ↓\textcolor{blue}{\text{GeoPhys ↓}}
ContPhy272.590.16650.6913‾\underline{0.6913}0.46160.44690.5530‾\textcolor{blue}{\underline{0.5530}}
IntPhys251.600.15560.65840.43840.42770.7761\textcolor{blue}{0.7761}
IntPhys2215.110.0951‾\underline{0.0951}0.68770.49350.47670.6818\textcolor{blue}{0.6818}
PhysBench216.170.10750.67990.46290.44910.5688\textcolor{blue}{0.5688}
Physion++193.82‾\underline{193.82}0.11850.64420.4370‾\underline{0.4370}0.4254‾\underline{0.4254}0.6790\textcolor{blue}{0.6790}
NewtPhys189.680.07830.69700.42870.42160.4918\mathbf{\textcolor{blue}{0.4918}}

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.