NewtPhys: Dissecting Newtonian Understanding In Foundation Models

Previous work has evaluated physics reasoning in foundation models using synthetic or semi-synthetic scenes and visual question-answering tasks. However, these benchmarks emphasize high-level events and lack the visual fidelity needed to assess low-level Newtonian understanding. We introduce **NewtPhys**, a 4D physically annotated dataset built from multiview images of real-world scenes with physics-grounded simulations. The dataset provides dense, fine-grained annotations across timesteps — including 3D forces and amodal per-pixel quantities covering physics, tracking, semantics and geometry — bridging the gap between simplistic synthetic setups and realistic visual complexity. Using NewtPhys, we systematically evaluate 57 VLMs, including 54 open models and 3 frontier closed models, and 10 VFMs and reveal limitations in low-level physics reasoning. Beyond benchmarking, our study enables future research in physics-grounded vision and next-generation physics-aware evaluation. Code and datasets will be public.