MPMWorlds: A Dataset of Executable Material-Point-Method Simulations for Inferring and Extrapolating Physical Dynamics

To study the ability to infer physical dynamics from videos and extrapolate them forward in time, we assemble a dataset of 2D Material Point Method (MPM) physical simulations covering rich physical phenomena such as deformable objects, fluids, kinematic objects, and emitters. Every simulation pairs executable source code, a structured scene configuration, and a rendered video, which lets code generation and video generation be trained and conditioned on the same scene and evaluated on the same video. We study code generation and video diffusion approaches on this dataset, identifying their strengths and weaknesses by varying the amount of physically relevant side information. Our contributions are the dataset and this study. The study does not rank the two approaches; it characterizes how each one fails. The code generation models, beyond giving a working demonstration of automatic synthesis of MPM simulations, reveal that such an approach struggles with inferring physical parameters from visual input, but relative to video diffusion, produces physically and temporally stable extrapolations forward in time, while the video diffusion models more strongly identify geometric properties from visual input but produce physically implausible extrapolations. We test the main findings across three VLMs and three VDMs, including a sample-matched comparison and physically grounded metrics.