- Rating
- 3
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces MPMWorlds, a dataset of 95,805 physics simulations with executable code, scene configurations, and rendered videos. The authors compare code generation models, which predict future dynamics by generating simulation programs, with video diffusion models, which directly generate future frames. Their results reveal complementary strengths: code generation models produce more temporally stable and physically consistent predictions but struggle to infer physical properties from visual input, whereas video diffusion models better preserve geometry and spatial layouts but often generate physically implausible dynamics. The findings highlight the challenge of achieving both visual fidelity and physical consistency in world models.
Critical strengths
Important and timely research question: The paper investigates an important question for world models: whether video generative models can learn physical dynamics from visual observations and reliably extrapolate them into the future, compared with approaches that explicitly construct the world through executable simulation code. The comparison between code generation and direct video generation is meaningful, as it helps characterize the different strengths and limitations of implicit visual modeling and explicit physical simulation.
Critical weaknesses
- Limited evidence of generalization to real-world physical understanding. The experiments are conducted entirely on synthetic 2D MPM simulations. Although this controlled setting provides access to accurate physical parameters and ground-truth dynamics, it remains unclear whether the findings generalize to real-world images and videos, where physical properties must be inferred under complex visual conditions. In particular, the paper does not provide evidence that the observed differences between code generation and video generation persist in realistic 3D environments. This limits the conclusions that can be drawn about the physical understanding capabilities of current world models in real-world settings.
- Limited novelty of the empirical findings. The main findings largely align with intuitive expectations: explicit simulation code can provide more temporally stable dynamics when the underlying physical model is correctly specified, whereas video diffusion models can better preserve visual appearance and spatial layout but may generate physically implausible motion. Similarly, providing explicit scene configurations substantially benefits code generation models. Although the paper provides a systematic empirical comparison, it is unclear whether the findings offer sufficiently novel insights beyond quantifying these expected trade-offs. The work would be stronger if it revealed less obvious failure mechanisms or provided deeper insights into how current models learn and represent physical dynamics.
- Limited coverage of physical phenomena. Although MPM supports a range of materials and interactions, the proposed benchmark is restricted to 2D MPM simulations and a specific set of physical processes. It does not adequately cover the diversity of real-world 3D dynamics, such as complex rigid-body interactions, articulated mechanisms, and more general deformable structures. Consequently, the benchmark may not sufficiently capture the physical reasoning challenges encountered in broader real-world applications, limiting the generality of the conclusions.
- Limited evidence of physical knowledge acquired during video pretraining. The video generation models are pretrained on general video data, and some are subsequently fine-tuned on the proposed synthetic physics dataset. Consequently, the observed performance does not clearly distinguish physical knowledge acquired during pretraining from capabilities learned through task-specific fine-tuning. Although the paper includes zero-shot evaluations, further controlled comparisons between pretrained and fine-tuned models would help clarify whether general-purpose video pretraining already provides meaningful physical understanding, or whether the observed capabilities primarily emerge from training on the synthetic simulation data.
Questions for authors
Refer to weakness.
2 comments
Thanks for your reply. I would keep my scores, but after reading all the other review comments, I'm not going to champion this paper.
We thank the reviewer for their thoughtful comments and suggestions.
Novelty of the Findings
The reviewer writes:
We emphasize that MPMWorlds pairs videos with their generating simulation programs and structured configurations across deformable and multi-material dynamics. The physics-video benchmarks discussed in Sec. 5 do not provide this paired code–video supervision. This lets us--for the first time--test these expectations rigorously by training both approaches on the same scenes, withholding specific physical information, and comparing pretrained and fine-tuned models (Tab. R4).
For example, explicit configurations raise the code models' mean mIoU from 0.235 to 0.384, while changing the video models' scores little (Tab. 4). The missing information also affects the code models differently: positions matter most for Qwen2.5-VL, and materials for Kimi K3. For Qwen, the median position error rises from 0.026 to 0.164 domain widths when the configuration is withheld. The new configuration baseline further separates physical inference from code generation (Tab. R3 in our response to xWS1). We will make these findings and the research questions the dataset enables clearer in Secs. 1 and 4 [C,D].
Real-World and 3D Scope
The reviewer writes:
Our experiments study how the two approaches infer and extrapolate dynamics in a controlled 2D setting; they do not establish transfer to real videos or 3D physics. MPMWorlds extends 2D data beyond rigid-body dynamics to fluids, sand, snow, elastic and plastic solids, multi-material interactions, and emitters, alongside colliders and kinematic objects. Pairing each video with its configuration and code lets us control the information given to a model and evaluate against the simulator state. We agree that extending this pairing to real videos and 3D simulations is an interesting direction for future work. We will state the scope explicitly in the introduction and Sec. 6 [C,D].
Pretraining vs. Fine-Tuning
The reviewer writes:
Following the reviewer's suggestion, we evaluated Wan and LongCat without fine-tuning on the 105 scenes of Tab. 4, using the same inputs, inference settings, and future-frame scoring as for the fine-tuned models (Tab. R4).
Table R4: Pretraining vs. Fine-Tuning. Pretrained → fine-tuned; Full Configuration / Frames Only
Without fine-tuning, LongCat and Wan already score 0.51 and 0.42 mIoU from frames alone, but have temporal-anomaly rates of 0.85--0.98. Fine-tuning improves mIoU in all four comparisons (all 95% CIs exclude zero), with larger relative gains in mass loss and flow agreement. In Full Configuration, mass loss falls from 0.54 to 0.24 for LongCat and from 0.91 to 0.31 for Wan, while flow agreement rises from 0.09 to 0.27 and from 0.07 to 0.18. Both metrics also improve from frames alone. Fine-tuning reduces Wan's temporal anomalies but not LongCat's. Some of the gains may reflect adaptation to the rendering style, so this comparison does not isolate physical learning. We will add it to App. J [E].