acceptodds
Under review as a conference paper at ICLR 2027

MPMWorlds: A Dataset of Executable Material-Point-Method Simulations for Inferring and Extrapolating Physical Dynamics

Abstract

To study the ability to infer physical dynamics from videos and extrapolate them forward in time, we assemble a dataset of 2D Material Point Method (MPM) physical simulations covering rich physical phenomena such as deformable objects, fluids, kinematic objects, and emitters. Every simulation pairs executable source code, a structured scene configuration, and a rendered video, which lets code generation and video generation be trained and conditioned on the same scene and evaluated on the same video. We study code generation and video diffusion approaches on this dataset, identifying their strengths and weaknesses by varying the amount of physically relevant side information. Our contributions are the dataset and this study. The study does not rank the two approaches; it characterizes how each one fails. The code generation models, beyond giving a working demonstration of automatic synthesis of MPM simulations, reveal that such an approach struggles with inferring physical parameters from visual input, but relative to video diffusion, produces physically and temporally stable extrapolations forward in time, while the video diffusion models more strongly identify geometric properties from visual input but produce physically implausible extrapolations. We test the main findings across three VLMs and three VDMs, including a sample-matched comparison and physically grounded metrics.

Primary area:
datasets and benchmarks
Keywords:
Physical simulation, program synthesis, intuitive physics, world models, multimodal models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PJ86OpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MPMWorlds, a dataset of 95,805 physics simulations with executable code, scene configurations, and rendered videos. The authors compare code generation models, which predict future dynamics by generating simulation programs, with video diffusion models, which directly generate future frames. Their results reveal complementary strengths: code generation models produce more temporally stable and physically consistent predictions but struggle to infer physical properties from visual input, whereas video diffusion models better preserve geometry and spatial layouts but often generate physically implausible dynamics. The findings highlight the challenge of achieving both visual fidelity and physical consistency in world models.

Critical strengths

Important and timely research question: The paper investigates an important question for world models: whether video generative models can learn physical dynamics from visual observations and reliably extrapolate them into the future, compared with approaches that explicitly construct the world through executable simulation code. The comparison between code generation and direct video generation is meaningful, as it helps characterize the different strengths and limitations of implicit visual modeling and explicit physical simulation.

Critical weaknesses
  1. Limited evidence of generalization to real-world physical understanding. The experiments are conducted entirely on synthetic 2D MPM simulations. Although this controlled setting provides access to accurate physical parameters and ground-truth dynamics, it remains unclear whether the findings generalize to real-world images and videos, where physical properties must be inferred under complex visual conditions. In particular, the paper does not provide evidence that the observed differences between code generation and video generation persist in realistic 3D environments. This limits the conclusions that can be drawn about the physical understanding capabilities of current world models in real-world settings.
  2. Limited novelty of the empirical findings. The main findings largely align with intuitive expectations: explicit simulation code can provide more temporally stable dynamics when the underlying physical model is correctly specified, whereas video diffusion models can better preserve visual appearance and spatial layout but may generate physically implausible motion. Similarly, providing explicit scene configurations substantially benefits code generation models. Although the paper provides a systematic empirical comparison, it is unclear whether the findings offer sufficiently novel insights beyond quantifying these expected trade-offs. The work would be stronger if it revealed less obvious failure mechanisms or provided deeper insights into how current models learn and represent physical dynamics.
  3. Limited coverage of physical phenomena. Although MPM supports a range of materials and interactions, the proposed benchmark is restricted to 2D MPM simulations and a specific set of physical processes. It does not adequately cover the diversity of real-world 3D dynamics, such as complex rigid-body interactions, articulated mechanisms, and more general deformable structures. Consequently, the benchmark may not sufficiently capture the physical reasoning challenges encountered in broader real-world applications, limiting the generality of the conclusions.
  4. Limited evidence of physical knowledge acquired during video pretraining. The video generation models are pretrained on general video data, and some are subsequently fine-tuned on the proposed synthetic physics dataset. Consequently, the observed performance does not clearly distinguish physical knowledge acquired during pretraining from capabilities learned through task-specific fine-tuning. Although the paper includes zero-shot evaluations, further controlled comparisons between pretrained and fine-tuned models would help clarify whether general-purpose video pretraining already provides meaningful physical understanding, or whether the observed capabilities primarily emerge from training on the synthetic simulation data.
Questions for authors

Refer to weakness.

2 comments

We thank the reviewer for their thoughtful comments and suggestions.

Novelty of the Findings

The reviewer writes:

The main findings largely align with intuitive expectations ... The work would be stronger if it revealed less obvious failure mechanisms or provided deeper insights into how current models learn and represent physical dynamics.

We emphasize that MPMWorlds pairs videos with their generating simulation programs and structured configurations across deformable and multi-material dynamics. The physics-video benchmarks discussed in Sec. 5 do not provide this paired code–video supervision. This lets us--for the first time--test these expectations rigorously by training both approaches on the same scenes, withholding specific physical information, and comparing pretrained and fine-tuned models (Tab. R4).

For example, explicit configurations raise the code models' mean mIoU from 0.235 to 0.384, while changing the video models' scores little (Tab. 4). The missing information also affects the code models differently: positions matter most for Qwen2.5-VL, and materials for Kimi K3. For Qwen, the median position error rises from 0.026 to 0.164 domain widths when the configuration is withheld. The new configuration baseline further separates physical inference from code generation (Tab. R3 in our response to xWS1). We will make these findings and the research questions the dataset enables clearer in Secs. 1 and 4 [C,D].

Real-World and 3D Scope

The reviewer writes:

it remains unclear whether the findings generalize to real-world images and videos, where physical properties must be inferred under complex visual conditions.

Our experiments study how the two approaches infer and extrapolate dynamics in a controlled 2D setting; they do not establish transfer to real videos or 3D physics. MPMWorlds extends 2D data beyond rigid-body dynamics to fluids, sand, snow, elastic and plastic solids, multi-material interactions, and emitters, alongside colliders and kinematic objects. Pairing each video with its configuration and code lets us control the information given to a model and evaluate against the simulator state. We agree that extending this pairing to real videos and 3D simulations is an interesting direction for future work. We will state the scope explicitly in the introduction and Sec. 6 [C,D].

Pretraining vs. Fine-Tuning

The reviewer writes:

the observed performance does not clearly distinguish physical knowledge acquired during pretraining from capabilities learned through task-specific fine-tuning.

Following the reviewer's suggestion, we evaluated Wan and LongCat without fine-tuning on the 105 scenes of Tab. 4, using the same inputs, inference settings, and future-frame scoring as for the fine-tuned models (Tab. R4).

Table R4: Pretraining vs. Fine-Tuning. Pretrained →\to fine-tuned; Full Configuration / Frames Only

MetricLongCatWan2.1-VACE
mIoU ↑\uparrow0.49 →\to 0.62 / 0.51 →\to 0.590.40 →\to 0.57 / 0.42 →\to 0.57
Anomaly ↓\downarrow0.88 →\to 0.92 / 0.85 →\to 0.850.98 →\to 0.76 / 0.97 →\to 0.71
Mass loss ↓\downarrow0.54 →\to 0.24 / 0.46 →\to 0.290.91 →\to 0.31 / 0.94 →\to 0.26
Flow ↑\uparrow0.09 →\to 0.27 / 0.09 →\to 0.250.07 →\to 0.18 / 0.07 →\to 0.19

Without fine-tuning, LongCat and Wan already score 0.51 and 0.42 mIoU from frames alone, but have temporal-anomaly rates of 0.85--0.98. Fine-tuning improves mIoU in all four comparisons (all 95% CIs exclude zero), with larger relative gains in mass loss and flow agreement. In Full Configuration, mass loss falls from 0.54 to 0.24 for LongCat and from 0.91 to 0.31 for Wan, while flow agreement rises from 0.09 to 0.27 and from 0.07 to 0.18. Both metrics also improve from frames alone. Fine-tuning reduces Wan's temporal anomalies but not LongCat's. Some of the gains may reflect adaptation to the rendering style, so this comparison does not isolate physical learning. We will add it to App. J [E].

Thanks for your reply. I would keep my scores, but after reading all the other review comments, I'm not going to champion this paper.

Reviewer_hYmrOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

MPMWorlds is a dataset of 2D Material-Point-Method physical simulations - 9,204 core scene templates (1,093 LLM-extended Parents + 8,111 Children from 35 human-written seeds, plus automated perturbations) - where every entry pairs executable Taichi source code, a structured YAML scene configuration, and a rendered video (512x512, 30 FPS, 10 s), covering fluids, deformable bodies, rigid bodies, emitters, motorized/kinematic objects, and Drucker-Prager sand. The paper uses the dataset to characterize, not rank, two families: code-generation VLMs (which infer physical parameters poorly from video but extrapolate stably in time) and video-diffusion VDMs (which recover geometry well but hallucinate dynamics and degrade over long horizons), across 3 VLMs and 3 VDMs with sample-matched comparisons and physically grounded mass/motion metrics calibrated against simulator state. The explicit goal is constructive: identify which physical properties each approach struggles to recover from video and carry forward in time.

Critical strengths
  1. The paired (code, config, video) triple structure is a genuinely useful resource: it enables controlled study of conditioning modalities and of explicit intermediate representations (code) on identical scenes, which existing physics-video datasets (mostly rigid-body or raw-video-only) do not offer. 2. The paper's stance - characterizing failure modes rather than ranking approaches - is scientifically honest and actionable, and the headline finding (code gen: stable but parameter-blind; video diffusion: geometric but hallucinating) is interesting and non-obvious.
Critical weaknesses
  1. The core experiments (which scenes fail, quantitative comparisons, physically grounded metric results) were not accessible at the time of this review; the headline conclusions rest on the abstract/intro statements, so the strength of the evidence (effect sizes, statistical care, metric validity) could not be verified. 2. All scenes are 2D synthetic simulations with a fixed rendering protocol (512x512/30 FPS/10 s); transferability of the findings to real videos and 3D physics is not addressed.
Questions for authors
  1. What are the headline numbers: on which scene classes does each approach fail most (materials, multi-body interactions, emitters), and how large are the gaps in the physically grounded metrics? 2. How is "physically grounded" defined for mass/motion metrics, and how tight is the calibration to simulator state? 3. How does side-information granularity (f(rho)) change each family's performance - which information most improves code gen vs VDM?
1 comment

We thank the reviewer for their thoughtful comments and suggestions.

The reviewer writes:

The core experiments (which scenes fail, quantitative comparisons, physically grounded metric results) were not accessible at the time of this review; the headline conclusions rest on the abstract/intro statements, so the strength of the evidence (effect sizes, statistical care, metric validity) could not be verified.

We would like to point the reviewer to the experiments supporting these conclusions in Secs. 4.1 and 4.2. Withholding positions and materials measures the code models' difficulty inferring physical properties (Figs. 3 and 5, Tab. 4). The code models' fewer temporal anomalies and the video models' higher spatial overlap are quantified in App. J (Tabs. 4 and 5), with long-horizon results in App. I (Figs. 12 and 13). App. H reports results by material (Fig. 11), and App. K calibrates the physical metrics against simulator state (Tab. 8). We will move the one-sample comparison (Tab. 4) into the main text [C].

Headline Numbers

The reviewer asks:

What are the headline numbers: on which scene classes does each approach fail most (materials, multi-body interactions, emitters), and how large are the gaps in the physically grounded metrics?

At one sample per model, video generation has higher mIoU and lower W-MAE and CTV, while code generation produces fewer temporal anomalies, loses less mass per material, and predicts speeds closer to the ground truth in all four input settings (Tab. 4). In the Full Configuration input regime, the code models produce 1.58--2.47 times fewer temporal anomalies, lose less mass per material (0.188--0.208 compared to 0.243--0.309), and predict speeds at 94.4% of the ground truth compared to 65.4--67.2% for the video models. They also violate the mass envelope less (0.044--0.094 compared to 0.098--0.159). We will add standard errors to Tab. 4 [C].

The approaches also differ across materials and scenes with several interacting objects. Code generation performs worst on sand, and video diffusion on elastic and plastic scenes (Fig. 11). With three or more objects, the code models score 0.253 mIoU and the video models 0.631, compared to 0.330--0.337 and 0.567--0.581 with one or two objects (Tab. R1 in our response to xWS1). On scenes with emitters, both approaches perform worse: code generation scores 0.286 compared to 0.321 without emitters, and video generation 0.467 compared to 0.607.

Physically Grounded Metrics

The reviewer asks:

How is "physically grounded" defined for mass/motion metrics, and how tight is the calibration to simulator state?

We use "physically grounded" because the metrics estimate mass and motion and we calibrate them against the simulator's particle state (App. K). Three measure whether material appears or disappears and whether its amount changes as in the ground truth; two measure how fast material moves and whether it moves in the correct direction. We compute all five from rendered videos so that both approaches can be evaluated. For executable programs, we also compute them from particle state. The video-based and particle-based measurements correlate at Pearson r = 0.904 (per-material mass loss) to 0.999 (mass-trajectory shape error; Tab. 8). For video models, these measurements remain approximations, as their predictions provide no particle state.

Side Information

The reviewer asks:

How does side-information granularity (f(rho)) change each family's performance - which information most improves code gen vs VDM?

Side information mainly helps code generation (Figs. 3 and 5, Tab. 4). From Frames Only to Full Configuration, the mean mIoU of the code models rises from 0.235 to 0.384, while that of the video models changes little (LongCat from 0.592 to 0.617, Wan from 0.574 to 0.565). Positions matter most for Qwen2.5-VL: its mIoU falls to 0.268 without positions, compared to 0.356 without materials. Materials matter most for Kimi K3, which falls to 0.317 without materials, compared to 0.351 without positions.

Real Video and 3D

The reviewer writes:

All scenes are 2D synthetic simulations…transferability of the findings to real videos and 3D physics is not addressed.

Our experiments study how the two approaches infer and extrapolate dynamics in a controlled 2D setting; they do not establish transfer to real videos or 3D physics. MPMWorlds extends 2D data beyond rigid-body dynamics to fluids, sand, snow, elastic and plastic solids, multi-material interactions, and emitters, alongside colliders and kinematic objects. Pairing each video with its configuration and code lets us control the information given to a model and evaluate against the simulator state. We agree that extending this pairing to real videos and 3D simulations is an interesting direction for future work. We will state the scope explicitly in the introduction and Sec. 6 [C,D].

Reviewer_xWS1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces MPMWorlds, a dataset of approximately 96K videos generated using Material Point Method (MPM) simulations, covering diverse physical phenomena such as fluids, deformable objects, and multi-material interactions. Each sample includes executable simulation code, structured scene configurations, and rendered videos. The authors evaluate video diffusion models and VLM-based simulation code generation approaches under different levels of available physical information, analyzing their respective strengths and failure modes in physical dynamics extrapolation.

Critical strengths
  1. The paired simulation code, scene configurations, and videos provide a useful foundation for studying physical dynamics inference and video prediction. The dataset also covers a reasonable variety of materials and interactions.
  2. Comparing explicit simulation-based prediction with direct video generation is meaningful. The experiments reveal different failure modes, particularly regarding spatial accuracy and temporal consistency.
  3. The authors put considerable effort into dataset construction and evaluation, including dataset validation, multiple model baselines, different input conditions, and additional physically grounded evaluation metrics.
Critical weaknesses
  1. The novelty and significance of the dataset are not sufficiently established. The dataset is primarily constructed using established physical simulation techniques and LLM-assisted scene generation. Although pairing executable programs with rendered videos is useful, the paper does not clearly demonstrate what unique research opportunities MPMWorlds offers compared with existing physical reasoning and video generation benchmarks.
  2. The experimental analysis does not fully exploit the potential of the dataset. Most experiments focus on comparing existing models under different input conditions, and several findings are relatively expected given the characteristics of the two approaches. More systematic analysis of physical complexity, material properties, and compositional generalization would provide deeper insights into the capabilities and limitations of current models.
  3. The practical value of the dataset scale remains unclear. Despite constructing approximately 96K simulations, the authors conduct their main experiments on a 10K subset. Without experiments examining the effects of training data scale and scene diversity, it is difficult to assess whether the larger dataset provides substantial additional benefits.
  4. The evaluation protocol raises some concerns about the fairness of the comparison. The reported mIoU includes observed frames, which are directly preserved by VDMs but must be reconstructed by VLMs, potentially affecting the comparison of spatial accuracy. Moreover, the main experiments select the best VLM result from 15 candidates separately for each metric, while VDMs use a single sample. Although additional comparisons are provided in the appendix, evaluating only future frames under more consistent sampling conditions would make the conclusions more convincing.
  5. The analysis does not clearly identify the bottlenecks of simulation-based prediction. VLM failures may arise from inaccurate inference of scene geometry and physical properties, or from errors in generating executable simulation code. The current experiments do not sufficiently distinguish these sources of error. A baseline that predicts structured scene configurations from videos and executes them using a fixed simulator could help clarify where the main difficulties lie.
Questions for authors
  1. Could the authors further clarify the unique contributions of MPMWorlds compared with existing simulation-based physical reasoning and video generation benchmarks? What research questions does this dataset uniquely enable?
  2. Could the authors provide a more systematic analysis of how model performance varies with scene complexity, material properties, and multi-material interactions? Such analysis could help reveal more generalizable insights into the limitations of current models.
  3. Given that MPMWorlds contains approximately 96K simulations, why are the main experiments conducted on only a 10K subset? Could the authors either provide scaling experiments to demonstrate the benefits of additional training data and scene diversity, or further justify why the 10K subset is sufficient to support the paper's conclusions?
  4. Since VDMs directly preserve the observed video prefix while VLMs must reconstruct it, could the authors report evaluation results computed exclusively on future frames? This would help clarify whether the observed differences in spatial accuracy remain consistent.
  5. How much do the main conclusions depend on the best-of-15 candidate selection used for VLMs? Although sample-matched results are provided in the appendix, could the authors further discuss the impact of different sampling and selection strategies on the comparison?
  6. Could the authors further disentangle errors in physical state inference from errors in simulation code generation? For example, evaluating a baseline that predicts structured scene configurations from videos and executes them using a fixed simulator could help identify the main performance bottlenecks.
4 comments

We thank the reviewer for their thoughtful comments and suggestions.

Distinctive Contribution

The reviewer asks:

Could the authors further clarify the unique contributions of MPMWorlds compared with existing simulation-based physical reasoning and video generation benchmarks? What research questions does this dataset uniquely enable?

We emphasize that MPMWorlds provides both a wider range of materials than rigid-body physics-video benchmarks and paired code–video training data. It covers fluids, elastic and plastic solids, snow, sand, emitters, and multi-material interactions. The physics-video benchmarks discussed in Sec. 5 and App. A do not pair their videos with the simulation programs that generated them. Among code-based benchmarks, VisPhyWorld studies rigid-body reconstruction, while PhysCodeBench generates simulations from text descriptions. Our paired programs, configurations, and videos let us train and compare code and video generation on the same future-prediction task, then withhold positions, materials, or the whole configuration to measure what each approach struggles to infer from video (Figs. 3, 5, and 6). The paired data also enable the configuration-prediction experiment suggested by the reviewer (Tab. R3). We believe this combination offers a unique opportunity to study how code and video models infer physical properties and predict future dynamics under controlled inputs. We will make these contributions clearer in Secs. 1 and 5 [C,D].

Scene Complexity and Materials

The reviewer asks:

Could the authors provide a more systematic analysis of how model performance varies with scene complexity, material properties, and multi-material interactions?

Following the reviewer's suggestion, we grouped the results of Tab. 4 by object count, material composition, emitters, and kinematic objects (Tab. R1). In each of the nine scene groups and all four input settings, both video models have higher mIoU and both code models produce fewer temporal anomalies. Code generation also loses less mass per material in 28 of these 36 comparisons.

The mIoU advantage of video generation grows by 0.137 on scenes with three or more objects and by 0.104 on scenes with multiple materials. The code models also have higher CTV with three or more objects (+0.147), and lower W-MAE (-0.144) and CTV (-0.084) with kinematic objects. These differences remain significant after correcting for multiple comparisons. By material, code generation performs worst on sand, and video diffusion on elastic and plastic scenes (Fig. 11). We will add this analysis to the appendix and limit the compositional-scaling caption of Fig. 6 to the illustrated example [C,E].

Table R1: Scene Structure. One sample per model; future frames; 105 scenes of Tab. 4; VLM and VDM values average both models of each type over the four input regimes; objects are dynamic bodies and emitters, and a kinematic object is a collider with prescribed motion.

SubgroupnmIoU ↑\uparrow VLMmIoU ↑\uparrow VDMW-MAE ↓\downarrow VLMW-MAE ↓\downarrow VDMCTV ↓\downarrow VLMCTV ↓\downarrow VDMAnom ↓\downarrow VLMAnom ↓\downarrow VDM
1 object370.3300.5670.4990.3010.3730.2370.3970.809
2 objects450.3370.5810.4530.2680.3480.2330.4030.838
3+ objects230.2530.6310.5750.2920.5060.2310.2760.824
Single material520.3210.5390.4730.2790.3700.2390.3890.800
Multiple materials530.3120.6340.5190.2910.4120.2280.3580.850
No emitter900.3210.6070.5150.2920.3880.2280.3700.822
Emitter150.2860.4670.3830.2400.4100.2690.3920.843
No kinematic object350.2960.6270.5920.3720.4480.2340.3250.774
Kinematic object700.3260.5670.4480.2410.3630.2340.3980.851

10K Subset

The reviewer asks:

Given that MPMWorlds contains approximately 96K simulations, why are the main experiments conducted on only a 10K subset?

Most of the dataset's scene diversity comes from its 9,204 scene templates, which contain 3,709 combinations of object types, materials, and mechanisms (App. N). The remaining simulations are parameter perturbations of these templates, such as object positions. We emphasize templates in the 10K subset to cover this diversity while limiting the cost of training and evaluating several models across four input settings. We agree that testing whether additional positional variants improve the inference of initial object positions is an interesting direction for future work [C,D].

Future Frames

The reviewer asks:

Since VDMs directly preserve the observed video prefix while VLMs must reconstruct it, could the authors report evaluation results computed exclusively on future frames?

All comparisons in the paper score only future frames. We exclude the observed prefix from scoring because it is provided to every model as input. We use the prefix only in the best-of-k selection procedure for VLM candidates. The video models retain their mIoU advantage when we score only future frames (Tab. R2). We will state the evaluation window explicitly in Sec. 3.4, App. E, and the relevant captions [C].

Table R2: Evaluation Window. mIoU; Full Configuration; 105 scenes; LongCat is shown for one run (five-seed mean 0.617 in Tab. 4)

ModelAll framesFuture only, as reported
Qwen2.5-VL (FT)0.4040.384
Kimi K30.4130.385
LongCat (FT)0.6860.611
Wan2.1-VACE (FT)0.6500.565

Sampling and Selection

The reviewer asks:

How much do the main conclusions depend on the best-of-15 candidate selection used for VLMs?

As discussed in Sec. 4.2 and App. J, the main findings hold with one sample per model (Tab. 4) and at a five-sample budget (Tab. 5). At both budgets, video generation has higher mIoU and lower W-MAE and CTV, while code generation produces fewer temporal anomalies and loses less mass per material. At one sample, the code models also predict speeds closer to the ground truth in all four input settings. Their stronger dependence on scene information also holds in the matched comparison (Sec. 4.2). Selecting among five programs improves the mIoU of Qwen2.5-VL from 0.309 to 0.402 and Kimi K3 from 0.324 to 0.447, still below the 0.570--0.604 of the video models.

The code models' advantage on W-MAE and CTV in Fig. 3 depends on selection over 15 programs; it disappears at both smaller budgets. Qwen2.5-VL also collapses less than both video models without selection (Tab. 4). We will move Tab. 4 into the main text, present Fig. 3 as a per-metric selection upper bound, and revise Sec. 4.1 to distinguish the advantages of each approach [C].

Scene Inference vs. Implementation

The reviewer asks:

Could the authors further disentangle errors in physical state inference from errors in simulation code generation?

Following the reviewer's suggestion, we executed model-inferred scene configurations with a fixed compiler on all 105 test scenes of Tab. 4 (Frames Only). We asked Kimi K3 to predict configurations directly from the frames and extracted Qwen2.5-VL's configurations from its generated programs. We execute each configuration with the deterministic compiler from Tab. 7, which scores 0.369 mIoU on the ground-truth configurations (Tab. R3).

Table R3: Configurations and Programs. Future frames; all 105 scenes; failed configuration pipelines scored as zero mIoU and full collapse; one sample per model

Executed simulationmIoU ↑\uparrowObject collapse ↓\downarrow
Compiler from Tab. 7, ground-truth configuration0.3690.169
Compiler from Tab. 7, Qwen2.5-VL (FT) configuration0.2640.233
Qwen2.5-VL (FT) program0.2280.337
Compiler from Tab. 7, Kimi K3 configuration0.2390.270
Kimi K3 program0.2590.434

Replacing the ground-truth configuration with the inferred one lowers mIoU by 0.105 for Qwen and 0.130 for Kimi. Executing the models' own programs instead lowers it by a further 0.036 for Qwen, with no detectable loss for Kimi (-0.020). For both models, the first drop is significantly larger than the second, by 0.069 (95% CI 0.023--0.115) for Qwen and 0.150 (0.095--0.211) for Kimi. These comparisons suggest that, relative to the compiled ground-truth configuration, most of the mIoU loss is attributable to physical inference.

On object collapse, the models' own programs collapse more than their compiled configurations, significantly for Kimi (0.164, 95% CI 0.042--0.285), but not for Qwen (0.104, -0.001 to 0.210). This suggests that coding ability also affects stability. We will add this analysis to App. J [E].

Thank you for the clarifications and additional experiments. The rebuttal partially addresses my concerns. Scoring only future frames resolves my question about the evaluation window, and the matched-sampling comparisons clarify which findings depend on candidate selection. Table R3 helps distinguish scene inference errors from code generation errors, while Table R1 shows how performance varies with object count and material composition. I will take these clarifications and results into account in my final assessment.

Comments on the paper

1 comment

We thank each of the reviewers for their constructive comments and suggestions, which we will use to improve the quality of our paper. They recognize that pairing simulation code, scene configurations, and videos is "a genuinely useful resource" (hYmr) and "a useful foundation for studying physical dynamics inference and video prediction" (xWS1). They find our research question "important and timely" (PJ86), the comparison between code generation and video generation "meaningful" (xWS1, PJ86), and our headline finding "interesting and non-obvious" (hYmr). We respond to each reviewer below and will update the paper accordingly, adding clarifications [C], discussions [D], and experiments [E] to the text.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.