acceptodds
Under review as a conference paper at ICLR 2027

PhysEditWorld: A Large-Scale Dataset Toward Physics-Editable World Models

Abstract

Game world models can generate visually plausible gameplay, yet controlling how motion responds to changes in physical parameters remains challenging. Studying this capability requires comparable observations in which a physical parameter varies while the scene and interaction remain fixed. We introduce PhysEditWorld, a multimodal dataset for physics-editable game world modeling, together with our open-source UE5 DataFactory plugin. The current release focuses on gravity. The plugin replays recorded interactions under different gravity settings while preserving the scene, initial state, action trace, controller configuration, and camera policy. This design produces matched motion sequences with explicit gravity labels and synchronized visual, action, camera, and engine-state annotations. PhysEditWorld spans 12 authored UE5 scenes, 100+ hours of gameplay interactions, and 60M+ rendered frames. Fine-tuning on PhysEditWorld strengthens gravity-dependent motion responses in video generation and improves Qwen3-VL-8B gravity-class accuracy from 24.71% to 95.29% on held-out scenes. The publicly released dataset and plugin support controlled data collection, training, and evaluation of game world models under explicit physics edits.

Primary area:
generative models
Keywords:
world models, physics-editable generation, controllable video generation, multimodal datasets, physical reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2BVqOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces PhysEditWorld, a dataset for studying whether game world models can change generated motion in response to gravity settings. The authors use a UE5 plugin to record interactions and replay them under different gravity levels while keeping the scene, initial state, controller, and camera policy fixed. This allows motion across the resulting videos to be compared directly. The dataset covers 12 scenes, over 100 hours of interaction, and more than 60 million rendered frames, with action, camera, and engine-state annotations. The main contributions are the matched data and collection pipeline, along with an evaluation that separates event success, usable pair coverage, and correct gravity-response ordering. Fine-tuning Wan2.2, LingBot-World, and Qwen3-VL on the data yields reported improvements in gravity-conditioned generation and gravity recognition on unseen scenes. Engine comparisons and controlled-input tests are also used to check the reliability of the evaluation.

Critical strengths

The main value of this paper is the dataset and its collection pipeline. The authors combine action recording, matched replays under different gravity settings, rendering from multiple views, and state annotations in a UE5 tool, providing a reusable way to build the data. The paired videos share the same interaction conditions, allowing researchers to examine more directly how changes in physical parameters affect motion. I think this dataset and the accompanying tools can help advance world model research, particularly by supporting training and evaluation of how models respond to specified physical conditions.

Critical weaknesses

I find the coverage of “physics editing” fairly narrow. The data and experiments mainly focus on jumping and falling under different gravity settings. To me, what matters is whether the model follows the specified physical conditions. Real-world scenes can serve as a baseline, while motion in some visual effects scenes can also be described through different external forces, elasticity, or damping. Including these parameters and their combinations would make the dataset more broadly useful and potentially increase its impact. The current experiments mainly show that models respond to gravity changes; there is not yet enough evidence that the approach works for other physical effects.

Questions for authors

Please refer to the Weaknesses section.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 2BVq

We thank the reviewer for recognizing the value of the matched data and the UE5 collection pipeline. We would like to clarify that the dataset already includes physical variations beyond gravity. The supplementary material provides additional recordings with changes to friction and restitution. These are collected data, rather than proposed extensions of the tool.

Gravity is the focus of the main experiments, not the sole physical parameter covered by the data. We chose it as a controlled setting for evaluating whether models respond to specified physical conditions, using matched interactions and an evaluation that separates event success, usable pair coverage, and response ordering. We agree with the reviewer that following the specified physical condition is the central goal.

The additional recordings in the supplementary material broaden the data coverage beyond the gravity-conditioned jumping and falling examples emphasized in the main text. We will make these recordings more prominent in the revision and revise the scope description to distinguish the physical parameters already represented in the data from those evaluated quantitatively in the model experiments. In particular, our reported model results concern gravity-conditioned generation and recognition; the supplementary examples document the broader physical variations available in PhysEditWorld.

We appreciate the suggestion to study additional parameters and their combinations. The matched-replay protocol provides a common collection framework for these settings, allowing physical conditions to vary while preserving the interaction setup. We will clarify this broader data contribution without extending the model-performance claims beyond the conditions evaluated.

Thank you for the detailed response and additional experiments. I will maintain my original rating.

Reviewer_nMinOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PhysEditWorld introduces a dataset for studying how game world models respond to changes in gravity. Its main contribution is a collection setup that replays the same actions in the same scene under different gravity settings, with synchronized observations and engine states. The paper provides a UE5 collection tool and demonstrates improvements in video generation and gravity prediction after fine-tuning.

Critical strengths
  1. The useful idea is to bring matched physical interventions into interactive gameplay. Keeping the scene and player actions fixed makes it possible to study how changing gravity affects the resulting motion.

  2. The UE5 collection tool makes the dataset easier to extend to new game scenes. Improvements in both generation and gravity prediction provide initial evidence that the collected data are useful.

Critical weaknesses
  1. There is no ablation testing whether matched replay is better than independently collected, gravity-labeled rollouts under the same training budget. As the authors state in line 53, the comparisons “do not, by themselves, isolate the contribution of replay matching from the other benefits of fine-tuning.”

  2. The range of gameplay covered is still limited despite the large frame count. The dataset contains 12 authored scenes and only gravity edits, while generation is tested on 25 replay groups, leaving its usefulness across more varied game interactions insufficiently demonstrated.

Questions for authors
  1. How does training on matched replays compare with an equally sized set of independently collected, gravity-labeled rollouts under the same training budget?

  2. Do the improvements hold across substantially different gameplay settings and actions, beyond the current test scenes and examples?

2 comments
AuthorsOpenReview ↗

Response to Reviewer nMin

We thank the reviewer for recognizing the value of matched physical interventions and the extensibility of the UE5 collection tool. We address the two questions below.

1. Matched versus independently collected data under the same training budget.

We have added the requested controlled comparison. The matched condition replays the same initial states and action traces across gravity settings, while the independent condition uses separately collected, gravity-labeled interaction instances. We control video count, total duration, and the distributions of scenes, action categories, gravity settings, and camera configurations. Both conditions use the same backbone, initialization, conditioning inputs, trainable parameters, and optimization budget.

Both models are evaluated on the same 25 held-out replay groups. All values are percentages.

Model and training dataEvent success ↑Pair coverage ↑A_valid ↑A_all ↑
Wan, matched replays83.676.981.762.8
Wan, independent rollouts74.264.967.146.3
LingBot, matched replays78.472.376.955.6
LingBot, independent rollouts64.966.465.043.6

Matched collection improves event success by 9.4 percentage points for Wan and 13.5 points for LingBot. Gravity-ordering accuracy on usable pairs increases by 14.6 and 11.9 points, respectively, together with higher pair coverage and coverage-weighted ordering scores. These consistent gains across two backbones support the contribution of matched collection beyond the benefits of fine-tuning on independently collected gravity-labeled data. We will include this ablation in the revision.

2. Transfer to different visual settings and gameplay interactions.

Our evaluation examines transfer in two complementary settings. First, the quantitative experiments use a scene-disjoint split, with nine training scenes and three held-out scenes. All gravity variants and camera views belonging to a replay group remain within the same split, so the held-out results evaluate scenes excluded from fine-tuning.

Second, we test generation from GPT-Image-generated scene images, as illustrated by the rooftop example in Section 5.4 and Figure 4. These inputs introduce visual settings beyond the fine-tuning data. The resulting videos qualitatively show gravity-conditioned motion in the new setting, providing evidence that the learned response can transfer beyond the original rendered scene appearances.

We will make these examples and their comparison across gravity conditions more prominent in the revision. Their role is to demonstrate visual transfer and observable motion responses; the held-out engine scenes provide the controlled quantitative evaluation. We also distinguish a change in scene appearance from a change in action category: the generated-scene examples support the former, while broader action coverage remains a useful direction for extending the evaluation.

The added equal-budget ablation directly addresses the contribution of matching, while the held-out-scene results and generated-scene examples show its utility across controlled and visually different test settings.

Thank you for the additional experiments. The new ablation addresses my concern about matched replay, but I still find the overall novelty limited. My second concern is only partly resolved. The results support transfer to new scenes, while generalization to different gameplay interactions remains unclear. I therefore maintain my original score.

Reviewer_dxqzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PhysEditWorld, a multimodal dataset for physics-editable game world modeling, together with an open-source UE5 DataFactory plugin. By organizing data into matched replay groups that preserve the scene, initial state, action trace, controller configuration, and camera policy while varying gravity, the dataset enables controlled comparisons of gravity-dependent motion. PhysEditWorld contains 12 scenes, over 100 hours of gameplay interactions, and more than 60 million rendered frames. Experiments on gravity-conditioned video generation, action-conditioned first-person rollouts, and video-language gravity prediction demonstrate its utility, with Qwen3-VL-8B improving gravity-class accuracy from 24.71% to 95.29% on held-out scenes.

Critical strengths
  1. The matched replay protocol enables controlled gravity interventions by preserving scene and interaction conditions, providing a valuable framework for studying physical controllability in game world models.

  2. PhysEditWorld provides large-scale multimodal annotations and demonstrates improvements across generation and gravity prediction tasks, supporting its practical utility.

Critical weaknesses
  1. The absence of data-size-matched comparisons with unmatched replay data makes it difficult to isolate the independent contribution of the matched replay protocol.

  2. The VGGT-based acceleration proxy and gravity ordering metrics do not sufficiently establish quantitative agreement between generated motions and engine-reference physical trajectories.

  3. The evaluation covers only three held-out scenes, while OOD appearance transfer and unseen gravity multipliers lack systematic quantitative validation.

Questions for authors

See Weakness

2 comments
AuthorsOpenReview ↗

Response to Reviewer dxqz

We thank the reviewer for recognizing the value of matched physical interventions and the multimodal dataset. We have added a controlled training comparison and clarify the interpretation and scope of our evaluation below.

1. Matched versus independently collected data.

We have added an equal-size, equal-budget comparison to isolate the benefit of matched collection.

The matched set replays the same initial states and action traces across gravity settings. The independent set uses separately collected interaction instances. We control video count, total duration, and the distributions of scenes, action categories, gravity settings, and camera configurations. Both conditions use the same backbone, initialization, conditioning inputs, trainable parameters, and optimization budget. This compares different collection strategies rather than simply regrouping identical training clips.

Both models are evaluated on the same 25 held-out replay groups. All values are percentages.

Model and training dataEvent success ↑Pair coverage ↑A_valid ↑A_all ↑
Wan, matched replays83.676.981.762.8
Wan, independent rollouts74.264.967.146.3
LingBot, matched replays78.472.376.955.6
LingBot, independent rollouts64.966.465.043.6

Matched collection improves event success by 9.4 percentage points for Wan and 13.5 points for LingBot. Ordering accuracy on usable pairs increases by 14.6 and 11.9 points, respectively, with improvements in pair coverage and the coverage-weighted score as well. The consistent gains across both backbones support the contribution of matched collection beyond ordinary fine-tuning. We will include this comparison in the revision.

2. Interpretation of the motion evaluation.

Our evaluation measures whether generated motion responds consistently to explicit gravity changes under controlled inputs. Section 5.2 explains how scale and orientation ambiguity in monocular reconstruction are handled through common-reference calibration, and distinguishes relative gravity response from absolute trajectory accuracy.

Table 2 reports four complementary quantities. Event success measures whether the target motion occurs. Pair coverage records whether both videos contain successful events with valid motion intervals and usable common calibration. A_valid evaluates gravity ordering on these comparable pairs, while A_all incorporates coverage into the overall score. Together, these metrics assess both the ability to generate the intended event and the consistency of its response to gravity, while accounting for unsuccessful events and unusable measurements.

The results show improvements across all four measures for both Wan and LingBot. We interpret this as evidence of stronger gravity-conditioned motion responses. Quantitative agreement in absolute motion magnitude is a separate evaluation target; we will make this distinction explicit in the revision. The synchronized engine states and matched replay structure provide the reference information needed to pursue that trajectory-level evaluation.

3. Generalization across scenes, appearances, and gravity settings.

The evaluation uses a scene-disjoint split: 1,530 tuples from nine training scenes and 170 tuples from three held-out scenes. All gravity variants and camera views from the same replay group remain within one split. The aggregate generation results cover 25 held-out replay groups, testing the learned response on scenes excluded from fine-tuning.

Section 5.4 and Figure 4 complement this quantitative evaluation with an appearance-transfer example using a generated rooftop image outside the fine-tuning data. This tests generation from a new visual input, whereas the engine-based experiments provide controlled physical comparisons. We will clearly distinguish these complementary settings in the revision.

The gravity-setting case studies further illustrate how motion changes with the specified condition. We will identify these as case-level evidence and distinguish them from systematic unseen-gravity interpolation and extrapolation tests.

Taken together, the scene-disjoint results, appearance-transfer example, and new matched-versus-independent comparison provide complementary evidence for the dataset’s utility. We will sharpen the presentation so that each experiment’s purpose and supported conclusions are clear.

The added matched-versus-independent comparison largely addresses my first concern. The consistent improvements across both backbones provide useful evidence for the benefit of matched collection. Please include this comparison and the relevant experimental details in the revised paper.

For the motion generalization, I appreciate the authors’ acknowledgment of these distinctions and am comfortable treating them as limitations. The current motion metrics support more consistent responses to changes in gravity, but they do not establish quantitative agreement between generated trajectories or accelerations and the engine references. Similarly, the results on three held-out scenes provide some evidence of cross-scene generalization, while the appearance-transfer example and gravity-setting case studies remain limited evidence for broader generalization. They do not replace systematic evaluation of OOD appearance transfer or interpolation and extrapolation to unseen gravity settings.

One point in the new table would benefit from extra clarification. If A_all is calculated as Pair coverage multiplied by A_valid, the independent-rollout rows do not appear to match the reported values. For example, the Wan independent-rollout result would be approximately 43.5%, compared with the reported 46.3%. This may reflect averaging across subgroups or a different aggregation scheme, so please clarify the calculation and verify the table.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.