acceptodds
Under review as a conference paper at ICLR 2027

WMEval: Evaluating Interactive World Models Grounded on Physical Simulations

18 views

Abstract

Interactive world models hold immense potential for high-stakes applications in robotics, game engines, and immersive digital twins, yet existing benchmarks primarily focus on perceptual similarity rather than the physical-causal coherence essential for real-world utility. We introduce WMEval, a comprehensive benchmark that evaluates interactive world models against Unity-based physical simulations. By leveraging Unity as a high-fidelity ground-truth oracle, we shift evaluation beyond perceptual visual assessment toward the objective state-based verification required for downstream applications. Spanning 9 diverse scenes, WMEval assesses action controllability, scene and character consistency, prompt adherence, kinematics, fluid mechanics, physical causality, and lighting. By executing matched action sequences in both the simulator and the world model, WMEval enables evaluation at both the perceptual level and the physical-causal level, using simulator-derived states as reference. Experiments on state-of-the-art models reveal significant rooms for improvements in dimensions that remain hidden by standard perceptual metrics but are vital for real-world deployments.

Primary area:
datasets and benchmarks
Keywords:
Interactive World Models, Video Diffusion Models, Physical Simulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZLKvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WMEval evaluates interactive world models using nine Unity scenes, with simulator states and event traces as references. It combines metrics for action control, consistency, physics, and causality with perceptual measures. Experiments on six models show that visual similarity and immediate action following do not reliably indicate correct physical or causal outcomes. The authors also assess evaluator reliability through simulator-based calibration and human checks.

Critical strengths
  1. Using Unity provides reproducible test conditions and access to physical states such as velocity, momentum, and collision events. This gives the evaluation a concrete reference beyond visual similarity or subjective judgments.
  2. The experiments show that visually plausible outputs can still fail to preserve interactions and causal chains. These examples help identify weaknesses that standard video-quality metrics may miss, making the benchmark useful for diagnosing current models.
Critical weaknesses

The simulation-grounded design provides controlled and reproducible evaluation, but the extent to which its findings generalize beyond the selected environments remains unclear. Although the nine Unity scenes cover diverse evaluation dimensions, they represent a limited set of simulated scenarios and dynamics. Without validation in real-world settings or across a broader range of simulators, it remains difficult to determine how well the reported failures reflect general physical limitations rather than mismatches with the specific environments and dynamics used for evaluation. This limits the broader conclusions about real-world applicability.

Questions for authors
  1. What evidence supports extending the findings from these nine Unity scenes to real-world physical behavior? Could some errors reflect a mismatch with the simulator’s specific dynamics rather than a violation of general physical laws?
  2. How does the evaluation distinguish failure to generate the required interaction from incorrect physics after the interaction occurs? Do the main conclusions hold when physical scores are evaluated separately for successful, measurable interactions and failed interactions?
  3. How are actions made comparable across models with different control interfaces, and to what extent could model-specific action mappings affect the reported physical-causality results?
2 comments
AuthorsOpenReview ↗

Rebuttal Response to Reviewer ZLKv

We thank the reviewer ZLKv for recognizing that Unity gives reproducible conditions and a concrete reference, and that the experiments expose failures that video-quality metrics miss. We address each concern below.


W1 & Q1: Generalization beyond nine Unity scenes, and simulator-specific dynamics

We would like to clarify WMEval is simulator-scoped by design. Unity is not meant to reproduce the full visual or physical complexity of the real world. We use it because simulator allows controlled interventions and exposes exact, synchronized physical states, which is sufficient to test first-order phenomena: motion, collisions, object persistence and causal transitions. This is the same scope as simulation-grounded benchmarks such as Habitat, iGibson, BEHAVIOR and Isaac Gym, and the revision states it explicitly. Two results show that the findings are not artefacts of Unity's particular dynamics.

By comparing against simulator states, WMEval can still separate today's world models well. For example:

  • Ball trajectory and momentum (BilliardGame). Each generation is scored against the best-matching of 26 Unity references that vary the strength, timing and direction of the strike. Ranges are over five seeds, and the two models do not overlap on either metric.
Mechanics error (%)Momentum error at contact (%)
Unity video5.02.8
Matrix-Game 2.04.2–4.486–88
Project Genie6.9–23.512–82
  • Liquid mask and area (LiquidContainer). Models differ in whether they keep the container at all, and in how stable the liquid area is where they do (ranges over reportable action profiles).
Clips with the container keptArea jitter
Unity video–0.002
HY-WorldPlay13 of 200.003–0.026
LingBot-World18 of 200.036–0.266
Project Genie12 of 200.060–0.274
Matrix-Game 2.09 of 190.082–0.088
  • Event trace (BeamDomino). Project Genie preserves the full causal chain in 3 of 15 rollouts, against 13 of 15 for the Unity video (p = 0.0007).

  • Car trajectory (RacingCar). Project Genie keeps the car tracked in 100% of frames, Matrix-Game 2.0 in 25–36%.


Q2: Failure to generate the interaction versus incorrect physics after it

WMEval separates the two by gating: physics is scored only where the object of the interaction is present, and the gate is reported as a result of its own (Appendix C.1; Appendix B.4.1). Both kinds of failure occur:

  • Interaction not generated. The object is not there to measure, so no physics score is given.
SceneModelTarget kept in view
LiquidContainerMatrix-Game 3.01 of 20 clips
RacingCarMatrix-Game 2.025–36% of frames
  • Interaction generated, physics scored. Position error against the best-matching reference (BilliardGame) and liquid area jitter (LiquidContainer).
SceneModelTarget kept in viewErrorUnity video
BilliardGameMatrix-Game 2.0100% of frames0.24 m0.008 m
BilliardGameProject Genie74–85% of frames0.38 m0.008 m
LiquidContainerHY-WorldPlay13 of 20 clips0.003–0.0260.002
LiquidContainerLingBot-World18 of 20 clips0.036–0.2660.002

Videos where interactions are not generated well barely change model ranking. Most video generations fail to produce interactions due to poor action following. We compared the models in 58 settings (one scene and one metric each) in three ways: leaving those videos out, keeping whatever score they produced, or giving them the worst score. In 43 settings the ranking of models is the same all three ways, almost always because no video in that setting was unscorable.

AuthorsOpenReview ↗

Rebuttal Response to Reviewer ZLKv (Part 2)

Q3: Comparability of actions across interfaces, and the effect on physical causality

Ideally every model would receive physically equivalent controls. Today that is not attainable: the models expose different interfaces (keys, camera poses, text) and each learned its own mapping from input to motion. There is little we can do on the modeling side, but we have tried our best to standardize in the inference setting.

We argue they can still be compared when actions are matched by semantics and timing (Section 3.1.2) and the effect of a control mismatch is measured. For example, we allow for a mismatch in force or timing by scoring each BilliardGame generation against the best-matching of 26 Unity references that vary the strength, timing and direction of the strike:

Error against the best-matching referencePosition (m)Mechanics (%)Momentum change at contact (%)
Unity video0.0085.02.8
Matrix-Game 2.00.244.388
Project Genie0.3811.556

Even with this allowance and the fact that the interfaces are different, WMEval separates the models today, from the simulator and from each other. Matrix-Game 2.0 has the lower mechanics error in all five seeds (4.2–4.4% against 6.9–23.5%), Project Genie has the lower contact error in all five (12–82% against 86–88%), and both stay far from the Unity video in position and at contact.

We expect the comparison to tighten as world models mature: as their capabilities and control interfaces converge, the same action can be delivered to every model more directly and the adapter gap shrinks. In the revision we will add a table listing each model's native input and control type per scene, and report best-match errors alongside the original ones.

Reviewer_WRjDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces WMEval, a benchmark for evaluating interactive world models beyond visual quality. The main idea is to use Unity simulations as physical references and compare how world models respond to the same action sequences. The benchmark includes nine scenes covering action controllability, scene consistency, motion and mechanics, fluid dynamics, physical causality, and lighting. The authors evaluate six existing world models and show that, while these models can often generate visually plausible videos and follow basic actions, they still struggle with accurate physical interactions, long-term consistency, and multi-step causal reasoning. Overall, the paper highlights the gap between generating realistic-looking videos and actually simulating a physically consistent world.

Critical strengths
  • The paper addresses an important gap in current world model evaluation. Moving beyond visual quality to physical consistency and causal behavior is a useful direction, especially for robotics and interactive simulation.
  • The nine Unity scenes cover a reasonable range of physical interactions. I also appreciate the effort to calibrate the evaluators and discuss measurement failures rather than simply reporting scores.
Critical weaknesses
  • Different models have very different control interfaces, and mapping actions at the semantic level does not necessarily mean they receive equivalent physical inputs. For example, controlling a ball with physical forces is quite different from prompting a model to generate the corresponding motion. This makes some cross-model comparisons less convincing.
  • Most physical quantities are recovered from generated videos using CV models rather than measured directly. Errors in detection, tracking, and segmentation could affect the results. This is particularly concerning for fluid mechanics, where 46 out of 99 clips fail the validity gate. Also, VLM-based causal QA does not necessarily reflect whether the underlying physical dynamics are correct.
Questions for authors
  • How do the authors verify that the action adapters produce physically equivalent controls across different models? Could differences in control interfaces explain part of the performance gap?
  • How sensitive are the reported physical scores and model rankings to errors from the CV evaluators? In particular, how should we interpret the fluid results given the large number of invalid clips?
  • Is there any evidence that better WMEval scores translate into better performance in downstream closed-loop tasks? This would help establish the practical value of the benchmark.
1 comment
AuthorsOpenReview ↗

Rebuttal Response to Reviewer WRjD

We thank the reviewer WRjD for recognizing the importance of evaluating physical and causal behaviour and for appreciating our evaluator calibration. We address each concern below.


W1 & Q1: Action adapters and physically equivalent controls

The reviewer has raised a valid point on control interface standardization. Ideally every model would receive physically equivalent controls. Today that is not attainable: the models expose different interfaces (keys, camera poses, text) and each learned its own mapping from input to motion. There is little we can do on the modeling side, but we have tried our best to standardize in the inference setting.

We argue they can still be compared when actions are matched by semantics and timing (Section 3.1.2) and the effect of a control mismatch is measured. For example, we allow for a mismatch in force or timing by scoring each BilliardGame generation against the best-matching of 26 Unity references that vary the strength, timing and direction of the strike:

Error against the best-matching referencePosition (m)Mechanics (%)Momentum change at contact (%)
Unity video0.0085.02.8
Matrix-Game 2.00.244.388
Project Genie0.3811.556

Even with this allowance and the fact that the interfaces are different, WMEval separates the models today, from the simulator and from each other. Matrix-Game 2.0 has the lower mechanics error in all five seeds (4.2–4.4% against 6.9–23.5%), Project Genie has the lower contact error in all five (12–82% against 86–88%), and both stay far from the Unity video in position and at contact.

We expect the comparison to tighten as world models mature: as their capabilities and control interfaces converge, the same action can be delivered to every model more directly and the adapter gap shrinks. In the revision we will add a table listing each model's native input and control type per scene, and report best-match errors alongside the original ones.


W2 & Q2: Sensitivity to CV evaluator errors, the fluid gate, and VLM causal QA

We measured each source of evaluator error. Most results are stable, and we will not rank the ones that are not.

Evaluator error is small against action trajectory differences measured. Each Unity video is scored against its own trajectory and against the other trajectories of its scene.

MetricOwn trajectoryOther trajectories
Position error, RacingCar0.14 m22.6 m
Mechanics error, RacingCar1.1%45.2%
Mechanics error, BilliardGame4.4%29.8%

Scores do not move with appearance. A Unity BilliardGame clip is recolored, blurred or resampled with its motion unchanged.

MetricOriginalColourHueBlurResample
Position error (m)0.0080.0070.0080.0100.012
Mechanics error (%)5.05.05.15.25.2

The shifts are one to two orders of magnitude smaller than the gap to models or to other trajectories

Fluid gate. A rejection is a property of the generated video, not a random evaluator error. A clip is rejected when the liquid mask is lost in any frame or the container is rendered at under half or over twice the reference size, and where the bottle is absent from most frames, inspection confirms the model lost it (Appendix B.4.1). Rejections therefore separate the models: LingBot-World 2 of 20 clips, HY-WorldPlay 7, Project Genie 8, Matrix-Game 2.0 10 of 19, Matrix-Game 3.0 19 of 20. On the profiles that remain reportable, area stability separates them too -- HY-WorldPlay has the lowest area jitter in every profile. Surface tilt and slosh spectrum do not separate the models, and the revision highlight the results.

VLM causal QA. We agree that a pass does not by itself show that the dynamics are right. The score measures whether a video visibly departs from the reference chain of events, and it also separates the models: Project Genie preserves 3 of 15 chains, against 13 of 15 for Unity (p = 0.0007). The judge matches the human answer key on 88.4% of questions (a human annotator: 89.5%) and rejects control videos with no motion or a different trajectory (0%).


Q3: Do better WMEval scores translate into better downstream closed-loop performance?

Applications of interactive world models to gaming and robotic simulation are at an early stage, and to our knowledge there is no established closed-loop task on which the evaluated models can be compared. WMEval is therefore an open-loop diagnostic. It measures properties those applications will need, such as following actions and keeping physical and causal consistency, and we do not claim that its scores predict closed-loop performance. We will list closed-loop validation as future work.

Reviewer_WJAdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces WMEval, a benchmark that uses Unity simulations as references to evaluate the physical behavior of interactive world models. The authors construct nine scenes, apply corresponding actions to the simulator and the world models, and compare motion, physical quantities, causal events, lighting, and scene consistency.

I consider this an important problem: a video can look realistic while depicting physically incorrect motion or interactions. Moving beyond visual similarity to examine specific behaviors is a valuable direction. However, I remain concerned that the evaluation does not sufficiently distinguish errors in the models’ physical predictions from differences in control inputs or errors in measuring motion from video.

Critical strengths
  1. I appreciate the paper’s focus on an important evaluation problem. World models should do more than generate convincing images; their actions should also lead to physically reasonable consequences. Evaluating motion and interactions is therefore more relevant to their practical use than assessing visual quality alone.

  2. the simulator is a useful source of reference data. The authors can directly access object positions, velocities, and collision events, rather than relying entirely on human or vision-model judgments of whether a video looks plausible. This creates opportunities to identify where a generated sequence goes wrong.

Critical weaknesses
  1. I am concerned that differences in visual appearance may be counted as physics errors. The evaluation relies on detection, segmentation, tracking, and optical flow to measure motion in generated videos. Even when the underlying motion is unchanged, differences in materials, style, apparent object size, or lighting can affect these measurements. The current experiments do not sufficiently rule out this influence. I would like to see evidence that the physics scores remain reasonably stable when only appearance changes.

  2. I do not think the same initial image and a “move forward” command necessarily imply a single correct future. A “move forward” command may correspond to different distances, speeds, or forces across scenes and models. A single image also generally does not reveal object mass, friction, initial velocity, or hidden parts of the scene. A model could therefore generate a physically plausible continuation that differs from the Unity reference. I believe the paper needs to explain why the model should predict that particular trajectory and which deviations can reasonably be treated as physics errors.

  3. The models do not always receive equivalent controls. In BeamDomino, Project Genie uses manual control, while LingBot-World (Cam) receives natural-language action prompts. Differences in control targets, strength, and timing make it difficult to attribute score differences specifically to physical modeling ability.

  4. The accuracy of motion measurements remains unclear. I would like stronger numerical evidence that tracking, camera correction, and coordinate conversion errors are small relative to the reported model differences. Matching momentum and energy calculated from estimated velocity also does not directly verify the underlying physical laws.

  5. The causal score may reward expected events without verifying their causes. Text-conditioned generation and whole-clip judging may reward a recognizable impact–collapse–cascade sequence even when it does not correctly respond to the action. Action interventions and videos with incorrect event order or missing contact are needed to test this distinction.

Questions for authors
  1. How much do the physics scores change when only appearance changes? Could the authors render identical simulated trajectories with different textures, materials, lighting, or styles and check whether the metrics remain stable?

  2. How do the authors ensure that commands such as “move forward” represent equivalent controls on both sides? How are movement speed, applied force, and duration aligned? What additional information does the model receive about physical properties that cannot be inferred from the initial image? If multiple physically plausible continuations are possible, why should a single Unity trajectory determine correctness?

  3. Which experiments use action inputs that can be directly matched? Could the authors list the actual control method for each model and scene, and report results from matched action replay, manual control, and text-driven generation separately?

  4. How much error does the video measurement pipeline introduce? Could the authors first apply the full pipeline to physically correct simulator videos and report errors in position, velocity, momentum, and kinetic energy under different camera conditions? I would also like to know how often tracking fails on generated videos and how those failures affect the reported scores.

  5. Can the causal metrics detect incorrect outcomes when an action changes or does not occur? Could the authors include controls with no action, a different action direction, or a different action strength, and vary text and action inputs independently where the model interface allows it? Could they also test videos with a collapse before contact, a collapse without contact, or events in the wrong order, to verify that merely including all expected events is not enough to obtain a high score?

2 comments
AuthorsOpenReview ↗

Rebuttal Response to Reviewer WJAd

We thank the reviewer WJAd for recognizing the importance of evaluating the physical consequences of actions and the value of simulator references. We address each of your questions and concerns below.

W1 & Q1: Appearance differences counted as physics errors

This is exactly the distinction we raise in Section 4.2.2 (Perceptual Quality and Physical Adherence): perceptual similarity captures appearance-level divergence and can hide physical failures. WMEval therefore intentionally scores the two separately. Appearance is scored by the perceptual metrics, and physics from tracked object states against simulator ground truth.

We provide further evidence with the control the reviewer proposes. We changed only the appearance of Unity clips (identical motion in every frame) and re-scored them against the unchanged ground truth. The kinematics and mechanics scores stay where they were:

Scene, eval metricOriginalColour shiftHue shiftBlur + compression2× down/up-sample
BilliardGame, final position error (m)0.0080.0070.0080.0100.012
BilliardGame, robust mechanics error (%)5.05.05.15.25.2
RacingCar, ego-corrected position error (m)0.390.380.38rejected by gate0.37
RacingCar, robust mechanics error (%)2.42.41.2rejected by gate2.4

The shifts are one to two orders of magnitude smaller than the gap to models or to other trajectories. Where heavy blur makes the camera estimate unreliable, the evaluator rejects the clip instead of reporting a value. We will update the controllability table and add this control to the appendix in the revision.


W2 & Q2: One first frame and "move forward" do not imply a single correct future

We agree that one first frame admits more than one valid future. The Unity simulation is still a useful reference, because the models' errors remain when the reference is widened from one future to many.

On BilliardGame we have 26 Unity rollouts that start from the same first frame and simulalte different futures in how hard, and in which direction the cue ball is struck. We scored each generated video against all 26 and kept the one with smallest error. Each cell below shows the error against the single original reference vs. against the best of the 26.

ModelFinal position error (m)Momentum and energy error over time (%)Error in momentum change at contact (%)
Unity video (measurement floor)0.0085.02.8
Matrix-Game 2.00.45 --> 0.2423.3 --> 4.3163 --> 88
Project Genie0.83 --> 0.3826.4 --> 11.5176 --> 56

which brings us to the following conclusions:

  • The errors are not a different valid future. Against the best of the 26, the position error is still 30–50 times the measurement floor, and the momentum transferred at contact is still off by 56–88%, against 2.8% for the Unity video.
  • The same failure mode for world models. Matrix-Game 2.0 repeats the same deviation in every seed (0.448 ± 0.006 m from the reference).

The simulator therefore serves as a reference in two ways: as the measurement floor, and as a family of valid futures that separates a control mismatch from a physics error. We will report best-match errors as well in the revision.


W3 & Q3: The models do not receive equivalent controls

The protocol is standardized by design, at the level of the action and not the input device. Interactive world models expose different native inputs, so WMEval gives every model the same first frame and the same semantic action trajectory, and an adapter maps that trajectory to the model's closest native control with matched timing (Section 3.1.2, Action Adaptation). A model is evaluated only on the dimensions its interface supports (Sections 3.1.2 and 4.1). We have documented the cases where a model's interface cannot carry the action. For example, LingBot-World (Act) and Matrix-Game 2.0/3.0 are reported as ineligible for physical causality (Appendix E.2, Conditioning-mismatched Models).

AuthorsOpenReview ↗

Rebuttal Response to Reviewer WJAd (Part 2)

W4 & Q4: Accuracy of the motion measurements

We have calibrated every evaluator against simulator states instead of treating its output as ground truth (Section 4.1.1; Appendix C.1, Evaluator Calibration and Validity). For detection and tracking on Unity renders it reports 98.6–100% recall, 0.9–7.2 px centroid error and 0–100 ms impact-timing error. We extend this evidence with additional analysis the reviewer asks for.

Calibration and errors on simulator videos. Each Unity video is scored against its own ground truth ("self") and against the ground truth of the other Unity trajectories in its family ("negative"). A sound measurement gives a small self error and a large negative error.

Family (trajectories)QuantityError-selfError-negativeP(Error-negative > Error-self)
RacingCar (29)Camera path from video0.12 m (0.4%)––
RacingCar (29)Ego-corrected position error0.14 m22.6 m1.00
RacingCar (29)Robust mechanics error1.1%45.2%1.00
BilliardGame (26)Robust mechanics error4.4%29.8%0.95
BilliardGame (26)Momentum change at contact2.2%154%0.71

We will add the analysis to the paper.


W5 & Q5: The causal score may reward expected events without verifying their causes

We would like to clarify the intended scope: a pass is not meant to verify the cause on its own. The score measures whether a generated video visibly departs from the reference chain of events, and we argue it remains useful for that.

  • It is calibrated against humans. The judge matches the human-reviewed answer key on 88.4% of questions, against 89.5% for a human annotator (Appendix C.1).
  • It separates models. Project Genie preserves 3 of 15 chains, against 13 of 15 for Unity (p = 0.0007).

The same controls also show the score's limit: when one of the causal event is off, it can still pass (80–100%). A stricter test would reflect causality better, but such test would sit at the floor for today's models (therefore could separate today's model less well), which do not yet pass this simpler one -- Project Genie preserves only 3 of 15 chains, and three other models cannot receive the ball action at all (Appendix E.2). In the revision we will state this scope, add test cases that examine causality in more detail, and sample frames densely around each event.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.