- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
WMEval evaluates interactive world models using nine Unity scenes, with simulator states and event traces as references. It combines metrics for action control, consistency, physics, and causality with perceptual measures. Experiments on six models show that visual similarity and immediate action following do not reliably indicate correct physical or causal outcomes. The authors also assess evaluator reliability through simulator-based calibration and human checks.
Critical strengths
- Using Unity provides reproducible test conditions and access to physical states such as velocity, momentum, and collision events. This gives the evaluation a concrete reference beyond visual similarity or subjective judgments.
- The experiments show that visually plausible outputs can still fail to preserve interactions and causal chains. These examples help identify weaknesses that standard video-quality metrics may miss, making the benchmark useful for diagnosing current models.
Critical weaknesses
The simulation-grounded design provides controlled and reproducible evaluation, but the extent to which its findings generalize beyond the selected environments remains unclear. Although the nine Unity scenes cover diverse evaluation dimensions, they represent a limited set of simulated scenarios and dynamics. Without validation in real-world settings or across a broader range of simulators, it remains difficult to determine how well the reported failures reflect general physical limitations rather than mismatches with the specific environments and dynamics used for evaluation. This limits the broader conclusions about real-world applicability.
Questions for authors
- What evidence supports extending the findings from these nine Unity scenes to real-world physical behavior? Could some errors reflect a mismatch with the simulator’s specific dynamics rather than a violation of general physical laws?
- How does the evaluation distinguish failure to generate the required interaction from incorrect physics after the interaction occurs? Do the main conclusions hold when physical scores are evaluated separately for successful, measurable interactions and failed interactions?
- How are actions made comparable across models with different control interfaces, and to what extent could model-specific action mappings affect the reported physical-causality results?
2 comments
Rebuttal Response to Reviewer ZLKv (Part 2)
Q3: Comparability of actions across interfaces, and the effect on physical causality
Ideally every model would receive physically equivalent controls. Today that is not attainable: the models expose different interfaces (keys, camera poses, text) and each learned its own mapping from input to motion. There is little we can do on the modeling side, but we have tried our best to standardize in the inference setting.
We argue they can still be compared when actions are matched by semantics and timing (Section 3.1.2) and the effect of a control mismatch is measured. For example, we allow for a mismatch in force or timing by scoring each BilliardGame generation against the best-matching of 26 Unity references that vary the strength, timing and direction of the strike:
| Error against the best-matching reference | Position (m) | Mechanics (%) | Momentum change at contact (%) |
|---|---|---|---|
| Unity video | 0.008 | 5.0 | 2.8 |
| Matrix-Game 2.0 | 0.24 | 4.3 | 88 |
| Project Genie | 0.38 | 11.5 | 56 |
Even with this allowance and the fact that the interfaces are different, WMEval separates the models today, from the simulator and from each other. Matrix-Game 2.0 has the lower mechanics error in all five seeds (4.2–4.4% against 6.9–23.5%), Project Genie has the lower contact error in all five (12–82% against 86–88%), and both stay far from the Unity video in position and at contact.
We expect the comparison to tighten as world models mature: as their capabilities and control interfaces converge, the same action can be delivered to every model more directly and the adapter gap shrinks. In the revision we will add a table listing each model's native input and control type per scene, and report best-match errors alongside the original ones.
Rebuttal Response to Reviewer ZLKv
We thank the reviewer ZLKv for recognizing that Unity gives reproducible conditions and a concrete reference, and that the experiments expose failures that video-quality metrics miss. We address each concern below.
W1 & Q1: Generalization beyond nine Unity scenes, and simulator-specific dynamics
We would like to clarify WMEval is simulator-scoped by design. Unity is not meant to reproduce the full visual or physical complexity of the real world. We use it because simulator allows controlled interventions and exposes exact, synchronized physical states, which is sufficient to test first-order phenomena: motion, collisions, object persistence and causal transitions. This is the same scope as simulation-grounded benchmarks such as Habitat, iGibson, BEHAVIOR and Isaac Gym, and the revision states it explicitly. Two results show that the findings are not artefacts of Unity's particular dynamics.
By comparing against simulator states, WMEval can still separate today's world models well. For example:
Event trace (BeamDomino). Project Genie preserves the full causal chain in 3 of 15 rollouts, against 13 of 15 for the Unity video (p = 0.0007).
Car trajectory (RacingCar). Project Genie keeps the car tracked in 100% of frames, Matrix-Game 2.0 in 25–36%.
Q2: Failure to generate the interaction versus incorrect physics after it
WMEval separates the two by gating: physics is scored only where the object of the interaction is present, and the gate is reported as a result of its own (Appendix C.1; Appendix B.4.1). Both kinds of failure occur:
Videos where interactions are not generated well barely change model ranking. Most video generations fail to produce interactions due to poor action following. We compared the models in 58 settings (one scene and one metric each) in three ways: leaving those videos out, keeping whatever score they produced, or giving them the worst score. In 43 settings the ranking of models is the same all three ways, almost always because no video in that setting was unscorable.