- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Addressing the gap in existing action-conditioned video world models (ACWMs) regarding the evaluation of complex physical interactions, this paper introduces the ACWM-Phys benchmark, which encompasses four major physical mechanisms, eight simulation environments, and a controlled out-of-distribution (OoD) test set. The authors propose the ACWM-DiT model and employ cross-architecture validation, discovering that performance degradation in OoD scenarios is primarily driven by task complexity rather than physical categories. Furthermore, the paper introduces physical metrics, such as trajectory tracking and mass conservation, demonstrating that traditional pixel-level perception metrics often fail to detect physical violations, thereby providing a diagnostic tool for future research into physically plausible world models.
Critical strengths
The paper highlights that visual realism does not imply physical correctness, and introduces a dedicated quantitative evaluation suite (e.g., trajectory tracking and conservation errors) to expose physical violations missed by traditional perceptual metrics.
Critical weaknesses
-
Successful rollout on out-of-distribution (OoD) data does not equate to a genuine understanding of physical laws. As revealed by the finding that "generalization capability depends on task complexity rather than physical category," OoD testing is inherently incapable of demonstrating true physical learning ability; the pattern of strong performance in low-dimensional tasks alongside failure in high-dimensional ones merely measures the relative difficulty for a neural network to fit visual manifolds of varying complexity. The model's so-called "generalization" relies entirely on statistical interpolation within the high-dimensional feature space and the learning of visual shortcuts; this form of black-box fitting, which masquerades as physically correct through visual illusions, bears no relation to actual causal physical mechanisms. In short, OoD rollout may not be a good task for physics learning evaluation.
-
The benchmark dataset is overly simplistic and "toy-like." The OoD splits (such as varying rope length, water volume, or cube counts) are merely idealized, single-variable laboratory cuts that fall far short of the multi-factor, noisy, and visually complex realities of the physical world.
Questions for authors
1. Diversity and Scalability of the Dataset: The paper claims over 15k simulated videos across 8 environments, yet the benchmark is restricted to only four major physical mechanisms. Given this confined scope, how are the 15k videos generated? Does this high volume imply that the trajectories are heavily repetitive or homogeneous?
2. Dependence on Visual Proxies in "Physics" Evaluation: The proposed "Track error" relies on CoTracker3 to compute point-track displacements. Given that CoTracker3 is itself a neural network driven by appearance features, how can the authors guarantee that tracking failures in OOD scenarios (e.g., due to blur or deformation) do not contaminate the "physical error" metric? To what extent does this measure actual physical discrepancy rather than mere visual representation drift?
3. Inadequate Baseline Comparison: The evaluation heavily relies on the single in-house baseline (ACWM-DiT), with only a partial and unmatched post-training test on Cosmos-Predict2.5-2B for two environments. Can you provide more comparisons with other existing open-source action-conditioned video world models or physics-aware video generation baselines across all 8 environments?
4. Missing Qualitative Video Demonstrations: Since ACWM-Phys deals with complex dynamic rollouts (such as fluid pouring, cloth deformation, and high-DoF kinematics), static grid figures in the paper are often insufficient to evaluate temporal consistency and physical plausibility. Do the authors provide or plan to release supplementary videos showcasing side-by-side comparisons of ground truth, model predictions, and failure cases (especially under OOD shifts)?