acceptodds
Withdrawn from ICLR 2027

ACWM-Phys: Investigating Generalized Physical Interaction in Action-Conditioned Video World Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Action-conditioned world models (ACWMs) have shown strong promise for video prediction and decision-making, yet existing benchmarks are largely restricted to egocentric navigation or narrow robot manipulation, covering only a small part of the physical interactions a general world model must capture. We introduce **ACWM-Phys**, a simulated benchmark spanning four physical regimes (rigid-body, deformable, particle, and kinematic dynamics) across eight environments, each paired with a controlled out-of-distribution (OoD) split that shifts a single physical factor so failures are attributable. Alongside it we release ACWM-DiT, a latent video diffusion transformer baseline. Because every environment is simulated, we score rollouts not only with perceptual metrics but directly against physics: point-track error, conservation of simulator-conserved quantities, and a zero-action counterfactual. Both families of metrics agree on the same trend: OoD degradation is governed by task complexity rather than physics category, with low-dimensional geometric tasks nearly unaffected and contact-rich deformation and high-DoF kinematics degrading sharply. A second, architecturally distinct model post-trained from a pretrained video foundation model shows the same degradation, indicating the gap is not an artifact of our backbone. The physics metrics additionally expose failures that perceptual metrics miss, such as mass that is not conserved under distribution shift. We release ACWM-Phys as a diagnostic instrument for physically grounded world modeling.

Primary area:
generative models
Keywords:
world model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eFMeOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Addressing the gap in existing action-conditioned video world models (ACWMs) regarding the evaluation of complex physical interactions, this paper introduces the ACWM-Phys benchmark, which encompasses four major physical mechanisms, eight simulation environments, and a controlled out-of-distribution (OoD) test set. The authors propose the ACWM-DiT model and employ cross-architecture validation, discovering that performance degradation in OoD scenarios is primarily driven by task complexity rather than physical categories. Furthermore, the paper introduces physical metrics, such as trajectory tracking and mass conservation, demonstrating that traditional pixel-level perception metrics often fail to detect physical violations, thereby providing a diagnostic tool for future research into physically plausible world models.

Critical strengths

The paper highlights that visual realism does not imply physical correctness, and introduces a dedicated quantitative evaluation suite (e.g., trajectory tracking and conservation errors) to expose physical violations missed by traditional perceptual metrics.

Critical weaknesses
  1. Successful rollout on out-of-distribution (OoD) data does not equate to a genuine understanding of physical laws. As revealed by the finding that "generalization capability depends on task complexity rather than physical category," OoD testing is inherently incapable of demonstrating true physical learning ability; the pattern of strong performance in low-dimensional tasks alongside failure in high-dimensional ones merely measures the relative difficulty for a neural network to fit visual manifolds of varying complexity. The model's so-called "generalization" relies entirely on statistical interpolation within the high-dimensional feature space and the learning of visual shortcuts; this form of black-box fitting, which masquerades as physically correct through visual illusions, bears no relation to actual causal physical mechanisms. In short, OoD rollout may not be a good task for physics learning evaluation.

  2. The benchmark dataset is overly simplistic and "toy-like." The OoD splits (such as varying rope length, water volume, or cube counts) are merely idealized, single-variable laboratory cuts that fall far short of the multi-factor, noisy, and visually complex realities of the physical world.

Questions for authors

1. Diversity and Scalability of the Dataset: The paper claims over 15k simulated videos across 8 environments, yet the benchmark is restricted to only four major physical mechanisms. Given this confined scope, how are the 15k videos generated? Does this high volume imply that the trajectories are heavily repetitive or homogeneous?

2. Dependence on Visual Proxies in "Physics" Evaluation: The proposed "Track error" relies on CoTracker3 to compute point-track displacements. Given that CoTracker3 is itself a neural network driven by appearance features, how can the authors guarantee that tracking failures in OOD scenarios (e.g., due to blur or deformation) do not contaminate the "physical error" metric? To what extent does this measure actual physical discrepancy rather than mere visual representation drift?

3. Inadequate Baseline Comparison: The evaluation heavily relies on the single in-house baseline (ACWM-DiT), with only a partial and unmatched post-training test on Cosmos-Predict2.5-2B for two environments. Can you provide more comparisons with other existing open-source action-conditioned video world models or physics-aware video generation baselines across all 8 environments?

4. Missing Qualitative Video Demonstrations: Since ACWM-Phys deals with complex dynamic rollouts (such as fluid pouring, cloth deformation, and high-DoF kinematics), static grid figures in the paper are often insufficient to evaluate temporal consistency and physical plausibility. Do the authors provide or plan to release supplementary videos showcasing side-by-side comparisons of ground truth, model predictions, and failure cases (especially under OOD shifts)?

Reviewer_pTpzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ACWM-Phys, a simulated benchmark for evaluating action-conditioned video world models. It contains eight environments grouped into four categories (rigid-body, deformable, particle/fluid, and kinematics). Besides in-distribution training and test splits, each environment has an out-of-distribution (OoD) test split that changes a single factor. The authors also provide a latent video diffusion baseline, ACWM-DiT, and, in addition to perceptual metrics, introduce three metrics computed from simulator ground truth: point-track error, conservation error, and a zero-action counterfactual. Experiments show that the model performs well in distribution but degrades out of distribution, and the authors conclude that the degree of degradation depends mainly on task complexity rather than physics category. Cosmos-Predict2.5, post-trained on two environments, shows a similar degradation.

Critical strengths
  1. Compared with existing datasets for action-conditioned world models, the benchmark covers a wider range of scenes, including deformable objects, particles, and fluids, which have received less attention. The research direction is meaningful.

  2. Using simulator ground truth to design physics-grounded metrics, and using a zero-action counterfactual to check whether predictions are actually driven by the actions, is a valuable evaluation idea that can reveal failures perceptual metrics alone would miss.

  3. The limitations are discussed candidly; for example, the authors explicitly acknowledge the limited coverage of the cross-architecture validation and the unmatched training budgets.

Critical weaknesses
  1. The OoD design is narrow and does not support the claim of generalized physical interaction. Each environment shifts along a single axis, and most shifts are changes in quantity or position, such as cube placement, stacking direction, joint range, goal region, water volume, and particle count. Physical properties that determine the dynamics, such as object shape, mass, friction, stiffness, and viscosity, are not varied. Under such shifts, the model mainly faces extrapolation in value ranges or spatial positions, which falls clearly short of the physical generalization emphasized in the title and abstract. As a reference, the PushObj experiment in DINO-WM (DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning, Zhou et al., 2024, arXiv:2411.04983) trains on some object shapes and tests on unseen ones. Splits by object shape of this kind may better probe whether a model actually understands the physics.

  2. The choice of split dimensions and the sufficiency of the training sets are not justified. Given the cost of data collection, gaps in training coverage are normal, but OoD splits should lie along dimensions where such gaps realistically occur, and the authors need to show that the training set is sufficient for a capable model to learn the underlying physics, so that OoD failures can be attributed to generalization itself. Currently, most environments have about two thousand training trajectories (Pour Water has one thousand), and the model, with about 200M parameters, is trained from scratch. Apart from trajectory counts and the value ranges of a few parameters, the paper does not describe how the training data cover each factor, nor does it report on overfitting. The authors' own ablations show that more training data and larger models both improve OoD performance, and Push Cube still improves noticeably when the data is doubled. This suggests that at least part of the observed gap comes from data size and model capacity, rather than being purely a property of the task.

  3. The validity of the benchmark is not demonstrated, and the baselines are insufficient. The paper evaluates only the authors' own ACWM-DiT, plus Cosmos on two environments, and there is no leaderboard. The value of a benchmark lies in whether its rankings reflect models' actual physical generalization: for example, models with stronger physical priors should score better here, and methods that generalize well here should keep their advantage when scaled up. The paper provides no evidence of this. The only second model, Cosmos, is heavily pretrained, yet it scores lower in absolute terms than the small model trained from scratch, and the two models disagree on which environment degrades more. In addition, simple baselines such as copying the first frame are missing as a lower bound. For Reacher, for instance, the links occupy only a small part of the frame, so full-frame metrics are likely dominated by the static background; without a lower bound, it is hard to tell how much of the score reflects learned dynamics.

  4. The proposed physics metrics are not themselves validated. The conservation error requires segmenting water, sand, rope, or cloth in the predicted frames, but the segmentation method is not described and its accuracy is not checked. Table 2 reports conservation errors for Robot Arm and Reacher, yet Appendix A.9 does not define the conserved quantity for these two environments. The track error relies on CoTracker3 applied to generated videos, and its reliability is likewise not examined: Reacher is described as nearly perfect, yet its in-distribution track error is the second highest among the eight environments, which suggests that the tracker may be unreliable on thin structures. Moreover, the claim that the two metric families give consistent rankings is based on five of the eight environments; when all environments are included, the position of Pour Water differs considerably between the two rankings.

  5. The paper lacks a clear storyline and a systematic structure, and the amount of work is insufficient to support its conclusions. Different environments shift along different factors and by different magnitudes, with no shared shift dimension, so results cannot be compared across environments and general patterns are hard to derive. The central claim that degradation depends on complexity rather than category is therefore confounded by the differing shift magnitudes. The ablations also use different subsets of environments, with no fixed set used throughout, which makes their conclusions hard to relate to one another. The authors mention that the data generators support further shifts, such as material and compositional ones, but these are not explored in this paper. Overall, the conclusions are largely as expected and do not provide deeper insight.

Minor issues:

  • All comparisons report only the mean over the test set, without the spread of per-trajectory metrics (e.g., standard deviations or confidence intervals). Without knowing the distribution of the metrics, the differences of a few tenths of a dB on which some ablation conclusions rely are hard to accept as distinguishable.
  • There are several inconsistencies. Section 4.1.1 states that Push Cube contains one to five cubes, whereas Appendix A.8 states that scenes with four or more cubes are unseen during training, and it is not stated which OoD setting the reported results correspond to. The OoD shift of Push Sand is described as an increased particle count in Section 4.1.2, Appendix A.7, and Figure 11, but as different random seeds in Appendix A.2. The results of Cloth Move with the 8-D action and AdaLN conditioning differ between Table 5 and Table 9. The two numbers behind the claim in the introduction that PSNR improves while pose error grows by 47% seem to come from different action settings. The Reacher action is a joint torque, yet its range is given in rad.
  • The paper states that code and data are provided as supplementary material, but no supplementary material is available.
Questions for authors

Questions that could affect my assessment:

  1. Can the authors provide statistics on how the training data cover each factor, together with learning curves or overfitting diagnostics, to show that the current training sets are sufficient for learning the physics of each environment? Does the OoD gap shrink substantially with more data?

  2. Have the authors evaluated models with stronger physical priors, or larger pretrained models under matched training budgets? Are their rankings on this benchmark consistent with their known physical generalization ability?

  3. What scores do simple baselines, such as copying the first frame, obtain on each environment and metric?

  4. How is the segmentation for the conservation error implemented, and how accurate is it? What are the conserved quantities for Robot Arm and Reacher?

Clarifications (not affecting my rating):

  1. Please clarify the OoD settings of Push Cube and Push Sand, and explain the discrepancy between Table 5 and Table 9.
Reviewer_CfAKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors introduce ACWM-Phys, a simulated benchmark that spans multiple physical regimes across different environments for testing whether action-conditioned video world models predict physical interactions and generalize to out of distribution scenarios. It covers environments from rigid and deformable objects, particles and articulated mechanisms, and also provides in-distribution and out-distribution sets. On the other hand, the paper also introduce ACW-DiT a diffusion transformer baseline that predicts future video from an initial image and a sequence of actions. According to the experiments the authors show multiple findings, among the most important ones I think is that, generalization failures relate more closely to task complexity than to the category of physics, and that the measurement of physics errors is what exposes failures that pixels do not.

Critical strengths

Strengths:

  • The proposed benchmark addresses a necessary and important gap in action-conditioned world models, by creating a benchmark that spans multiple simulated scenarios such as cloths, ropes, sand, water, and robot arms. Its simulated setting also allows repeatable experiments and deliberate changes to test conditions.

  • The authors provide experiments that show useful evidence about their model design, for example model size, action conditioning, video encoding, training-data volume, and action dimensionality, I think these comparisons are useful and give future users concrete starting points beyond a single baseline result.

Critical weaknesses

Weaknesses:

  • Related to Appendix A9. I believe that the conservation scores need stronger validation. An object’s visible size or shape can change without violating physics, so image-based measurements may flag false errors. So I think is unclear whether the scores reliably detect physical mistakes.

  • Some experiment descriptions and results are inconsistent. The main text and appendix describe different test conditions for Push Sand and Push Cube. Additionally, the paper highlights a case where Cloth Move’s PSNR improves while physical error worsens, yet Table 1 reports PSNR falling from 19.72 to 16.23. On the other hand, a different action configuration does show improving PSNR in later tables, but the paper does not establish that the physical scores use that configuration. These issues affect the claims about controlled shifts and disagreement between image and physical metric.

Questions for authors

Please refer to weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.