acceptodds
Withdrawn from ICLR 2027

MotionCore: A Latent Neural Simulator for Video Generation with Rigid-Body Dynamics

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video generation models can produce highly realistic videos while generating physically inconsistent object motion. This limitation suggests that visual modeling alone may not provide a transferable representation of physical dynamics. We introduce MotionCore, a framework for learning physical dynamics via a learnable partial differential equation solver and focuses on rigid-body motion. We introduce a latent neural simulators learning to predict the evolution of object-level physical states from observed motion. Its predictions are then converted into spatially aligned latent controls for a controllable video diffusion model. The generator serves as a visual interface: it provides appearance and scene detail, while the learned dynamics model specifies the physical trajectory. To preserve this trajectory through the interface, MotionCore combines a diffusion-matching objective with a low-capacity decodability constraint on object identity and position. We evaluate MotionCore on public physical-consistency benchmarks and on paired data with known trajectories. The latter setting measures obedience to a specified physical rollout, while the former evaluates broader physical and semantic consistency. Results indicate that MotionCore improves physical consistency and trajectory obedience in the evaluated rigid-body settings, while maintaining comparable visual quality. Ablation studies examine the roles of spatial alignment, temporal grouping, and decodability supervision.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
latent space, neural operator, rigid-body motion, physics-grounded video generation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_NmJZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I found the paper extremely challenging to follow. My understanding is that it proposes a learned rigid-body simulator that predicts future object states and trajectories. These predictions are then converted into spatially aligned control signals for a pretrained video generator, which renders the predicted motion as photorealistic video.

Critical strengths

None

Critical weaknesses

Weaknesses

  1. Table 1 is incomplete. The paper introduces RigidBench as a main benchmark but does not report MotionCore’s results on it. No explanation is provided.

  2. The CLEVRER experiment is unclear. The paper reports results from one CLEVRER scene without introducing the dataset, citing it, or explaining the experimental setup.

  3. Figures 1 and 2 conflict with the text and contains a lot of incorrectness.

  4. Unclear mathematical presentation. Equation (1) uses a backslash instead of a correct derivative. Several other equations use unclear notation or are not clearly connected to the implemented method and other equations.

Questions for authors

Could the authors clarify the extent to which generative AI was used in preparing this paper, including its role in writing, theoretical derivations and experimental analysis? What procedures were used to independently verify the AI-generated content and reported results?

Reviewer_pi5gOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

MotionCore separates rigid-body dynamics prediction from video appearance generation. It encodes scene geometry using an occupancy encoder and predicts object states through a latent neural operator, a free-motion module, and a collision impulse module with learnable parameters. The predicted geometry is then rendered into spatially aligned control videos that guide Wan2.1-VACE. According to Section 5.2, the final pipeline uses staged training with physical-state supervision; the simulator is not trained through the video generation objective, and the learned video-conditioning adapter is primarily studied as an ablation. Experiments on synthetic rigid-body scenes and a rigid-body subset of Physics-IQ-Verified demonstrate improvements in trajectory control. However, the final system configuration remains insufficiently clear, and the advantages of the proposed latent simulator over existing approaches are not yet adequately established.

Critical strengths
  1. Separating dynamics prediction from appearance generation allows physical trajectories to be inspected independently and edited through changes to initial states. The initial-velocity rotation experiment demonstrates this controllability.

  2. Table 3 compares interfaces using the same ground-truth states and frozen video model. Rendered conditioning achieves a hit rate of 0.85, compared with 0.52 for the token-based cross-attention interface, supporting spatial alignment within the evaluated system.

  3. Evaluating generated trajectories against both ground truth and the states actually supplied to the generator helps separate simulator error from conditioning error. The appendix also reports negative findings, including unsuccessful joint training and identity-binding failures, which provide useful evidence about the system’s limitations.

Critical weaknesses
  1. Figure 1 states that video diffusion losses backpropagate into the simulator, whereas Section 5.2 explicitly excludes this. The generator is repeatedly described as frozen, yet Table 2 includes “no LoRA on the control branch” as an ablation. The learned adapter highlighted in the abstract is also presented as an ablation. These discrepancies obscure the implemented contribution and training requirements.

  2. PhysGen and PSIVG already combine physical simulation with video generation. Comparisons with learned rigid-body simulators such as HOPNet and RigidFormer, and explicit-state models under matched training settings, are missing. The appendix also acknowledges that its analytic baseline uses only one impulse update per frame and does not represent stronger classical simulation methods.

  3. The source of initial velocities, 3D geometry, and camera parameters, the number of observed frames, and input equivalence across methods are unclear. Although RigidBench is presented as a main benchmark, all corresponding MotionCore results in Table 1 are missing.

  4. In Table 4, the learned simulator’s contact F1 is 0.508, below the analytic solver’s 0.651. In Figure 10, rendered conditioning achieves an identity-free hit rate of 0.94, but only 0.48 when the correct identity is required. High positional accuracy therefore does not establish that the correct objects follow the intended physical trajectories.

  5. In Theorem 4.1, epsilon is slack introduced when selecting a near-optimal representing measure, rather than encoder–decoder representation error as described in the text. The connection between this finite-width approximation result and the implemented attention network and collision module is also insufficiently explained.

Questions for authors
  1. Which modules are trainable in the main experiments? Does the simulator receive video-loss gradients, does the video control branch use LoRA, and is the learned adapter used in the main results? Please reconcile Figure 1, the training section, and the conclusion, and clarify why Table 1 reports a Physics-IQ score of 26.2 while the text reports 24.53.

  2. With input information and the video interface held fixed, could the authors compare against a stronger classical physics engine, an existing learned rigid-body simulator, and a model that directly predicts explicit states while retaining the same collision module? How should the lower trajectory error but slightly lower video hit rate of the fixed simulator in Table 2 be interpreted?

  3. How are initial positions, velocities, geometry, material properties, and camera parameters obtained? Are additional observation frames, manual annotations, or ground-truth states used? Could the authors provide the exact evaluation subsets, baseline rerun settings, and results on RigidBench?

  4. Could the authors report identity-aware trajectory errors and evaluate collision timing, rotation, and post-contact motion? How do simulation errors and identity swaps change with increasing occlusion, object count, and shape complexity?

  5. How does the operator class in Theorem 4.1 correspond to the implemented network? Which quantities capture encoding and representation errors? Could the authors correct the interpretation of epsilon and distinguish conditional approximation guarantees from empirically validated conclusions?

Reviewer_qwR3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

Video generation models can generate visually plausible videos, but the motion of objects is often physically inconsistent. The authors argue that visual modeling alone is insufficient to fully learn physical dynamics, and explicitly decouple the roles of appearance and physical motion: the video generator handles appearance, while a neural physical model governs motion. Specifically, a latent neural simulator autoregressively predicts rigid-body trajectories from the object states in the first frame. The predicted trajectories are rendered into spatially aligned control videos, which are then provided as conditioning inputs to a frozen video generation model to synthesize appearance. The simulator is trained step by step in the physical state space rather than using video-level losses. The authors claim that this approach enables the generation of physically more consistent videos of rigid-body scenes that follow the predicted trajectories.

Critical strengths
  • The idea of addressing the limitations of video generation models in capturing physics by decoupling motion and appearance is well-motivated.
  • The proposed pipeline is coherently designed and well-aligned with the motivation.
Critical weaknesses
  1. Clarity and Completeness
  • The reported score is 26.2 in Table 1, whereas the main text reports 24.53.
  • The paper introduces RigidBench as a benchmark but reports no results for it in Table 1, marking it as “not evaluated” instead. Is there a specific reason for not evaluating on this benchmark? If the evaluation was simply left incomplete at the time of submission, this would be a major red flag.
  • The paper does not describe how 3D geometry and initial physical states are obtained for real videos.
  1. Limited Improvements Relative to the Claims
  • Results are missing for one of the two benchmarks, while the improvements on the other appear marginal. Even though the gain over the unconditioned backbone is clear, overall peformance remains below several baselines.
  • The authors justify their results by emphasizing the relatively small number of parameters in their model. However, reported figure of 1.3B refer only to the video generator, without accounting for the additional components introduced by the proposed pipeline. Thus, the claim of achieving comparable performance with fewer parameters is not sufficiently substantiated.
  • Fig. 3(c) suggests that error accumulation primarily occurs in the explicit object-state stream rather than in the occupancy latent. Thus, Theorem 4.1, which focuses on approximating the latent transition, does not appear to address the main bottleneck observed in the results.
  • Despite being a video generation paper, the submission provides neither video samples nor inference code for generating videos.
  1. Concerns Regarding Novelty
  • Section 4 largely reiterates existing theories, and the LNO used in the method appears to closely follow prior work. Section 5 seems to contain the main contribution, suggesting that Section 4 occupies a disproportionate amount of space relative to its contribution. The novel contribution appears to lie in the proposed learning framework for free-motion proposals and a hybrid collision operator, but these components are not supported by sufficient analysis or theoretical justification.
  • The training data are restricted to spheres, and the methodology itself also appears to assume spherical objects. However, the evaluation datasets include non-spherical objects as well. How are such objects handled, and to what extent does the method generalize beyond spheres?
  1. Regarding the scope of the task
  • Despite being presented as a video generation method, the text prompt controls only appearance, while motion is dictated entirely by the initial object states and a deterministic simulator rollout. Consequently, users cannot directly control motion through textual prompts, and each initial state yields only a single trajectory. This makes the approach more akin to physics-conditioned image-to-video prediction than general-purpose video generation, and the scope of its claims should be adjusted accordingly.
Questions for authors
  1. What is the difference in inference time between the proposed method and the baselines?
  2. The experiments in Table 3 seem somewhat circular. The authors argue that the frozen backbone cannot learn the transformation, but is this not an expected outcome given that the backbone is frozen?
  3. Given the substantial difference in the amount of input information provided, how does the proposed method perform compared to controllable video generation models that receive trajectory or mask conditions as inputs? In particular, how does it compare with methods mentioned in the related work, such as MotionCraft and PhysGen?
Reviewer_Mmt3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

MotionCore proposes a learned physics framework based on latent neural operators (LNOs) to improve rigid-body physics in video generation. The main idea is to explicitly learn how rigid bodies move and interact, rather than relying on a video generation model to learn these dynamics implicitly. The system uses a 3D occupancy autoencoder to represent scene geometry in a compact latent space, together with object-level physical states such as 3D positions, velocities, and angular velocities, and physical attributes including mass, friction, and restitution. A latent neural operator learns to predict how these states evolve over time, while a collision operator handles rigid-body interactions using impulse-based physics. The predicted 3D states are then rendered into spatially aligned control videos that guide a pretrained Wan2.1-VACE-1.3B video generation model. One of the interesting findings is that spatially aligned conditioning works significantly better than directly providing physical state tokens. The method is evaluated on simulated rigid-body datasets and physical video generation benchmarks.

Critical strengths

The paper addresses an important problem in video generation, namely physical consistency, which is also a core challenge in robotics and embodied AI. Unlike methods that rely primarily on VLM reasoning or 2D motion guidance, the proposed approach explicitly models 3D rigid-body dynamics and collision handling, which is useful for physics-based world modeling applications. I also find the idea of separating learned physical simulation from visual generation useful. The quantitative results show improvements over the base video generator without physical guidance, and the experiments demonstrate that spatially aligned conditioning is more effective than physical state tokens. The use of simulated data with known physical ground truth is also very good.

Critical weaknesses

My main concern is the limitation to relatively simple rigid-body dynamics and geometries, particularly spheres and basic collisions. While the idea of learning physics is interesting, the advantages and limitations compared with using a conventional physics engine are not entirely clear to me, especially since the method has not been convincingly demonstrated on more complex dynamics and geometries. In fact, the fixed simulator achieves lower trajectory errors than the learned simulator in some experiments, even though the generated video results are slightly better. Since one of the main motivations for learning physics is to generalize beyond predefined physical systems, I think this limitation is quite important. It is also unclear how practical the method would be for real-world scenes where accurate 3D geometry and physical properties are not readily available.

Questions for authors

1/ Why does the learned simulator sometimes produce higher trajectory errors than the fixed physics simulator, while achieving slightly better video results? Does this mean that the learned simulator is better aligned with the video generator rather than producing more accurate physical simulations?

2/ How does the method generalize to more complex rigid-body dynamics, interactions, and object geometries beyond spheres and simple collisions? What are the main limitations compared with conventional physics engines? Would be great to have a more comprehensive discussion on this.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.