acceptodds
Withdrawn from ICLR 2027

Phys4D: Injecting Physical Structure into Video Diffusion via Lifted RGB-D-Motion 4D Interfaces

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent video diffusion models generate increasingly realistic videos, yet visual plausibility alone does not imply physical understanding. Physical laws govern evolving 3D scene states, while videos observe only their 2D projections. We introduce Phys4D, a structure-focused, appearance-constrained physics injection framework that transfers simulation-derived physical structure into pretrained video diffusion models while preserving their real-video generative prior. Phys4D operationalizes physics-aware video generation through a prior-preserving RGB-D-motion interface. Our key principle is simulation teaches physics, not appearance: simulation provides dense structural supervision, including depth, motion, and masks, while adaptation remains anchored to the pretrained real-video prior rather than drifting toward simulator-specific visual statistics. To this end, Phys4D exposes geometry and motion with lightweight prediction heads, injects local physical structure through gated geometry-motion supervision, and aligns sequence-level evolution using a simulation-grounded trajectory objective. Supported by a scalable coupled-physics simulation system, Phys4D improves external RGB physics scores and controlled geometry-motion diagnostics across multiple video generation backbones while maintaining visual quality. The simulated diagnostic split tests unseen templates within supported physical regimes. We plan to release code and datasets upon acceptance. Project page: https://sensational-brioche-7657e7.netlify.app

Primary area:
generative models
Keywords:
Video Diffusion Models, Physics-Aware Generation, Simulation-Grounded Learning, RGB-D-Motion Representation, 4D Physical Dynamics, Prior-Preserving Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xbh5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Phys4D, a three-stage post-training recipe that uses simulation data to make pretrained video diffusion models more physically consistent while keeping their real-video appearance prior. The authors also build an asynchronous multi-solver simulation pipeline. Validation covers CogVideoX-5B, Wan2.2-5B and Open-Sora 1.2 on Physics-IQ, VBench-2.0 Physics and PhyGenBench. The paper adds geometry/motion diagnostics on 20 held-out simulated templates and a Physics-IQ ablation that separates structural supervision from simulator-RGB flow matching.

Critical strengths
  • The core design choice is reasonable, and the RQ4 control is the right experiment. Using simulation for structure (depth, flow, masks, trajectories) rather than as an appearance target is a sensible way.

  • The evaluation is broad. Three backbones, three external physics benchmarks plus Physics-IQ Verified, and the RQ4 comparison against a direct simulator-RGB LoRA (Table 4) all address the obvious worry that the model just learns simulator appearance.

  • The scalable simulation pipeline and the dataset it produces. The authors build an asynchronous, tiled-rendering data engine on Isaac Sim / GarmentLab. The coupled-physics simulation pipeline covers rigid, deformable, fluid, cloth and granular cases and exports aligned depth, flow, masks and trajectories. If it is released as promised, people working on physics in video models could use it.

Critical weaknesses
  • The pipeline is overly complex, and its necessity is never shown. It stacks three stages (pseudo-label head pretraining, noise- and mask-gated adapter training with six losses, then PPO on a Chamfer reward). That costs about 864 H100 GPU-hours plus a 1.25M-video simulator corpus. The paper never shows that a simpler recipe, such as Stage II alone or the structural losses without the gating machinery, would not recover most of the gain.

  • The training simulations may overlap with the Physics-IQ test scenes. The supplementary configs include weight_on_pillow, light_on_block, duck_and_domino, potato_in_water, domino_in_juice and others, which closely mirror Physics-IQ scenes. The held-out split is only among the authors' own templates.

  • No comparison with the closest physics post-training methods. The main results only compare Base with +Phys4D. Several methods are directly comparable: they use the same backbones and have public code and checkpoints such as PISA (Li et al., ICML 2025), VideoREPA (Zhang et al., NeurIPS 2025), WISA (Wang et al., 2025) or simply Flow-GRPO with a physics reward at matched compute.

  • Figure 3 needs a redesign. The pipeline figure looks largely AI generated. That is fine in itself, but the result is crowded, the text is too small to read at print size, and it does not match the method text.

Questions for authors
  • Do your training templates replicate Physics-IQ scenarios?

  • Can you provide evidence that all three stages are necessary? For example, train the heads from scratch in Stage II without Stage I, and compare Stage III against Stage II trained longer with the same compute.

  • Can you ablate the noise gate, the learned spatial gate and the mask weighting? Please also report the loss ablations at the Stage II checkpoint, before Stage III, so the two stages are not mixed.

  • Can you compare against PISA on Open-Sora 1.2 and against VideoREPA / WISA on CogVideoX-5B, under your benchmarks?

Reviewer_9Q95OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Phys4D post-trains pretrained video backbones in three stages: stage 1 introduces auxiliary heads to expose explicit depth and flow, stage 2 applies supervision from simulation data with dynamics-aware gating, stage 3 further optimizes on lifted 4D trajectory alignment of this data. Quantitative results show consistent improvements over the base models with ablations demonstrating the contribution of each stage. Qualitative results are a bit less convincing in terms of quality.

Critical strengths
  • The post-training strategy is well structured, with each stage addressing a distinct objective. Ablations are thorough and provide good support for the effectiveness of the proposed training stages.
Critical weaknesses
  • Quality of post-trained qualitative results seem to be suboptimal despite being better than base model. Supplementary has quantitative results that state this is better than other physics-grounded video generation models (RealWonder, etc.) but no qualitative results to support. Side-by-side comparison with such baselines would be appreciated to understand why.
Questions for authors
  • Could the authors add discussion and/or compare with existing physics-aware post-training methods, such as PISA Experiments, PhyGDPO, PhyWorld, etc., to better establish the advantages of the proposed three-stage pipeline for post-training?

  • How diverse is the training data, do authors think the limited diversity or the simulation-to-real gap would explain the remaining qualitative artifacts mentioned in weakness section?

  • The main paper provides no qualitative comparisons, with all examples in supplementary materials. Could the authors move representative visual comparisons in the main paper, especially given the focus on video generation?

Reviewer_9wUXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Phys4D, a framework for transferring simulation-derived physical structure into pretrained video diffusion models while constraining appearance drift. The framework first trains auxiliary depth and optical-flow heads using pseudo-labeled real and self-generated videos. It then jointly trains structural heads and gated adapters using simulation-derived depth, motion, and temporal-consistency supervision, together with restricted high-noise flow matching on simulator-video latents. Finally, a simulator-paired policy-optimization stage updates the adapters using a reward that compares dynamic-region 3D point sets within a local temporal neighborhood. The pretrained backbone remains frozen throughout training. Supported by a coupled-physics simulation pipeline, the method is evaluated on CogVideoX, Wan2.2, and Open-Sora. The reported results show improvements on external RGB-level physics benchmarks and geometry–motion diagnostics, while largely preserving the measured visual quality.

Critical strengths
  1. Consistent improvements in observable physics across multiple backbones. Phys4D reports improvements on Physics-IQ, VBench-2.0 Physics, and PhyGenBench across CogVideoX, Wan2.2, and Open-Sora. For example, the Wan2.2 Physics-IQ Original score increases from 16.8 to 30.9. These evaluations assess generated RGB videos without relying on the proposed depth and motion heads, providing meaningful evidence that the method improves observable generation behavior. The consistency across backbones and benchmarks supports the practical value of the framework.

  2. Useful evidence for transferring physical structure while retaining measured visual quality. Removing simulator-video RGB flow matching still achieves a Physics-IQ score of 25.7, retaining approximately 81% of the full Stage-II improvement over the base model. Meanwhile, the complete framework largely maintains the evaluated visual quality; for Wan2.2, VBench Visual Avg. changes from 0.7521 to 0.7519. These results support the value of structural supervision and suggest a practical approach to benefiting from simulation without substantially degrading the measured appearance quality.

Critical weaknesses
  1. The additional contribution over closely related physics post-training methods is not sufficiently established. PhyCo already combines simulation-based physical supervision, trainable adaptation of a frozen video backbone, and reward-based refinement. Phys4D differs in its structural interfaces, gating mechanisms, and geometric feedback, but the manuscript should more clearly explain and demonstrate what these differences contribute. The quantitative comparisons with PhysGen3D, TesserAct, and RealWonder in Appendix F.6 are useful, although their different input interfaces limit conclusions about the proposed post-training strategy itself. Furthermore, the comparative figures and project videos focus on the base Wan2.2 model; side-by-side visualizations against these competing methods are missing. This limits direct assessment of the claimed advantages in contact behavior, deformation, and temporal consistency. Clearer differentiation from close prior work, focused controls isolating the proposed gates, and representative cross-method visual comparisons would strengthen the evidence for the framework’s additional value.

  2. Key details of the training and structural evaluation protocols remain underspecified. The manuscript does not clearly explain which denoising states supply features to the depth and motion heads, how the generated RGB frames used by the Stage-II warp objective are obtained, or how gradients propagate through these computations. It also needs to specify how external monocular depth is converted to the scale used in simulator-reference evaluations, since this affects the interpretation of the structural comparisons. Additionally, the project page describes a global 4D Chamfer reward, whereas the manuscript specifies locally time-relaxed frame matching with a bounded reward. The authors should identify the formulation and configuration used for the reported results. These ambiguities hinder reproducibility and make the structural gains harder to assess. Training-update pseudocode, the actual configurations, and a consistent evaluation specification would substantially address this concern.

Questions for authors
  1. What is the principal methodological advance over closely related physics post-training approaches? Please clarify the differences from PhyCo in the roles of simulation supervision, backbone adaptation, and reward-based refinement, while accounting for the different conditioning requirements and objectives. Which existing results most directly establish the additional value of Phys4D’s structural interface and gating mechanisms? The project page reports noise-interval ablations; were their data, optimization budgets, and other objectives held fixed? Is there a controlled comparison that isolates the learned spatial gate from the remaining components?

  2. Could you specify the complete computation used for a Stage-II training update? Appendix C describes the losses and parameter-level gradient routing, but the underlying forward computation remains unclear. At which denoising timesteps or latent states are the features supplied to the depth and motion heads extracted? Are the generated RGB frames in the warp objective obtained from a single-step clean-latent estimate or a multi-step rollout, and how are their associated flow predictions obtained? Pseudocode identifying the forward passes, decoding operations, and any stop-gradient operations would clarify how structural supervision updates the generator.

  3. What exact depth-scale conversion is used in the structural evaluations? Appendix H.1 states that the same external-depth checkpoint and scale conversion are used across OTF rows. Please specify this conversion, including whether scale or shift is estimated using training data, initial-frame information, or test-sequence ground truth. If reference-assisted alignment is used, at what granularity is it performed, and how is it handled for the native Phys4D interface? This information is necessary to interpret the geometry and trajectory comparisons in Tables 21–23.

  4. Which alignment reward was used for the reported Stage-III results? The project page describes a global 4D Chamfer objective with a negative-distance reward, whereas the PDF specifies locally time-relaxed frame matching and a bounded reward. Is the website an outdated or simplified description? Please confirm the implemented formulation, temporal matching window, temporal penalty, and reward normalization used for the main results and ablations, and clarify whether these settings were otherwise held fixed across the reported variants.

  5. Could you provide qualitative comparisons from the existing cross-method evaluation? Appendix F.6 reports quantitative comparisons with PhysGen3D, TesserAct, and RealWonder, but the comparative figures and project videos focus on the base Wan2.2 model. Representative side-by-side clips from the already evaluated AC35 cases, using the same initial observation and event description through each method’s native interface, would help assess the differences in contact behavior, deformation, and temporal consistency. Please explain how the examples are selected and include representative failure cases, so that the visual evidence can be interpreted alongside the aggregate scores.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.