acceptodds
Withdrawn from ICLR 2027

Joint Modeling of Visual and Physical Fields for Consistent Video Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Text-to-video generation models have achieved remarkable progress in visual fidelity and temporal coherence, yet generated videos can still exhibit implausible motion dynamics and inconsistent interactions. Existing efforts incorporate physical information (e.g., structured motion or geometric knowledge) mainly via external guidance or constraints, failing to directly consider it within the visual generative representation. In this paper, we proposed to employ physical field signals (e.g., velocity, acceleration, and depth) as a quantitative outcome of motion which naturally serves as unified structured representation across different motion scenarios, and then simultanesouly optimize phyical and visual signals in representation space. However, physical field signals is sparse and noisy in real videos, posing siginificant difficulties in the joint optimization. To tackle these issues, we propose a novel VideoPF method that jointly models video data and physical field signals within a unified generative framework, enabling the synthesis of videos with improved motion consistency. Specifically, we first analyze the degree to which generative models capture physical dynamics. Based on this analysis, we design an adaptive physical-aware data pipeline. To facilitate effective fusion of visual and physical information, we introduce a Gated Physical DiT that selectively integrates physical field signals into the visual generation process. To enhance learning stability and encourage physically plausible dynamics, we propose a Physical Consistency Refinement mechanism and dynamically assess the model’s understanding of physical information. Extensive experiments demonstrate the effectiveness of our proposed VideoPF framework.

Primary area:
generative models
Keywords:
Video Generation, Conditional Generation, Physics-Informed Generation, Joint Modeling, Temporal Consistency

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zQrYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper jointly generates video, velocity, acceleration, and depth representations, integrating this information through a gated DiT. It combines real-video filtering, simulated data, and error-weighted training to improve the plausibility of dynamics. The experiments evaluate video quality, physical commonsense, and trajectories in simple scenes.

Critical strengths

Using simulated data to supplement inaccurate physical-field supervision in real videos is a valuable idea and provides more reliable training signals for some dynamic scenes. Improvements over the original backbone and the trajectory results in the appendix support the effectiveness of this training scheme in some scenarios.

Critical weaknesses
  1. “Physics-consistent refinement” supervises reconstruction errors within the physical-field branches without directly checking whether these fields correspond to the actual motion in the generated video. Shared hidden states allow information exchange, but the relationship between more accurate physical-field predictions and more consistent video motion remains unverified. The trajectory experiments in the appendix support improved motion in some scenarios, but do not resolve this question of correspondence across branches.
  2. The positioning relative to prior work is inaccurate. VideoJAM already jointly models appearance and motion, so moving physical information from external guidance into joint generation is insufficient by itself to establish the main novelty. The paper does not clearly explain what new modeling capabilities arise from adding acceleration, depth, and gating.
Questions for authors

Are the jointly generated physical fields consistent with the actual motion in the generated video? Compared with existing joint appearance–motion modeling, what specific, previously unresolved problem does this work address?

Reviewer_1cs7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VideoPF, which improves physical plausibility in text-to-video generation by jointly modeling visual content and physical field signals (velocity, acceleration, depth)

Critical strengths

Incorporates physical field signals directly into the generative representation space rather than external guidance, providing a scalable, architecturally integrated solution. The Gated Physical DiT and adaptive data pipeline thoughtfully address sparse and noisy physical signals in real videos.

Critical weaknesses
  1. Base model (1.3B) is significantly smaller than baselines. Table 12 shows random sampling achieves competitive results (0.778 vs 0.839), making it unclear whether improvements stem from physical modeling or data augmentation. Lacks controlled comparisons on identical data.

  2. VideoPhy-Bench physical commonsense only 0.215, identical to CogVideoX and worse than VideoREPA (0.220). Even on best-performing VideoJAM-Bench (0.262), improvement is only ~6-7%. Absolute errors remain substantial (e.g., RMSE 1.2046 for buoyancy).

  3. Lacks comparison with recent physics-aware methods like WISA. Without this comparison, it's unclear whether improvements are competitive with state-of-the-art in physics-aware video generation specifically.

Questions for authors

See weaknesses.

Reviewer_pjZgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces VideoPF, a method that jointly models video, velocity, acceleration and depth. The proposed method combines 3 components: 1) a data pipeline that uses loss-based filtering and VLM annotations to identify videos with physics-related phenomena, 2) a "Gated Physical DiT" architecture, where a pretrained DiT is augmented with an attention gating mechanism and is trained to jointly generate visual and physical information via expansions of its input and output projections, and 3) "Physical Consistency Refinement", which involves using reconstruction errors on the physical fields to reweight the training loss. VideoPF demonstrates improvements on motion consistency, visual fidelity, and semantic adherence over its Wan2.1 backbone across a variety of benchmarks.

Critical strengths
  1. The experiments demonstrate that the proposed method, taken as a whole, has improved performance relative to the Wan2.1 base model.

Based on a variety of benchmarks, a user-preference study, and comparisons to simulator predictions, the proposed VideoPF approach demonstrates strong empirical performance and conclusively outperforms the Wan2.1 base model. This suggests that at least some components of the method may be broadly useful for improving the generation quality and physical understanding of video diffusion models.

Critical weaknesses
  1. Experimental weaknesses.

Although table 12 is stated to ablate the effect of the data pipeline, the results for the "random" baseline ("results obtained by randomly sampling an equal amount of data for experimentation") seems to exactly match the results for the "Wan2.1" baseline in table 2, as well as table 13's "w/o all" results. This makes it unclear whether these ablations represent actual controlled comparisons in which only one component is removed at a time, or may be a reporting error.

Next, although the provided baselines of Wan 2.1 and VideoREPA are reasonable, there are several notable baselines which were not compared to, such as WISA (Wang et al. 2025), NewtonGen (Yuan et al. 2026), and PHANTOM (Shen et al. 2026). In particular, PHANTOM has some methodological similarities with VideoPF, as both involve adapting a WAN DiT to jointly generate both video and physical variables. Providing a discussion and controlled experimental comparison between the two methods would be valuable for establishing both the novelty of VideoPF and its performance relative to SOTA.

  1. Method design

It is stated that the gating mechanism of the Gated Physical DiT "adaptively modulates the interaction among visual and physical representations, allowing the model to control the contribution of heterogeneous physical signals during joint modeling". However, as written, the visual and physical representations are mixed in the model's input, while the gating scales intermediate features of attention. Nothing in this attention gate requires the inputs to contain physical fields. Indeed, similar attention gating mechanisms are applied as a generic architectural improvement, unrelated to physics, in existing transformer models (e.g. Qiu et al. 2025). The lack of a controlled comparison for physical alignment with and without the attention gate makes it unclear whether it actually provides any kind of meaningful advantage for modulating visual and physical representations. In particular, the presented experiments fail to rule out the possibility that gated attention is an orthogonal and non-novel architectural change that generically improves performance while being unrelated to the paper's overarching approach of modeling physical signals. Combined with the possibility of uncontrolled ablations as described above, it is impossible to precisely determine which components of the proposed method actually lead to better physical understanding.

References:

  • Wang, J., Ma, A., Cao, K., Zheng, J., Feng, J., Zhang, Z., Pang, W., & Liang, X. (2025). WISA: World simulator assistant for physics-aware text-to-video generation. In D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, & N. Chen (Eds.), Advances in neural information processing systems (Vol. 38, pp. 6144–6172)
  • Yuan, Y., Wang, X., Wickremasinghe, T., Nadir, Z., Ma, B., & Chan, S. H. (2026). NewtonGen: Physics-consistent and controllable text-to-video generation via neural Newtonian dynamics. In The Fourteenth International Conference on Learning Representations.
  • Shen, Y., Xiong, J., Yu, T., & Lourentzou, I. (2026). PHANTOM: Physics-infused video generation via joint modeling of visual and latent physical dynamics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 11185–11194).
  • Qiu, Z., Wang, Z., Zheng, B., Huang, Z., Wen, K., Yang, S., Men, R., Yu, L., Huang, F., Huang, S., Liu, D., Zhou, J., & Lin, J. (2025). Gated attention for large language models: Non-linearity, sparsity, and attention-sink-free. In D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, & N. Chen (Eds.), Advances in neural information processing systems (Vol. 38, pp. 110931–110957)
Questions for authors
  1. Why do the Wan2.1 baseline results in table 2 exactly match table 12's "random" baseline results, and table 13's "w/o all" results? Are the baseline variants ones where the studied component (the data mixture for table 12, and the physical fields for table 13) is removed relative to the full, proposed method (as suggested by the table captions), or are they variants where the studied components are added on top of the baseline Wan2.1 model?

  2. Is the adaptive physical-aware data pipeline "adaptive" in the sense that the data mixture is modified online during training, or does data selection only occur once at the beginning of training? For physical consistency refinement, are the physical error estimates used for loss weighting detached in the objective, or are gradients backpropogated through them as well?

  3. Could further details be provided for reproducibility? E.g. the convention for diffusion timesteps / noise levels, the per-GPU batch size, optimizer momentum, the number of Muon orthogonalization steps and their coefficients, etc. How are the physical fields actually preprocessed before being fed into the model, in terms of normalization or scaling (which affects their relative contribution to the loss compared to pixel MSE)? How was the acceleration physical field constructed, when there is occlusion or camera motion? Which variant of DepthAnything was used to produce depth estimates? Could more details be provided about the "Hilbert Curve" described in Appendix F?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.