- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes LPhyVideo, a physics-guided video prediction framework that follows a video-to-physics-to-video paradigm. It tracks system-relevant keypoints from observed video, constructs interpretable physical states, learns a state-space dynamics model (via parameter identification, Neural ODEs, or hybrid models), rolls the model forward, maps predicted states back to image space to build a preliminary video and edit mask, and uses a VACE-based video diffusion model for constrained refinement. Experiments on six physical systems (Double Pendulum, Rolling Orange, Spring Mass, Torricelli Motion, Forced 3D Robot Arm, Moving Camera 3D Robot Arm) show improvements over generic video generation and guided baselines on Physics-IQ component metrics and Trajectory Error
Critical strengths
1.Well-motivated paradigm. Decoupling physical evolution from appearance synthesis—determining “what happens” before “how it looks”—is a sensible way to improve physical reliability in video prediction.
2.Modular and inspectable pipeline. The video-to-physics-to-video formulation is clear, and the intermediate physical states are interpretable. The use of preliminary video and edit mask as a bridge to generative refinement is a neat design.
3.Strong empirical results. LPhyVideo achieves the best or near-best scores on Spatial IoU, Weighted Spatial IoU, MSE, and competitive Spatiotemporal IoU / Trajectory Error across six systems. The “before refinement” variant already outperforms guided baselines, showing the value of the physical rollout.
Critical weaknesses
1.Limited generality and scalability. The method requires system-specific tracking, known geometry, camera calibration, and hand-designed physical state representations. It only applies to systems with compact interpretable states and stable backgrounds. This severely limits applicability to arbitrary videos. The paper acknowledges this, but it remains a major limitation for a general video prediction framework.
2.Simple physical systems. The six systems are relatively low-dimensional and well-controlled. There are no complex scenes, deformable objects, fluids, or multi-body interactions. Claims about physical consistency may not generalize to more complex dynamics.
3.Novelty and positioning. The idea of using physical simulation to guide video generation is not entirely new (e.g., PhysGen, Neural Gaussian Force Fields, Feng et al.). The specific combination of video-to-physics-to-video with preliminary video and edit mask is a contribution, but the individual components are standard. The paper should better differentiate from prior work and clarify what is fundamentally new.
Questions for authors
Please see weakness.
3 comments
Response to W3 (2/3)
W3:
It is true that several individual components of LPhyVideo, such as physical modeling and diffusion-based video generation, have been explored in prior work. Our contribution lies in establishing an explicit and geometry-consistent connection between video-derived physical dynamics and future video generation, rather than simply combining a physics simulator with a video diffusion model.
The key methodological distinction is the physical representation and the role of dynamics prediction in the generation pipeline. As summarized in the following Table, existing approaches either use externally simulated physical states, learn dynamics over scene representations, or extrapolate image-space trajectories. In contrast, LPhyVideo reconstructs interpretable physical states from observed videos, learns their coupled state-space dynamics, and maps the predicted physical configurations back to image space to construct structured visual guidance.
| Method | Physical Representation | Dynamics Modeling | Video Generation |
|---|---|---|---|
| PhysGen (2024) | 2D rigid-body states | Physical simulation | Rendering + diffusion |
| NGFF (2026) | 3D Gaussian representations | Neural force fields | Dynamic Gaussian rendering |
| Feng et al. (2025) | Image-space trajectories | Symbolic coordinate extrapolation | Trajectory-guided I2V |
| LPhyVideo | Video-derived physical states | State-space dynamics learning | Geometry-consistent reconstruction; Preliminary video + VACE |
Specifically, PhysGen (Liu et al., 2024) uses rigid-body simulation to guide image-to-video generation, whereas LPhyVideo reconstructs physical states directly from observed videos and learns their subsequent evolution. NGFF (Li et al., 2026) models 3D dynamics through neural force fields on Gaussian scene representations, while LPhyVideo operates on compact, interpretable physical states and explicitly incorporates system geometry and camera projection. Feng et al. (2025) discovers symbolic functions for tracked image-space coordinates and extrapolates these trajectories for video guidance. In contrast, LPhyVideo models the coupled evolution of physically interpretable state variables before projecting the predicted configurations back into image space.
To further distinguish physical state dynamics modeling from image-space trajectory extrapolation, we have added a direct experimental comparison with Feng et al. (2025) on the Double Pendulum (Feng et al. (2025) can not handle 3D even Moving Camera situations), as detailed in the newly added Appendix A.6. Since the official implementation is unavailable, we independently implemented their equation discovery approach following the published methodological description. Both methods use the same observation interval, prediction horizon, VACE-1.3B backbone, reference image, and generation settings, with five independent video generation runs.
Importantly, we first evaluate the predicted trajectories independently of video generation. As shown in Figure 22, although both methods capture the dominant horizontal oscillations, Feng et al.'s independently discovered coordinate functions exhibit noticeable deviations in the vertical trajectories, particularly overestimating the second endpoint's oscillation amplitude. LPhyVideo better preserves the coupled physical evolution of the two pendulum links. This comparison highlights the difference between extrapolating individual image-space coordinates and modeling coupled physical states.
Response to W3 (3/3)
The video-level results provide further evidence. As reported in Appendix A.6, Table 9, LPhyVideo outperforms Feng et al. across all five evaluation metrics under the matched generation configuration. Specifically, Spatial IoU improves from 0.70278 to 0.80072, Spatiotemporal IoU from 0.44997 to 0.51232, and Weighted Spatial IoU from 0.57637 to 0.65995. Meanwhile, MSE decreases from 0.01135 to 0.00238, and Trajectory Error decreases from 0.01075 to 0.00905. Figure 23 and the corresponding video comparison further illustrate the resulting differences in generated physical motion.
Finally, the backbone scale experiment in Table 4 further supports this distinction. LPhyVideo with VACE-1.3B on a single A40 achieves comparable or better overall performance than VACE-14B on eight A40 GPUs, and increasing the generative backbone scale does not consistently improve the five evaluation metrics. This supports our design principle that physical dynamics modeling establishes the predicted evolution, while the diffusion model primarily refines visual appearance and local details.
That is, the novelty of LPhyVideo is not the individual use of physical models or video diffusion, but the explicit video-to-physics-to-video formulation that connects learned physical state dynamics with geometry-consistent future video reconstruction. The additional comparison with Feng et al. provides direct empirical evidence for the importance of this physical state-space formulation.
References
Liu, S., Ren, Z., Gupta, S., Wang, S. (2025). PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15140. Springer, Cham. https://doi.org/10.1007/978-3-031-73007-8_21
Li, S., Shen, R., Ni, J., Pan, C., Zhang, C., & Zhu, Y. (2026, April). Learning physics-grounded 4D dynamics with neural Gaussian force fields. In International Conference on Learning Representations (Vol. 2026, pp. 81165-81207). https://proceedings.iclr.cc/paper_files/paper/2026/hash/8365c06948c91169bb74b1b0629c68a2-Abstract-Conference.html
Feng, T., Zhao, X., Chen, Z., Wong, T. T., Rezatofighi, H., Haffari, G., & Qu, L. (2025). Physics-Grounded Motion Forecasting via Equation Discovery for Trajectory-Guided Image-to-Video Generation (arXiv:2507.06830). arXiv. https://doi.org/10.48550/arXiv.2507.06830
Response to W1 and W2 (1/3)
We sincerely thank the reviewer for the thoughtful assessment and constructive feedback. We appreciate the recognition of our physical dynamics modeling paradigm and its empirical performance. In the revised manuscript (with all modifications highlighted in red), we have made several targeted improvements to address the concerns regarding generality, experimental coverage, and methodological novelty (in updated Figure 1). In particular, we have added a direct experimental comparison with Feng et al. (2025), including both physical trajectory prediction and video generation under matched settings. We have also clarified the supported system scope, camera-motion handling, and tracking procedure in the revised manuscript. Our responses are detailed below.
W1
Indeed, LPhyVideo is not intended as a universal predictor for arbitrary videos. It targets structured physical systems whose observable configurations can be reconstructed into compact physical states using system-specific geometric and kinematic relationships. Importantly, the same video-to-physics-to-video formulation is consistently applied across all six systems in the manuscript, while the state representations and dynamics models are adapted to their respective physical structures.
Regarding visual tracking, we have clarified the feature selection and tracking procedure in Appendix A.4. Specifically, tracked features are selected according to their physical relevance, visual distinguishability, and geometric structure. System-specific combinations of color-based segmentation, template matching, and geometry-constrained tracking are used, with only a small number of points or regions manually initialized when necessary. The same tracking pipeline is consistently applied to both ground truth and generated videos for evaluation.
A key capability beyond fixed-camera planar settings is demonstrated by our Moving Camera 3D Robot Arm experiment. LPhyVideo reconstructs physical configurations in a common world coordinate frame and learns dynamics independently of camera poses. During inference, the predicted physical configurations can be projected using a specified future camera trajectory, without retraining the dynamics model for different camera routes. This is detailed in Appendix A.1 and experimentally evaluated in Appendix A.5.7.
We have also clarified the scope and limitations in the revised Introduction and Conclusion. The current framework focuses on compact, interpretable physical states, while contact-rich interactions, deformable bodies, and spatially distributed dynamics remain beyond its current scope.
W2
Although our current benchmark does not cover highly complex contact-rich or deformable systems, the six experiments are designed to evaluate substantially different physical dynamics and observation conditions within a unified framework. Specifically, they cover nonlinear coupled multi-degree-of-freedom (MDOF) articulation (Double Pendulum), translation-rotation coupling under a no-slip constraint (Rolling Orange), dissipative oscillation (Spring Mass), nonlinear free-surface evolution (Torricelli Motion), externally forced 3-DOF robotic articulation (Forced 3D Robot Arm), and 3-DOF physical motion under continuous camera movement (Moving Camera 3D Robot Arm). Importantly, the framework is not restricted to single-degree-of-freedom or simple translational motion: it includes coupled nonlinear MDOF systems and 3D articulated dynamics, providing a broader evaluation of physical state reconstruction and dynamics prediction.
These experiments evaluate the framework across different physical state representations, dynamical mechanisms, and camera configurations, rather than repeatedly testing the same type of motion. The results include both recovered physical state trajectories (Figure 2 and Appendix A.5) and generated video comparisons (Table 2 and Figure 3). The detailed per-system results are additionally provided in Appendix A.5.
Furthermore, the full video comparisons in the Supplementary Material show the continuous predicted evolution across all six systems and provide visual evidence complementary to the quantitative metrics and selected frames in the manuscript.