acceptodds
Under review as a conference paper at ICLR 2027

Physics-Guided Video Prediction via Learned Dynamics from Video Observations

Abstract

Future video prediction for physical systems must preserve both visual realism and physical dynamic evolutions. Existing video generators often model motion non-physically or rely on externally user-defined manipulations, while methods that infer physical dynamics from video typically focus on identifying parameters or trajectory prediction, with limited integration into future video generation. This leaves a research gap between explicit physical prediction and physically grounded future video generation. We introduce LPhyVideo ("L" stands for "Learned"), a physics-guided video prediction framework via learned physical dynamics from the video, within a video-to-physics-to-video formulation for systems whose low-dimensional physical states are reconstructed from visual observations. LPhyVideo maps tracked key pixel locations to physically interpretable states, and rolls out via a learned state-space dynamics model; subsequently, LPhyVideo reconstructs the predicted states back into image space, and with the aid of a reference frame, converted into a video and edit mask for a constrained refinement of final generation. Across six diverse physical systems, LPhyVideo achieves the highest average Spatial IoU, Spatiotemporal IoU, Weighted Spatial IoU and the lowest MSE and Trajectory Error among comparisons.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Physics-guided Video Prediction, Dynamics Learing from Videos, Controllable Video Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VBVzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LPhyVideo, a physics-guided video prediction framework that follows a video-to-physics-to-video paradigm. It tracks system-relevant keypoints from observed video, constructs interpretable physical states, learns a state-space dynamics model (via parameter identification, Neural ODEs, or hybrid models), rolls the model forward, maps predicted states back to image space to build a preliminary video and edit mask, and uses a VACE-based video diffusion model for constrained refinement. Experiments on six physical systems (Double Pendulum, Rolling Orange, Spring Mass, Torricelli Motion, Forced 3D Robot Arm, Moving Camera 3D Robot Arm) show improvements over generic video generation and guided baselines on Physics-IQ component metrics and Trajectory Error

Critical strengths

1.Well-motivated paradigm. Decoupling physical evolution from appearance synthesis—determining “what happens” before “how it looks”—is a sensible way to improve physical reliability in video prediction.

2.Modular and inspectable pipeline. The video-to-physics-to-video formulation is clear, and the intermediate physical states are interpretable. The use of preliminary video and edit mask as a bridge to generative refinement is a neat design.

3.Strong empirical results. LPhyVideo achieves the best or near-best scores on Spatial IoU, Weighted Spatial IoU, MSE, and competitive Spatiotemporal IoU / Trajectory Error across six systems. The “before refinement” variant already outperforms guided baselines, showing the value of the physical rollout.

Critical weaknesses

1.Limited generality and scalability. The method requires system-specific tracking, known geometry, camera calibration, and hand-designed physical state representations. It only applies to systems with compact interpretable states and stable backgrounds. This severely limits applicability to arbitrary videos. The paper acknowledges this, but it remains a major limitation for a general video prediction framework.

2.Simple physical systems. The six systems are relatively low-dimensional and well-controlled. There are no complex scenes, deformable objects, fluids, or multi-body interactions. Claims about physical consistency may not generalize to more complex dynamics.

3.Novelty and positioning. The idea of using physical simulation to guide video generation is not entirely new (e.g., PhysGen, Neural Gaussian Force Fields, Feng et al.). The specific combination of video-to-physics-to-video with preliminary video and edit mask is a contribution, but the individual components are standard. The paper should better differentiate from prior work and clarify what is fundamentally new.

Questions for authors

Please see weakness.

3 comments
AuthorsOpenReview ↗

Response to W1 and W2 (1/3)

We sincerely thank the reviewer for the thoughtful assessment and constructive feedback. We appreciate the recognition of our physical dynamics modeling paradigm and its empirical performance. In the revised manuscript (with all modifications highlighted in red), we have made several targeted improvements to address the concerns regarding generality, experimental coverage, and methodological novelty (in updated Figure 1). In particular, we have added a direct experimental comparison with Feng et al. (2025), including both physical trajectory prediction and video generation under matched settings. We have also clarified the supported system scope, camera-motion handling, and tracking procedure in the revised manuscript. Our responses are detailed below.

W1

Indeed, LPhyVideo is not intended as a universal predictor for arbitrary videos. It targets structured physical systems whose observable configurations can be reconstructed into compact physical states using system-specific geometric and kinematic relationships. Importantly, the same video-to-physics-to-video formulation is consistently applied across all six systems in the manuscript, while the state representations and dynamics models are adapted to their respective physical structures.

Regarding visual tracking, we have clarified the feature selection and tracking procedure in Appendix A.4. Specifically, tracked features are selected according to their physical relevance, visual distinguishability, and geometric structure. System-specific combinations of color-based segmentation, template matching, and geometry-constrained tracking are used, with only a small number of points or regions manually initialized when necessary. The same tracking pipeline is consistently applied to both ground truth and generated videos for evaluation.

A key capability beyond fixed-camera planar settings is demonstrated by our Moving Camera 3D Robot Arm experiment. LPhyVideo reconstructs physical configurations in a common world coordinate frame and learns dynamics independently of camera poses. During inference, the predicted physical configurations can be projected using a specified future camera trajectory, without retraining the dynamics model for different camera routes. This is detailed in Appendix A.1 and experimentally evaluated in Appendix A.5.7.

We have also clarified the scope and limitations in the revised Introduction and Conclusion. The current framework focuses on compact, interpretable physical states, while contact-rich interactions, deformable bodies, and spatially distributed dynamics remain beyond its current scope.

W2

Although our current benchmark does not cover highly complex contact-rich or deformable systems, the six experiments are designed to evaluate substantially different physical dynamics and observation conditions within a unified framework. Specifically, they cover nonlinear coupled multi-degree-of-freedom (MDOF) articulation (Double Pendulum), translation-rotation coupling under a no-slip constraint (Rolling Orange), dissipative oscillation (Spring Mass), nonlinear free-surface evolution (Torricelli Motion), externally forced 3-DOF robotic articulation (Forced 3D Robot Arm), and 3-DOF physical motion under continuous camera movement (Moving Camera 3D Robot Arm). Importantly, the framework is not restricted to single-degree-of-freedom or simple translational motion: it includes coupled nonlinear MDOF systems and 3D articulated dynamics, providing a broader evaluation of physical state reconstruction and dynamics prediction.

These experiments evaluate the framework across different physical state representations, dynamical mechanisms, and camera configurations, rather than repeatedly testing the same type of motion. The results include both recovered physical state trajectories (Figure 2 and Appendix A.5) and generated video comparisons (Table 2 and Figure 3). The detailed per-system results are additionally provided in Appendix A.5.

Furthermore, the full video comparisons in the Supplementary Material show the continuous predicted evolution across all six systems and provide visual evidence complementary to the quantitative metrics and selected frames in the manuscript.

AuthorsOpenReview ↗

Response to W3 (2/3)

W3:

It is true that several individual components of LPhyVideo, such as physical modeling and diffusion-based video generation, have been explored in prior work. Our contribution lies in establishing an explicit and geometry-consistent connection between video-derived physical dynamics and future video generation, rather than simply combining a physics simulator with a video diffusion model.

The key methodological distinction is the physical representation and the role of dynamics prediction in the generation pipeline. As summarized in the following Table, existing approaches either use externally simulated physical states, learn dynamics over scene representations, or extrapolate image-space trajectories. In contrast, LPhyVideo reconstructs interpretable physical states from observed videos, learns their coupled state-space dynamics, and maps the predicted physical configurations back to image space to construct structured visual guidance.

MethodPhysical RepresentationDynamics ModelingVideo Generation
PhysGen (2024)2D rigid-body statesPhysical simulationRendering + diffusion
NGFF (2026)3D Gaussian representationsNeural force fieldsDynamic Gaussian rendering
Feng et al. (2025)Image-space trajectoriesSymbolic coordinate extrapolationTrajectory-guided I2V
LPhyVideoVideo-derived physical statesState-space dynamics learningGeometry-consistent reconstruction; Preliminary video + VACE

Specifically, PhysGen (Liu et al., 2024) uses rigid-body simulation to guide image-to-video generation, whereas LPhyVideo reconstructs physical states directly from observed videos and learns their subsequent evolution. NGFF (Li et al., 2026) models 3D dynamics through neural force fields on Gaussian scene representations, while LPhyVideo operates on compact, interpretable physical states and explicitly incorporates system geometry and camera projection. Feng et al. (2025) discovers symbolic functions for tracked image-space coordinates and extrapolates these trajectories for video guidance. In contrast, LPhyVideo models the coupled evolution of physically interpretable state variables before projecting the predicted configurations back into image space.

To further distinguish physical state dynamics modeling from image-space trajectory extrapolation, we have added a direct experimental comparison with Feng et al. (2025) on the Double Pendulum (Feng et al. (2025) can not handle 3D even Moving Camera situations), as detailed in the newly added Appendix A.6. Since the official implementation is unavailable, we independently implemented their equation discovery approach following the published methodological description. Both methods use the same observation interval, prediction horizon, VACE-1.3B backbone, reference image, and generation settings, with five independent video generation runs.

Importantly, we first evaluate the predicted trajectories independently of video generation. As shown in Figure 22, although both methods capture the dominant horizontal oscillations, Feng et al.'s independently discovered coordinate functions exhibit noticeable deviations in the vertical trajectories, particularly overestimating the second endpoint's oscillation amplitude. LPhyVideo better preserves the coupled physical evolution of the two pendulum links. This comparison highlights the difference between extrapolating individual image-space coordinates and modeling coupled physical states.

AuthorsOpenReview ↗

Response to W3 (3/3)

The video-level results provide further evidence. As reported in Appendix A.6, Table 9, LPhyVideo outperforms Feng et al. across all five evaluation metrics under the matched generation configuration. Specifically, Spatial IoU improves from 0.70278 to 0.80072, Spatiotemporal IoU from 0.44997 to 0.51232, and Weighted Spatial IoU from 0.57637 to 0.65995. Meanwhile, MSE decreases from 0.01135 to 0.00238, and Trajectory Error decreases from 0.01075 to 0.00905. Figure 23 and the corresponding video comparison further illustrate the resulting differences in generated physical motion.

Finally, the backbone scale experiment in Table 4 further supports this distinction. LPhyVideo with VACE-1.3B on a single A40 achieves comparable or better overall performance than VACE-14B on eight A40 GPUs, and increasing the generative backbone scale does not consistently improve the five evaluation metrics. This supports our design principle that physical dynamics modeling establishes the predicted evolution, while the diffusion model primarily refines visual appearance and local details.

That is, the novelty of LPhyVideo is not the individual use of physical models or video diffusion, but the explicit video-to-physics-to-video formulation that connects learned physical state dynamics with geometry-consistent future video reconstruction. The additional comparison with Feng et al. provides direct empirical evidence for the importance of this physical state-space formulation.

References

Liu, S., Ren, Z., Gupta, S., Wang, S. (2025). PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15140. Springer, Cham. https://doi.org/10.1007/978-3-031-73007-8_21

Li, S., Shen, R., Ni, J., Pan, C., Zhang, C., & Zhu, Y. (2026, April). Learning physics-grounded 4D dynamics with neural Gaussian force fields. In International Conference on Learning Representations (Vol. 2026, pp. 81165-81207). https://proceedings.iclr.cc/paper_files/paper/2026/hash/8365c06948c91169bb74b1b0629c68a2-Abstract-Conference.html

Feng, T., Zhao, X., Chen, Z., Wong, T. T., Rezatofighi, H., Haffari, G., & Qu, L. (2025). Physics-Grounded Motion Forecasting via Equation Discovery for Trajectory-Guided Image-to-Video Generation (arXiv:2507.06830). arXiv. https://doi.org/10.48550/arXiv.2507.06830

Reviewer_GRhvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents LPhyVideo, a framework for future video prediction that follows a video-to-physics-to-video pipeline. It first tracks keypoints and reconstructs interpretable physical states, then predicts future motion through parameter identification or a Neural ODE. The predicted motion is used to construct a preliminary video and local editing masks, which guide a pretrained video generation model in refining the appearance. Experiments on four real-world planar physical systems and two synthetic robotic-arm scenarios show better overall visual agreement and trajectory accuracy than the selected baselines.

Critical strengths
  1. The separation between physical prediction and appearance generation is well motivated. Interpretable states allow predicted motion to be examined independently, while the preliminary video and local editing masks provide explicit structural guidance for the generation model.
  2. The experiments include useful controls. The paper reports results before refinement, provides some generation baselines with the same predicted trajectories or preliminary videos, and includes appropriate component ablations.
Critical weaknesses
  1. The contribution of the proposed method has not been sufficiently established. The approach relies on system-specific state definitions, known geometry, and custom rendering procedures. The robotic-arm experiments also use known control inputs, while the moving-camera experiment assumes a prescribed future camera trajectory. Without comparisons against directly related dynamics prediction methods using the same inputs and generation backbone, it is difficult to determine whether the gains come primarily from the proposed design or from additional prior information and implementation choices.
  2. The method lacks an explicit mechanism to ensure consistency between physical prediction and generative refinement. Even after the dynamics model determines future motion, the generation model may alter object structure and position; local editing masks alone do not guarantee that the final video follows the predicted trajectories. Table 6 shows that the mean trajectory error increases after refinement for all six systems, suggesting that refinement can compromise the physical accuracy established earlier in the pipeline.
Questions for authors
  1. What is the key new mechanism introduced by this work compared with existing methods that use dynamics predictions to guide video generation?
  2. Are non-editable regions strictly preserved through pixel compositing, or are they only supplied as conditioning to the generation model?
  3. Table 6 shows an increase in mean trajectory error after generative refinement for all six systems. What benefits justify this loss in trajectory accuracy?
  4. Are the reported means and standard deviations for the two IoU metrics in the Torricelli scenario in Table 7 correct? They appear inconsistent with the stated statistics over five nonnegative scores.
  5. Please provide the five individual IoU scores for the Torricelli scenario without the preliminary video in Table 7, and clarify the unit of aggregation used to compute the standard deviations. Do these discrepancies affect the aggregate results in Table 3?

I would raise my score if the authors satisfactorily address these concerns.

5 comments
AuthorsOpenReview ↗

Response to W1 and W2 (1/3)

We thank the reviewer for the thoughtful evaluation and constructive feedback. In response, we have revised the manuscript (with all modifications highlighted in red) to clarify the methodological contributions and scope of LPhyVideo, and added a new comparative experiment with Feng et al. (2025) to directly examine the difference between image-space trajectory extrapolation and physical state dynamics modeling. We have also clarified the role and limitations of generative refinement, including its effect on trajectory accuracy, and corrected the statistical results for the Torricelli experiments. The corresponding changes, additional quantitative results, and qualitative comparisons are provided in the revised manuscript and Supplementary Material (comparison videos), as detailed below.

W1:

The key contribution of LPhyVideo is not simply using predicted trajectories to guide video generation, but establishing an explicit video-to-physics-to-video formulation that reconstructs interpretable physical states, learns their coupled dynamics, and translates the predicted physical evolution into geometry-consistent visual guidance.

LPhyVideo uses system-specific geometry and kinematic structures, and known external inputs and future camera trajectories are required in the corresponding controlled experiments. These are explicit assumptions of our current framework, rather than claims of learning arbitrary physical systems without prior information. Our objective is to model physical evolution independently of camera-induced image motion and visual appearance. The revised manuscript clarifies these assumptions and the corresponding scope in Secs. 1, 3.1, 3.2, and Appendix A.1.

To provide a more direct comparison with related work, we have added a new experiment against Feng et al. (2025) in Appendix A.6. Both approaches use the same Double Pendulum (Feng et al. (2025) can not handle 3D even Moving Camera situations) observation interval (10 s), prediction horizon (5 s), reference image, and VACE-1.3B generation backbone. Importantly, we distinguish their prediction representations: Feng et al. discover symbolic functions directly over image-space trajectories, whereas LPhyVideo reconstructs physically interpretable link angles and angular velocities and models their coupled evolution in state space.

The new results show that LPhyVideo achieves better future trajectory predictions (Figure 22) and outperforms Feng et al. across all five video evaluation metrics under the shared generation configuration (Table 9). For example, Spatial IoU improves from 0.70278 to 0.80072, while MSE decreases from 0.01135 to 0.00238. The qualitative video comparison is provided in Figure 23 and the Supplementary Material.

Furthermore, the overall results (Table 2) and component ablations (Table 3) support the effectiveness of physical dynamics rollout and geometry-consistent visual guidance. The backbone scale experiment (Table 4) also shows that increasing VACE from 1.3B to 14B does not consistently improve performance. Together, these additional experiments support the contribution of explicitly modeling coupled physical state evolution, rather than relying solely on image-space trajectory extrapolation or a larger generative backbone.

W2:

LPhyVideo provides two complementary guidance signals during generative refinement. The preliminary video explicitly encodes the predicted physical configurations and their temporal evolution, while the edit mask identifies local regions surrounding the predicted moving structures that require visual refinement. Both are constructed from the predicted geometric trajectory and the appearance reference, providing temporally aligned structural and spatial guidance to VACE.

We clarify that these signals serve as conditioning rather than strict pixel-level or trajectory-level constraints. Consequently, VACE may introduce local changes to object appearance and landmark positions during refinement. We have explicitly clarified this limitation in the revised Sec. 3.3. The preliminary video remains independently available as the direct rendering of the predicted physical evolution, whereas the refined video provides an improved visual realization.

AuthorsOpenReview ↗

Response to Q1, Q2 and Q3 (2/3)

Q1:

The key distinction of LPhyVideo is its explicit reconstruction and prediction of physical states, rather than directly extrapolating image-space trajectories to guide video generation. This allows physical evolution to be modeled independently of camera-dependent image motion before being translated into visual guidance.

To substantiate this distinction, we have added a direct comparison with Feng et al. (2025) in Appendix A.6. As shown in Figure 22, LPhyVideo produces more accurate future landmark trajectories on the Double Pendulum. Under the same VACE-1.3B backbone and generation configuration, it also outperforms Feng et al. across all five evaluation metrics (Table 9), with MSE reduced from 0.01135 to 0.00238. The corresponding qualitative comparison is presented in Figure 23 and the Supplementary Material.

The component ablations further demonstrate the importance of the proposed guidance mechanism. Removing the preliminary video reduces Spatial IoU from 0.75827 to 0.21600 and Weighted Spatial IoU from 0.67094 to 0.17593 (Table 3). These additional results provide direct evidence that both physical state-space prediction and geometry-consistent visual guidance contribute to the final performance.

Q2:

The editable mask is constructed frame by frame from the predicted geometric trajectory G^fut\hat{\mathcal{G}}_{\mathrm{fut}} and the appearance reference frame IrefI_{\mathrm{ref}}, as defined in Eq. (11). The mask operator Ψ\Psi identifies local regions surrounding the predicted moving structures that require generative refinement.

The resulting mask is supplied to VACE together with the appearance reference, preliminary video, and text prompt as an explicit spatial conditioning signal. Therefore, non-editable regions are encouraged to remain unchanged through spatial conditioning, but are not strictly preserved through pixel-level compositing. We have clarified this distinction in the revised Sec. 3.3.

Q3:

The increase in Trajectory Error after refinement is caused by changes introduced during visual synthesis, rather than deterioration of the learned dynamics model. Before refinement, the preliminary video achieves an average Trajectory Error of 0.00900 across the six systems, compared with 0.00996 after refinement (Table 2).

Generative refinement serves a complementary purpose: improving visual appearance, texture, and local details. As shown in Table 2, it improves Spatial IoU from 0.74569 to 0.75827, Weighted Spatial IoU from 0.66546 to 0.67094, and reduces MSE from 0.00221 to 0.00194. Meanwhile, Spatiotemporal IoU changes slightly from 0.59931 to 0.59605. Thus, refinement provides improved spatial and pixel-level agreement at the cost of a small increase in trajectory error.

Importantly, the Supplementary Material provides full video comparisons across all six physical systems (Double Pendulum video is a good example to show the difference between LPhyVideo before Refinement and LPhyVideo), allowing the visual improvements to be directly inspected. In particular, the refined videos exhibit improved appearance, texture, and local visual details compared with the preliminary videos, while retaining the predicted overall motion.

We therefore distinguish physical prediction accuracy from visual refinement quality: the preliminary video provides the direct realization of the learned physical dynamics, whereas the refined video enhances its visual presentation without guaranteeing strict trajectory preservation.

AuthorsOpenReview ↗

Response to Q4 and Q5 (3/3)

Q4:

Thank you for identifying this issue. We rechecked all five independent runs and found an error in the previously reported standard deviations for Spatial IoU and Weighted Spatial IoU in Table 7. The means were correct. The corrected statistics are:

  • Spatial IoU: 0.00622±0.002860.00622 \pm 0.00286.
  • Weighted Spatial IoU: 0.00104±0.000830.00104 \pm 0.00083.

The Spatiotemporal IoU statistics remain unchanged. All corrections have been incorporated into the revised Table 7.

Q5:

The five individual runs for Torricelli Motion w/o Preliminary Video, corresponding to random seeds 2025–2029, are reported below.

SeedSpatial IoUSpatiotemporal IoUWeighted Spatial IoU
20250.006519350.035079370.00109085
20260.003986230.097879920.00041933
20270.003668580.176438630.00033206
20280.006121920.136280190.00241016
20290.010803910.110321890.00094760
Mean0.006220.111200.00104
Sample Std.0.002860.052100.00083

The reported standard deviations are the sample standard deviations across the five independent generative runs (n−1n-1 denominator). The aggregation unit has been clarified in the revised Table 7 caption.

Regarding Table 3, the aggregate results are obtained by averaging the per-system mean values across the six physical systems. Since the corrections affect only the standard deviations and all three means remain unchanged, the aggregate results in Table 3 are unaffected.

Reference

Feng, T., Zhao, X., Chen, Z., Wong, T. T., Rezatofighi, H., Haffari, G., & Qu, L. (2025). Physics-Grounded Motion Forecasting via Equation Discovery for Trajectory-Guided Image-to-Video Generation (arXiv:2507.06830). arXiv. https://doi.org/10.48550/arXiv.2507.06830

Reviewer_UJdmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes LPhyVideo, a “video-to-physics-to-video” framework for predicting future videos of physical systems. It tracks system-specific keypoints using OpenCV segmentation and template matching with manual initialisation, then maps them to low-dimensional, interpretable physical states through a hand-designed mapping. A continuous-time dynamics model is fitted to these trajectories by estimating parameters of known equations or training a Neural ODE. At inference, the model predicts future states from an observed or user-specified initial state and projects them back into image space. Components from a reference frame are repositioned according to the predicted geometry to produce a preliminary video and an edit mask, which guide refinement by the VACE-1.3B video diffusion model.

Critical strengths
  • The system architecture is reasonable and lays a good foundation for future research. The method first maps video frames to keypoint trajectories and then to a low-dimensional physical state. It models the transitions in that state space as a continuous-time ODE and maps the predicted states back to frames.

  • The evaluated systems are reasonably diverse and the averaged results are consistent. The six systems cover chaotic articulated motion, rolling with a no-slip constraint, damped oscillation, a draining liquid, forced 3D articulation and a moving camera. Averaged over them, the LPhyVideo variants beat all baselines on all five metrics (Table 2).

Critical weaknesses
  • The scope is narrower than the framing, and the assumptions should be stated explicitly. All six systems are single, isolated, low-dimensional systems governed by classical mechanics and described by lumped parameters. There are no contacts, collisions or interactions between objects, and the backgrounds are static. The authors do note a scope restriction (lines 94–96, 484–485). However, the title, the abstract and the repeated phrase “systems’ true dynamics” suggest more generality than the evidence supports.

  • Using keypoints as the physical interface is not justified, and the evaluation is partly circular. Everything downstream depends on the tracker T. That tracker relies on colour segmentation, template matching and manual initialisation, and the robot scenes need synthetic colour fiducials. The paper does not explain why this design was chosen over.

  • The choice of dynamics model is neither justified nor compared with alternatives. An ODE follows naturally from the low-dimensional state assumption, but the paper never discusses that link or its consequences. Systems that need spatially distributed states (fluids, deformable objects, cloth) would require PDE-based models or particle and mesh simulators [1] [1]. A. Sanchez-Gonzalez et al. Learning to simulate complex physics with graph networks. ICML, 2020.

  • The novelty is not clearly positioned. The pipeline combines established components: classical tracking, system identification or Neural ODEs, projective geometry, and VACE. The new parts are the integration and the preliminary-video and edit-mask interface. The paper should separate its contribution more sharply from PhysGen [2] and Feng et al. [1], both of which predict motion explicitly before generating video.

[1] T. Feng et al. Physics-grounded motion forecasting via equation discovery for trajectory-guided image-to-video generation. arXiv:2507.06830, 2025. [2] S. Liu et al. PhysGen: Rigid-body physics-grounded image-to-video generation. ECCV, 2024.

  • The value of the diffusion refinement stage is not demonstrated, and comparisons with trajectory-conditioned generators are incomplete. The paper’s own results suggest that, for these relatively simple scenes, rendering the predicted geometry does most of the work.
Questions for authors

The value of the diffusion refinement stage is not demonstrated, and comparisons with trajectory-conditioned generators are incomplete. The paper’s own results suggest that, for these relatively simple scenes, rendering the predicted geometry does most of the wor

3 comments
AuthorsOpenReview ↗

Response to W1 and W2 (1/3)

We thank the reviewer for the thoughtful evaluation and constructive comments. In response, we have revised the manuscript (with all modifications highlighted in red) to clarify the scope and methodological contributions of LPhyVideo, and added a new comparative experiment with Feng et al. (2025) to directly examine the difference between image-space trajectory extrapolation and physical state dynamics modeling. We have also expanded the experimental analysis of the generative refinement stage. The corresponding changes and some results are included in the revised manuscript and Supplementary Material (comparison videos), as detailed below.

W1:

We thanks for mentioning that the applicability of LPhyVideo should be clearly distinguished from general physical world modeling. LPhyVideo targets structured physical systems whose observable configurations can be represented by compact, interpretable physical states and mapped back to image space through system-specific geometry. The term ``systems' true dynamics'' refers to the governing physical evolution represented in this state-space, rather than image-space motion patterns. As illustrated in updated Figure 1 of the revised manuscript, physical dynamics are learned independently of camera-induced image motion.

The six evaluated systems cover substantially different dynamical characteristics, including nonlinear multi-body coupling, translation--rotation coupling, damped oscillation, free-surface evolution, externally forced 3D articulation, and physical motion under changing viewpoints. In particular, the Moving Camera 3D Robot Arm demonstrates that physical state prediction can remain independent of camera motion. These experiments validate the proposed formulation across different physical dynamics and observation conditions, rather than claiming universal applicability to arbitrary physical systems.

We have further clarified the scope in the revised Abstract, Introduction, and Conclusion. Contact-rich interactions, deformable bodies, and spatially distributed physical dynamics are explicitly identified as outside the current scope, and the title is retained to reflect the central video-to-physics-to-video objective.

W2:

We use system-relevant keypoints as an intermediate visual measurement interface because the considered systems can be characterized by a small number of identifiable geometric landmarks, such as joint positions, link endpoints, and object centers. Importantly, the tracked pixel coordinates are not directly used as the variables for dynamics prediction. They are converted through system-specific geometric and kinematic relationships into physically interpretable states, such as link angles, angular velocities, translational positions, and joint configurations. The subsequent dynamics model operates entirely on these reconstructed physical states.

The tracking procedure combines colour-based segmentation, template matching, and geometry-constrained tracking. Manual initialization is only required when necessary, after which tracking proceeds automatically. For the Robot Arm experiments, synthetic visual fiducials provide identifiable joint observations and are interpreted using known robot kinematics and calibrated camera parameters. These fiducials are removed from the appearance reference before video generation and do not serve as inputs to the dynamics model.

Regarding evaluation, the four Physics-IQ component metrics are computed directly from generated and ground truth video frames, independently of the keypoint tracking pipeline. The same system-specific tracker is applied to all generated videos and ground truth only for the additional Trajectory Error metric, ensuring consistent landmark definitions and measurement procedures across methods. Thus, the evaluation does not rely exclusively on the tracker-based metric. We have expanded the tracking rationale, feature selection criteria, and evaluation description in Appendix A.4 of the revised manuscript.

AuthorsOpenReview ↗

Response to W3 and W4 (2/3)

W3:

The ODE formulation follows directly from the finite-dimensional physical state representation adopted in LPhyVideo. The considered systems are represented using interpretable variables such as positions, velocities, angles, and angular velocities, whose temporal evolution can be described by continuous-time state-space equations. Depending on the available physical knowledge, we use parameter identification to estimate unknown parameters in known governing equations or Neural ODEs to learn unknown dynamics from observed physical trajectories.

For example, Torricelli Motion is represented through a low-dimensional liquid height state that characterizes the evolution of the free surface, rather than explicitly modeling a spatially distributed fluid velocity field. This is appropriate for the specific observable prediction task considered here.

We agree that deformable bodies, cloth, and complex fluid interactions require richer spatial representations. Such systems would require corresponding PDE-based, particle-based, or mesh-based dynamics models, rather than the current finite-dimensional ODE formulation. We have explicitly clarified this connection and its limitations in Section 3.2 of the revised manuscript.

W4:

The novelty of LPhyVideo lies in learning and predicting the coupled physical dynamics in an interpretable state space, and translating the predicted physical evolution into geometry-consistent visual guidance for future video generation. Unlike directly extrapolating image-space coordinates, LPhyVideo reconstructs physical variables and models their coupled temporal evolution. Camera geometry is used for state reconstruction and subsequent image projection, but is excluded from the dynamics model itself. This separation also enables prediction under specified future camera trajectories, including changing viewpoints.

The methodological differences from PhysGen (Liu et al., 2024) and Feng et al. (2025) are summarized in the following Table. PhysGen combines rigid-body simulation with generative rendering, whereas Feng et al. discover time-dependent symbolic functions over tracked image-space coordinates for trajectory-guided generation. In contrast, LPhyVideo models the coupled evolution of physically interpretable state variables rather than independently extrapolating image-space landmark pixel coordinates.

AspectPhysGen (2024)Feng et al. (2025)LPhyVideo
Physical Representation2D rigid body statesImage space landmark trajectoriesInterpretable physical states
Dynamics ModelingPhysical simulationCoordinate-wise symbolic functionsCoupled state space dynamics
Camera HandlingImage-conditioned generationImage space trajectory predictionCamera-independent dynamics rollout
Visual GuidanceSimulated physical statesExtrapolated pixel trajectoriesGeometry consistent preliminary video and edit mask
Video GenerationGenerative renderingTrajectory guided I2VVACE based refinement

To further examine this distinction empirically, we have added a direct comparative experiment with Feng et al. (2025) in Appendix A.6. Since their official implementation is unavailable, we independently implemented their equation discovery approach following the original methodological description. Both methods use the same Double Pendulum (Feng et al. (2025) can not handle 3D even Moving Camera situations) video, 10 s observation interval, and 5 s prediction horizon.

Figure 22 in the revised manuscript compares the predicted landmark trajectories before generative refinement. While both methods capture the dominant horizontal oscillations, Feng et al. exhibit noticeable deviations in the vertical trajectory of the first endpoint and substantially overestimate the oscillation amplitude of the second endpoint. LPhyVideo more closely follows the ground truth trajectories. This comparison directly demonstrates the difference between coordinate-wise image-space extrapolation and coupled physical state dynamics prediction.

AuthorsOpenReview ↗

Response to W4, W5, and Q (3/3)

We further conduct a controlled video generation comparison using the same VACE-1.3B backbone, reference image, and generation settings, with five independent random seeds. As reported in the newly added Table 9, LPhyVideo achieves higher Spatial IoU (0.80072 vs. 0.70278), Spatiotemporal IoU (0.51232 vs. 0.44997), and Weighted Spatial IoU (0.65995 vs. 0.57637), together with lower MSE (0.00238 vs. 0.01135) and Trajectory Error (0.00905 vs. 0.01075).

LPhyVideo outperforms the independently implemented approach across all five metrics under the same generative backbone and evaluation protocol. Figure 23 additionally provides qualitative video comparisons. Together, these new trajectory-level and video-level experiments provide direct empirical evidence supporting the distinction and effectiveness of our physical state dynamics modeling.

W5:

It is true that the preliminary video already captures the predicted physical motion and geometry, but this is an intentional design of LPhyVideo rather than a limitation. In our work, the physical prediction capability is established before generative refinement and does not depend on the video diffusion model.

As shown in Table 2, LPhyVideo before Refinement achieves the best average performance among all compared methods on all five metrics, with Spatial IoU of 0.74569, Spatiotemporal IoU of 0.59931, Weighted Spatial IoU of 0.66546, MSE of 0.00221, and Trajectory Error of 0.00900. Generative refinement further improves Spatial IoU (0.74569 to 0.75827), Weighted Spatial IoU (0.66546 to 0.67094), and MSE (0.00221 to 0.00194), although Spatiotemporal IoU and Trajectory Error exhibit small trade-offs. This confirms that VACE primarily enhances visual realization rather than determining the underlying physical evolution.

We have also added two complementary experimental analyses. First, the new guided-generation comparisons with Wan-Move and LTX-2.3 + EditAnything use the same predicted physical evolution as LPhyVideo. LPhyVideo achieves better average performance than both methods across all five metrics (Table 2), supporting the effectiveness of our preliminary video and edit mask guidance. Second, the backbone scale experiment (Table 4) shows that increasing VACE from 1.3B to 14B parameters does not consistently improve performance. The 1.3B backbone achieves better average results across all five metrics while requiring only one A40 GPU, compared with eight A40 GPUs for the 14B backbone.

These results establish that physical state dynamics modeling provides the primary predictive capability, while constrained generative refinement improves visual quality without requiring a substantially larger video generation model. Detailed per-system video results and qualitative comparisons are provided in Supplementary Material (Double Pendulum video is a good example to show the difference between LPhyVideo before Refinement and LPhyVideo) and Appendix A.5.

Q:

The diffusion refinement stage serves a complementary role rather than determining the physical evolution. LPhyVideo already produces physically grounded future predictions through explicit dynamics rollout and geometry-based reconstruction, even without diffusion refinement. VACE subsequently improves visual appearance, textures, and local details while following the predicted motion and geometry. LPhyVideo separates physical prediction from visual synthesis, allowing the physical evolution to be determined explicitly while retaining generative refinement for improved visual realization. The corresponding quantitative results are reported in Tables 2 and 4, and qualitative video comparisons are provided in the Supplementary Material.

References

Sanchez-Gonzalez, A., Godwin, J., Pfaff, T., Ying, R., Leskovec, J., & Battaglia, P. (2020, November). Learning to simulate complex physics with graph networks. In International conference on machine learning (pp. 8459-8468). Pmlr. https://proceedings.mlr.press/v119/sanchez-gonzalez20a

Liu, S., Ren, Z., Gupta, S., Wang, S. (2025). PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15140. Springer, Cham. https://doi.org/10.1007/978-3-031-73007-8_21

Feng, T., Zhao, X., Chen, Z., Wong, T. T., Rezatofighi, H., Haffari, G., & Qu, L. (2025). Physics-Grounded Motion Forecasting via Equation Discovery for Trajectory-Guided Image-to-Video Generation (arXiv:2507.06830). arXiv. https://doi.org/10.48550/arXiv.2507.06830

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.