acceptodds
Withdrawn from ICLR 2027

PhysV2V: Simulation-Controlled Generative Video Editing of Object Physics

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent video-to-video (V2V) generative models excel at coarse semantic editing, but their lack of explicit physical grounding causes them to hallucinate incorrect motions when tasked with editing object physics, such as altering object mass or velocity. Conversely, although recent physics-integrated image-to-video (I2V) methods provide such explicit grounding, their reliance on single-image inputs prevents them from accurately recovering the full dynamic state needed to preserve a source scene's physical properties. To address this gap, we introduce PhysV2V, a framework for editing object physics in videos that couples a differentiable physics simulator with a conditional video diffusion model to enable explicit control over physical parameters. Given an input video featuring object dynamics, PhysV2V first estimates 3D scene geometry using SAM-3D to extract object meshes. Second, it solves an inverse problem to optimize physical properties of both rigid and deformable objects (e.g., initial velocities, restitution, Young's Modulus), establishing a physically grounded proxy whose simulated trajectories faithfully replicate the observed video. Finally, after applying user-specified physics edits, PhysV2V projects points from the updated 3D simulation into 2D to condition a foundation video generator. Evaluations across simulated and real-world videos demonstrate that PhysV2V significantly improves physical realism, visual quality, and editing fidelity over state-of-the-art baselines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
computer vision, physics simulation, video generation, diffusion models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sAu7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents PhysV2V, a framework for editing object physics in videos. Given an input video, it reconstructs object geometry, initializes physical properties using a vision-language model (VLM), and optimizes these properties by matching projected simulation trajectories to tracked video points. Rigid and deformable objects are handled using Brax and the material point method (MPM) implementation from PAC-NeRF, respectively. After user-specified parameter changes, the scene is simulated again, and the resulting point trajectories condition a frozen LTX-Video backbone adapted with IC-LoRA. The evaluation includes synthetic physical edits, qualitative real-video examples, and ablations of geometry estimation and physical-property optimization.

Critical strengths
  1. The problem is relevant, and using source-video motion to support physical editing is well motivated. Fitting an editable physical proxy to observed motion provides a useful way to connect the dynamics of the source video with subsequent parameter changes. This addresses an important limitation of approaches that initialize a scene from a single image.

  2. The experiments provide initial evidence for the usefulness of per-video physical-property optimization. The comparisons and ablations show improvements in output-video similarity, including over a variant without property optimization that receives ground-truth initial velocities. The real-video examples also illustrate edits to rigid-body motion and elastic deformation.

Critical weaknesses
  1. The scope of compatibility with rigid-body and continuum dynamics is unclear.

The paper claims to design a per-scene property optimization framework that is “compatible with both rigid-body and continuum dynamics.” However, it does not explain whether this means supporting the two object types separately or handling scenes containing interacting rigid and deformable objects.

Section 4.2 uses Brax for rigid bodies and the PAC-NeRF MPM simulator for deformable objects, while the method and appendix describe these simulations separately. Joint simulation would require explicit handling of contact, momentum exchange, time stepping, and gradient propagation between the solvers; their individual differentiability does not automatically establish such a coupled system. No coupling mechanism or mixed-interaction experiment is provided. The authors should clarify the intended scope and, if mixed scenes are supported, describe and validate the interaction mechanism.

  1. The evaluation provides insufficient direct evidence for reliable physical-parameter editing across scenes.

The main quantitative experiments in Tables 2 and 3 evaluate the final videos using PSNR, SSIM, and LPIPS. These comparisons provide evidence about edited-video similarity, but do not directly quantify parameter-estimation accuracy or how accurately the generated motion realizes the requested physical intervention. Moreover, the synthetic evaluation consists of parameter variations of a single two-ball collision setup.

Direct parameter-level evidence is mainly in Appendix A.2: Figure 7 shows optimization curves for one example, while Table 5 does not specify the dataset, number of scenes or samples, repeated runs, or aggregation protocol. Table 5 also initializes parameters at scaled ground-truth values, rather than evaluating the default VLM initialization across diverse inputs.

Although exact parameter recovery is not the authors’ stated objective, the paper should more directly establish that the inferred proxy supports accurate physical edits. Quantitative trajectory and physical-response comparisons before and after edits, across multiple independent scenes and edit magnitudes, would better support this claim. The issue is the limited scope and specificity of the evidence, rather than the placement of results in the appendix itself.

  1. The methodological novelty appears limited.

The reconstruction, tracking, simulation, and generation stages rely on existing components. The generation stage adopts existing IC-LoRA adapters. The paper’s main contribution appears to be the integration of video-based parameter fitting, physical interventions, and conditional video generation. The novelty of this integration is insufficient.

Questions for authors
  1. Does the framework support scenes containing interacting rigid and deformable objects? If so, how are contact, momentum exchange, time stepping, and gradient propagation handled between Brax and MPM?

  2. What dataset, scenes, and sample counts are used in Table 5?

  3. Can the authors quantify trajectory and physical-response errors under different edit magnitudes across multiple scenes, and evaluate these errors for both the simulator output and the final generated video?

  4. What specific methodological advance is introduced beyond combining existing reconstruction, tracking, simulation, and IC-LoRA-based generation components? Which controlled comparisons establish the benefit of the proposed design choices?

Reviewer_WDxmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents PhysV2V, a framework for editing object physics in videos. It reconstructs scene geometry, estimates physical parameters through differentiable trajectory matching, applies user-specified edits, and conditions a video diffusion model on the simulated motion. Experiments include synthetic collision sequences and qualitative real-world examples.

Critical strengths
  1. The framework provides an explicit and interpretable interface for editing physical parameters.
  2. Using the source video to refine physical properties is a useful improvement over single-image initialization.
  3. The geometry, optimization, and point-sampling ablations help identify the pipeline’s main bottlenecks.
Critical weaknesses
  1. The contribution appears mainly incremental. The pipeline combines existing reconstruction, tracking, simulation, and diffusion components.
  2. Evaluation is too narrow for the broader claims. The 250 synthetic outputs are parameter variations of a single two-ball reference scene, while deformable editing is evaluated qualitatively. Include diverse scenes and quantitative deformation tests, and measure whether generated trajectories follow the simulation rather than relying mainly on image-similarity metrics.
  3. Reproducing the original video does not necessarily mean that the estimated physical parameters are correct. Different parameter combinations may explain the same observed motion but produce different results after an edit. Please verify whether mass edits remain accurate across such alternative parameter estimates.
Questions for authors

Please see Weaknesses.

Reviewer_FUGbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PhysV2V achieves physics-grounded video editing by first reconstructing 3D scene geometry and physical properties from a given video, then simulating the reconstructed scene under new user-specified physical properties, finally using the simulation rollout to guide video generation to obtain the edited video. Demos on synthetic and real-world videos show clear response to different user edits, and quantitative results demonstrate improvements over existing V2V and physics-based I2V methods.

Critical strengths
  • Task of physics-grounded V2V editing is practical and the overall pipeline is straightforward and an effective combo of inverse physics optimization and forward simulation-controlled generation.

  • The results seem promising, with qualitative examples demonstrating clear response to different types of physics edits.

Critical weaknesses
  • As the authors acknowledge, 3D reconstruction and physical parameter estimation remain bottlenecks for PhysV2V. The quantitative evaluation of this is in Table 3 is limited to a simple two-ball collision scenario. Comparisons with video-to-simulation methods such as Vid2Sim or PhysTwin on more diverse objects/physical interactions would better establish the accuracy and robustness of the recovered parameters.
Questions for authors
  • The method assumes motion driven by passive forces only, but some examples involve external force, such as a golf club striking a ball. Since these forces are approximated through initial velocities without explicitly modeling the interacting objects, it seems from the videos that the contact events are not quite accurate/realistic? Could the authors discuss this potential limitation?

  • Qualitative results involving changes in angular velocity or object rotation seem to exhibit more noticeable artifacts and unrealistic motion. Discussion appreciated.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.