PhysV2V: Simulation-Controlled Generative Video Editing of Object Physics

Recent video-to-video (V2V) generative models excel at coarse semantic editing, but their lack of explicit physical grounding causes them to hallucinate incorrect motions when tasked with editing object physics, such as altering object mass or velocity. Conversely, although recent physics-integrated image-to-video (I2V) methods provide such explicit grounding, their reliance on single-image inputs prevents them from accurately recovering the full dynamic state needed to preserve a source scene's physical properties. To address this gap, we introduce PhysV2V, a framework for editing object physics in videos that couples a differentiable physics simulator with a conditional video diffusion model to enable explicit control over physical parameters. Given an input video featuring object dynamics, PhysV2V first estimates 3D scene geometry using SAM-3D to extract object meshes. Second, it solves an inverse problem to optimize physical properties of both rigid and deformable objects (e.g., initial velocities, restitution, Young's Modulus), establishing a physically grounded proxy whose simulated trajectories faithfully replicate the observed video. Finally, after applying user-specified physics edits, PhysV2V projects points from the updated 3D simulation into 2D to condition a foundation video generator. Evaluations across simulated and real-world videos demonstrate that PhysV2V significantly improves physical realism, visual quality, and editing fidelity over state-of-the-art baselines.