acceptodds
Withdrawn from ICLR 2027

VideoPhysEdit: Physical Counterfactual Video Editing via Rigid-Body Physical Scene Reconstruction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video editing has advanced substantially in recent years, with methods increasingly accounting for the visual consequences of edits, such as changes to shadows and occlusions. However, the physical consequences of edits, including changes to subsequent motion and interactions, remain less explored. We formulate this problem as (PCVE), which aims to generate a counterfactual video depicting the resulting motion and interactions given a source video, a physical edit, and its execution frame. PCVE is challenging because it requires understanding scene physics and inferring the downstream motion and interactions induced by a physical intervention, while paired factual and counterfactual data and dedicated evaluation metrics are lacking. We introduce , a new training-free pipeline for PCVE in rigid-body scenes. It makes physical reasoning explicit through a novel physical scene reconstruction method that recovers a scene reproducing the observed motion and interactions under simulation, enabling the pipeline to apply physical edits as interventions and use the resulting trajectories to guide counterfactual video generation. We further construct , a synthetic benchmark with paired source and counterfactual target videos and physical ground truth, and introduce the Physical Edit Score. VideoPhysEdit achieves substantially higher physical edit accuracy than open-source methods and commercial models while maintaining competitive visual fidelity. Its Physical Edit Score is 0.376, the only positive score among the compared methods. Qualitative comparisons on real videos further show that VideoPhysEdit applies to real-world scenes and better depicts the downstream motion and interactions induced by the edits than the compared methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Edit, Video Generation, Physical Scene Reconstruction, Physical Edit

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_hLMJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on physical counterfactual scenarios in video editing, where an edit should change not only the visual content but also the subsequent motion and interactions in the scene. In this sense, the task also evaluates whether existing video generation and editing models have meaningful physical priors. The authors propose VideoPhysEdit, a training-free pipeline that explicitly reconstructs a physical scene, applies the requested physical edit, and uses the simulated results to guide video generation. The paper also introduces a dedicated benchmark for this task. The experiments show that existing open-source and commercial models generally perform poorly on this challenging setting, while VideoPhysEdit achieves much better physical edit accuracy.

Critical strengths

One critical strength is that the carefully designed pipeline is effective both in theory and in practice. It essentially lets the generation model focus only on rendering, while using the reconstructed physical environment for reasoning and simulation. Its performance is therefore clearly better than that of the best current generation models. When the prompt is not sufficiently detailed, current video generation models do not have strong physical reasoning ability. They lack too much context to understand the physical parameters and perform counterfactual reasoning.

Critical weaknesses
  1. The method is not very novel, as explicit reconstruction is already relatively common in the video generation field. A pipeline that reconstructs a physical environment and then performs reasoning will also naturally be more accurate than implicit generation.

  2. Ideally, generation models should learn general physical laws and reasoning abilities about the physical world. This approach does not seem very scalable or generalizable, and it appears to work only for specific physical scenarios.

Questions for authors
  1. The full pipeline takes much longer than a single video generation pass. With more streaming video generation models emerging, do the authors think this approach could be adapted to low-latency or interactive settings?

  2. Is the comparison with other generation models fully fair when they only receive the source video and a short counterfactual editing instruction, while VideoPhysEdit explicitly reconstructs the physical environment before reasoning? The other models may lack enough scene-specific context to infer the physical parameters and downstream effects. Appendix C.5 partly studies this issue on four tasks, but could the authors provide broader evidence or discussion about this potential input-context disadvantage?

5 comments

Thank you for your careful review and recognition of the value of separating physical reasoning from visual generation. We address two weaknesses and two questions.

W1. Methodological novelty.

We agree reconstruction has precedents. We introduce physical counterfactual video editing (PCVE), a new task, a new editing framework and dynamics-prediction pipeline, as no ready-made solution directly covers source-motion reproduction and post-intervention physical rollouts. Stages 2–7 each have a new task-specific design: Motion Observation Analysis (2), Canonical and Anchor Scene Reconstruction (3), Motion Prior Reconstruction (4), Physical Inversion (5), Physical Intervention (6), and Counterfactual Video Generation (7). We also introduce PCVE-RigidBench, a new benchmark with paired targets and physical ground truth, and PES for physical evaluation.

W2. Scope and scalability.

VideoPhysEdit covers falling, sliding, rolling, bouncing, collisions and collision chains, with edits to mass, friction, restitution, velocity and object addition/removal. Fixed cameras, planar supports and contacts adequately approximated by sphere/box proxies define a reusable family of rigid-body systems.

These camera, geometry and motion settings have broad precedents:

Generation: Static cameras: PISA (ICML 2025), PhysStream (SIGGRAPH Asia 2026), Generating Videos with Scene Dynamics (NeurIPS 2016); fixed-view data: TASTE-Rob (CVPR 2025); substantial camera-motion filtering: Generative Image Dynamics (CVPR 2024); fixed-camera test prompts: NewtonGen (ICLR 2026); static-view temporal branch: DimensionX (ICCV 2025). Simple 2D shapes: How Far Is Video Generation from World Model? (ICML 2025); 2D rigid proxies/meshes: PhysGen (ECCV 2024)/PhysAnimator (CVPR 2025); controlled training motions including rolling balls: Force Prompting (NeurIPS 2025).

Reconstruction/inference: Δynamics (CVPR 2026) uses primitives and fixed cameras; IRIS (ECCV 2026) tests balls, cones and pendulums under stable camera placement. AGILE (SIGGRAPH 2026), Consistent4D (ICLR 2024) and V-SysId (L4DC 2022) use static cameras. Physics-as-Inverse-Graphics (ICLR 2020), CoPhy (ICLR 2020) and Visual De-animation (NeurIPS 2017) study controlled systems. Neural Implicit Representations (WACV 2023) assumes static cameras and planar motion; Physically Based Video Editing (Pacific Graphics 2016) uses fixed cameras and static backgrounds. Learning Physics From Video (CVPR 2025) estimates parameters for systems such as pendula, sliding and free fall; VRDP (NeurIPS 2021) models CLEVRER primitives; Galileo (NeurIPS 2015) uses boxes, cylinders and tori.

Compared with single-motion generation or system-specific parameter estimation, we address unified multi-object rigid-body editing across motions, combined interactions, parameter edits and object addition/removal, offering broad applicability and control. Jointly reconstructing source motion, predicting new contacts and cascading effects, and maintaining visual consistency makes the task more demanding.

Extensions could use richer proxies and support surfaces, add moving-camera motion estimation, and expand the motion rules. Stronger model components could substantially improve robustness.

Q1. Latency and interaction.

Table 15 gives approximately 8.24 minutes for reconstruction and 11.47 per edit, including 11.09 for video generation, the main per-edit cost. Reusing the source scene with faster or streaming generators could enable real-time editing.

Q2. Fairness and context.

Our method and general video-editing baselines receive the same source video and quantitative instruction (Sec. 5.1). We infer the scene without extra ground-truth scene parameters or target trajectories. Recovering scene-specific physical information is part of the task and our contribution. We compare complete systems' ability to infer this information from shared inputs and use it for editing; Q1 reports the added cost.

In C.5, baselines receive explicit downstream-motion and interaction descriptions, while VideoPhysEdit retains the original instruction. Results on four tasks show no consistent improvement (Table 16): models still struggle to reliably generate the described physical changes. We believe the main bottleneck is translating physical edits into correct dynamics, beyond prompt detail alone. Our pipeline addresses this by recovering physical information from shared inputs, computing motion through explicit interventions and simulation, and guiding video generation. This motivates our dedicated reconstruction and dynamics-prediction design and is a core contribution.

These results demonstrate our contributions to physical scene reconstruction, intervention and paired evaluation. We respectfully ask you to reassess them in light of these clarifications and results. Thank you for your careful review and discussion.

Thanks to the authors for their detailed response and efforts. While I appreciate the clarifications, my concerns about the novelty and generalizability to complex real-world physical scenarios remain.

Thank you for your review, and we respect any decision you make.

We would still like to reiterate that physical editing in complex real-world scenes is also the goal we pursue, yet even the most advanced current models completely fail on the simplest scenes, whereas our method at least already takes a big step forward. In addition, regarding novelty, we emphasize again that we are the first to explicitly reconstruct kinematics with collision interactions, that every stage is our new design with extensive theoretical and experimental support, and that the design of each stage addresses a challenging problem in 3D.

Finally, thank you again for your review.

Reviewer_3zysOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents a pipeline for editing the physical events in a video. It reconstructs a simplified 3D scene and fits its physical parameters so that a simulation reproduces the observed motion. It then applies the requested change in the simulator and uses the resulting trajectories, together with a reference image, to guide Wan-Move in generating the edited video. The paper also introduces a benchmark with 129 editing tasks across 20 synthetic scenes and reports improvements over the compared video editors.

Critical strengths

The pipeline is coherent and produces useful results in the tested setting. The reported results show improved physical edit accuracy over the evaluated baselines.

The benchmark makes sense. Paired source and counterfactual videos make it possible to evaluate whether an edit correctly changes later motion and interactions.

Critical weaknesses

The demonstrated scope is too narrow to justify the complexity convincingly. The method assumes a static camera, planar supports, and rigid objects represented by spheres or boxes. Although the benchmark contains 129 edits, these come from only 20 synthetic scenes, and the real-video evaluation is qualitative. The system also requires substantial processing time. Overall, the paper demonstrates a complex solution for a restricted set of relatively simple physical events, with limited evidence of broader practical value.

Questions for authors
  1. How would the method compare with a coding agent that builds and edits a simulated scene from the source video, followed by the same motion-guided video generator?

  2. Does the method’s narrow range of applications justify the system’s complexity?

3 comments

Thank you for your careful review and comments on scope, complexity and coding-agent comparisons. They help clarify our designs and benefits. We address one weakness and two questions, with added reconstruction results.

W1. Scope and value.

VideoPhysEdit unifies free fall, sliding, rolling, bouncing, collisions, collision chains and their combinations, with edits to mass, friction, restitution, velocity and object addition/removal. Sphere/box contact proxies handle different shapes, such as curling stones and toy cars, when main contacts are adequately approximated.

Fixed views and controlled geometry/dynamics have broad precedents.

Generation: PISA (ICML 2025) and PhysStream (SIGGRAPH Asia 2026) use static cameras; TASTE-Rob (CVPR 2025) uses fixed-view data, while Generative Image Dynamics (CVPR 2024) filters substantial camera motion. How Far Is Video Generation from World Model? (ICML 2025) studies simple 2D shapes; PhysGen (ECCV 2024) and PhysAnimator (CVPR 2025) use 2D rigid proxies and meshes, respectively. Force Prompting (NeurIPS 2025) learns from controlled training scenes including rolling balls. NewtonGen (ICLR 2026) specifies fixed cameras in test prompts; DimensionX (ICCV 2025) trains a static-view temporal branch; T2Bs (ICCV 2025) requests static cameras for expression videos; Generating Videos with Scene Dynamics (NeurIPS 2016) uses a static camera.

Reconstruction/inference: Δynamics (CVPR 2026) uses primitives and fixed cameras; IRIS (ECCV 2026) evaluates balls, cones and pendulums under stable camera placement. AGILE (SIGGRAPH 2026), Consistent4D (ICLR 2024) and V-SysId (L4DC 2022) use static cameras. Neural Implicit Representations (WACV 2023) also assumes planar motion. Physically Based Video Editing (Pacific Graphics 2016) uses fixed cameras and static backgrounds. Physics-as-Inverse-Graphics (ICLR 2020) studies bouncing balls, springs and pendula; CoPhy (ICLR 2020) uses block towers, balls and collisions; VRDP (NeurIPS 2021) models CLEVRER primitives; Visual De-animation (NeurIPS 2017) studies billiards and block towers; Galileo (NeurIPS 2015) uses boxes, cylinders and tori.

Beyond single-motion generation or system-specific parameter estimation, VideoPhysEdit offers unified multi-object rigid-body editing, covering motions, interactions, parameter edits and object addition/removal with broader applicability and control. Joint source-motion reconstruction, prediction of new contacts and cascading effects, and visual consistency make the task more demanding.

Q1. Coding-agent comparison.

Our novelty is reconstructing a source-calibrated, executable rigid-body scene from monocular video without task-specific training. Simulation reproduces source motion and recomputes motion and interactions after parameter edits or object addition/removal for counterfactual video generation. Few ready-made baselines directly cover source-motion reproduction and post-intervention physical rollouts.

We added a PhysCodeGym-based reconstruction baseline, retaining its workflow while adapting it to our videos and Mask IoU evaluation. Agents run on an RTX 4090 server with medium reasoning, at most 20 Blender calls and a 3-hour budget per scene. Results are:

MethodTime (min) ↓Mask IoU ↑
GPT-5.6 Sol23.620.3232
Astra27.520.5627
VideoPhysEdit (reconstruction, Stages 1–5)8.240.6780

VideoPhysEdit exceeds the stronger Astra agent by 0.1153 IoU, demonstrating dedicated reconstruction's accuracy advantage.

Q2. Complexity and benefits.

Broad coverage and explicit physical control are the central benefits. Within fixed-camera scenes with planar support and adequately approximated rigid-body contacts, we cover falling, sliding, rolling, bouncing and multi-object collisions, with parameter edits and object addition/removal. Compared with relying on learned priors to interpret edits, explicit simulation provides testable physical predictions for these interventions. Final-video trajectory error is 54.0% lower than the strongest general video-editing baseline (Table 1), with higher physical edit scores for other affected objects (Table 9).

Compared with parameter estimation for specific equations, as in Learning Physics From Video (CVPR 2025), our framework combines multiple rigid-body motions in one executable scene without retraining a dynamics model for each edit. Modularity provides clear extensions: richer collision proxies and support geometry, moving-camera motion estimation, and additional motion priors. Reusing one reconstruction for multiple edits amortizes its cost.

The results demonstrate broad rigid-body editing capabilities within explicit modeling boundaries. We respectfully ask you to reassess our contributions to executable scene reconstruction, counterfactual video editing and paired evaluation. Thank you for your careful review and constructive discussion.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts, particularly the new comparison with coding-agent-based reconstruction. However, my main concern remains that the applicability of the proposed pipeline is substantially limited by its underlying assumptions. Given the considerable complexity of the pipeline, I remain unconvinced that its demonstrated scope and methodological contributions are sufficient to justify acceptance. Therefore, I maintain my original assessment.

Thank you for your reply and for recognizing our additional experiments.

We acknowledge your concern about the applicability, but our setting follows a large body of recent frontier work and is a reasonable one. Moreover, even in this "simple" setting, the most advanced current models almost completely fail. Existing methods can hardly solve this problem, whereas we devoted substantial effort to completely redesigning all the underlying reconstruction algorithms and conducted extensive experiments to verify the effectiveness of the new algorithms. Like you, we also hope to achieve more comprehensive editing capabilities in general complex scenes, but to our knowledge, this still requires more effort from the field.

Thank you again for your review and the discussion.

Reviewer_tZQTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VideoPhysEdit addresses physical counterfactual video editing: modifying an object, motion state, or physical parameter and generating the resulting downstream motion. Its training-free pipeline reconstructs a simulation-ready rigid-body scene from the source video, estimates physical parameters through analytic constraints and simulation search, applies the requested intervention, and uses simulated point trajectories to guide a pretrained video generator.

The paper introduces PCVE-RigidBench, containing 129 editing tasks across 20 synthetic scenes, and a Physical Edit Score. Results show substantially better trajectory accuracy than the evaluated video editors, with additional qualitative demonstrations on real videos.

Critical strengths
  • An executable physical model provides a meaningful mechanism for downstream consequences. Changes to mass, friction, or scene composition are propagated through simulation rather than inferred solely through visual generation. The stage-wise analysis helps establish that the resulting trajectories are substantially retained in the generated video.

  • Paired counterfactual targets enable more direct evaluation. The benchmark and trajectory-based metrics distinguish successful physical edits from visually convincing outputs that retain the original motion. Including an unchanged-video baseline is particularly useful for interpreting editing success.

Critical weaknesses
  • The formulation is restricted to simplified scenes, substantially limiting its significance. The method assumes a static camera, planar supports, and sphere- or box-based object geometry, and uses manually specified motion models, contact constraints, and filtering thresholds. These assumptions exclude many common physical scenes and interactions. Even within the synthetic benchmark, reconstruction fails on two source scenes covering 13 editing tasks, while closely spaced contacts produce substantial physical-inversion errors. This suggests that the restrictions are central to the method’s current capability rather than minor implementation details.

  • The evaluation does not sufficiently establish a generalizable methodological advance. Quantitative evidence comes from only 20 synthetic scenes generated with PyBullet, the same simulator used for reconstruction and intervention. Of the 129 tasks, 117 intervene at the first frame, and only seven involve insertion. Real-video evidence is qualitative. Furthermore, comparisons primarily against video editors demonstrate the benefit of adding explicit simulation, but do not sufficiently isolate the contribution of the proposed reconstruction procedure relative to simpler simulation-fitting alternatives. The results therefore establish a useful proof of concept more strongly than a broadly applicable counterfactual editing method.

Questions for authors
  • Which assumptions—primitive object geometry, planar supports, static cameras, and the prescribed motion models—are essential to the reconstruction formulation? What classes of rigid-body scenes cannot be represented adequately without changing the method itself?

  • Reproducing factual motion does not uniquely identify the underlying physical scene. How sensitive are counterfactual predictions to alternative parameter configurations that fit the source video similarly well, particularly when an intervention introduces previously unobserved contacts?

  • How does the proposed reconstruction compare with a simpler physical-parameter fitting baseline using the same geometric observations, simulator, and Wan-Move generator? This would help separate the contribution of the reconstruction method from the advantage of explicit simulation.

  • My central concern remains that the method’s restrictive assumptions substantially limit its applicability; this appears to be an inherent limitation of the formulation, rather than a gap that additional experiments alone would resolve.

2 comments

Thank you for your careful review and recognition of executable physics and paired counterfactual evaluation. We address two concerns and four questions, with added reconstruction results.

W1. Scope and significance.

We cover free fall, sliding, rolling, bouncing, collisions, collision chains and their combinations, with edits to mass, friction, restitution, velocity and object addition/removal. Sphere/box collision proxies simulate trajectories through contact approximations; visible objects can have different shapes, e.g., curling stones, toy cars and bowling balls. Simplified geometry does not restrict us to a few preset motions.

Fixed views, controlled geometry and defined dynamics have broad precedents.

Generation: Static cameras: PISA (ICML 2025), PhysStream (SIGGRAPH Asia 2026), Generating Videos with Scene Dynamics (NeurIPS 2016); fixed-view data: TASTE-Rob (CVPR 2025); camera-motion filtering: Generative Image Dynamics (CVPR 2024); fixed-camera prompts: NewtonGen (ICLR 2026, tests), T2Bs (ICCV 2025, expression videos); static-view temporal branch: DimensionX (ICCV 2025). Simple 2D shapes: How Far Is Video Generation from World Model? (ICML 2025); 2D rigid proxies/meshes: PhysGen (ECCV 2024)/PhysAnimator (CVPR 2025); controlled training motions including rolling balls: Force Prompting (NeurIPS 2025).

Reconstruction/inference: Fixed cameras: Δynamics (CVPR 2026, primitives), AGILE (SIGGRAPH 2026), Consistent4D (ICLR 2024), V-SysId (L4DC 2022), Neural Implicit Representations (WACV 2023, planar motion), Physically Based Video Editing (Pacific Graphics 2016, static backgrounds). Controlled systems: IRIS (ECCV 2026, stable cameras; balls/cones/pendula), Learning Physics From Video (CVPR 2025, pendula/sliders/free fall), Physics-as-Inverse-Graphics (ICLR 2020, bouncing balls/springs/pendula), CoPhy (ICLR 2020, block towers/balls/collisions), VRDP (NeurIPS 2021, CLEVRER primitives), Visual De-animation (NeurIPS 2017, billiards/block towers), Galileo (NeurIPS 2015, boxes/cylinders/tori).

Beyond single-motion generation or system-specific parameter estimation, we address unified multi-object rigid-body editing with broad applicability and control. Joint source-motion reconstruction, prediction of new contacts and cascading effects, and visual consistency make the task more demanding.

Appendix C.2.3 traces bowling and domino-chain failures to SAM instance segmentation and VGGT correspondences, respectively. These affect reliability; stronger components can substantially improve robustness.

W2. Evaluation and contribution.

Real counterfactual targets are rarely obtainable, so we provide qualitative generalization evidence, also indicating independence from PyBullet. The 20 scenes and 129 tasks cover diverse scenarios, addition/removal and parameter edits. Current editors' difficulty with physical instructions supports our method's generality and the benchmark's value. Our explicit scope covers general rigid-body motion and these edits.

Q1. Key assumptions.

These assumptions are necessary, but motion modeling can extend through richer proxies and support surfaces, moving-camera motion estimation, and additional motion rules.

Q2. Parameter ambiguity.

Video editing is ill-posed: a 2D video admits multiple explanations. We recover a scene explaining observations and supporting interventions, not unique true parameters. New-contact predictions may depend on underconstrained materials and geometry. Paired evaluation tests the chosen solution, not variation across equally well-fitting parameters. This sensitivity needs validation; we will clarify that factual fit does not guarantee unique counterfactual predictions.

Q3. Simple fitting baseline.

Our novelty is reconstructing a source-calibrated, executable rigid-body scene from monocular video without task-specific training. Simulation reproduces source motion and recomputes motion and interactions after parameter edits or object addition/removal for counterfactual video generation. Few ready-made baselines directly cover source-motion reproduction and post-intervention physical rollouts.

To assess explicit reconstruction, we added a PhysCodeGym agent baseline using medium reasoning, at most 20 Blender calls and a 3-hour budget per scene:

MethodTime (min) ↓Mask IoU ↑
GPT-5.6 Sol23.620.3232
Astra27.520.5627
VideoPhysEdit (reconstruction, Stages 1–5)8.240.6780

The contribution is also reflected in our substantial gains over current video editors in the paper.

Q4. Limitations and value.

Our boundaries are clear: reasonable results can be expected for general rigid-body motion, with broad editing support. The extensions in Q1 are future directions.

These responses and added results clarify our scope, assumptions and contributions. Thank you again for your careful review and discussion.

Thank you for the clarifications and additional experiments.

However, my concerns about the method’s assumptions and generalizations remain, so I keep my score.

Comments on the paper

1 comment
Area_Chair_Mhr9OpenReview ↗

Reminder for Dicussion

Dear Reviewers,

We’re about halfway through the discussion period. This is a friendly reminder to check all of your assigned papers. If the authors have answered questions or responded to critiques in reviews, please leave a comment for the authors!

Thank you for your contribution to the community!

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.