acceptodds
Withdrawn from ICLR 2027

Reimagine Video Dynamics

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Most video editing methods focus on changing the appearance of the source video, while offering limited control over its dynamics. We introduce Reimagine Video Dynamics (RVD), a framework that disentangles a compact, editable dynamics token from visual context. We learn this token through self-supervised reconstruction: given the first frame as visual context, a renderer must recover the original video from the dynamics token, encouraging it to capture how the scene evolves rather than how it looks. This disentanglement allows video dynamics to be edited directly while preserving visual context. We develop a language-guided dynamics-token editor that transforms source dynamics into target dynamics, and train it with a scalable counterfactual video-pair pipeline and a two-stage training strategy. Extensive experiments show that RVD enables effective video dynamics editing, training-free retiming, and appearance-controlled re-rendering.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video dynamics editing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Jfi7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript newly introduced Reimagine Video Dynamics (RVD), which learns a compact dynamics token through video reconstruction conditioned on the first frame. The authors combined compressed V-JEPA 2 features with a Wan-based renderer and trains a language-guided dynamics editor using counterfactual video pairs. The same representation also supports appearance re-rendering and retiming without additional training.

Critical strengths

Separating dynamics editing from rendering provides a clear interface for multiple video manipulation tasks. Beyond the 900 edits, including 600 appearance edits and 300 dynamics edits, the paper includes an optical-flow-based motion-preservation evaluation and fixed-target-context comparisons, providing meaningful evidence of functional control.

Critical weaknesses

The contribution of the learned dynamics token is not sufficiently isolated. The functional controls in Tables are already useful, but Tab. 2 reports only reconstruction metrics and does not establish an advantage over equally compact V-JEPA features. A comparison measuring motion preservation and appearance leakage under the same renderer is needed, together with a clearer distinction from reconstruction-based motion representations. Furthermore, the Tab. 3 evaluates the two-stage training and loss design through token-space statistics, several of which are directly optimized by the proposed losses. Rendering these ablations, including a simple token-MSE baseline, under fixed target contexts would show whether they improve edit completion, motion timing, and appearance consistency.

Furthermore, the evaluation needs closer baselines and independent validation. The fourth table (Tab.4) omits the cited DynaEdit, which uses the same Wan I2V backbone model and supports target-frame/caption conditioning, so a comparison using RVD’s planned target context is needed. Edit-success scores should also be checked with blind human evaluation or an independent model, because the Qwen-based judge shares its model family with the RVD pipeline.

Questions for authors

Please carefully address all the concerns mentioned in weakness section.

Reviewer_A8Z1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a way of editing the dynamics of a video by disentangling the appearance and dynamics, with the dynamics being represented by a compact dynamics token. Then, an editor is trained (based on generated counterfactual video pairs for motion) to edit this dynamics token according to a given prompt, with the goal of ensuring that the overall appearance stays unchanged. The method is evaluated on the tasks of video dynamics editing, video appearance re-rendering, and visually on training-free retiming. Across these tasks, the method shows strong results.

Critical strengths
  1. Visually, the results seem quite strong, with good dynamics editing and good appearance preservation. They do seem to outperform the existing baselines, in many cases quite strongly.

  2. The editor training strategy and counterfactual video pair generation seem quite useful and more broadly applicable, and overall the ablations provided are meaningful with the controlled interventions properly demonstrating the claims.

Critical weaknesses
  1. One of my main concerns is regarding the positioning and novelty of this work. The paper is very much written as if one of the core contributions is the idea to disentangle appearance and motion into these tokens in order to allow for better editing. While the specific models used for this are new (Wan for the video generation and VJEPA-2 for the dynamics encoder), this idea has been heavily explored in the past, both for video generation [1,2, 3] and video editing [4,5,6]. This paper does not contain any citations or remarks to any of the significant number of prior methods using this idea. Much of the core idea can be obtained from some collection of these papers: decomposition into content + dynamics [1, 2, 3, 5], editing the dynamics [4, 5, 6], the use of compact dynamics tokens (all of them), image animation based on the first frame [4, 5, 6], and swapping the dynamics of two videos [5]. Due to this, I believe that the paper needs a significant rewrite to acknowledge which parts are actually new and which parts widely exist in the literature and are just repackaged with new models.

  2. I also have concerns with the evaluation setup, in 3 separate parts. Appendix D.2 mention that Qwen-Image-Edit edits the first frame in 192 out of the 300 cases, while the remaining samples use the source frame. However, the baselines do not receive these edited frames, or even have access to Qwen-Image-Edit, but instead use the source frame in all samples. This discrepancy raises some concerns regarding the fairness of this specific stage's comparisons, and makes attributing the advantage to the approach more difficult. Additionally, in Appendix E, Table 7, on the controlled target, target context alone achieves 90% success, versus 95% with the edited dynamics token. This raises concerns regarding the sole contribution of the edited dynamics token to the task. Finally, the evaluation of the physical correctness of the edited video seems limited, as it mainly relies on the Qwen models for judgement, which is not necessarily always reliable.

[1] Tulyakov, Sergey, et al. "Mocogan: Decomposing motion and content for video generation." 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2018.

[2] Wang, Yaohui, et al. "G3AN: Disentangling appearance and motion for video generation." 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2020.

[3] Denton, Emily L. "Unsupervised learning of disentangled representations from video." Advances in neural information processing systems 30 (2017).

[4] Zhao, Rui, et al. "Motiondirector: Motion customization of text-to-video diffusion models." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.

[5] Ali, Moayed Haji, et al. "Vidstyleode: Disentangled video editing via stylegan and neuralodes." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[6] Wang, Xiang, et al. "Videocomposer: Compositional video synthesis with motion controllability." Advances in Neural Information Processing Systems 36 (2023): 7594-7611.

Questions for authors

The main question i have which could make an impact is regarding the evaluation setup, specifically the points I noted in the weaknesses.

First, given that Qwen-Image-Edit edits the first frame for the current approach but uses the original first frame for all the baselines, how can we be sure that this is even a fair comparison? What happens if we use the edited first frame for all baselines as well?

Second, why does target context alone achieve 90% success, and how can we necessarily say that the editing actually doe smuch?

Third, do you have any validation that the Qwen models are actually good at judging the things you want them to judge? Can we trust their results? Can we get some more direct quantitative validation, rather than just judgement by Qwen?

Reviewer_ng51OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Reimagine Video Dynamics (RVD), a framework for editing video dynamics through a compact spatiotemporal representation.

RVD's central idea separates visual context, supplied by the first frame and caption, from a compact dynamics representation extracted from the full video. A language-guided transformer edits this representation, and a pretrained video generator renders the result. The representation is learned through video reconstruction, encouraging it to capture temporal information complementary to the first frame.

The renderer is trained on approximately 169,000 real video clips. The dynamics editor is first pretrained on static-to-real pairs constructed from these clips, then trained on approximately 20,000 directed synthetic editing examples covering 17 transformations. These examples pair real videos with generated counterfactual versions and are filtered using a vision-language model.

Evaluation. The paper evaluates dynamics editing on 300 cases across 17 edit directions and appearance editing on 600 cases, comparing against existing video editors. It reports automated instruction-following and edit-success scores, motion similarity, and visual-quality metrics. Reconstruction ablations and controlled dynamics-token substitutions examine the representation and editor’s contribution; temporal retiming is also demonstrated.

Critical strengths

1.The compact dynamics representation provides a useful interface for video manipulation. The same representation supports learned dynamics editing, temporal resampling, and appearance re-rendering. The retiming results are particularly helpful in showing that the representation has an operationally meaningful temporal structure.

  1. The controlled experiments help isolate the dynamics editor’s contribution. Appendix E fixes the target context, renderer, and sampling conditions while varying the dynamics input. This is more informative than complete-system comparisons alone and provides evidence that edited source dynamics improve over both unedited source tokens and matched donor tokens.
Critical weaknesses
  1. The additional value of the dynamics editor over strong target-context conditioning is not sufficiently established. In Table 7, the target first frame and caption alone already achieve 90.2% success, compared with 95.5% using the edited source token. This is a measurable improvement, and the accompanying appearance-consistency improvement is encouraging, but much of the requested behavior can already be produced without dynamics conditioning. Moreover, the complete pipeline includes target-caption planning and, in 192 of the 300 benchmark cases, first-frame editing. Consequently, the 98.3% complete-system success rate should not be interpreted as evidence for the dynamics editor alone.

  2. The evidence supports a narrower scope than the general dynamics-editing motivation suggests. The editor is trained on synthetic pairs covering 17 predefined transformation directions, and the reported editing benchmark also covers 17 directions. This does not establish generalization to unseen transformation types or more complex combinations of edits.

Questions for authors
  1. Under identical target-frame and caption preparation, how does RVD compare with a strong source-video-conditioned generative baseline, like H3, in preserving source-specific trajectories, timing, and scene content? What evidence shows that the dynamics editor’s incremental gains justify its additional training and complexity?

  2. Does the editor generalize to transformation categories excluded from training, or to compositions of known transformations? A held-out transformation evaluation would materially affect my assessment of the method’s generality.

  3. How does performance change under substantial camera motion and disocclusion?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.