acceptodds
Under review as a conference paper at ICLR 2027

Diverse Motion Customization via Control-based Dynamic Optimization

Abstract

Despite recent advances in video generation, motion customization remains challenging due to content leakage, where appearance attributes from the reference video unintentionally propagate into the generated output. We identify this issue as a consequence of the generative process collapsing toward the reference video, which arises from formulating the learning objective as a direct regression on the reference. To address this, we propose Control-based Motion Customization (CMC), a principled training framework that is structurally robust to content leakage. Our key idea is to steer generative dynamics toward desired motion while avoiding collapse toward the reference video, which we formalize using Stochastic Optimal Control (SOC). Under this formulation, customized videos acquire the target motion yet remain within the pre-trained model's prompt-conditional distribution, where appearance is determined by the text prompt rather than the reference video. Furthermore, to improve efficiency, we tailor the SOC formulation to motion customization by eliminating the need for an explicit reward and introducing a timestep-adaptive motion cost that focuses only on early generative stages, accelerating training by . Extensive experiments demonstrate that CMC effectively mitigates content leakage and achieves competitive motion fidelity while preserving the diversity of the base model across diverse scenarios. Video results are available at https://control-based-mc.github.io/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Motion Customization, Stochastic Optimal Control, Content Leakage

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_LY4kOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  1. The paper addresses a practically relevant problem, and the experiments show encouraging results. Motion transfer requires not only motion fidelity but also flexibility to change the subject and scene. Table 1 on page 9 shows that CMC improves motion fidelity over the base model and achieves the highest diversity score among the evaluated motion customization methods. These results indicate that the method can learn reference motion while retaining substantial variation in the generated outputs. Table S4(a) in the appendix shows that reducing the cutoff step for motion constraints from 28 to 10 decreases training time from approximately 945 to 349 minutes and reduces GPU memory usage, without worsening the motion metric in this experiment. This is a practical improvement over the method’s own version that uses the full denoising trajectory for training.
Critical strengths
  1. Motion transfer requires not only motion fidelity but also flexibility to change the subject and scene. Table 1 on page 9 shows that CMC improves motion fidelity over the base model and achieves the highest diversity score among the evaluated motion customization methods. These results indicate that the method can learn reference motion while retaining substantial variation in the generated outputs.
  2. Table S4(a) in the appendix shows that reducing the cutoff step for motion constraints from 28 to 10 decreases training time from approximately 945 to 349 minutes and reduces GPU memory usage, without worsening the motion metric in this experiment. This is a practical improvement over the method’s own version that uses the full denoising trajectory for training.
Critical weaknesses
  1. In Table 1, CMC has a motion distance of 0.1093, compared with 0.0781 for MotionDirector, where lower is better. Thus, although CMC produces more diverse outputs, its motion is less faithful to the reference than that of some baselines. Could the proposed method simply be relaxing the motion constraints, thereby allowing greater visual variation? A comparison at similar levels of motion fidelity is needed to determine which method copies less of the reference appearance and background.
  2. Moreover, the text fidelity metric mainly checks whether the target object appears, while the diversity metric measures pixel-level differences across generated videos. Neither directly indicates whether the reference background or appearance is retained. From a theoretical standpoint, “remaining close to the base model” does not imply “not copying reference content.”
  3. Equation (20) in the appendix uses a hard argmax to select matching positions and then computes the motion loss from positional differences. However, the training procedure in Eq. (23) requires gradients of this loss with respect to the input. As currently formulated, the discrete position selection cannot provide useful motion gradients through standard backpropagation. The authors should clarify which differentiable approximation or gradient estimator is used in practice. This affects whether the method can be reproduced as described, rather than being merely an issue of mathematical presentation.
Questions for authors

At similar levels of motion fidelity, does CMC still copy less of the reference appearance and background? Could the authors provide a comparison that directly evaluates content leakage, rather than reporting diversity alone?

How is the argmax in Eq. (20) handled during training? Please provide the loss formulation and gradient computation used in the actual implementation.

3 comments

We thank the reviewer for the constructive feedback, in particular on the comparison at similar levels of motion fidelity, the direct measurement of content leakage, and the implementation of Eq. (22). All revisions are shown in blue in the revised PDF, and those addressing your comments are marked with LY4k in orange.

W1, Q1 & W2. Direct leakage metrics, and comparison at similar levels of motion fidelity

(1) [W2, Q1] Direct leakage metrics. We thank the reviewer for this point, and we agree that neither text fidelity nor diversity directly measures whether the reference background or appearance is retained. We therefore add direct leakage metrics and show that CMC achieves the lowest subject leakage, the highest subject-centric diversity, and background leakage comparable to that of the backbone.

Specifically, we evaluate the 7 object-motion references of Table 1 (105 videos per method) with three metrics: subject leakage (DINOv2 similarity between the subject crops of the generated and reference videos), background leakage (CLIP similarity between the backgrounds after removing the subject with SAM), and subject-centric diversity on Grounding DINO crops; the 3 camera-motion references contain no subject and are excluded.

Table A. Direct measures of content leakage (CogVideoX-2B, 105 videos per method).

MethodSubj. leak ↓BG leak ↓Subj. div ↑
Backbone (CogVideoX-2B)0.0830.77399.5
SMM0.0810.78093.3
MOFT0.0930.77789.4
MotionClone0.0850.78092.9
DiTFlow0.0820.76797.0
MotionDirector0.2840.85583.9
MotionInversion0.1770.85179.2
DeT0.1060.80081.8
CMC (Ours)0.0790.774104.4

The tuning-based methods (MotionDirector, MotionInversion, DeT) show the highest leakage in both the subject and the background, together with the lowest subject-centric diversity. These results support our content-leakage claim more rigorously: CMC enables diverse motion customization by mitigating content leakage.

These metrics are feature-similarity proxies for retained appearance and background rather than complete measures of copying, which is why we report them relative to the backbone for the same prompts. (Revised PDF: Sec. 5.1 Evaluation Protocol, Table 1, Sec. 5.2 Quantitative Comparison; definitions in App. D.3)

(2) [W1, Q1] Comparison at matched motion fidelity. Following the suggestion, we compared content leakage at matched motion fidelity, and CMC copies less of the reference than all three tuning-based methods, which have a lower motion distance than CMC in Table 1 (Table B). For each (reference, prompt), we match the 5 baseline videos 1:1 to the 5 CMC videos with an optimal assignment (Hungarian algorithm) that maximizes the number of pairs whose per-video motion distance differs by at most 0.01 and, among those, minimizes the total difference; videos without a partner are excluded, which yields 43–46 pairs out of 105.

Table B. Content leakage at matched motion fidelity (pairs within the same reference and prompt, |ΔMF| ≤ 0.01).

Baseline# pairsMF (baseline / CMC)Subj. leak ↓ (baseline / CMC)BG leak ↓ (baseline / CMC)
MotionDirector460.033 / 0.0370.242 / 0.0580.865 / 0.801
MotionInversion430.048 / 0.0490.147 / 0.0460.870 / 0.798
DeT460.044 / 0.0450.081 / 0.0630.832 / 0.807

The difference is large for MotionDirector and MotionInversion, and the 95% bootstrap intervals of the difference (CMC − baseline, resampled over references) lie below zero for all three methods in both subject and background leakage. The lower motion distance of the tuning-based methods in Table 1 therefore comes with copying the reference, i.e., the overfitting we described, now measured over the whole benchmark. (Revised PDF: App. E.1 "Content Leakage at Matched Motion Fidelity", Table S5)

(3) [Q1] CMC does not relax the motion constraint. Tables A and B show that the diversity of CMC does not come from following the motion less closely: at the same motion distance, CMC still copies less. CMC nevertheless has a higher average motion distance than the tuning-based methods because MF measures how closely a video reproduces the reference trajectories, including their spatial extent, whereas CMC transfers the motion to a different layout. On the examples of Fig. S2, the motion distance between the reference and baseline outputs with content leakage (0.0090) is lower than that between the reference and CMC (0.0200), while unrelated videos with different motion score far worse (0.3250); copying the reference layout thus lowers the distance. The videos are shown in the Rebuttal section Motion fidelity metric.

W2 (cont.). Closeness to the base model and copying

To show more clearly why staying close to the base model prevents copying, we make the link explicit. The key is that CMC stays close to the prompt-conditional base distribution pbase(⋅∣c)p^{\rm base}(\cdot\mid c), and copying the reference requires appearance or background that the prompt does not describe. Let AA be the set of such videos and L(X)=∫01ℓ(Xt,t)dtL(X)=\int_0^1\ell(X_t,t)\mathrm{d}t the accumulated motion cost, and assume that (a) p1base(A∣c)≤εp_1^{\rm base}(A\mid c)\le\varepsilon for a small ε\varepsilon, i.e., the prompt alone rarely produces the reference-specific content (e.g., "a panda turning its head" is very unlikely to produce the background of a specific wolf video), and (b) ρ:=inf⁡x0Epbase[e−L(X)∣X0=x0,c]\rho:=\inf_{x_0}\mathbb E_{p^{\rm base}}[e^{-L(X)}\mid X_0{=}x_0,c] is non-negligible, i.e., the desired motion is reachable from every initial noise. Since ℓ≥0\ell\ge 0, Eq. (7) gives ϕ(x)≤1/ρ\phi(x)\le 1/\rho for every xx, hence p1⋆(A∣c)≤p1base(A∣c)ρ≤ερ.p_1^\star(A\mid c)\le\frac{p_1^{\rm base}(A\mid c)}{\rho}\le\frac{\varepsilon}{\rho}. CMC can therefore raise the probability of copied content by at most a factor of 1/ρ1/\rho. When the reference and the prompt share content, the bound weakens, but the shared content is not leakage either. We do not estimate ρ\rho, since it is a worst case over all initial noises; instead, the direct measurements in W1 & W2 show that this amplification does not occur in practice. We have revised Sec. 4.3 accordingly (Properties 1–2) and added the proof to App. B. (Revised PDF: Sec. 4.3 Properties 1–2; proof in App. B)

W3, Q2. Handling of the argmax in Eq. (20) (Eq. (22) in the revised PDF)

The hard argmax in Eq. (22) is used only for the reference: the reference motion field v(Xref,1)v(X_{\rm ref},1) is a fixed target and requires no gradient. For the generated video, whose motion field enters the loss and must be differentiated, we replace the argmax with a soft-argmax over the same local window N(i)\mathcal{N}(i) (9×99\times 9 tokens, Table S1): p^i=∑j∈N(i)exp⁡(sij/κ)∑k∈N(i)exp⁡(sik/κ)pj,\hat p_i=\sum_{j\in\mathcal{N}(i)} \frac{\exp(s_{ij}/\kappa)}{\sum_{k\in\mathcal{N}(i)}\exp(s_{ik}/\kappa)}p_j, where sijs_{ij} is the cosine similarity of Eq. (22) between the block-15 features of token ii in frame ff and token jj in frame f+1f{+}1, pjp_j is the position of token jj, and κ\kappa is a temperature. The displacement p^i−pi\hat p_i-p_i replaces the argmax displacement in the motion loss (Eq. (23)). The reference field is computed once with the hard argmax and detached, so the gradient flows only through the generated video: from the loss through the softmax weights and the similarities sijs_{ij} to the block-15 features of the generated latent, and from there to the input XtX_t, as required by Eq. (25). As κ→0\kappa\to 0, the soft-argmax recovers the argmax of Eq. (22) when the maximum is unique. We use κ=\kappa= 0.25 (Table S1), and we have added this formulation to App. C.2. (Revised PDF: App. C.2, Table S1)

We hope these clarifications and additional experiments address the reviewer's concerns.

We sincerely thank the reviewer for the thoughtful discussion and for raising the score. Your comments have helped us strengthen our paper.

Reviewer_K4rnOpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CMC, a control-based framework for reference-video motion customization in text-to-video generation. The paper argues that content leakage mainly arises because existing methods directly optimize generated trajectories toward the reference video. CMC instead formulates motion customization as a stochastic optimal control problem, where the generation process is encouraged to follow the reference motion while remaining close to the pretrained model distribution. The method further restricts motion guidance to early denoising steps and uses feature-space motion matching as the supervision signal. Experiments on CogVideoX-2B show competitive motion fidelity with improved text fidelity and diversity.

Critical strengths
  1. The paper provides an interesting perspective on content leakage by analyzing it from the viewpoint of generative dynamics and output distributions rather than only motion and appearance disentanglement. The stochastic optimal control formulation gives a principled way to preserve the pretrained generation distribution while incorporating reference motion.
  2. The paper also considers efficiency through early-stage motion control. The analysis suggests that motion information is largely formed during early denoising steps, and the proposed truncation reduces training cost while maintaining competitive performance.
Critical weaknesses
  1. The effectiveness and generality of CMC are not sufficiently validated on modern T2V backbones. The full CMC method is evaluated only on CogVideoX-2B. Although the appendix includes experiments on models such as Wan2.2 to analyze motion behavior during generation, these experiments do not directly demonstrate that CMC itself remains effective on recent T2V architectures. Since CMC is presented as a general framework, I believe validation on at least one recent flow-matching-based backbone is important. It may not be necessary to reproduce all competing methods on the new backbone. For example, experiments on the smaller Wan2.1-1.3B model with a limited set of comparisons would already provide useful evidence for the generality of the method.
  2. The baseline comparison does not sufficiently cover recent related work. The current comparisons mainly focus on methods from 2025 or earlier, such as MotionClone and DiTFlow. For an ICLR 2027 submission, I believe more recent approaches, such as EffiVMT, FastVMT, and FlowMotion, should also be compared against or sufficiently discussed. Without positioning CMC against these recent motion-transfer methods, it is difficult to assess the value of the proposed method relative to the current state of the field.
  3. The current evaluation may not fully capture content leakage and motion fidelity. I am not fully convinced that the current Diversity metric provides a sufficiently direct measure of content leakage. Diversity computed over the entire video can also be affected by variations in the background and other scene-level content that are unrelated to the appearance of the main subject. The authors should provide stronger justification for why the current metric is appropriate. One possible alternative would be to use Grounding DINO to localize the main subject and evaluate diversity on subject-centric video regions after appropriate padding or resizing, rather than directly using the full video. In addition, I noticed that in some examples on the anonymous project page, the transferred motion appears noticeably faster than the motion in the reference video. Since motion speed and temporal alignment are important aspects of motion cloning, I believe this discrepancy should be reflected either in the Motion Fidelity metric or through an additional evaluation metric.
  4. The accuracy of the latent-space motion fields used for supervision is not directly validated. The method relies on motion-field tracking in the latent feature space to construct the motion loss. However, the current probing experiments mainly provide evidence that intermediate features contain motion-related information; they do not directly demonstrate that the motion fields estimated from these features accurately correspond to the actual motion in the original video. I believe a direct evaluation of the consistency between latent-space motion-field estimates and pixel-space feature tracking would provide stronger justification for using latent-space motion estimation as the supervision signal.
Questions for authors

See weaknesses above.

5 comments

We thank the reviewer for the detailed review and concrete suggestions, in particular on validation with a recent flow-matching backbone, comparison with recent methods, and direct measures of content leakage. All revisions are shown in blue in the revised PDF, and those addressing your comments are marked with K4rn in green.

W1 & W2. Validation on a modern flow-matching backbone and comparison with recent methods

(1) Setup. Following the reviewer's suggestion, we trained CMC on Wan2.1-T2V-1.3B and compared it with three recent methods built on the same backbone: FlowMotion (CVPR'26), FastVMT (ICLR'26), and EffiVMT (ICLR'26). CMC uses the same hyperparameters as on CogVideoX-2B without re-tuning, and each baseline is run with its official settings. All methods are evaluated with the metrics of Table 1 on a smaller subset: 10 references with one prompt and five seeds each (50 videos per method).

Table A. Comparison on Wan2.1-T2V-1.3B.

MethodMF ↓TF ↑TC ↑Div ↑Subj. leak ↓BG leak ↓Subj. div ↑
Backbone (Wan2.1-1.3B, text only)0.13360.83160.97810.21670.0840.77975.4
FlowMotion (training-free)0.11510.74350.95360.14050.1660.79864.3
FastVMT (training-free)0.10070.67530.94330.17800.2090.82067.7
EffiVMT (training-based)0.09110.78080.95950.22310.3580.84763.9
CMC (Ours)0.10930.81120.96990.28760.0770.78587.5

(2) Results. The trend on CogVideoX-2B holds on Wan2.1: CMC achieves a comparable motion distance while outperforming all baselines on every other metric in Table A, particularly diversity and subject leakage. The two methods with a lower motion distance obtain it at the cost of the other metrics: EffiVMT tunes LoRA weights on the reference video and thus learns it directly (L463–464), and FastVMT, in its official setting, starts generation from the noised reference latent (denoising strength 0.75) rather than from pure noise. Both have lower text fidelity, video quality, and diversity, and higher leakage, than CMC. The leakage metrics are those of W3, computed on the 7 object-motion references of this subset. (Revised PDF: App. E.1 "Additional Backbone", Table S4)

(3) Relation to the new baselines. All three methods regress directly to the reference or to a motion representation extracted from it, so the analysis in Sec. 4.1 applies to them as well: the minimizer of their objective is the reference trajectory. They improve efficiency (FastVMT), applicability to flow models without training (FlowMotion), or the disentanglement of spatial and temporal adaptation (EffiVMT) within that objective, whereas CMC changes the objective itself; these improvements are orthogonal to our framework.

W3. Direct measures of content leakage, and motion speed

(1) Direct measures of content leakage. We agree that full-video pixel diversity measures leakage only indirectly (L430); we used it because samples that inherit the appearance and layout of the reference also become similar to each other (Fig. S2). Following the reviewer's suggestion, we now measure leakage directly, and CMC shows the lowest subject leakage and the highest subject-centric diversity among all methods, with background leakage comparable to the backbone (Table B). We evaluate the Table 1 samples on the 7 object-motion references (3 prompts × 5 seeds, 105 videos per method); the 3 camera-motion references contain no subject and are excluded.

  • Subject leakage: DINOv2 cosine similarity between the subject crops of the generated and reference videos (Grounding DINO box with 10% padding, resized to 224×224).
  • Background leakage: CLIP cosine similarity between the generated and reference backgrounds after removing the subject with SAM (filled with the mean color).
  • Subject-centric diversity (the reviewer's proposal): for each group of 15 videos per (method, reference), the mean L2 distance of the concatenated subject crops to the group centroid, normalized by k\sqrt{k}, where kk is the minimum number of frames with a detected subject in the group.

Δ denotes the difference from the backbone for the same prompts, so prompt-consistent changes in subject and background are not penalized.

Table B. Direct measures of content leakage (CogVideoX-2B, 105 videos per method). Δ: difference from the backbone.

MethodSubj. leak ↓ΔBG leak ↓ΔSubj. div ↑Δ
Backbone (CogVideoX-2B)0.083–0.773–99.5–
SMM0.081−0.0020.780+0.00693.3−6.2
MOFT0.093+0.0090.777+0.00489.4−10.1
MotionClone0.085+0.0010.780+0.00692.9−6.6
DiTFlow0.082−0.0010.767−0.00697.0−2.5
MotionDirector0.284+0.2010.855+0.08283.9−15.6
MotionInversion0.177+0.0940.851+0.07879.2−20.3
DeT0.106+0.0230.800+0.02781.8−17.6
CMC (Ours)0.079−0.0040.774+0.001104.4+5.0

The tuning-based methods (MotionDirector, MotionInversion, DeT) show the highest leakage in both the subject and the background, together with the lowest subject-centric diversity, which supports our overfitting claim with a direct measurement. The training-free methods have leakage comparable to CMC but lower subject-centric diversity. (Revised PDF: Sec. 5.1 Evaluation Protocol, Table 1, Sec. 5.2 Quantitative Comparison; definitions in App. D.3)

(2) Motion speed in the reported metric. We agree that speed and temporal alignment are important aspects of motion cloning, and that some samples differ from their references in them. The reported MF already reflects such discrepancies in part: it compares the camera and object motion of the generated and reference videos after resampling both to 12 time steps. Zoom, roll, scale, and rotation are compared step by step, so faster, slower, larger, or smaller motion increases the distance, and pan and object path (Fréchet distance) increase when the extent of the path differs (App. D.3).

(3) An additional timing evaluation. As the additional evaluation the reviewer suggested, we measure how closely the timing of the generated motion follows the reference. For the "panda turning head" example, we generate 100 samples, track the most-moving subject points with CoTracker3, and compare, between each sample and the reference, the fraction of the total motion completed at each time step. 84 of the 100 samples follow the timing of the reference closely, with a gap below 0.13, i.e., within 13% of the total motion on average (median 6%). We have also added samples that similarly follow the speed of the reference, even with different ranges (Rebuttal section Motion speed).

(4) The trade-off. The remaining variation arises because CMC transfers the motion within the prompt-conditional base distribution rather than replicating the reference: the match it targets is a semantic motion match, realized in a form that fits the new subject. Methods that pull the generated video toward the reference, such as the tuning-based baselines, also pull it toward the reference appearance, as their leakage in Table B shows, whereas CMC transfers the motion to subjects that differ from the reference without copying its appearance (Rebuttal section Different object). We have made this trade-off explicit in the Limitations (Sec. 6), and we hope the reviewer will appreciate this benefit. (Revised PDF: MF description in App. D.3; trade-off in Sec. 6 Limitations)

W4. Accuracy of the latent-space motion fields

As requested, we conducted the direct comparison between the target motion field computed in the latent space and pixel-space tracking. The target motion field closely follows the actual motion of the video: 90.1% of the keypoints propagated by this field stay within 0.1⋅max⁡(H,W)0.1\cdot\max(H,W) of their pixel-space tracks (PCK@0.10.1).

Setup. We take 10 DAVIS videos, sampled to 16 frames at 480×720 as in training, and manually annotate 10 keypoints per video on the first frame. The keypoints are placed on moving regions of the subject, so that the evaluation measures how well the field tracks the motion rather than static background. Their trajectories from CoTracker3, which matches human-annotated tracks on TAP-Vid-DAVIS within 1–2 px, serve as the reference, averaged over the four pixel frames encoded by each latent frame. The target motion field is computed exactly as for the reference video in training (CogVideoX-2B, hidden states of block 15 for the clean video, i.e., t=1t=1 in the notation of Eq. (23), cosine similarity within a 9×99\times 9 token window, hard argmax), and each keypoint is propagated by the displacement of the token it occupies. Following DIFT [1] and COVE [2], we report PCK@0.10.1, the fraction of propagated keypoints within 0.1⋅max⁡(H,W)0.1\cdot\max(H,W) (72 px, about four 16-px tokens) of the reference.

This agrees with prior work that estimates motion from correspondences between diffusion features (e.g., DIFT [1], COVE [2], TokenFlow [3], DiTFlow [4], FastVMT [5], SMM [6], and MOFT [7]); our Eq. (22) follows the nearest-neighbor feature matching of TokenFlow [3] and COVE [2]. (Revised PDF: App. C.2 "Accuracy of the Target Motion Field")

References

[1] L. Tang et al. Emergent Correspondence from Image Diffusion. NeurIPS 2023.

[2] J. Wang et al. CoVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editing. NeurIPS 2024.

[3] M. Geyer et al. TokenFlow: Consistent Diffusion Features for Consistent Video Editing. ICLR 2024.

[4] A. Pondaven et al. Video Motion Transfer with Diffusion Transformers. CVPR 2025.

[5] Y. Ma et al. FastVMT: Eliminating Redundancy in Video Motion Transfer. ICLR 2026.

[6] D. Yatim et al. Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfer. CVPR 2024.

[7] Z. Xiao et al. Video Diffusion Models are Training-free Motion Interpreter and Controller. NeurIPS 2024.

We hope these clarifications and additional experiments address the reviewer's concerns.

Reviewer_cXLCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CMC transfers motion from one reference video into text-conditioned generation by learning a correction to the pretrained generative dynamics. It combines a quadratic control penalty with a feature-motion running cost, sets the terminal cost to zero, and truncates Adjoint Matching to early denoising steps. Experiments on CogVideoX-2B report competitive motion fidelity, increased sample variability, and reduced training cost.

Critical strengths
  • The SOC objective provides a principled way to encourage reference-motion alignment while penalizing deviation from the pretrained generative dynamics
  • Quantitative and visual comparisons show convincing motion transfer with diverse, prompt-consistent appearances
Critical weaknesses
  • Preserving support still permits rare, prompt-inconsistent appearances to receive much higher probability. The expected KL/control penalty is not a pointwise bound on the reweighting factor, so the stated guarantee needs additional assumptions or a narrower claim (§4.3, Properties 1–2).

  • The distinctive contribution needs clearer attribution: DiTFlow[1] already matches patch displacements during an initial denoising window, leaving learned regularized dynamics as CMC's important difference. Table S3 neither isolates the control penalty nor directly measures leakage, and its MotionClone variant gains pixel variance while worsening motion distance (0.188→0.212), leaving objective-specific protection unestablished.

  • In video demos, some samples are not following the motion well, or it's very hard to tell how "match" we're expecting. For example, the "panda turning head" sample, the generated panda head motion appears to have a speed and range gap compared with the original video. Same happens in some gallery samples, such as "a bear looking ahead" is not following the motion trajectory of the reference head.

  • In Table 1, the quantitative metrics is also suggesting similar results, and the authors attributes it to overfitting of previous method and supports it with a specific case. Are there evaluation metrics better suited to independently quantify reference-motion matching and unwanted reference-appearance transfer, while allowing prompt-consistent changes in subject and background?

[1] Video Motion Transfer with Diffusion Transformers. CVPR 2025.

Questions for authors

See weaknesses

4 comments

We thank the reviewer for the careful reading and constructive comments, in particular on the scope of the leakage guarantee and on metrics that separate motion matching from appearance transfer. All revisions are shown in blue in the revised PDF, and those addressing your comments are marked with cXLC in purple.

W1. The control penalty bounds the reweighting factor only in expectation (Sec. 4.3, Properties 1–2)

We thank the reviewer for this point. We would like to clarify the conditions under which the guarantee in Sec. 4.3 holds.

The guarantee relies on two assumptions: (a) the base model assigns negligible probability to prompt-inconsistent appearances (L299), and (b) the desired motion lies within the base distribution (L259).

To clarify, let AA be a set of videos with a prompt-inconsistent appearance and L(X)=∫01ℓ(Xt,t)dtL(X)=\int_0^1\ell(X_t,t)\mathrm{d}t. The two assumptions are

  • (a) p1base(A∣c)≤εp_1^{\rm base}(A\mid c)\le\varepsilon for a small ε\varepsilon;
  • (b) ρ:=inf⁡x0Epbase[e−L(X)∣X0=x0,c]=inf⁡x0e−V(x0,0)\rho:=\inf_{x_0}\mathbb E_{p^{\rm base}}[e^{-L(X)}\mid X_0{=}x_0,c]=\inf_{x_0}e^{-V(x_0,0)} is non-negligible, i.e., the desired motion is reachable under the base model from every initial noise.

Then Property 1 holds pointwise. Since ℓ≥0\ell\ge0 and eV(x0,0)≤1/ρe^{V(x_0,0)}\le1/\rho, Eq. (7) gives ϕ(x)≤1/ρ\phi(x)\le1/\rho for every xx, hence p1⋆(A∣c)≤p1base(A∣c)ρ≤ερ.p_1^\star(A\mid c)\le\frac{p_1^{\rm base}(A\mid c)}{\rho}\le\frac{\varepsilon}{\rho}. That is, CMC can amplify the probability of any appearance by at most 1/ρ1/\rho, for any ℓ≥0\ell\ge0.

We do not estimate ρ\rho directly, since it is a worst case over all initial noises, which cannot be verified by sampling. Instead, we check the amplification it bounds, which is not observed in practice: CMC's leakage stays at the backbone level (W4, Table B), and with the same running cost, direct optimization collapses toward the reference while CMC does not (W2).

In summary, the revised paper makes explicit which parts are assumed and which are measured: (a) and (b) are assumptions, the guarantee holds under them (i.e., for a non-negligible ρ\rho), and the non-negligibility of ρ\rho is supported empirically rather than proven. Specifically, we have revised Sec. 4.3 to state (a) and (b) as the conditions of Properties 1–2 and to replace the claim that the control penalty bounds ϕ\phi pointwise (L303–304 of the submission) with the bound above, and we have added the derivation to App. B. (Revised PDF: Sec. 4.3; proof in App. B)

W2. Attribution relative to DiTFlow, and what Table S3 establishes

(1) Relation to DiTFlow. Although our running cost was derived from optical flow, computed in the feature space (App. C.2), we agree that its final form is close to DiTFlow's patch-displacement matching. However, our contribution is the objective, which changes the minimizer, not the running cost (L360–361, App. E). DiTFlow minimizes the motion loss directly, so its minimizer is the reference trajectory, whereas CMC solves a KL-regularized SOC problem whose solution is the base distribution reweighted toward the reference motion (Eq. (7)). Since the bound in W1 holds for any ℓ≥0\ell\ge 0, the protection comes from the objective, not from the choice of ℓ\ell. Indeed, with DiTFlow's own AMF cost, CMC also yields higher diversity and lower subject leakage than optimizing the same cost directly (Table A). As the motion cost and our training framework are orthogonal, we would be grateful if the reviewer could regard the framework itself as our contribution.

(2) Table S3 isolates the control penalty. The running cost ℓ\ell can be designed freely, and the protection against content leakage comes from the CMC objective rather than from a particular design of ℓ\ell. For each running cost, Table S3 compares optimizing the cost directly (fine-tuning) with optimizing it under the CMC objective. We have revised Table S3 to state the objective and settings of each row.

With the simple feature-matching cost ℓ\ell, Table S3 separates the effect of learning from that of the control penalty: Direct (training-free) optimizes each sample at inference, Direct (fine-tuning) trains the model on ℓ\ell (adding learning), and CMC uses the same training setup but adds the control penalty. Fine-tuning directly on ℓ\ell lowers diversity from 0.197 to 0.171, whereas adding the control penalty raises it to 0.206 at a comparable motion distance (0.140 vs. 0.134). The protection therefore comes from the objective, not from training.

The same holds for the other running costs: with the MotionClone cost (0.254 → 0.282) and DiTFlow's AMF cost, CMC again raises diversity over direct fine-tuning on the same cost. For the AMF cost, we also measure leakage directly with the metrics of W4 (Table A): direct fine-tuning reaches a lower motion distance but with higher subject leakage and lower subject-centric diversity than CMC. Across all three running costs, CMC preserves the diversity that direct optimization of the same cost loses by collapsing toward the reference. (Revised PDF: App. E.1 "Effect of the CMC Objective across Running Costs", Table S3)

Table A. DiTFlow's AMF cost: same running cost, different objective. Leakage metrics: 7 object-motion references, 35 videos per method.

Running costObjectiveMF ↓Div ↑Subj. leak ↓BG leak ↓Subj. div ↑
AMF (DiTFlow cost)Direct (fine-tuning)0.1190.1880.1680.77073.4
AMF (DiTFlow cost)CMC (Ours)0.1290.2490.1560.78186.1

(3) Trade-off between motion distance and diversity. We agree that, in Table S3, the higher diversity of CMC can come with a higher motion distance, and this follows from what CMC generates and what Motion Fidelity (MF) measures. MF compares the camera motion and object motion of the generated video with those of the reference, so it measures how closely a video reproduces the reference trajectories, including their spatial extent. CMC transfers the motion to a different layout and therefore does not replicate the reference tracklets exactly. As a result, MF can favor methods that copy the reference over methods that transfer its motion to a new scene. We have clarified this description of MF in App. D.3.

To quantify this, we test whether MF favors videos that copy the reference by computing it on four pairs for the examples in Fig. S2: (a) ref vs. ref, (b) ref vs. the baseline outputs, which exhibit content leakage, (c) ref vs. CMC outputs for the same prompts, and (d) ref vs. unrelated videos with different motion. The motion distance (lower is better) is (a) 0.0000 < (b) 0.0090 < (c) 0.0200 < (d) 0.3250. Copying the reference layout lowers the distance ((b) better than (c)), while CMC remains far closer to the reference motion than unrelated motion ((c) clearly better than (d)). The videos used in this comparison are shown in the Rebuttal section Motion fidelity metric of the project page for visual inspection.

W3. Speed and range of the transferred motion in the demo samples

(1) Speed and range. We thank the reviewer for the careful inspection of the demos. We agree that speed and range are important components of motion, and that some samples differ from their references in how closely they follow them; we acknowledge this as a trade-off of how CMC generates videos.

Nevertheless, the generated motion generally follows the reference. For the "panda turning head" example raised by the reviewer, 84 of 100 samples follow the timing of the reference closely, each with a timing gap below 0.13, i.e., within 13% of the total motion on average (median 6%). The timing gap compares, between a sample and the reference, the fraction of the total motion completed at each time step, measured by tracking the most-moving subject points with CoTracker3. We have also added samples that similarly follow the speed of the reference, even with different ranges (Rebuttal section Motion speed).

(2) The trade-off: a semantic motion match without copying. This trade-off arises because CMC transfers the motion within the prompt-conditional base distribution rather than replicating the reference. The match CMC targets is therefore a semantic motion match: the same motion, realized in a form that fits the new subject, which can differ from the reference in speed or range. When the generated subject differs from the reference subject, an objective that directly pulls the generated video toward the reference, as in the tuning-based methods, also pulls it toward the shape and layout of the reference, consistent with their high leakage in W4, both overall (Table B) and at matched motion distance (Table C).

(3) What this trade-off enables. The benefit is clearest when the reference and generated subjects differ significantly in appearance: the tuning-based baselines tend to carry the reference appearance along with the motion, whereas CMC realizes the motion in a form that fits the new subject (Rebuttal section Different object). This empirically supports our analysis in W1: CMC selects among videos that are valid for the prompt, so prompt-inconsistent leakage remains rare. We have made this trade-off explicit in the paper, and we hope the reviewer will appreciate this benefit, which methods that replicate the reference cannot achieve. (Revised PDF: Sec. 6 Limitations)

W4. Metrics that separate motion matching from appearance transfer

(1) Measuring appearance transfer directly. We thank the reviewer for this suggestion. We now measure the transferred reference appearance directly, and CMC still shows the lowest subject leakage and the highest subject-centric diversity among all methods, with background leakage comparable to the backbone (Table B). Specifically, we measure it separately for the subject and the background. We re-evaluate the Table 1 samples on the 7 object-motion references (3 prompts × 5 seeds, 105 videos per method); the 3 camera-motion references contain no subject and are excluded.

  • Subject leakage: DINOv2 cosine similarity between the subject crops of the generated and reference videos (Grounding DINO box with 10% padding, resized to 224×224).
  • Background leakage: CLIP cosine similarity between the generated and reference backgrounds after removing the subject with SAM (filled with the mean color).
  • Subject-centric diversity: for each group of 15 videos per (method, reference), the mean L2 distance of the concatenated subject crops to the group centroid, normalized by k\sqrt{k}, where kk is the minimum number of frames with a detected subject in the group.

Δ denotes the difference from the backbone for the same prompts, so prompt-consistent changes in subject and background are not penalized.

Table B. Direct measures of content leakage. Δ: difference from the backbone.

MethodSubj. leak ↓ΔBG leak ↓ΔSubj. div ↑Δ
Backbone (CogVideoX-2B)0.083–0.773–99.5–
SMM0.081−0.0020.780+0.00693.3−6.2
MOFT0.093+0.0090.777+0.00489.4−10.1
MotionClone0.085+0.0010.780+0.00692.9−6.6
DiTFlow0.082−0.0010.767−0.00697.0−2.5
MotionDirector0.284+0.2010.855+0.08283.9−15.6
MotionInversion0.177+0.0940.851+0.07879.2−20.3
DeT0.106+0.0230.800+0.02781.8−17.6
CMC (Ours)0.079−0.0040.774+0.001104.4+5.0

Among all methods, CMC has the lowest subject leakage (0.079) and the highest subject-centric diversity (104.4), and its background leakage (0.774) is comparable to that of the backbone. The tuning-based methods (MotionDirector, MotionInversion, DeT) show the highest leakage in both the subject and the background, together with the lowest subject-centric diversity. (Revised PDF: Sec. 5.1, Table 1, Sec. 5.2; definitions in App. D.3)

(2) Separating motion matching from appearance transfer. The metrics in (1) measure appearance transfer, but the tuning-based baselines also match the reference motion more closely than CMC (lower motion distance in Table 1), so their higher leakage could simply accompany closer motion matching. To additionally separate the two, we compare content leakage between CMC and these baselines on pairs of videos with matched motion distance.

For each (reference, prompt), we match the 5 baseline videos 1:1 to the 5 CMC videos based on motion distance alone, with an optimal assignment (Hungarian algorithm) that maximizes the number of pairs whose per-video motion distance differs by at most 0.01 and, among those, minimizes the total difference.

Table C. Content leakage at matched motion fidelity (pairs within the same reference and prompt, |ΔMF| ≤ 0.01).

Baseline# pairsMF (baseline / CMC)Subj. leak ↓ (baseline / CMC)BG leak ↓ (baseline / CMC)
MotionDirector460.033 / 0.0370.242 / 0.0580.865 / 0.801
MotionInversion430.048 / 0.0490.147 / 0.0460.870 / 0.798
DeT460.044 / 0.0450.081 / 0.0630.832 / 0.807

At matched motion distance and prompt, CMC copies less of the reference than all three tuning-based methods, by a large margin for MotionDirector and MotionInversion; the 95% bootstrap intervals of the difference (CMC − baseline, resampled over references) lie below zero for all three methods in both subject and background leakage. The lower motion distance of the tuning-based methods in Table 1 therefore comes with copying the reference, i.e., the overfitting we described, now measured over the whole benchmark rather than a single case. (Revised PDF: App. E.1, Table S5)

We hope these clarifications and additional experiments address the reviewer's concerns.

Comments on the paper

2 comments

Summary of revisions

We thank all reviewers and the AC. All revisions are shown in blue in the revised PDF, and each change is tagged with the reviewer it addresses (cXLC purple, K4rn green, LY4k orange). The main changes are:

  • Leakage guarantee with explicit assumptions (cXLC, LY4k; Sec. 4.3, App. B): Property 1 is restated as a pointwise bound under two explicit assumptions, replacing the claim that the control penalty bounds ϕ\phi pointwise (L303–304 of the submission).
  • Direct leakage metrics (all reviewers; Sec. 5.1, Sec. 5.2, Table 1, App. D.3): subject leakage, background leakage, and subject-centric diversity. CMC has the lowest subject leakage and the highest subject-centric diversity, with background leakage at the backbone level.
  • Leakage at matched motion fidelity (cXLC, LY4k; App. E.1, Table S5): at matched MF, CMC copies less of the reference than all tuning-based baselines.
  • Additional backbone and recent baselines (K4rn; App. E.1, Table S4; Sec. 2): CMC on Wan2.1-T2V-1.3B against FlowMotion, FastVMT, and EffiVMT, with comparable MF while outperforming all baselines on all other metrics.
  • Running-cost ablation (cXLC; App. E.1, Table S3): each running cost is listed with its objective to isolate the control penalty, and DiTFlow's AMF cost is added.
  • Accuracy of the target motion field (K4rn; App. C.2): 90.1% PCK@0.10.1 against pixel-space tracks.
  • Gradient of the motion cost (LY4k; App. C.2, Table S1): soft-argmax formulation, gradient path, and temperature κ.
  • Motion speed and MF (cXLC, K4rn; App. D.3, Sec. 6): MF description clarified, and the trade-off between exact trajectory matching and content leakage discussed.

Videos referenced in the responses are in the Rebuttal section of the original project page: https://control-based-mc.github.io/

We sincerely thank the Area Chair for handling our submission and all reviewers for taking the time to engage in the discussion despite their busy schedules. The reviewers' constructive feedback has greatly helped us improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.