acceptodds
Under review as a conference paper at ICLR 2027

From Static to Dynamic: Motion-Refined On-Policy Distillation from Image to Video Diffusion Models

Abstract

On-policy distillation (OPD) specializes pretrained video diffusion models through teacher supervision along the student's own generation trajectory. Although large video models are natural teachers, developing specialized video experts can require costly video data and training, while querying them incurs substantially higher latency than querying image experts. More readily available and cheaper to query, image experts offer a cost-effective alternative, particularly for largely temporal-agnostic capabilities such as aesthetics and OCR that admit frame-level supervision. However, heterogeneous image and video latent spaces prevent direct supervision of intermediate student states, while image experts lack cross-frame motion supervision, making temporal consistency vulnerable to frame-level improvements. In this paper, we propose **MILD**, a **M**otion-Preserving **I**mage-to-Video **L**atent **D**istillation framework that transfers specialized image expertise while preserving pretrained video dynamics. MILD uses a learnable linear connector that aligns student latent states and predicted updates with those of image experts, enabling supervision transfer across heterogeneous latent spaces. We further constrain image-guided corrections around the pretrained student's predictions to preserve video dynamics and incorporate an optical-flow-based motion reward to improve motion quality and temporal consistency. Across specialized image experts and multiple video-student backbones, our method consistently outperforms video-teacher OPD baselines, with further studies demonstrating effective transfer across connector designs and heterogeneous architectures. These results establish image-to-video distillation as an effective route to improving video generation by drawing on the diverse and evolving capabilities of the image-generation ecosystem.

Primary area:
generative models
Keywords:
Video Diffusion, On-Policy Distillation, Cross-Architecture Transfer

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ApLuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MILD, an on-policy distillation framework that transfers capabilities from image diffusion model to video diffusion models. The method uses a learnable connector to align heterogeneous image and video latent spaces and pulls image-expert scores back into video coordinates through the connector’s Jacobian transpose. To control the effects of frame-level supervision on temporal generation, MILD anchors its corrections to a frozen video model, constrains their magnitude and inter-frame variation, and incorporates a motion reward from a separately trained optical-flow discriminator. The experiments show improved

Critical strengths
  1. The paper addresses a practically relevant problem and provides a reusable interface for supervision across models.
  2. The improvements are consistent across backbones, with meaningful evidence attributing gains to the proposed components.
Critical weaknesses
  1. The central assumptions underlying cross-latent-space transfer lack direct validation, particularly regarding the compatibility of noisy states.

The connector aligns clean latents using statically repeated video clips and adapts to student trajectories through velocity alignment. However, these objectives do not directly ensure that mapped states follow the noise distribution expected by the image teacher at the same diffusion time. For a linear connector, for example, F(xt)=(1−t)Ax0+tAϵ,F(x_t)=(1-t)Ax_0+tA\epsilon, where the noise covariance is AA⊤AA^\top, which generally differs from the identity covariance expected by the image teacher. Although the gradient pullback in Eq. (5) is mathematically valid, this alone does not establish that the teacher’s predictions at the mapped states provide reliable supervision. The paper does not report diagnostics of state compatibility or teacher predictions across noise levels. Consequently, the conditions under which this central interface works, and the limits of its generalization, remain unclear.

  1. The experiments do not sufficiently isolate the mechanism underlying the central motion-preservation claim.

Table 5 provides useful ablations of image supervision and motion refinement, but it does not isolate the contributions of temporal filtering and inter-frame correction bounds: the image-guided variants retain both the frozen video anchor and the temporal correction machinery. Consequently, the results do not establish whether these temporal constraints provide benefits beyond conservative anchoring, magnitude clipping, and motion refinement. A controlled comparison of spatial improvement and temporal quality is needed.

Questions for authors
  1. How are noisy states and diffusion times aligned?

How are latent scaling, noise covariance, and diffusion-time parameterization matched between the image and video models? Are any constraints imposed on AA⊤AA^\top or the noise statistics of mapped states? Could the authors provide diagnostics across diffusion times showing that the image teacher’s predictions at these states still reflect the intended visual content?

  1. Do the temporal correction constraints improve the trade-off between spatial quality and temporal quality beyond anchoring and magnitude control alone?

Could you compare full MILD against a variant that removes temporal filtering and inter-frame correction bounds while retaining other settings unchanged? This would clarify whether the proposed constraints specifically prevent temporal degradation from image supervision.

5 comments

We thank the reviewer for the constructive feedback. We respond to each question below.

Q1&W1. How are latent scaling, noise covariance, and diffusion time matched across image and video models? What constraints are imposed on the connector or mapped noise, and do teacher predictions preserve visual content across diffusion times?

Answer: We evaluate the Wan2.1 video student with three SD3.5-Medium image experts specialized in GenEval, aesthetics, and OCR. The experts share a VAE and one video-to-image connector.

Latent scaling and diffusion time. The connector maps Wan-normalized video latents, zV=(zraw,V−μV)/σVz_{V}=(z_{\mathrm{raw},V}-\mu_{V})/\sigma _ {V}, into SD3.5-Medium’s normalized image-latent space, zI=(zraw,I−0.0609)×1.5305z _ {I}=(z_ {\mathrm{raw},I}-0.0609)\times1.5305. Wan normalization is applied once within its VAE interface. An eight-image audit confirms matching fresh and cached Wan encodings, with inverse-normalization error of approximately 10−710^{-7}.

Under xt=(1−t)x0+tϵx_{t}=(1-t)x_{0}+t\epsilon, t=0t=0 denotes clean data and t=1t=1 noise (Appendix A.1). We query the image teacher at τI=1000t\tau _ {I}=1000t, using the student scheduler’s actual flow time, not the step index. Its sigma shift of 5 is already reflected in tt. For the mapped state yt=Axty _ {t}=Ax _ {t}, the teacher’s clean prediction is

y^0=yt−t uICFG(yt,1000t,c),\hat{y}_{0}=y _{t}-t\,u _ {I}^{\mathrm{CFG}}(y _ {t},1000t,c),

where uICFGu _ {I}^{\mathrm{CFG}} denotes its CFG-guided velocity prediction.

Connector constraints and noise covariance. The shared connector is bias-free and linear, comprising fixed latent-slice selection, bilinear resizing, and spatial convolutions. Training uses content alignment, velocity alignment, and regularization toward its near-identity initialization (Eq. 6; Appendix B), without explicit orthogonality or noise-whitening constraints.

For ϵV∼N(0,I)\epsilon_{V}\sim\mathcal{N}(0,I), mapped noise has covariance AA⊤AA^\top. We measure its deviation from isotropic covariance as Ecov=∥AA⊤−I∥Fdout,E _ {\mathrm{cov}}=\frac{\lVert AA^\top-I\rVert _ {F}}{\sqrt{d _ {\mathrm{out}}}}, where doutd_{\mathrm{out}} is the output dimension. For the evaluated Wan2.1 connector configuration, the mean over three trials with 32 independent Gaussian output probes each is 0.00547, indicating approximate isotropic-noise preservation by the tested operator under this metric.

Content correspondence across diffusion times. We use the original Wan2.1 checkpoint-250 student and trained connector without recalibration or parameter updates. We generate videos for 32 prompts with 50 sampling steps and student CFG 5, arranging them into 16 disjoint pairs.

For each pair, we query the image teacher with one video’s mapped donor state and the other’s recipient prompt. Both directions and three latent slices yield 96 comparisons per teacher and time. Teacher CFG is 4.5. Donor identification succeeds when the decoded teacher clean prediction has higher DINOv2 similarity to the corresponding donor final-video frame than to the recipient frame.

Flow time ttGenEval identificationAesthetic identificationOCR identification
0.24193587.50% [78.10, 95.83]88.54% [80.21, 95.83]87.50% [78.10, 95.83]
0.48780572.92% [62.50, 83.33]71.88% [61.46, 82.29]70.83% [59.38, 82.29]
0.79710152.08% [45.83, 58.33]53.12% [44.79, 60.42]51.04% [45.83, 57.29]

All three teachers retain identifiable donor content at low and intermediate noise despite the recipient prompt. At high noise, identification approaches the 50% reference level.

To assess the teacher-velocity contribution, let yD,yRy _ {D},y _ {R} be the mapped donor and recipient states at the same time, and cRc _ {R} the recipient prompt. We compare y^0state=yD−t uICFG(yD,1000t,cR)\hat{y} _ {0}^{\mathrm{state}}=y _ {D}-t\,u _ {I}^{\mathrm{CFG}}(y _ {D},1000t,c _ {R}) with a control using the recipient-state velocity:

=y _ {D}-t\,u _ {I}^{\mathrm{CFG}}(y _ {R},1000t,c _ {R}).$$ Both retain $y _ {D}$ and $c _ {R}$; only the velocity-query state changes. We report the paired improvement $\Delta m$ in the donor-minus-recipient DINO similarity margin: |Flow time $t$|GenEval $\Delta m$|Aesthetic $\Delta m$|OCR $\Delta m$| |-|-:|-:|-:| |0.241935|0.0953 [0.0632, 0.1247]|0.0971 [0.0681, 0.1224]|0.0959 [0.0657, 0.1238]| |0.487805|0.1119 [0.0770, 0.1564]|0.1212 [0.0886, 0.1580]|0.1219 [0.0820, 0.1676]| |0.797101|0.0912 [0.0608, 0.1197]|0.0693 [0.0271, 0.1130]|0.0625 [0.0327, 0.0957]| Both tables report 95% confidence intervals from 1,000 pair-level bootstrap resamples. Donor-state queries improve content correspondence over the mismatched-velocity control in all nine conditions, with every pointwise interval above zero. We additionally provide qualitative examples in Appendix C.8, showing frames from the final student videos alongside image-teacher clean predictions at low, intermediate, and high noise levels. These examples visualize content correspondence across diffusion times and complement the quantitative diagnostics above.

Q2&W2. Do the temporal correction constraints improve the trade-off between spatial quality and temporal quality beyond anchoring and magnitude control alone? Could the authors compare full MILD against a variant that removes temporal filtering and inter-frame correction bounds while retaining other settings unchanged, to clarify whether these constraints specifically prevent temporal degradation from image supervision?

Answer: Following the reviewer’s suggestion, we jointly remove temporal filtering and inter-frame correction bounds on Wan2.2-5B, while retaining the frozen video anchor, global and per-frame magnitude constraints, image-expert supervision, and motion refinement. All other training and evaluation settings remain unchanged. We evaluate both overall video quality and the temporal metrics in EvalCrafter.

EvalCrafter results.

MethodVisual Quality ↑Text–Video Alignment ↑Motion Quality ↑Temporal Consistency ↑
Pretrained student61.4855.6354.0061.93
w/o temporal filtering and inter-frame bounds62.3658.3254.2561.98
MILD63.7659.1254.4362.79

Compared with the ablated variant, full MILD improves Temporal Consistency by 0.81 and Motion Quality by 0.18, alongside gains of 1.40 in Visual Quality and 0.80 in Text–Video Alignment. These results support an additional benefit from temporal constraints beyond anchoring and magnitude control, without sacrificing visual quality or prompt alignment. The ablated variant remains slightly above the pretrained student on temporal consistency, so the evidence supports improved temporal quality rather than degradation below the pretrained baseline.

VBench-2.0 results.

MethodCreativity ↑Commonsense ↑Controllability ↑Human Fidelity ↑Physics ↑AVG. ↑
w/o temporal filtering and inter-frame bounds46.4460.4917.2982.6349.6351.30
MILD49.5062.8421.8084.8954.2954.66

Full MILD also improves the VBench-2.0 average by 3.36 points, with gains across all five categories, providing complementary evidence of improved overall video quality.

Magnitude control and temporal constraints address different properties of the correction. For example, adjacent frames can receive corrections of equal, bounded magnitude but opposite directions. Both satisfy the magnitude constraint, yet their difference can still be large. Temporal filtering and inter-frame bounds explicitly regulate this variation. The joint ablation supports the value of this temporal control within the full training framework.

The VBench-2.0 ablation is reported in Table 5 (p. 9), with its setting and analysis in Section 4.4, “Temporal filtering and correction magnitude” (p. 10, lines 495–503).

Dear Reviewer,

We would like to kindly follow up on our rebuttal and ask whether our responses have addressed your concerns. We would greatly appreciate any further feedback or questions you may have.

Thank you for your time and consideration.

Best,

Authors

Reviewer_2QUyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MILD, an image-to-video on-policy distillation framework that transfers specialized capabilities from image diffusion teachers to video diffusion students. The main motivation is to leverage cheaper and more accessible image experts instead of computationally expensive video teachers, particularly for capabilities such as visual aesthetics and text rendering. To address the mismatch between image and video latent spaces, the authors introduce a learnable connector trained with content and velocity alignment objectives. Image-teacher guidance is then mapped into the video latent space and constrained using a frozen pretrained video model to preserve temporal dynamics.

Critical strengths
  1. Clear and convincing motivation. Leveraging specialized image diffusion models to improve video generation is intuitive and well-motivated, especially considering the computational cost of video diffusion teachers.

  2. Reasonable and theoretically grounded approach. The proposed method addresses the challenges of heterogeneous latent spaces and temporal consistency in a principled way, supported by mathematical derivations.

  3. Clear presentation. The paper is well-organized and easy to follow. The method and its individual components are clearly explained.

Critical weaknesses
  1. Limited comparison with equally specialized video teachers. The V2V OPD baselines use either general-purpose pretrained video teachers or video teachers fine-tuned only for aesthetics, whereas MILD combines multiple image experts specializing in aesthetics, compositional generation, and OCR. This makes it difficult to isolate whether the improvements come from using image teachers or from combining more diverse specialized capabilities.

  2. Missing ablations for connector alignment objectives. The paper introduces both content alignment and velocity alignment to train the connector, but does not evaluate their individual contributions. Ablations removing each objective would help clarify their necessity and effectiveness.

  3. Missing Related work: It would be helpful to discuss Cross-Space Distillation (Nguyen, Anh, et al., ECCV 2026) Both methods learn a mapping network between heterogeneous latent spaces, but use different alignment objectives. In particular, a discussion of how the proposed content and velocity alignment losses differ from those in the prior work would help clarify the contribution of the connector design.

Questions for authors
  1. Evaluation protocol in Figure 4: Could the authors clarify how the image and video models were evaluated on GenEval, OCR, and Aesthetic metrics? In particular, were the video models evaluated using a single frame or multiple frames, and were the evaluation prompts and sampling conditions consistent across image and video models?

  2. Linear connector assumption: The default connector uses a linear mapping between heterogeneous VAE latent spaces. Is there any theoretical or empirical justification for assuming that a linear mapping is sufficient? Although Table 5 compares linear and nonlinear connectors, it is unclear whether the performance differences arise from linearity itself or other architectural and optimization factors.

6 comments

We thank the reviewer for recognizing the strengths of our work. We are grateful for the constructive feedback and address your questions below.

W1. MILD combines image experts in aesthetics, composition, and OCR, whereas V2V OPD uses general-purpose or aesthetic-only video teachers. Do the gains arise from image-to-video transfer or more diverse teacher capabilities?

Answer: We compare MILD with one aesthetic image expert against V2V OPD with one aesthetic-SFT video teacher, controlling teacher count and specialization domain. Tables 2 and 5 report the following Wan2.2-5B results:

MethodCreativity ↑Commonsense ↑Controllability ↑Human Fidelity ↑Physics ↑AVG. ↑
V2V OPD (aesthetic-SFT video teacher)48.6461.6919.0783.7550.6252.75
MILD (aesthetic image expert only)49.2661.0919.7084.5251.5853.23

MILD achieves a 0.48-point higher average and improves four of five categories, showing that its gains are not solely attributable to expert diversity. This comparison matches teacher count and specialization, although teacher strength and training data are not fully matched. Separately, our efficiency evaluation shows teacher-query latency decreasing from 4.59 s to 1.83 s on Wan2.1-1.3B (Figure 1), alongside training-memory reductions across all three student backbones (Section 4.2; Figure 5).

Access to diverse specialized image capabilities is also a motivation for MILD. The image domain provides established supervision resources for aesthetics, composition, and text rendering, including the LAION aesthetic predictor [1], GenEval [2], and Flow-GRPO’s text-rendering training pipeline [3]. DiffusionOPD [4] further demonstrates specialized image-expert construction and transfer. These resources enable image-expert development without corresponding task-specific video datasets; equivalent video specialization would require additional video-data curation or reward-based post-training, including video-generation and optimization costs. MILD exploits this available expertise: extending from one aesthetic expert to three experts raises AVG from 53.23 to 54.66 (Table 5).

The single-expert results appear in Tables 2 and 5, with expert-configuration analysis in Section 4.4 (p. 10, lines 518–523).

W2. What are the individual contributions of content and velocity alignment? Could each objective be ablated separately?

Answer: Content alignment matches paired clean image/video representations; velocity alignment matches denoising directions along student trajectories. We separately remove each objective on Wan2.2-5B, retaining the other objective and all remaining training settings. Both ablations use the iteration-250 EMA checkpoint and the same evaluation protocol.

MethodCreativity ↑Commonsense ↑Controllability ↑Human Fidelity ↑Physics ↑AVG. ↑
w/o content alignment47.2658.2018.4083.7850.0551.54
w/o velocity alignment47.9259.9419.1183.8652.6652.70
MILD (content + velocity)49.5062.8421.8084.8954.2954.66

Removing content alignment reduces AVG by 3.12 points, most notably Commonsense (−4.64) and Physics (−4.24), showing that velocity supervision alone does not replace explicit clean-content correspondence. Removing velocity alignment reduces AVG by 1.96, including Commonsense (−2.90) and Controllability (−2.69), supporting the benefit of aligning denoising directions beyond clean representations. Both ablations underperform across all five categories, supporting the objectives’ complementary contributions.

These ablations appear under “Connector alignment objectives” in Table 5 (p. 9), with settings and analysis in the corresponding paragraph of Section 4.4 (p. 10, lines 510–517). All additions are highlighted in blue.

W3. Please discuss Cross-Space Distillation (CSD; Nguyen et al., ECCV 2026), particularly how MILD’s content and velocity alignment differ from its alignment objectives.

Answer: Thank you for highlighting CSD [5]. CSD transfers modern image teachers into compact one-step image students across heterogeneous latent spaces. MILD instead transfers image expertise to video students during on-policy post-training, requiring alignment at states visited by an evolving student.

Alignment objectives. CSD trains its Bridge with latent reconstruction and attention fidelity (Eqs. 7–9), matching reference teacher latents and preserving frozen-teacher attention responses. MILD’s content alignment similarly matches paired clean video/image VAE representations (Eq. 6). The key distinction is velocity alignment along student trajectories, rather than the reconstruction norm: at student-visited xtx_t, we match JFψ(xt)vθ(xt,t,c)J_{F_\psi}(x_t)v_\theta(x_t,t,c) to image-expert predictions at Fψ(xt)F_\psi(x_t), aligning denoising directions alongside clean content.

Connector updates. CSD pretrains and freezes its Bridge during one-step distillation. MILD updates the connector alongside the video student to track evolving trajectories (Sections 3.2 and 3.4), using separate objectives and detaching student velocities and expert predictions during connector updates.

Removing content or velocity alignment reduces AVG from 54.66 to 51.54 or 52.70, respectively, supporting their complementary contributions (Table 5; Answer W2).

We have added this comparison to Section 2, “On-policy diffusion distillation” (p. 3, lines 140–144).

Q1. How were image and video models evaluated on GenEval, OCR, and Aesthetic? Were single or multiple video frames used, and were prompts and sampling settings consistent?

Answer: Figure 4 and Table 4 evaluate videos using one predetermined middle frame: index 40 of 81 frames (zero-based), without best-frame selection or multi-frame averaging. Image models are evaluated on directly generated images.

Prompts and scoring. Both modalities use identical benchmark prompt files, evaluators, and aggregation procedures, without prompt rewriting or added positive-prompt prefixes/suffixes.

BenchmarkEvaluation samples per modelScoring
GenEval553 prompts × 4 samples = 2,212Mask2Former instance segmentation and CLIP-based color assessment
OCR1,018 prompt entriesEnglish EasyOCR; one minus normalized Levenshtein distance to the quoted target text
Aesthetic200 DrawBench prompts × 5 samples = 1,000CLIP ViT-L/14 features and LAION aesthetic predictor

Sampling settings. Generation settings are fixed across baseline and adapted checkpoints within each family, but differ across modalities:

SettingImage models (SD3.5-M family)Video models (Wan family)
Resolution (height × width)512 × 512480 × 832
Frames181
Denoising steps4050
CFG4.55.0
SchedulerDefault SD3 configurationWan configuration; sigma shift 5.0
Negative promptNoneShared default Wan negative prompt
SeedNot explicitly fixedGlobal evaluation sample ID

Video models share sample-index seeds; image and video outputs are not seed-paired. Figure 4 thus compares spatial capabilities under these settings, rather than identical cross-modal sampling conditions.

Frame-position sensitivity. We additionally evaluate all four Wan2.2-5B variants separately at fixed indices 20, 40, and 60, using the same videos, benchmark samples, and evaluators. Frame 40 remains the primary result.

Aesthetic↑

MethodFrame 20Frame 40Frame 60MeanSD
Pretrained student4.72384.78554.59374.70100.0799
V2V OPD (aesthetic-SFT teacher)4.89324.91694.88234.89750.0144
V2V OPD (Wan2.2-A14B teacher)4.81234.82174.79424.80940.0114
MILD5.02555.03055.02945.02850.0021

GenEval↑

MethodFrame 20Frame 40Frame 60MeanSD
Pretrained student0.32330.32520.31620.32160.0039
V2V OPD (aesthetic-SFT teacher)0.39270.40230.40160.39890.0044
V2V OPD (Wan2.2-A14B teacher)0.40740.40670.40720.40710.0003
MILD0.44230.44540.44160.44310.0017

OCR↑

MethodFrame 20Frame 40Frame 60MeanSD
Pretrained student0.25190.25220.25020.25140.0009
V2V OPD (aesthetic-SFT teacher)0.26130.26170.26070.26120.0004
V2V OPD (Wan2.2-A14B teacher)0.25390.25560.25430.25460.0007
MILD0.26830.26970.26750.26850.0009

MILD ranks first at every tested position on all three metrics. Its position-averaged gains over the strongest corresponding baseline are 0.1310 Aesthetic, 0.0360 GenEval, and 0.0073 OCR, indicating that the improvements are not specific to the middle frame. SD measures variation across the three position-level aggregate scores, not within-video temporal consistency.

Protocols and sampling settings appear in Appendix B, “Spatial capability evaluation” (pp. 22–23, Tables 8–9). We will add the frame-position results to the same appendix.

Reviewer_zG7NOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MILD, which adopts a specially optimized image diffusion model as the teacher and transfers the supervision from the image expert to the video diffusion model via a learnable latent connector. The method uses a frozen video model as the anchor, constrained image guidance, and optical-flow-based motion reward to improve spatial quality while preserving temporal dynamics. Evaluated on Wan2.1-1.3B, Wan2.2-5B and LTX-Video-2B, MILD outperforms the compared V2V OPD baselines in overall video generation metrics and partially reduces computational overhead.

Critical strengths
  1. The research has practical value. The authors leverage optimized image models to provide supervision for video generation, which can reduce the high training cost of large video teachers in practical application scenarios.

  2. The experiments in many video model architectures provide more useful results. The method on Wan2.1-1.3B, Wan2.2-5B and LTX-Video-2B shows the improvements, demonstrating that the method is not limited to a single model.

  3. The details are provided thoroughly. The appendix provides some details such as derivations of the loss function, connector optimization, and correction magnitude constraints, which facilitates understanding and reproducibility of the method.

  4. This paper aligns the image and video latent spaces via a learnable connector, and maps video velocity to the image space using the Jacobian, providing a supervised transfer scheme applicable to heterogeneous models.

Critical weaknesses
  1. The contribution of cross-modal distillation has not been sufficiently isolated. The image expert has advantages in tasks such as aesthetics, composition, and OCR. I think the experiments have not fully demonstrated whether comparable results can be obtained via a simpler video post-training approach using the same image supervision objective.

  2. The AUROC results in Appendix C.1 show that the discriminator can distinguish specific real, generated, and temporally perturbed samples. However, I would like to ask whether these results can prove that it understands semantic correctness or genuine physical motion.

  3. The generalization ability of the motion reward. In the Appendix C.1, the authors say their dataset is only split at the video-file level, and caption-matched positive and generated videos can appear in different splits. Frame rates are also not standardized before flow extraction. Therefore, I think the existing experiments cannot prove the generalization ability of the motion discriminator to unseen caption groups, and the experiments also cannot prove the influence of frame rate differences.

  4. This paper has not reported the user study results.

  5. The paper analyzes the effectiveness of the image expert and motion refinement separately. However, I think the authors should also prove the effectiveness of the frozen video anchor, temporal filtering, and correction magnitude constraint.

  6. The temporal consis. in Tab 2 on the Wan2.2-5B is lower than the V2V OPD results. I am concerned that the temporal consistency may be influenced by the proposed method.

  7. The connector, the score pullback, the frozen video model, and the motion refinement all exist in many recent works, so it is hard to judge whether the technical contribution of this paper is only an engineering attempt.

Questions for authors

Most of my concerns are listed in the Weaknesses, and there are some questions:

  1. Can the authors conduct some experiments only use the image expert, the video anchor, and the motion refinement, without cross-modal distillation in MILD, I think it can prove the improvement are from the proposed framework or the image expert.

  2. If remove the frozen video anchor, how will the performance change?

  3. I would like to see some failure cases such as fast motion, complex interactions, and camera movement, which is helpful to understand the boundary of the method.

  4. In my understanding, MILD also needs to train the image expert and motion discriminator, and the method also needs to decode and calculate the flow. Therefore, I would like to see the end-to-end training cost.

6 comments

We thank the reviewer for recognizing the strengths of our work. We are grateful for the constructive feedback and address your questions below.

Q1&W1. To isolate MILD’s cross-modal distillation from the image experts’ capabilities, could the authors compare against simpler video post-training using the same image supervision, video anchor, and motion refinement, but without cross-modal distillation?

Answer: Following the reviewer’s suggestion, we evaluate expert-image target regression on Wan2.2-5B, retaining the same image experts, prompt-dependent expert weights, frozen video anchor, and motion refinement, while replacing MILD’s cross-modal guidance with direct image-target supervision. Expert-generated images are repeated into static clips and encoded with the student’s video VAE to define velocity targets at student-rollout states, supervising only the middle temporal latent slice without a connector or score pullback. We test both aesthetic-only and three-expert configurations. We additionally evaluate frame-level aesthetic reward fine-tuning using the same LAION aesthetic scorer used to train the aesthetic expert, while retaining the video anchor and motion refinement.

MethodCreativity ↑Commonsense ↑Controllability ↑Human Fidelity ↑Physics ↑AVG. ↑
Expert-image target regression (aesthetic)48.9462.5421.6183.7849.1353.20
Frame-level aesthetic reward fine-tuning47.9961.0721.6382.8451.0352.91
MILD (aesthetic expert only)49.2661.0919.7084.5251.5853.23
Expert-image target regression (three experts)48.9861.9422.2582.6051.4753.45
MILD (three experts)49.5062.8421.8084.8954.2954.66

With three experts, MILD exceeds target regression by 1.21 points on average and scores higher in four of five categories, supporting the benefit of its state-conditioned cross-modal transfer beyond access to the same experts. With aesthetic supervision alone, MILD and target regression achieve comparable averages (53.23 vs. 53.20), while direct reward fine-tuning achieves 52.91.

Complementary evidence of spatial capability transfer. Section 4.3 and Table 4 show that MILD outperforms the pretrained student and both V2V OPD baselines on Aesthetic, PickScore, GenEval, and OCR. These results demonstrate spatial capability gains, while the new same-expert comparison above supports the benefit of MILD’s transfer mechanism over direct image-target supervision.

We have added the baseline results to Table 5 (p. 9) and the analysis to Section 4.4 (p. 10, lines 524–532). Detailed baseline protocols, including the objectives, loss weights, and sampling settings, appear in Appendix C.2.2 (pp. 25–26, lines 1348–1349 and 1375–1390; Eq. 17). All additions are highlighted in blue.

W2. The AUROC results in Appendix C.1 show that the discriminator can distinguish specific real, generated, and temporally perturbed samples. However, I would like to ask whether these results can prove that it understands semantic correctness or genuine physical motion.

Answer: AUROC is a standard threshold-independent discrimination metric. Here, it measures how well Motion-D separates real-video motion from generated or temporally corrupted motion. Motion-D provides auxiliary motion feedback, rather than verifying semantic correctness or physical laws (Appendix C.1). To assess alignment with human judgments, five CS PhD students independently rated motion naturalness for 400 full videos from four models on 100 shared prompts, with model identities and automatic scores hidden. We correlate Motion-D scores with mean human ratings, using 1,000 prompt-level bootstrap resamples for 95% confidence intervals.

AnalysisSpearman ρ95% CI
All videos0.45[0.30, 0.58]
Within pretrained-model videos0.35[0.14, 0.53]
Within A14B-teacher OPD videos0.40[0.20, 0.57]
Within aesthetic-SFT-teacher OPD videos0.33[0.12, 0.51]
Within MILD videos0.38[0.17, 0.56]
Within-prompt MILD−Base differences0.32[0.10, 0.51]

Motion-D shows a moderate positive correlation with human ratings overall, with positive correlations within each model and between matched-prompt MILD−Base score and rating differences. This supports alignment with perceived motion naturalness beyond differences between model averages.

The existing ablation in Table 5 shows that motion refinement improves the VBench-2.0 average by 1.21 points and Physics by 0.61 points, providing complementary evidence of training benefits. Together, these findings support Motion-D’s value as an auxiliary motion signal.

We have added the correlation results to Table 13 (p. 25), with the protocol and interpretation in Appendix C.1, “Alignment with human motion judgments” (p. 24, lines 1291–1295; p. 25, lines 1316–1326). All additions are highlighted in blue.

W3. Appendix C.1 allows caption overlap across video-file-level splits and uses native frame rates for flow extraction, leaving generalization to unseen caption groups and frame-rate effects untested.

Answer: Following the reviewer’s suggestion, we tested generalization to unseen caption/source groups and sensitivity to frame rate.

Group-disjoint evaluation and frame-rate control. We rebuilt the splits so that each real video, its caption-matched generated video, and temporal-corruption variants remain together; videos sharing a source or normalized caption are also grouped. We retrained Motion-D from scratch with the same architecture and training budget, selected the checkpoint on validation groups, and evaluated held-out test groups. Using the same grouped split, we additionally resampled all video classes to 12 fps according to encoded timestamps, constructed temporal corruptions after resampling, recomputed optical flow, and retrained.

SettingSplitTraining fpsEvaluation fpsAUROC ↑Paired accuracy ↑
Original protocolVideo-file-disjointNativeNative0.92880.9736
Group-disjointCaption/source-group-disjointNativeNative0.91500.9550
Group-disjoint + standardized fpsCaption/source-group-disjoint12120.90500.9450

Paired accuracy measures how often a real video scores above its temporally corrupted counterpart. Both new settings use the same held-out groups; the original is a reference. Standardizing fps reduces both metrics by 0.01, while retaining strong discrimination.

Sampling-rate sensitivity. We then fixed the 12-fps-trained discriminator and evaluated the same held-out groups at 8, 10, and 12 fps, recomputing optical flow without retraining.

Training fpsEvaluation fpsAUROC ↑Paired accuracy ↑
1280.8900.930
12100.8950.935
12120.9050.945

The largest decrease relative to 12-fps evaluation is 0.015 for both metrics. These experiments support generalization to unseen caption/source groups and limited sensitivity within the tested 8–12 fps range. Optical flow follows established motion-representation practice [1, 2]; the controlled experiments directly address the reviewer’s concerns.

These experiments appear in Appendix C.1, Tables 11–12 (pp. 24–25), with protocols and analysis on p. 24, lines 1269–1289. All additions are highlighted in blue.

W4. This paper has not reported the user study results.

Answer: Following the reviewer’s suggestion, we conducted a human evaluation with five CS PhD students, who independently watched 400 full videos on 100 shared prompts from pretrained Wan2.2-5B, V2V OPD with the A14B teacher, V2V OPD with the aesthetic-SFT teacher, and MILD. Videos had anonymous labels, with model identities and automatic scores hidden. Participants rated visual quality, motion naturalness, prompt alignment, and overall quality on a 1–5 scale, and made pairwise comparisons on visual quality, motion naturalness, and overall quality.

Human ratings.

MethodVisual Quality ↑Motion Naturalness ↑Prompt Alignment ↑Overall Quality ↑
Pretrained model3.793.233.023.16
V2V OPD (A14B teacher)3.653.203.053.18
V2V OPD (aesthetic-SFT teacher)3.883.253.103.28
MILD4.053.403.303.55

Direct pairwise preferences. Each comparison contains 500 judgments per dimension. Preference scores equal the win rate plus half the tie rate, with 95% confidence intervals computed by prompt-level bootstrap resampling.

BaselineDimensionMILD winsTiesMILD losesPreference score [95% CI]
Pretrained modelVisual quality50%30%20%65.0% [58.0, 72.0]
Pretrained modelMotion naturalness40%35%25%57.5% [51.0, 64.0]
Pretrained modelOverall quality50%25%25%62.5% [55.0, 70.0]
V2V OPD (A14B teacher)Visual quality60%25%15%72.5% [66.0, 79.0]
V2V OPD (A14B teacher)Motion naturalness45%35%20%62.5% [56.0, 69.0]
V2V OPD (A14B teacher)Overall quality58%25%17%70.5% [64.0, 77.0]
V2V OPD (aesthetic-SFT teacher)Visual quality40%40%20%60.0% [53.0, 67.0]
V2V OPD (aesthetic-SFT teacher)Motion naturalness35%45%20%57.5% [51.0, 64.0]
V2V OPD (aesthetic-SFT teacher)Overall quality43%35%22%60.5% [53.0, 68.0]

MILD achieves the highest mean ratings across all four dimensions, exceeding aesthetic-SFT-teacher OPD, the strongest baseline, by 0.17, 0.15, 0.20, and 0.27, respectively. All pairwise preference scores and their 95% confidence intervals exceed 50%, supporting perceptible gains on the evaluated prompts.

Results appear in Tables 15–16 (p. 29), with protocols, evaluation criteria, and statistical analysis in Appendix C.5 (pp. 29–30, lines 1550–1582), and a summary in Section 4.2, “Human evaluation” (p. 8, lines 421–426). All additions are highlighted in blue.

Q2&W5. What are the individual contributions of the frozen video anchor, temporal filtering, and correction magnitude constraint, beyond image-expert supervision and motion refinement?

Answer: We separately ablate the frozen video anchor, temporal filtering, and correction magnitude constraint (magnitude clipping) on Wan2.2-5B. Each variant retains image-expert supervision, motion refinement, and all other components and training settings.

MethodCreativity ↑Commonsense ↑Controllability ↑Human Fidelity ↑Physics ↑AVG. ↑
w/o frozen video anchor48.4660.2418.4282.9153.2852.66
w/o correction magnitude constraint48.8459.3618.5682.4154.2652.69
w/o temporal filtering48.7861.0919.5283.4749.0852.39
MILD49.5062.8421.8084.8954.2954.66

Removing the video anchor reduces AVG by 2.00 points, most notably Controllability (−3.38). Removing the magnitude constraint reduces AVG by 1.97, with declines in Commonsense (−3.48), Controllability (−3.24), and Human Fidelity (−2.48), while Physics is nearly unchanged. Removing temporal filtering reduces AVG by 2.27, with the largest decline in Physics (−5.21). Every ablation scores lower across all five categories, supporting each component’s contribution beyond image supervision and motion refinement.

We have added these results to Table 5 (p. 9), with analyses in Section 4.4, under “Supervision and optimization strategy” (p. 10, lines 486–489) and “Temporal filtering and correction magnitude” (p. 10, lines 495–503). All additions are highlighted in blue.

W6. The temporal consis. in Tab 2 on the Wan2.2-5B is lower than the V2V OPD results. I am concerned that the temporal consistency may be influenced by the proposed method.

Answer: MILD’s temporal consistency is lower than A14B-teacher OPD, but higher than both the pretrained student and aesthetic-SFT-teacher OPD. Table 2 reports:

MethodTemporal Consistency ↑Motion Quality ↑
Pretrained student61.9354.00
V2V OPD (A14B teacher)63.1054.07
V2V OPD (aesthetic-SFT teacher)62.7354.11
MILD62.7954.43

MILD improves temporal consistency by 0.86 points over the pretrained student, while remaining 0.31 points below A14B-teacher OPD. Thus, this evaluation shows no temporal-consistency degradation relative to the pretrained student. Moreover, the image experts provide spatial supervision for aesthetics, composition, and text rendering, while the video anchor, temporal filtering, and motion refinement help preserve the student’s dynamic capabilities. The full framework achieves temporal consistency close to A14B-teacher OPD and higher motion quality (54.43 vs. 54.07). Our human evaluation further supports its motion naturalness (see Answer W4): MILD receives a higher mean rating (3.40 vs. 3.20) and a tie-adjusted motion preference score of 62.5% [56.0, 69.0] against A14B-teacher OPD.

Together, these results support competitive dynamic performance while transferring specialized image capabilities. MILD does not surpass the stronger video-teacher baseline on every temporal metric. Closing the remaining temporal-consistency gap while retaining spatial gains is a direction for future work.

Comments on the paper

1 comment

Dear Area Chair,

Thank you for your time and efforts in handling our paper.

Our paper introduces MILD, an on-policy distillation framework that transfers specialized image-generation capabilities to video diffusion models. Reviewers recognize the practical value of this research direction (Reviewers zG7N and ApLu), the clear motivation and theoretically grounded approach (Reviewer 2QUy), and the consistent improvements across different video backbones (Reviewers zG7N and ApLu). They also appreciate the reusable interface for cross-model supervision (Reviewer ApLu), the clear presentation (Reviewer 2QUy), and the detailed derivations supporting reproducibility (Reviewer zG7N).

The reviewers’ main concerns center on the fairness of the comparisons, the validity of cross-latent-space transfer, and the individual contributions of the connector and motion-preservation components. They also requested further evidence on motion quality, generalization, and training efficiency. In response, we provided additional baseline comparisons and component ablations, clarified the connector design, and conducted diagnostics of mapped noise statistics and teacher predictions across diffusion times. We further added a human evaluation, controlled tests of the motion discriminator’s generalization and frame-rate sensitivity, and a detailed training-cost breakdown.

We have updated the manuscript with the additional experiments and clarifications, highlighting the revisions in blue. Reviewer 2QUy found our rebuttal convincing and maintained their positive recommendation, highlighting the value of transferring specialized image capabilities to video models without specialized video teachers and the additional evidence supporting our framework’s effectiveness. However, we have not yet received follow-up responses from Reviewers zG7N and ApLu. As the discussion period is nearing its end, we would greatly appreciate your help in encouraging further discussion so that we can clarify any remaining questions.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.