- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MILD, an on-policy distillation framework that transfers capabilities from image diffusion model to video diffusion models. The method uses a learnable connector to align heterogeneous image and video latent spaces and pulls image-expert scores back into video coordinates through the connector’s Jacobian transpose. To control the effects of frame-level supervision on temporal generation, MILD anchors its corrections to a frozen video model, constrains their magnitude and inter-frame variation, and incorporates a motion reward from a separately trained optical-flow discriminator. The experiments show improved
Critical strengths
- The paper addresses a practically relevant problem and provides a reusable interface for supervision across models.
- The improvements are consistent across backbones, with meaningful evidence attributing gains to the proposed components.
Critical weaknesses
- The central assumptions underlying cross-latent-space transfer lack direct validation, particularly regarding the compatibility of noisy states.
The connector aligns clean latents using statically repeated video clips and adapts to student trajectories through velocity alignment. However, these objectives do not directly ensure that mapped states follow the noise distribution expected by the image teacher at the same diffusion time. For a linear connector, for example, where the noise covariance is , which generally differs from the identity covariance expected by the image teacher. Although the gradient pullback in Eq. (5) is mathematically valid, this alone does not establish that the teacher’s predictions at the mapped states provide reliable supervision. The paper does not report diagnostics of state compatibility or teacher predictions across noise levels. Consequently, the conditions under which this central interface works, and the limits of its generalization, remain unclear.
- The experiments do not sufficiently isolate the mechanism underlying the central motion-preservation claim.
Table 5 provides useful ablations of image supervision and motion refinement, but it does not isolate the contributions of temporal filtering and inter-frame correction bounds: the image-guided variants retain both the frozen video anchor and the temporal correction machinery. Consequently, the results do not establish whether these temporal constraints provide benefits beyond conservative anchoring, magnitude clipping, and motion refinement. A controlled comparison of spatial improvement and temporal quality is needed.
Questions for authors
- How are noisy states and diffusion times aligned?
How are latent scaling, noise covariance, and diffusion-time parameterization matched between the image and video models? Are any constraints imposed on or the noise statistics of mapped states? Could the authors provide diagnostics across diffusion times showing that the image teacher’s predictions at these states still reflect the intended visual content?
- Do the temporal correction constraints improve the trade-off between spatial quality and temporal quality beyond anchoring and magnitude control alone?
Could you compare full MILD against a variant that removes temporal filtering and inter-frame correction bounds while retaining other settings unchanged? This would clarify whether the proposed constraints specifically prevent temporal degradation from image supervision.
5 comments
Q2&W2. Do the temporal correction constraints improve the trade-off between spatial quality and temporal quality beyond anchoring and magnitude control alone? Could the authors compare full MILD against a variant that removes temporal filtering and inter-frame correction bounds while retaining other settings unchanged, to clarify whether these constraints specifically prevent temporal degradation from image supervision?
Answer: Following the reviewer’s suggestion, we jointly remove temporal filtering and inter-frame correction bounds on Wan2.2-5B, while retaining the frozen video anchor, global and per-frame magnitude constraints, image-expert supervision, and motion refinement. All other training and evaluation settings remain unchanged. We evaluate both overall video quality and the temporal metrics in EvalCrafter.
EvalCrafter results.
| Method | Visual Quality ↑ | Text–Video Alignment ↑ | Motion Quality ↑ | Temporal Consistency ↑ |
|---|---|---|---|---|
| Pretrained student | 61.48 | 55.63 | 54.00 | 61.93 |
| w/o temporal filtering and inter-frame bounds | 62.36 | 58.32 | 54.25 | 61.98 |
| MILD | 63.76 | 59.12 | 54.43 | 62.79 |
Compared with the ablated variant, full MILD improves Temporal Consistency by 0.81 and Motion Quality by 0.18, alongside gains of 1.40 in Visual Quality and 0.80 in Text–Video Alignment. These results support an additional benefit from temporal constraints beyond anchoring and magnitude control, without sacrificing visual quality or prompt alignment. The ablated variant remains slightly above the pretrained student on temporal consistency, so the evidence supports improved temporal quality rather than degradation below the pretrained baseline.
VBench-2.0 results.
| Method | Creativity ↑ | Commonsense ↑ | Controllability ↑ | Human Fidelity ↑ | Physics ↑ | AVG. ↑ |
|---|---|---|---|---|---|---|
| w/o temporal filtering and inter-frame bounds | 46.44 | 60.49 | 17.29 | 82.63 | 49.63 | 51.30 |
| MILD | 49.50 | 62.84 | 21.80 | 84.89 | 54.29 | 54.66 |
Full MILD also improves the VBench-2.0 average by 3.36 points, with gains across all five categories, providing complementary evidence of improved overall video quality.
Magnitude control and temporal constraints address different properties of the correction. For example, adjacent frames can receive corrections of equal, bounded magnitude but opposite directions. Both satisfy the magnitude constraint, yet their difference can still be large. Temporal filtering and inter-frame bounds explicitly regulate this variation. The joint ablation supports the value of this temporal control within the full training framework.
The VBench-2.0 ablation is reported in Table 5 (p. 9), with its setting and analysis in Section 4.4, “Temporal filtering and correction magnitude” (p. 10, lines 495–503).
Dear Reviewer,
We would like to kindly follow up on our rebuttal and ask whether our responses have addressed your concerns. We would greatly appreciate any further feedback or questions you may have.
Thank you for your time and consideration.
Best,
Authors
We thank the reviewer for the constructive feedback. We respond to each question below.
Q1&W1. How are latent scaling, noise covariance, and diffusion time matched across image and video models? What constraints are imposed on the connector or mapped noise, and do teacher predictions preserve visual content across diffusion times?
Answer: We evaluate the Wan2.1 video student with three SD3.5-Medium image experts specialized in GenEval, aesthetics, and OCR. The experts share a VAE and one video-to-image connector.
Latent scaling and diffusion time. The connector maps Wan-normalized video latents, zV=(zraw,V−μV)/σV, into SD3.5-Medium’s normalized image-latent space, zI=(zraw,I−0.0609)×1.5305. Wan normalization is applied once within its VAE interface. An eight-image audit confirms matching fresh and cached Wan encodings, with inverse-normalization error of approximately 10−7.
Under xt=(1−t)x0+tϵ, t=0 denotes clean data and t=1 noise (Appendix A.1). We query the image teacher at τI=1000t, using the student scheduler’s actual flow time, not the step index. Its sigma shift of 5 is already reflected in t. For the mapped state yt=Axt, the teacher’s clean prediction is
y^0=yt−tuICFG(yt,1000t,c),
where uICFG denotes its CFG-guided velocity prediction.
Connector constraints and noise covariance. The shared connector is bias-free and linear, comprising fixed latent-slice selection, bilinear resizing, and spatial convolutions. Training uses content alignment, velocity alignment, and regularization toward its near-identity initialization (Eq. 6; Appendix B), without explicit orthogonality or noise-whitening constraints.
For ϵV∼N(0,I), mapped noise has covariance AA⊤. We measure its deviation from isotropic covariance as Ecov=dout∥AA⊤−I∥F, where dout is the output dimension. For the evaluated Wan2.1 connector configuration, the mean over three trials with 32 independent Gaussian output probes each is 0.00547, indicating approximate isotropic-noise preservation by the tested operator under this metric.
Content correspondence across diffusion times. We use the original Wan2.1 checkpoint-250 student and trained connector without recalibration or parameter updates. We generate videos for 32 prompts with 50 sampling steps and student CFG 5, arranging them into 16 disjoint pairs.
For each pair, we query the image teacher with one video’s mapped donor state and the other’s recipient prompt. Both directions and three latent slices yield 96 comparisons per teacher and time. Teacher CFG is 4.5. Donor identification succeeds when the decoded teacher clean prediction has higher DINOv2 similarity to the corresponding donor final-video frame than to the recipient frame.
All three teachers retain identifiable donor content at low and intermediate noise despite the recipient prompt. At high noise, identification approaches the 50% reference level.
To assess the teacher-velocity contribution, let yD,yR be the mapped donor and recipient states at the same time, and cR the recipient prompt. We compare y^0state=yD−tuICFG(yD,1000t,cR) with a control using the recipient-state velocity:
=y _ {D}-t\,u _ {I}^{\mathrm{CFG}}(y _ {R},1000t,c _ {R}).$$ Both retain $y _ {D}$ and $c _ {R}$; only the velocity-query state changes. We report the paired improvement $\Delta m$ in the donor-minus-recipient DINO similarity margin: |Flow time $t$|GenEval $\Delta m$|Aesthetic $\Delta m$|OCR $\Delta m$| |-|-:|-:|-:| |0.241935|0.0953 [0.0632, 0.1247]|0.0971 [0.0681, 0.1224]|0.0959 [0.0657, 0.1238]| |0.487805|0.1119 [0.0770, 0.1564]|0.1212 [0.0886, 0.1580]|0.1219 [0.0820, 0.1676]| |0.797101|0.0912 [0.0608, 0.1197]|0.0693 [0.0271, 0.1130]|0.0625 [0.0327, 0.0957]| Both tables report 95% confidence intervals from 1,000 pair-level bootstrap resamples. Donor-state queries improve content correspondence over the mismatched-velocity control in all nine conditions, with every pointwise interval above zero. We additionally provide qualitative examples in Appendix C.8, showing frames from the final student videos alongside image-teacher clean predictions at low, intermediate, and high noise levels. These examples visualize content correspondence across diffusion times and complement the quantitative diagnostics above.