acceptodds
Under review as a conference paper at ICLR 2027

Revisiting Score Distillation for Video Generation: A Temporal-Frequency Perspective

Abstract

Score distillation is a leading paradigm for compressing a many-step video diffusion or flow teacher into a few-step student. Existing practice is organized around two divergences that expose a semantics-dynamics trade-off: reverse KL divergence anchors semantics but attenuates dynamics, whereas Fisher divergence retains dynamics but shifts semantics. In this work, we show the trade-off is intrinsic to the divergences, for they respond differently to two errors in mode selection and variance retention: Fisher charges the former weakly, and reverse KL the latter. The two errors, however, separate along the temporal-frequency axis: the former is settled in the lower band that carries global semantics, while the latter is recorded in the upper band that carries local dynamics. These findings dictate the design of Temporal-Frequency Routed Distillation (TFRD), which routes the lower band through reverse KL and the upper band through Fisher, and uses a gradient projector to keep the two bands' updates spectrally disjoint. Applied to the 4-step distillation of Wan2.1-T2V-1.3B, TFRD consistently improves the total, quality, semantic, and dynamic scores on VBench over prior methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
score distillation, video generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YadoOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates the semantics–dynamics trade-off in video score distillation through temporal-frequency decomposition. Analyses of mode-mass drift and variance attenuation motivate TFRD, which assigns DMD-style updates to lower frequencies and SGMD-style updates to higher frequencies. Bandwise normalization and a post-Jacobian projection implement this division while preserving full-context forward computation. Experiments study four-step distillation of Wan2.1-T2V-1.3B against the corresponding full-band baselines.

Critical strengths

1. A meaningful conceptual contribution with a clear algorithmic consequence. The principal novelty is using the complementary sensitivities of the two objectives to determine where each should act. The mixture and variance analyses provide a substantive rationale for this assignment. Importantly, moving the cutoff produces the predicted semantic–dynamic trade-off, giving the perspective empirical content rather than leaving it as an illustrative explanation. This combination of insight and actionable design is my strongest reason for acceptance.

2. An elegant realization with demonstrated practical value. The two-sided projection addresses a genuine complication: a high-frequency discrepancy need not produce a high-frequency gradient after the fake-model Jacobian. TFRD resolves this without changing inference. The projector ablation supports the design, while the blinded preference study provides evidence that the improvements matter perceptually, beyond aggregate benchmark scores.

Critical weaknesses

The respective contributions of routing and adaptive scaling are not yet clearly distinguished.

TFRD changes both frequency assignment and sample-dependent normalization. Sweeping a constant α\alpha does not isolate the latter, so the conclusion that the gains arise from where the objectives act rather than their effective strength is premature. This matters for identifying the method’s transferable design principle.

Relatedly, Section 5.2 argues that interpolating the objectives would place performance between the two baseline endpoints. This does not follow: nonlinear optimization and evaluation need not preserve such an ordering. A direct full-band combination is the relevant comparator for that particular argument.

Questions for authors
  1. How much does independent band normalization contribute when routing is held fixed? A comparison using a shared full-band RMS scale would help distinguish frequency assignment from residual-dependent reweighting.

  2. Have the authors evaluated a full-band combination of the same DMD-style and SGMD-style updates with comparable normalization? Its behavior would materially inform the assessment of selective routing relative to ordinary objective combination.

Reviewer_UMQBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the loss of semantic alignment and motion diversity when distilling a video diffusion model into a few-step generator. It argues that two existing objectives have complementary weaknesses: reverse KL distillation preserves prompt-relevant modes but tends to suppress temporal variation, while Fisher divergence better preserves dynamics but can shift probability mass between semantic modes.

The proposed Temporal-Frequency Routed Distillation (TFRD) applies a temporal discrete cosine transform to split video latents into low-frequency and high-frequency bands. It uses a DMD-style reverse KL objective for the low-frequency band, which carries global semantics, and an SGMD-style Fisher objective for the high-frequency band, which carries local motion. A gradient projector keeps the high-frequency update within its assigned band. Distilling Wan2.1-T2V-1.3B from 100 teacher network evaluations to a four-step student, TFRD reports higher VBench total, quality, semantic, and dynamic scores than the DMD and SGMD baselines.

Critical strengths

TFRD delivers a substantial improvement in motion quality while maintaining strong performance on the other VBench metrics. Its dynamic score rises to 81.94, compared with 69.44 for SGMD and 52.78 for DMD, while it also achieves the highest semantic, quality, and total scores among the three methods. This is compelling evidence that the proposed frequency routing helps address the usual trade-off between preserving motion and maintaining prompt alignment.

Critical weaknesses
  1. Limited evidence of generalization. The experiments use only Wan2.1-T2V-1.3B in a text-to-video setting. Results on a different model scale or on image-to-video and autoregressive generation would strengthen the claim that the temporal-frequency split applies broadly. The paper should also examine whether its band cutoff needs to change across models and tasks.
  2. The practical baseline could be stronger. The reported DMD baseline uses a multi-step DMD2 implementation without its adversarial term. A comparison with the full DMD2 method, including its discriminator loss, would better establish TFRD’s advantage over a stronger practical baseline. The current comparison is useful for isolating the distillation objective, but does not resolve that question. From my knowledge, the adversarial loss is important for the performance of the DMD2.
  3. Comparison with recent diversity-preserving methods is missing. DP-DMD[1], DFD[2] and PDD[3] also address mode collapse in few-step distillation. Comparing against these methods would help clarify TFRD’s contribution relative to recent alternatives.

[1] Diversity-Preserved Distribution Matching Distillation for Fast Visual Synthesis

[2] Data-Forcing Distillation: Restoring Diversity and Fidelity in Few-Step Video Generation

[3] Parallel Decoding Distillation for Fast Image and Video Generation

Questions for authors
  1. Can you evaluate TFRD on additional video models and generation settings, such as image-to-video or autoregressive generation? How sensitive is performance to the temporal-frequency cutoff, and would the cutoff need to change across models or settings?
  2. Your DMD baseline uses a multi-step DMD2 implementation without its adversarial term. How does TFRD compare with the full DMD2 method, including the discriminator loss?
  3. Could you expand the discussion of recent alternatives such as DP-DMD and DFD? In particular, how does TFRD differ in its treatment of diversity loss, and how do its results compare with these methods where comparable video benchmarks are available?
Reviewer_LjSQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes temporal frequency routed distillation (TFRD) for distilling video diffusion models. The core hypothesis is that reverseKL goal better preserves global sementics while Fisher-based distillation better preserves motion. TFRD claims that the temporal frequency decomposition can combine the two strengths.

Critical strengths

The idea of Temporal frequency routing of reverse KL and fisher divergences is novel .

Critical weaknesses

1, The theretical analysis does not establish the behavior of the proposed training updates. It seems to provide a post-hoc motivation for the proposed method.

2, Missing ablation.

The baseline, a full-band combination of DMD and SGMD is not compared. It's hard to tell the contribution of "Temporal-Frequency Routed"

TFRD changes the normalization schemes without isolating its contribution.

TFRD does not ablate the effect of critic update ratio. Why 1 is adopted not 5, since researchers usually admits the contribution of higher critic update ratio proposed in DMD2.

3, The divergence analysis is not video-specific, but the paper does not distinguish general divergence behavior from effects specifically from temporal structure. Experiments on image generation tasks are required to expain the contribution of the two divergences.

4, The experiments is restricted on a 1.3B video model. It's not clear whether it scales and generalise to other base models like 14B wan and 33B H3 model.

Questions for authors

Please see the weakness

1 comment

The divergence analysis should be validated on both image and video generation tasks, because its implications extend beyond temporal effects.

DMD is widely used in industry because it is simple and effective across image, audio, and video generation. Higher scores on video benchmarks don't necessarily indicate an improvement over DMD, since the metrics can diverge from human assessment. SGMD, one of the baselines, illustrates this issue. It reports gains over DMD on some metrics, yet in my assessment, DMD produces much better results.

This discrepancy points to a flaw in the theoretical analysis and the flaw will be apparent in image generation. Without validation on image generation tasks, it's unclear whether the method is genuinely effective.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.