acceptodds
Withdrawn from ICLR 2027

NoiseFlow: Warp-Guided Video Generation with Distilled Temporal Priors

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Temporal coherence in video diffusion models is often imposed using motion correspondences estimated in video space, implicitly assuming that pixel-space motion defines an appropriate transport in latent noise space. We challenge this assumption and introduce NoiseFlow, a framework that learns the temporal noise structure directly from a pretrained video diffusion model. NoiseFlow distills the denoiser-induced local geometry into diffusion-timestep-conditioned autoregressive warp fields that transport Gaussian noise across video frames. We then train a student diffusion model on the resulting family of warped noise priors. During inference, the warp is inferred from the evolving denoising trajectory, and the student provides warped-trajectory guidance to the base model, enabling video generation without a reference video or externally estimated optical flow. Experiments show that NoiseFlow reduces FVD by up to 18%, improves SSIM by up to 43%, and PSNR by up to 19%, and improves several metrics of VBench and VBench2 compared to the unguided base model. These results suggest that denoiser-native temporal priors provide a useful alternative to prescribing video-space correspondences in generative noise space.\footnote{Models and code will be released.

Primary area:
generative models
Keywords:
video generation, diffusion, generative models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_39TDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes NoiseFlow, a video generation framework that learns diffusion-timestep-dependent temporal noise correlations from a frozen pretrained denoiser. A warp network is trained using teacher-derived sensitivity signals, and a stochastic transport process constructs correlated noise while preserving standard-Gaussian frame marginals. A student denoiser is then fine-tuned on these noise priors and used to guide the base model during inference, without requiring a reference video or external optical flow. Experiments based on CogVideoX-2B demonstrate improvements in FVD and several temporal metrics, alongside trade-offs in dynamic content and instance preservation.

Critical strengths
  1. An interesting approach to learning temporal noise structure. Learning noise-space transport from the denoiser provides a meaningful alternative to prescribing correspondences from pixel-space optical flow. The formulation directly connects the temporal prior to the generative model and supports generation without reference videos at inference.
  2. A statistically motivated construction with measurable empirical benefits. The transport process is supported by an analysis of Gaussian marginal preservation, addressing an important issue when warping noise. The full method also improves FVD over both the original backbone and the no-warp self-guidance baseline, suggesting practical value beyond ordinary fine-tuning or the simplest self-guidance configuration.
Critical weaknesses
  1. The evaluation is limited to CogVideoX-2B. While the authors acknowledge the model-specific nature of the learned prior, further evidence would be useful to establish whether the framework provides consistent benefits when trained on other backbones.
Questions for authors
  1. Is student adaptation necessary? Could the authors evaluate retaining the learned warp while using the frozen teacher, rather than the trained student, for the second denoising prediction under the same guidance rule? The existing self-guidance ablation uses i.i.d. noise and therefore does not test this configuration. This comparison would clarify whether student training is essential to benefiting from the learned noise prior.
  2. What motivates the warp supervision design? Could the authors explain the intuition behind the magnitude target in Eq. (15)? I am also curious whether directly regressing a complete warp field, using a calibrated target derived from the same teacher signals, would perform similarly to separately supervising shape and magnitude.
Reviewer_zqRhOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes NoiseFlow, which learns a diffusion-time-dependent noise transport field from the local sensitivity of a pretrained video denoiser. The field introduces cross-frame dependencies while preserving a standard Gaussian marginal for each frame. The authors then train a student model adapted to this noise structure and use orthogonal trajectory guidance to improve text-to-video generation. Experiments on CogVideoX-2B show improvements in FVD and several temporal-consistency metrics, although stronger guidance also reduces dynamic degree and instance preservation.

Critical strengths
  1. A valuable perspective. Treating cross-frame structure as a learnable noise prior, rather than borrowing pixel-level optical flow, is a meaningful direction for video generation.
  2. A relatively complete framework. Gaussian-marginal-preserving stochastic transport, field learning, and student guidance together form a coherent pipeline, and inference requires neither a reference video nor external optical flow.
  3. Reasonable empirical support. The method outperforms the base model on several metrics, and at low guidance strength it improves motion smoothness and dynamic degree at the same time.
Critical weaknesses
  1. The paper argues that cross-frame structure has to be built by the denoising network during generation. I do not find this convincing. In earlier work such as Go-with-the-Flow, temporal correlations across frames are injected directly through warped noise, and this noise-level structure already gives the model a very strong temporal signal. The denoising or velocity-field learning step mainly fits the data distribution; it is not where temporal coherence originates. The authors should either revise this claim or give evidence, for example a controlled comparison, that the network contributes cross-frame structure beyond what the structured noise already supplies.

  2. The algorithm for transporting noise while keeping its Gaussian marginals does not appear to add much conceptually over How I Warp Your Noise and Go-with-the-Flow. All three methods transport noise in essentially the same way. The authors should state clearly what is technically new compared with these methods and show, with direct experiments against them, that the difference matters in practice.

  3. The ablations do not fully separate what the student training contributes from what the noise structure contributes. They show that simply calling the base model more often does not match the full method. They do not rule out that the gains come from the extra training of the student model.

  4. For transport under a fixed field, the construction in the main text has a clear justification for preserving Gaussianity. In practice, however, the field is predicted from the noisy input. If the field depends on the same noise it transports, the field is no longer independent of the noise, and the proof for a fixed operator does not carry over directly. The authors should extend the analysis to this setting or show empirically that the transported noise remains approximately Gaussian, for example through statistical tests on its marginals and correlations.

  5. The results in the supplementary material show limited visual quality compared with existing video generation models.

Questions for authors

Please refer to the weaknesses.

Reviewer_AhKrOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

NoiseFlow learns temporally correlated noise priors from a pretrained video denoiser. A warp generator is trained using a denoiser-pullback objective, and its transport-and-innovation construction preserves standard-Gaussian frame marginals under the stated assumptions. A student denoiser is then trained on the resulting teacher-timestep-dependent paths. At inference, the student supplies a correction projected orthogonally to the base model's text-CFG direction, enabling generation without an external reference video or motion field. Experiments on CogVideoX-2B report improvements in several distributional and temporal metrics, with a guidance-dependent tradeoff.

Critical strengths
  1. The method presents a technically interesting connection between denoiser geometry and temporal noise structure. Learning transport from local denoiser sensitivity, conditioning the prior on diffusion time, and coupling transport with stochastic innovation form a coherent approach. The appendix appropriately distinguishes the idealized score-based motivation for guidance from the implemented inference procedure.

  2. The same-backbone results provide meaningful evidence of effectiveness. At guidance strength 0.5, FVD decreases from 345.618 to 283.307, approximately 18%, alongside improvements in several temporal metrics (Table 2). The no-warp self-guidance and conventional fine-tuning controls are useful, and the paper acknowledges the tradeoff in dynamics and instance preservation.

Critical weaknesses
  1. Go-with-the-Flow and EquiVDM provide precedents for distribution-preserving noise warping and training video denoisers with warped noise. APG also uses projected guidance, although with different vectors and a different reference direction. The contribution needing clearer evidence is the denoiser-pullback objective and its diffusion-time dependence. The no-warp control changes student training and conditioning, while conventional fine-tuning does not match the full pipeline. These controls cannot establish the advantage of the learned transport over a simpler prior under matched conditions.

  2. Appendix F reports 65 seconds per video versus 32 seconds for base CogVideoX on a B200. At guidance 0.8, Dynamic Degree falls from 0.611 to 0.533 and Instance Preservation from 0.825 to 0.790. Given this cost and tradeoff, comparisons against the default base sampler alone do not establish a practical advantage over other uses of the extra computation.

Questions for authors
  1. With student architecture, training budget, and guidance fixed, how much do pullback learning and diffusion-time conditioning each contribute? Compare the learned prior with a simple correlated Gaussian prior or a field estimated from the base model's own clean prediction. Please also explain how the projected correction differs from APG.

  2. At comparable wall-clock cost, how does NoiseFlow compare with a compatible reference-free method such as FreqPrior, FreeInit, or STG? Use the same backbone and evaluation protocol, and report FVD, text alignment, dynamics, instance preservation, and peak memory.

  3. For Table 2, please specify the sample count, reference-pairing protocol for PSNR/SSIM/LPIPS, and uncertainty of the main FVD and faithfulness differences. Figure 5's seed-variability analysis covers the guidance-scale experiment rather than this table.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.