acceptodds
Withdrawn from ICLR 2027

Harnessing Video Diffusion Model as Prior and Supervision for Video Restoration

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Adversarial transfer has proven an effective way of exploiting pretrained diffusion priors for image restoration: the restoration model is initialized from a diffusion prior and optimized adversarially against a discriminator that is itself a strong pretrained image model, which yields single step restoration at high perceptual quality. Extending this recipe to video is not straightforward. A pretrained video diffusion model can initialize the restoration network, but pretrained image discriminators act frame by frame, and a comparably capable pretrained video discriminator is not readily available to judge spatial appearance and temporal dynamics jointly. We, therefore, use distribution matching with the score of a pretrained video generation model to supervise temporal dynamics without requiring a video discriminator. This also avoids VSR-specific diffusion pretraining, reducing training costs and complexity. For spatial supervision, we apply image adversarial training to one randomly sampled frame per clip, whose gradient largely supports the full-frame image adversarial update. Extensive experiments demonstrate that the proposed HYPVR achieves state-of-the-art spatial quality and temporal consistency with a single forward pass, making effective use of the prior for efficient video restoration.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Restoration, Real-World Video Super-Resolution, Diffusion Priors, Single-Step Inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_58TJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HYPVR, a single-step video restoration method initialized from a pretrained video diffusion model. The framework can be viewed as three complementary components: a video distribution matching (Video DM) loss that uses a general-purpose pretrained video model as the teacher to promote temporal consistency; a fidelity objective (MSE + LPIPS) that keeps the output faithful to the input content; and a 2D image adversarial loss, applied to one randomly sampled frame per clip, that improves per-frame texture and perceptual quality. Experiments on four synthetic benchmarks and the real-world VideoLQ dataset show that HYPVR achieves the best DOVER score on all five benchmarks with single-step inference.

Critical strengths
  1. Strong perceptual quality. HYPVR achieves the best DOVER on all five benchmarks, indicating strong overall perceptual video quality.

  2. Good quality–consistency balance. The method strikes a reasonable balance between enhancing generated detail and preserving temporal consistency. The ablations show that Video DM substantially reduces the temporal degradation introduced by per-frame adversarial training.

  3. Clear decomposition of objectives. Temporal consistency, fidelity, and texture quality are each assigned to a dedicated objective, which makes the design easy to understand and analyze.

Critical weaknesses

1. Limited novelty and missing discussion of DUO-VSR.

The overall framework is not particularly novel. Its core idea, jointly combining distribution matching with adversarial supervision to train a one-step VSR model from a pretrained video diffusion backbone, is very similar to DUO-VSR (Lv et al., CVPR 2026). There are essential differences, for example HYPVR's use of a general-purpose T2V teacher without VSR-specific pretraining and its single-stage training, but the paper neither cites nor discusses DUO-VSR. A clear comparison is needed to establish the paper's actual contribution.

2. The rationale for using a 2D discriminator is not well supported.

HYPVR argues that no sufficiently strong pretrained video discriminator is readily available, and therefore adopts a 2D image discriminator. I find this rationale insufficiently supported, both by the existing literature and by the paper's own evidence.

  • (a) Existing evidence. DUO-VSR shows that a discriminator with a strong video prior can be obtained simply by extracting intermediate features from the two video DiTs already used in distribution matching (the real-score and fake-score networks). Notably, both networks are already present in HYPVR's own training pipeline, so such a discriminator could be built at little additional cost. The paper does not discuss this alternative.

  • (b) The paper's own ablation. The only empirical support for preferring the 2D discriminator is Table 2(c), which compares the HYPIR image discriminator, built on pretrained 2D features, with a "video-latent 3D discriminator." However, the architecture and initialization of the 3D discriminator are not described, and the authors themselves attribute its inferior results to slower, less stable convergence within the 20K-step ablation schedule. The comparison therefore confounds 2D vs. 3D with pretrained vs. potentially weakly initialized features and with insufficient training length, and it does not establish that a video discriminator is inherently less effective.

Since this rationale is central to the paper's design, I consider the lack of a fair comparison an important omission.

3. No video results are provided to verify temporal quality.

This is fundamentally a video restoration work, yet I could not find any supplementary video results; the paper presents only still frames, temporal profiles, and quantitative metrics, which makes it difficult to judge how well temporal consistency is actually maintained. If I have overlooked supplementary material containing videos, I would appreciate it if the authors could point me to it.

Questions for authors
  1. Could the authors discuss the differences between HYPVR and DUO-VSR, both conceptually and empirically?

  2. Could the authors describe the architecture and initialization of the 3D discriminator in Table 2(c)? Does the gap persist with longer training, or with a 3D discriminator built on pretrained video features?

  3. Could the authors compare their 2D image discriminator with a DUO-VSR-style discriminator built on the features of the teacher and fake-score networks already in HYPVR? Clarifying whether such a video-prior discriminator is more effective, or whether the video prior is truly redundant and a 2D discriminator suffices, would substantially strengthen the paper.

  4. For this comparison, could the authors also report the computational cost of both options (e.g., training time per step and GPU memory)?

Reviewer_1FDjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HYPVR, a single-step video restoration framework initialized from a pretrained video diffusion model. The main idea is to use video distribution matching (Video DM) with a frozen video diffusion teacher to supervise the spatiotemporal distribution of restored videos, without training a video discriminator. To improve spatial quality efficiently, the authors apply image adversarial and fidelity supervision to one randomly sampled frame per clip, while Video DM operates on the full latent sequence.

Experiments on four synthetic benchmarks and the real-world VideoLQ dataset show strong perceptual quality and competitive temporal consistency. The paper also provides ablation studies, inference comparisons, and a human evaluation.

Critical strengths
  1. I think using a pretrained video diffusion model as a distributional supervisor is a reasonable alternative to explicit flow-based consistency losses or training a video discriminator. The comparison in Table 2(b) is useful: Video DM achieves a better perceptual-temporal trade-off than the evaluated deterministic temporal losses, which tend to oversmooth the output.

  2. The method achieves the highest DOVER scores across all five benchmarks, and the user study in Appendix F provides additional evidence of perceptual quality. I also appreciate the sampled-frame supervision analysis. Table 2(d) shows that supervising one frame reduces training time per step from 9.38s to 3.62s compared with 29 frames, with relatively small changes in the evaluated metrics.

Critical weaknesses
  1. The evidence for improved temporal consistency is not fully convincing. I think this is the main weakness. Although Video DM improves E-Warp when added to image adversarial training (5.40 to 3.98 in Table 2), the fidelity-only baseline already achieves a lower E-Warp of 3.34. In Table 1, HYPVR also does not achieve the best E-Warp on any of the four synthetic benchmarks. The improvements in DOVER are encouraging, but DOVER measures overall video quality rather than temporal consistency alone. I am therefore not fully convinced by the claim of state-of-the-art temporal consistency. More importantly, the current ablation does not isolate whether the benefit comes specifically from the temporal information in the video diffusion prior. I think a comparison with frame-wise distribution matching would help establish the contribution of modeling the joint video distribution.

  2. The improvement in perceptual quality comes with a noticeable loss of reconstruction fidelity. Appendix Table 3 shows that HYPVR performs worse than SeedVR2 on PSNR, SSIM, and LPIPS across all four synthetic benchmarks. For example, on POND25, HYPVR achieves 21.74 PSNR compared with 23.70 for SeedVR2, while its LPIPS is 0.303 compared with 0.266. A similar trade-off appears in Table 2(a), where adding perceptual and distributional supervision reduces PSNR and worsens LPIPS compared with fidelity-only training. I understand that generative restoration involves a perception-distortion trade-off. However, since restoration should also preserve the original video content, I think the paper should better discuss the extent to which the generated details remain faithful to the underlying scene. The current user study evaluates overall visual quality but does not directly assess temporal correctness or content preservation.

Questions for authors
  1. Have the authors compared full-clip Video DM against frame-wise distribution matching using an image diffusion teacher or another spatial-only distributional objective? I think this would help distinguish the benefit of video-specific temporal supervision from general perceptual regularization.

  2. Table 2 shows that the fidelity-only baseline has lower E-Warp than the full objective, while Table 1 shows that HYPVR does not lead E-Warp on the evaluated synthetic datasets. Could the authors clarify which experimental evidence most directly supports the claimed improvement in temporal consistency?

  3. HYPVR achieves strong perceptual scores but lower PSNR, SSIM, and LPIPS than SeedVR2. Have the authors evaluated whether the additional generated details preserve the original scene content, rather than introducing visually plausible but inaccurate textures?

Reviewer_LNezOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes HYPVR, a single-pass video restoration model built from a pretrained video diffusion model. Its contributions are mainly in how it uses and combines existing components: 1) Video DM applies DMD-style distribution matching to the full video latent sequence, which can effectively ensure the generation ability of video and maintain the temporal consistency at the same time. 2) Sample-frame spatial supervision applies image adversarial and fidelity losses to one randomly chosen frame per clip, thus enhancing spatial detail

For validation, they compare against video restoration baselines on four synthetic datasets and one real-world dataset, using perceptual, fidelity, video-quality, and temporal metrics. They also ablate the training objectives, temporal supervision, discriminator, noise range, and number of supervised frames. The results support a balance between quality, consistency and cost, though HYPVR does not win every individual fidelity or warping metric.

Critical strengths
  1. The authors conduct sufficient experiments to show the effectiveness of Video DM and sample-frame discrimination.
Critical weaknesses
  1. Although the use of Video DM and single-frame discrimination has achieved effective results, the techniques they employed are already common in the field of video super-resolution, lacking insight and novelty. For example, the Video DM closely follows the DMD loss introduced in the paper "One-step diffusion with distribution matching distillation", and there are already some works using this techniques, such as FlashVSR and InfVSR.
  2. The comparison with other methods is not convincing enough. HYPVR generally does not win the best or the second best on the referenced metrics such as PSNR
Questions for authors
  1. In Table 2(a), using both Image Adv. and Video DM causes the worst LPIPS, can you explain the reasons and clarify how they assess the trade-off among fidelity, perceptual quality, and temporal consistency?
  2. Table 2(d) shows a counterintuitive trend: MUSIQ decreases as the number of frames used for image adversarial supervision increases. This result seems conflicting with the motivation in Section 3.4 that sampled-frame gradients are aligned with the full-frame gradient. Could the authors clarify why full-frame supervision lowers MUSIQ?
  3. The evaluation would be more convincing if the authors reported results on more real-world datasets
Reviewer_xWm3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new technique for video restoration via utilizing a pre-trained video diffusion-based network as a prior provider.

Critical strengths
  • Interesting idea and good overall execution.

  • The writing is overall clear, although there are many details missing in Section 3.

Critical weaknesses
  • Inconclusive results. The numbers in the tables look ok but not impressive. Much more importantly, there are no video results we could look at unless I missed any supplementary files.

  • The method claims to be a general video restoration method and yet the space of degradations is limited.

  • There is a fundamental trade-off in such a distributional approach between spatial detail and temporal stability, as also pointed out in the paper. I want to see an in-depth analysis of what we are losing for each dimension as we traverse the trade-off space.

Questions for authors

Please see above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.