acceptodds
Under review as a conference paper at ICLR 2027

Recurrent Second-Order Learning for Efficient Video Super-Resolution

Abstract

Video super-resolution (VSR) seeks to reconstruct high-resolution frames from low-resolution inputs. While diffusion-based methods have substantially improved perceptual quality, extending them to video remains challenging for two reasons: strong generative priors can introduce temporal instability, and multi-frame diffusion pipelines are often too expensive for practical deployment. To address both challenges simultaneously, we propose InstaVSR, a lightweight diffusion framework for efficient video super-resolution. InstaVSR combines three ingredients: (1) a pruned one-step diffusion backbone that removes several costly components from conventional diffusion-based VSR pipelines, (2) recurrent training with flow-guided temporal regularization to improve frame-to-frame stability, and (3) dual-space adversarial learning in latent and pixel spaces to preserve perceptual quality after backbone simplification. On an NVIDIA RTX 4090, InstaVSR processes a 30-frame video at 2K2K resolution in under one minute with only 7 GB of memory usage, substantially reducing the computational cost compared to existing diffusion-based methods while maintaining favorable perceptual quality with significantly smoother temporal transitions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video super-resolution

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8tA1OpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a video super-resolution method based on a pruned one-step diffusion model, which incorporates dual-space adversarial learning and a recurrent second-order learning scheme that progressively utilizes first-order SR predictions. The problem is relevant, but I find the contribution and supporting evidence insufficient. Compared to the baseline, the proposed method yields no performance improvement.

Critical strengths
  1. The paper is clearly written, providing sound reasoning and comprehensive experiments.
  2. By integrating model pruning, dual-space adversarial learning in latent and pixel spaces, and recurrent second-order learning, the authors achieve moderate performance for one-step diffusion-based VSR.
Critical weaknesses
  1. Most of the adopted modules are not novel and are already standard practices in video enhancement, which limits the overall technical contribution of this paper. For instance, the strategy of using predicted SR results for subsequent reconstruction has already been adopted in 'AnimeSR', while generative adversarial learning is also commonly used in low-level vision tasks.
  2. The paper lacks sufficient details on how the pruned student model is derived, which severely hinders reproducibility. The authors must clarify the pruning strategy with corresponding ablation studies.
  3. As reported in Table 1, the performance improvements are generally insignificant compared to existing methods, especially on the YouHQ and UDM10 datasets, which undermines the claimed effectiveness of the proposed modules.
  4. Table 3 only presents the quantitative performance of the ablation studies. The authors need to provide the corresponding qualitative visualizations for each ablated variant to better understand the role of each module.
  5. There is a lack of ablation studies regarding the proposed pruning mechanism. While Section 3.4 merely mentions the adoption of the pixel unshuffle module, it fails to investigate how structured pruning affects the overall performance.
Questions for authors

For detailed concerns, please refer to the Weaknesses section.

Reviewer_Bk3jOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes InstaVSR, an efficient diffusion-based video super-resolution method. It combines a lightweight, pruned one-step generator with optical-flow-aligned multi-frame inputs. The proposed recurrent second-order learning strategy writes reconstructed frames back into a temporal buffer and revisits an earlier frame using the updated context. Flow-guided temporal regularization encourages consistency between neighboring predictions, while dual-space adversarial supervision in latent and pixel spaces preserves perceptual quality. Experimental results demonstrate substantially reduced inference time and memory consumption compared with the evaluated methods, while maintaining competitive perceptual quality.

Critical strengths
  1. The method combines a lightweight one-step generator, flow-aligned multi-frame inputs, second-order learning, and dual-space adversarial supervision. The design considers the characteristics of the VSR task and focuses on computational efficiency.
  2. Under the reported evaluation protocol, InstaVSR demonstrates clear advantages in memory consumption and inference speed over the methods compared in Table 2.
Critical weaknesses
  1. (a) Quantitative evaluation is limited to synthetic-degradation benchmarks. Results on commonly used real-world datasets, such as VideoLQ and MVSR4×, are not reported, leaving the method’s generalization to real-world degradations insufficiently validated. (b) The quantitative results do not demonstrate a clear performance advantage over competing methods. Apart from MANIQA and DOVER on SPMCS, the proposed method falls behind state-of-the-art methods on multiple metrics across the evaluated datasets.
  2. The paper introduces second-order learning but does not specify whether second-order reconstruction is also performed during inference. Using it would introduce additional inference overhead, while omitting it would create a training–inference mismatch. The authors should clarify this choice and discuss its implications.
Questions for authors
  1. How does InstaVSR perform on real-world datasets such as VideoLQ and MVSR4×?
  2. Why does removing recurrent learning, temporal loss, or flow alignment improve Ewarp, even though these components are intended to enhance temporal consistency?
  3. How does the proposed method compare with competing methods in terms of fidelity metrics PSNR and SSIM?
  4. Please discuss the implications of second-order learning for inference.
Reviewer_xuLzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents an efficient framework for real-world video super-resolution. It adapts a pretrained diffusion-based image restoration model into a compact, single-step network that reconstructs each high-resolution frame from optical-flow-aligned neighboring low-resolution frames. To reduce computational cost, the architecture replaces the VAE encoder with pixel unshuffle and removes text cross-attention and timestep embeddings. Its main training contribution is recurrent second-order learning: restored frames are written back into an observation buffer and reused for another reconstruction, enabling temporal consistency supervision across reconstruction orders. The framework also combines latent-space and pixel-space adversarial learning to enhance perceptual quality. Experiments on SPMCS, YouHQ, and UDM10 demonstrate competitive perceptual quality with reduced runtime and memory consumption.

Critical strengths
  1. The lightweight backbone, combined with prediction feedback and second-order reconstruction, achieves competitive perceptual quality at a low computational cost. Reusing reconstructed frames provides a practical way to enrich the input context of the pruned network.

  2. Second-order reconstruction explicitly supervises predictions produced under contexts containing the model’s own outputs. This is a useful design for encouraging consistency across different reconstruction states.

  3. The method offers lower inference latency and memory consumption than the one-step baselines included in the comparison.

Critical weaknesses
  1. The technical contribution beyond AdcSR appears incremental, primarily involving multi-frame input adaptation, prediction feedback, and an additional reconstruction pass.

  2. Given the emphasis on efficiency, comparisons with recent efficient VSR methods, such as FlashVSR, would strengthen the evaluation and clarify the method’s position.

  3. The evaluation relies on synthetically degraded videos. Results on real-world degradation benchmarks such as VideoLQ are needed to substantiate the method’s practical effectiveness.

Questions for authors
  1. More discussion about the technical novelty is welcomed.

2.The comparisons with recent efficient VSR methods on real-world degradation benchmarks are needed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.