acceptodds
Under review as a conference paper at ICLR 2027

RelayVSR: Large-Small Model Collaboration for Efficient Real-World Video Super-Resolution

Abstract

Large generative models can recover realistic detail in real-world video super-resolution (VSR), but processing an entire video with them is computationally expensive. In this work, we present , a streaming VSR framework built on the mechanism. A large generative model generates reference latents for sparse keyframes, while a lightweight VSR network uses these references and low-resolution video to super-resolve every frame. The lightweight VSR network, implemented as a Dual-Memory Video Transformer, reuses keyframe information across frames and updates recent video context, supporting first-keyframe conditioning and dual-endpoint conditioning with bounded lookahead. However, errors in shared keyframes can propagate and accumulate across output frames, making keyframe quality alone an insufficient optimization target. We address this collaboration gap with Video-Aware Reference Optimization (VARO), which uses reinforcement learning to update the large generative model with two reward levels: a system-level reward evaluates videos produced by the fixed lightweight VSR network, while a reference-level reward evaluates decoded keyframe quality. VARO improves final video quality over direct joint training, and its dual-level rewards outperform a system-level reward alone. At 1080p on a single NVIDIA A100 80GB, dual-endpoint RelayVSR with a 15-frame keyframe interval reaches 29.29 FPS, 13.82 GB peak GPU memory, and 0.327 s first-frame model latency, compared with 7.80 FPS, 24.447 GB, and 2.83 s for FlashVSR-Tiny. Our code and video demo are available at https://bucket.anony.work/RelayVSR/code.zip and https://relayvsr.anony.work, respectively.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Real-world video super-resolution, Diffusion models, Reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_dfRNOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RelayVSR, a streaming video super-resolution framework based on large–small model collaboration. A large generative model restores sparse keyframes, while a lightweight Dual-Memory Video Transformer propagates their information to the remaining frames using persistent reference memory and rolling video memory. The authors further introduce Video-Aware Reference Optimization (VARO), which optimizes the large model using both reference-level and system-level rewards. Experiments demonstrate competitive restoration quality and improved inference efficiency over existing generative VSR methods.

Critical strengths
  1. Practical and interesting motivation. The paper explores sparse invocation of a large generative model combined with lightweight temporal propagation, offering a promising direction for efficient generative VSR.
  2. Comprehensive experimental validation. The paper includes comparisons with recent VSR methods, component ablations, quality-efficiency trade-offs, long-video evaluations, and a subjective user study. The provided video demonstrations further support the practical effectiveness of the proposed framework.
Critical weaknesses
  1. RelayVSR uses a fixed keyframe interval regardless of motion complexity or scene dynamics. Since videos may exhibit significantly different motion patterns, this fixed strategy may limit restoration quality and the quality-efficiency trade-off across diverse scenarios.
  2. VARO directly optimizes perceptual metrics such as MUSIQ and DOVER++, while the same or closely related metrics are used for final evaluation. This raises concerns about metric overfitting and potential evaluation bias.
Questions for authors

Can the authors provide more details about the approximately 0.5M videos and 1M images used for training (e.g., sources, preprocessing and procedures, quality, and public availability)? Given the substantial scale of the training data, clarifying its contribution to the reported performance and ensuring reproducibility would be valuable.

Reviewer_V5cKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RelayVSR, which establishes a large-small model collaboration mechanism for efficient video super-resolution. A large generative model is applied to sparse keyframes, and a lightweight VSR network uses the generated keyframe references to super-resolve all video frames, achieving higher efficiency than FlashVSR while maintaining strong performance.

Critical strengths
  1. The idea of using a large generative model to process keyframes and then efficiently processing the remaining frames with a lightweight VSR network is intuitive.
  2. The proposed method is highly efficient and demonstrates an advantage in throughput.
  3. Considering its high efficiency, the method achieves competitive performance.
Critical weaknesses
  1. Objective temporal consistency metrics, such as optical-flow warping error, should be reported on the evaluated datasets.
  2. The authors could provide more intuition on why the reference-level reward is necessary in addition to the system-level reward, which already evaluates the quality of the reconstructed video.
  3. The authors could provide more details about the sources of the training data to improve reproducibility.
  4. The results for DOVE in Table 11 could be further explained. RelayVSR performs worse than DOVE in terms of both content fidelity and temporal stability, yet achieves better overall quality. What other aspects, beyond content fidelity and temporal stability, contribute to the overall quality assessment?
Questions for authors

Why does VARO only update the large generative model while keeping the lightweight VSR network fixed?

Reviewer_ApS9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a large–small model collaboration framework for real-world video super-resolution. A large generative model produces high-quality reference latents only at sparse keyframes, while a lightweight Dual-Memory Video Transformer performs frame-by-frame restoration using persistent reference memory and rolling video memory. The authors further introduce Video-Aware Reference Optimization (VARO), which freezes the small model and optimizes the large model using reference-level and final video-level rewards, making the generated references more useful for downstream reconstruction. The system supports a first-keyframe mode without future inputs and a dual-endpoint mode with bounded lookahead.

Critical strengths
  1. The system design is well motivated, and the efficiency gains are practically meaningful. Restricting expensive generation to sparse keyframes and reusing reference features provides a clear mechanism for reducing computation. Table 2 reports that, at 1080p on a single A100 80GB GPU with a keyframe interval of 15, the dual-endpoint mode achieves 29.29 FPS with 13.82 GB peak GPU memory, compared with 7.80 FPS and 24.45 GB for FlashVSR-Tiny and 23.48 FPS and 31.36 GB for SwiftVR. The 3.76× throughput improvement over FlashVSR is substantial.

  2. Both reconstruction quality and the architectural choices receive experimental support. RelayVSR outperforms FlashVSR in PSNR, SSIM, and LPIPS on three paired benchmarks. The ablations in Table 3 separately remove reference memory and video memory, supporting their complementary roles: reference information primarily improves detail quality, while video memory further improves video quality and motion smoothness.

  3. The evaluation includes long videos and human judgments. LongVSR60 contains 60 videos of approximately 1,000 frames each. The appendix also presents a blinded paired-comparison study involving 12 participants and 36 videos. Relative to FlashVSR, the reported GSB scores are −3.3% for overall quality, +9.0% for fidelity, and +17.0% for temporal stability. These results support the efficiency advantage while maintaining comparable overall perceptual quality. The authors also explicitly disclose the scope of model timing and the lookahead used in the dual-endpoint mode, making the evaluation conditions reasonably clear.

Critical weaknesses

W1. The central claim that VARO outperforms direct joint training lacks a corresponding experiment The abstract and introduction explicitly claim that VARO outperforms direct joint training. The introduction also states that adapting the small model to generated references remains insufficient to prevent error propagation. However, Table 4 only compares four variants: no post-training, reference-level rewards, system-level rewards, and both rewards. All of these variants freeze the small model. Neither the main paper nor the appendix provides the corresponding joint-training or small-model adaptation results. The current evidence therefore supports the effectiveness of combining rewards within the chosen post-training framework, but does not establish the necessity of VARO relative to more direct training alternatives. This issue directly concerns the central contribution of addressing the “collaboration gap.” The authors should provide the relevant comparisons and specify which parameters are updated, which losses are used, and whether the training data and computational budgets are matched. If these experiments are unavailable, claims that go beyond the evidence should be revised. W2. VARO’s incremental benefits in independent perceptual quality and fidelity have not been isolated VARO uses rewards based on MUSIQ, the DOVER++ technical branch, RGB L1, and optical-flow warping, while the main evaluation metrics in Table 4 are MUSIQ, DOVER, and motion smoothness. MUSIQ directly overlaps with the training objective, and DOVER is related to the reward model, although the scores are not identical. The VARO ablations do not report PSNR, SSIM, or LPIPS, nor do they include independent human comparisons before and after VARO or across the reward variants. The user study comparing the final system with external baselines does not isolate VARO’s incremental contribution. It therefore remains unclear whether the improvements in quality scores consistently translate into better content fidelity and independently assessed perceptual quality. I recommend reporting fidelity metrics and independent human evaluations before and after VARO, with particular attention to keyframes, non-keyframes, and reference-update boundaries. This is a gap in evaluation independence; the current results are insufficient to conclude that reward hacking has occurred. W3. The data composition and operating limits lack sufficient detail for verification The paper reports training on approximately 500,000 videos and one million images, but does not sufficiently describe their sources, filtering criteria, splits, or procedures for removing overlap between training and test data. LongVSR60 likewise lacks a verifiable list of sources and sample-construction details. These omissions affect the assessment of generalization and reproducibility. In addition, the long-video evaluation mainly involves single-shot videos, whereas direct risks of sparse references include large motion, occlusion, erroneous references, and shot changes. I recommend reporting performance by scenario or providing representative failure cases. This does not require the authors to develop an additional adaptive keyframe algorithm, but the paper should clarify when the fixed-interval approach fails.

Questions for authors
  1. Which experiment supports the abstract’s claim that VARO outperforms direct joint training? Which parameters are updated in joint training, and what losses and training budget are used?
  2. How does VARO compare with adapting the small model to generated references, or freezing the small model and directly backpropagating video L1/LPIPS losses?
  3. How do PSNR/LPIPS, independent human preferences, and errors at reference-update boundaries change before and after VARO and under the two single-reward variants?
  4. What are the sources, splits, and deduplication procedures for the training data and LongVSR60? What are the typical failure modes under large motion and erroneous references?

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We are delighted that all three reviewers have given positive ratings. We sincerely thank all reviewers for their thoughtful and constructive feedback. We are encouraged by their recognition of RelayVSR’s practical efficiency, competitive video quality, and comprehensive evaluation.

In our revised manuscript, we have incorporated additional experiments, analyses, and implementation details to further strengthen our work. Below, we summarize our core contributions and the main updates included in the revision.


Core Contributions of Our Work

  1. Efficient Large–Small Model Collaboration: RelayVSR applies a large generative model to sparse keyframes and uses a lightweight VSR network with persistent reference memory and rolling video memory to efficiently super-resolve every frame.
  2. Video-Aware Reference Optimization: VARO combines reference-level and system-level rewards to optimize generated references for downstream video quality, improving perceptual quality over direct joint training and either reward level alone.
  3. Strong Quality–Efficiency Trade-off: Extensive experiments demonstrate competitive video quality, including on long videos. At 1080p on a single A100 GPU, dual-endpoint RelayVSR achieves 29.29 FPS with 13.82 GB peak GPU memory, providing 3.76× the model throughput of FlashVSR-Tiny.

Updates in the Revised Manuscript

  • Main Text, Table 4: Extended the VARO ablation with PSNR and reward-independent perceptual metrics: LPIPS, CLIP-IQA, and MANIQA.
  • Appendix C.1, Table 8: Added comparisons with small-model adaptation, direct video-loss backpropagation, and direct joint training, including their objectives and training settings.
  • Appendix C.6, Table 13: Added a blinded human study comparing full VARO with pre-VARO and both single-reward variants on overall quality, content fidelity, and reference-update continuity.
  • Appendix B.1 and Main Text, Sec. 4.1: Detailed training data sources and selection procedures, and added a pointer to LongVSR60 source information in the supplementary material.

We believe these additions and clarifications address the reviewers’ concerns and further strengthen our manuscript. Revisions are highlighted in dark-red text for ease of reference.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.