acceptodds
Withdrawn from ICLR 2027

ReCaVSR: One-Step Streaming Diffusion Video Super-Resolution with Recycled Latents and Learned Cache Routing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Real-time diffusion-based video super-resolution (VSR) is in high demand for online streaming, yet stringent latency requirements often compromise generative fidelity. We propose ReCaVSR, a Wan2.2-based, one-step framework for streaming VSR that builds on two observations: recycled SR latents retain local temporal context, reducing the need for full historical Key-Value (KV) caches; and individual transformer layers benefit from distinct temporal scopes. ReCaVSR combines three complementary designs: layer-wise cache routing with recycled SR latents: each DiT layer learns its KV-cache temporal scope under a cache budget and exports a static inference schedule, while recycled SR latents propagate local context by conditioning each new block on the model's own preceding predictions. Multi-Scope Query (MSQ) Discriminator: a compositional discriminator combining global, spatial-window, and temporal-tube feedback for holistic realism, local texture generation, and temporal stability. LR-conditioned adaptation of FlashDecoder: a VAE decoder that incorporates LR observations for efficient latent decoding. ReCaVSR enables streaming VSR without iterative sampling or full historical KV-cache materialization. Experiments on synthetic and real-world VSR benchmarks show better perceptual quality, temporal consistency, and streaming efficiency than representative VSR baselines. At output resolution on a single NVIDIA A100-80GB, ReCaVSR achieves 21.20 FPS with 15.16 GB peak allocated GPU memory, running 2.72 faster while using 38.0% less peak allocated memory than FlashVSR Tiny. Our code and video demo are available at https://bucket.anony.work/ReCaVSR/code.zip and https://recavsr.anony.work, respectively.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Diffusion Models, Video super-resolution, layer-wise KV-cache routing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ETaUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReCaVSR, a one-step streaming video super-resolution framework based on Wan2.2. The main idea is to reuse the SR latents from the previous block to provide local temporal information, and to learn a separate KV-cache policy for each transformer layer. The learned policies are fixed during inference to reduce computation and memory costs. The paper also introduces a Multi-Scope Query discriminator for global, spatial, and temporal supervision, and adds LR conditioning to FlashDecoder for efficient decoding. Training includes VSR adaptation, cache-route learning, and sequential self-rollout adversarial training. Experiments cover synthetic and real-world benchmarks, long videos, ablation studies, and a user study. The reported results show a favorable balance between perceptual quality and inference efficiency.

Critical strengths

The method uses recycled SR latents to provide local temporal information, and keeps additional historical KV states only in selected layers. Table 4 shows that, compared with Uniform (3R), the proposed configuration reduces KV memory from 6.44 GB to 2.17 GB and historical-attention FLOPs to 0.28×, while DOVER only decreases from 0.5601 to 0.5587. These results show that the method can substantially reduce cache costs while largely maintaining perceptual quality.

In Table 5, the total number of queries, discriminator backbone, and training settings are kept the same. Using all three query scopes gives better NIQE, MUSIQ, and DOVER than using only global queries or two query scopes. This supports the usefulness of combining global supervision with local spatial and temporal supervision.

Critical weaknesses

ReCaVSR and FlashVSR share a similar overall pipeline: sparse attention, a lightweight decoder, and streaming inference. However, ReCaVSR uses Wan2.2-TI2V-5B, whereas the original FlashVSR Tiny is based on Wan2.1-1.3B. Besides the differences in model version and size, Wan2.2-TI2V-5B uses a VAE with higher spatial compression: its temporal–spatial downsampling factors are 4×16×16, compared with 4×8×8 for Wan2.1. With the same patch size, this results in roughly one quarter as many spatial DiT tokens at the same output resolution. Therefore, the backbone change also affects the computational workload. Although the paper provides component ablations, it remains unclear how much of the reported quality improvement and overall speedup comes from the proposed method, and how much comes from the backbone and its more compressed latent representation. Moreover, the cache ablation changes both layer-wise allocation and anchor access relative to the uniform-window baselines, leaving the specific contribution of learned allocation insufficiently isolated.

I watched the demo videos on the project page, but did not see an obvious visual-quality advantage over FlashVSR Tiny. The paper reports improvements in several metrics, but it remains unclear whether these improvements correspond to a noticeable difference in the videos. In the user study, the reported gains over FlashVSR Tiny are 0.06, 0.09, and 0.18 points for overall quality, detail quality, and temporal stability, respectively, without uncertainty estimates. In addition, the MSQ ablation does not directly evaluate texture flicker or discontinuities between generated blocks, and the temporal metrics on LongVSR60 do not consistently favor ReCaVSR. Overall, the magnitude and reliability of the claimed perceptual and fine-detail temporal improvements over FlashVSR Tiny remain insufficiently established. Nevertheless, maintaining comparable quality at substantially lower runtime and memory would itself be a meaningful benefit.

I acknowledge the value of the reported quality–efficiency tradeoff. My main reservation is that the current controls do not sufficiently establish the benefit of learning the layer-wise cache assignment beyond simpler matched-budget alternatives, while the magnitude of the claimed temporal-quality improvement remains uncertain. These unresolved questions about the central contribution are the main reason for my weak-reject recommendation.

Questions for authors

Can the authors compare against a FlashVSR-style baseline using the same Wan2.2 backbone, latent compression ratio, training data, and training budget, or conduct ablations by replacing the corresponding components in ReCaVSR with those from FlashVSR under these matched conditions? A runtime breakdown for the DiT, including historical attention, and the decoder would also help clarify how much of the efficiency gain comes from the proposed designs rather than the backbone’s higher latent compression.

Since the visual difference from FlashVSR Tiny is not obvious in the demo videos, can the authors provide confidence intervals for the differences in user ratings, together with an evaluation that directly measures texture flicker or block-boundary discontinuities? This would help determine whether the reported quality improvements are reliable and perceptible.

Reviewer_zBpcOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

ReCaVSR is a one-step Wan2.2-based VSR system that conditions each block on previously generated SR latents, learns a static per-layer KV-cache policy, trains with global, spatial, and temporal adversarial queries, and decodes with an LR-conditioned FlashDecoder. On an A100-80GB at 1080p it reports better FPS and peak allocated memory versus FlashVSR Tiny on 201 frames. It obtains the highest DOVER in the four main benchmark columns, and a 35-video human study favors it on quality, detail, and temporal stability.

Critical strengths
  • The system is evaluated end-to-end under a causal protocol, with first-output latency, sustained FPS, and peak memory measured on the same hardware. Compared with FlashVSR Tiny at 1080p, the improvements are substantial.
  • The method is supported by targeted ablations: at a matched cache budget, learned routing improves UDM10 DOVER over uniform recent-window history, while removing recycled latents lowers.
Critical weaknesses
  • The paper provides the training procedure but gives insufficient detail on the sources, composition, redistribution rights, and separation of the self-collected training videos from evaluation footage. This limits independent reproduction of training and prevents a clear audit for possible overlap.
  • Several components change together: route learning, latent recycling, adversarial query scopes, and a separately trained LR-conditioned decoder. The reported cache ablation emphasizes DOVER and memory on UDM10 rather than a matched full-system fidelity/temporal/latency matrix.
Questions for authors
  • Can the authors test scene-cut sequences with explicit state reset and report post-cut recovery, error accumulation, first-output latency, and quality over time?
  • Can the authors provide a factorial ablation of routing, recycling, MSQ, and LR-conditioned decoding under matched training/data budgets, with confidence intervals for the human study?
Reviewer_BvhqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReCaVSR, a one-step streaming diffusion framework for video super-resolution based on Wan2.2-TI2V-5B. The method introduces three main components: (1) recycled SR latents and layer-wise cache routing, where previously generated SR latents provide local temporal context and each DiT layer learns a cache retention policy under a memory budget; (2) a Multi-Scope Query (MSQ) Discriminator that combines global, spatial, and temporal queries for adversarial training; and (3) an LR-conditioned FlashDecoder for efficient latent-to-pixel reconstruction. Experiments on synthetic and real-world VSR benchmarks demonstrate competitive perceptual quality and streaming efficiency.

Critical strengths
  1. The paper focuses on improving the efficiency of diffusion-based streaming VSR while preserving quality by reducing the overhead of maintaining historical temporal information. Combining recycled SR latents with selective historical KV access is a reasonable design for exploiting temporal redundancy in streaming VSR.
  2. Comprehensive evaluation of perceptual quality and efficiency. The paper reports results on multiple synthetic and real-world benchmarks, including long-video evaluation, and provides comparisons of perceptual quality and inference throughput.
Critical weaknesses
  1. The ablation studies are conducted only on UDM10 with very limited evaluation metrics. Specifically, Table 4 lacks reconstruction and perceptual quality metrics, making it difficult to comprehensively assess the impact of the proposed designs on generation quality, while Table 5 lacks direct temporal consistency metrics to substantiate the claimed improvements in temporal consistency across frames. Moreover, the improvements over FlashVSR may stem from a different backbone and a larger training dataset, rather than from the proposed methods.
  2. Limited end-to-end efficiency gains and decoder novelty. The practical efficiency benefits of the proposed cache routing are insufficiently demonstrated, as the paper only reports reductions in KV-cache memory and attention FLOPs without measuring their actual contribution to end-to-end acceleration. Furthermore, the decoder largely follows the existing FlashDecoder design, limiting its methodological novelty.
Questions for authors
  1. How are the cache allocation strategies determined? The candidate cache scopes appear to be manually predefined. How stable are the learned layer-wise allocations across different training runs with different random seeds or training data?
  2. Why is an explicit cache-budget constraint necessary? If different attention heads require different temporal contexts, why does the method enforce a shared layer-wise routing policy rather than allowing each head to learn its own optimal cache scope? Without the budget constraint, would the learned policies naturally favor larger cache windows?
  3. How much does the training data contribute to the performance improvements? Given the large-scale self-collected dataset, I would like to see an analysis of how the scale of training data affects performance. Do the authors plan to release the training dataset?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.