acceptodds
Under review as a conference paper at ICLR 2027

SwiftVR: Real-Time One-Step Generative Video Restoration

Abstract

Real-time video restoration (VR) for live streams requires high-resolution outputs under strict per-frame latency constraints. Existing one-step diffusion-based VR models remain difficult to deploy on consumer-grade GPUs due to two main bottlenecks: quadratic spatial attention at high resolutions and the latency-memory overhead of large video autoencoders. We present SwiftVR, a streaming one-step generative VR framework that reduces both bottlenecks under a causal chunk-wise protocol. For attention, mask-free shifted-window self-attention uses deterministic indexing to gather spatial windows into dense tensors. This design keeps all attention operations on the dense scaled dot-product attention path, eliminating masks, cyclic shifts, padding, and hardware-specific sparse kernels. Consequently, the trained model transfers directly to consumer-grade GPUs without retraining or custom kernels. For autoencoding, a lightweight Restoration-aware Autoencoder enables fast chunk-wise decoding while preserving reconstruction quality. On a single H100, SwiftVR sustains 31 FPS at and 14 FPS at (4K), whereas all compared diffusion-based VR baselines exceed the memory limit at 4K. On a consumer-grade RTX 5090 GPU, SwiftVR reaches 26 FPS at (1080p). To our knowledge, SwiftVR is the first generative VR model to achieve real-time 1080p streaming on a consumer-grade GPU, while attaining strong no-reference perceptual quality with lower inference cost.

Primary area:
generative models
Keywords:
video restoration, generative model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UJf1OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SwiftVR, a one-step generative video restoration framework designed for high-resolution real-time streaming. The method introduces Mask-Free Shifted-Window Attention (MFSWA) to reduce spatial attention costs using standard dense attention kernels, and a lightweight Restoration-aware Autoencoder (ReAE) to accelerate video encoding and decoding. A three-stage training pipeline combines flow matching, attention distillation, and adversarial fine-tuning. Experiments demonstrate improved throughput, achieving 31 FPS at 2K on an H100 and 26 FPS at 1080p on an RTX 5090.

Critical strengths
  1. Practical motivation and strong efficiency. The paper addresses important computational bottlenecks in generative video restoration and demonstrates impressive high-resolution streaming throughput on both server-class and consumer-grade GPUs.
  2. Comprehensive experimental evaluation. The paper provides comparisons with recent video restoration methods, detailed runtime analysis, component ablations, and qualitative results across multiple benchmarks.
Critical weaknesses
  1. Limited methodological novelty (the primary reason for my rejection recommendation). MFSWA is primarily an implementation-level modification of existing shifted-window attention, while ReAE follows the lightweight autoencoder strategy already adopted by FlashVSR[1]. The training pipeline also combines established flow matching, distillation, and adversarial fine-tuning techniques. Although the engineering improvements are meaningful, the methodological novelty appears relatively limited.
  2. Insufficient attribution of performance improvements. SwiftVR adopts Wan2.2-TI2V instead of the smaller Wan2.1-1.3B backbone used by FlashVSR, together with a more compressed latent representation. The reported 3.3× throughput improvement over FlashVSR may therefore partly originate from the backbone and latent compression differences rather than the proposed modules. This makes the paper's actual technical contribution less substantial than its claimed breakthrough suggests.

[1] Zhuang, Junhao, et al. "Flashvsr: Towards real-time diffusion-based streaming video super resolution." In CVPR, 2026.

Questions for authors
  1. Can the authors provide full-resolution visual comparisons (rather than only small cropped patches), along with video demonstrations? Given the relatively weak full-reference metrics and the lack of temporal consistency metrics, the current evidence is insufficient to convincingly assess the restoration quality.
  2. SwiftVR does not use cross-chunk DiT attention or KV caching, relying solely on ReAE boundary states to maintain temporal continuity. It remains unclear whether such low-level state propagation is sufficient to preserve generated details across chunks, especially under fast motion. Moreover, a short KV cache may already provide effective temporal context in VSR with relatively limited computational overhead. Can the authors further justify this design choice and provide stronger evidence that ReAE boundary states alone are sufficient?
4 comments

Response to Reviewer UJf1

We thank the reviewer for the clear summary of concerns. We address the novelty question first, since the reviewer names it as the primary reason for rejection. New results are given in Tables R1–R6 (R1–R5 are in our response to rBep and R6 in our response to Lst9). Paper table numbers refer to the revised PDF, whose wording changes are marked in blue.

(Part 1/2)

W1. Limited methodological novelty.

SwiftVR contributes a new attention design, MFSWA, and a restoration-adapted autoencoder, ReAE, and combines them into the first generative VR system that runs in real time on a consumer GPU. Each part has a measured, independent effect:

  1. It solves an open deployment problem. SwiftVR is the first generative VR method shown to sustain real-time 1080p streaming on a consumer GPU (26 FPS on an RTX 5090). It is also the only diffusion-based method in our evaluation that runs at 4K on a single H100. The closest streaming method, FlashVSR, reports 17 FPS at 768×1408768{\times}1408 on an A100 and depends on a dedicated sparse kernel.

  2. MFSWA is a new window design that outperforms existing ones. Its defining property is that every window has the same shape at every resolution, obtained by boundary clamping and priority scatter. This differs from Swin (cyclic shift and mask), from SeedVR (variable-sized windows with varlen kernels), and from plain non-shifted windows. In a new controlled comparison on the same backbone and budget (Table R6), MFSWA gives the best quality of all windowed variants (LPIPS 0.2508 vs. 0.2618–0.2637) while running within 3% of the fastest variant on standard dense SDPA only:

Table R6. Same backbone and budget, (16,16)(16,16) windows, 2K.

VariantPSNRLPIPSFPS
Non-shifted windows25.520.261832.01
SW-MSA (Swin)25.340.263727.47
SeedVR-style varlen (flash_attn)25.400.262132.22
MFSWA (ours)25.580.250831.32

Because no custom kernel is involved, the same weights run unchanged on five attention backends (Table 8) and on Blackwell consumer GPUs, where sparse-kernel methods such as FlashVSR currently have no supported acceleration path.

  1. ReAE turns a tiny autoencoder into a restoration-grade streaming codec. ReAE widens the decoder of TAEHV, is fine-tuned jointly with the DiT, and carries cross-chunk states. The wider decoder gives +1.39+1.39 dB over a recipe-matched control (Table R2), and removing the cross-chunk states raises boundary tLPIPS 4.7×4.7\times (Table R5). ReAE decodes 27×27\times faster than Wan2.2-VAE.

W2. Attribution of the performance gains (Wan2.2-TI2V-5B and latent compression).

The 3.3×3.3\times figure over FlashVSR-Tiny compares complete systems, and Sec. 4.3 of the final version will state this explicitly. The gain comes from MFSWA and ReAE, not from the backbone’s latent compression. Fig. 2 profiles the unmodified Wan2.2-TI2V-5B (original VAE and full-attention DiT) on an H100. It takes 8.37 s per 1080p chunk, VAE decoding alone takes 2.70 s at 1080p and 25.8 s at 4K, and its memory forces tiled inference at 2K and above. The compressed latent alone is therefore far from real-time.

The contribution of each module on this same backbone is isolated as follows:

  • ReAE versus Wan2.2-VAE (Table 5): decoding takes 99 instead of 2714 ms (27×27\times faster) and encoding 34 instead of 818 ms. This also frees enough memory to run the DiT without tiling.

  • MFSWA versus full attention (Table 4): the DiT time drops from 1039 to 566 ms at 2K, giving 1.62×1.62\times end-to-end throughput.

Starting from the unmodified 5B pipeline, these two modules bring it from 8.37 s per chunk to 54 FPS at 1080p on the H100.

Q1. Full-resolution visuals and video demos.

The Supplementary Material now includes comparison_visual.zip, with three full-resolution side-by-side comparison videos, one each from VideoLQ, SPMCS, and YouHQ40. Temporal consistency is also quantified in Tables R1 and R5.

(Part 2/2)

Q2. Are ReAE boundary states alone sufficient? Why not a KV cache?

We now provide direct evidence (Table R5, repeated from our response to rBep):

Table R5. Temporal consistency at chunk boundaries (20 videos at 2K, L=24L{=}24; tLPIPS ×10−2\times10^{-2}).

VarianttLPIPS allboundaryinteriorPSNRLPIPSFPS
w/o ReAE cross-chunk states3.4116.852.8427.890.193631.35
ReAE states only (default)2.373.612.3228.450.186631.35
+ 1-latent DiT overlap2.342.902.3128.450.186327.51
+ 2-latent DiT overlap2.332.872.3128.450.186124.57

The ReAE states are necessary. Removing them raises boundary tLPIPS 4.7×4.7\times and costs 0.56 dB PSNR, at no change in speed.

The ReAE states are sufficient. Overlapping one or two latent frames gives the DiT the same kind of short cross-chunk context that a short KV cache would provide. It lowers boundary tLPIPS from 3.61 to 2.90, but overall tLPIPS moves only from 2.37 to 2.33, and PSNR and LPIPS are unchanged. The cost is 12–22% of throughput.

A KV cache conflicts with the throughput target. Each chunk’s attention would then span the current tokens plus the cached ones, adding compute and memory to every chunk. With full temporal visibility inside each window, a cache of even one latent frame enlarges every window’s sequence. With ReAE states alone, whole-video temporal quality is already within 0.04×10−20.04{\times}10^{-2} tLPIPS of the overlapped variants, so the cache-free design is the right default for real-time streaming, and overlap is available as an option.

Each point above is answered with new measurements, and we are happy to provide further details during the discussion.

Thank you for the detailed rebuttal and additional experiments. Several of my concerns have been addressed, but I still have some remaining questions.

W1. I acknowledge the novelty and effectiveness of MFSWA. However, I still have concerns regarding ReAE. The authors describe it as a module that "turns a tiny autoencoder into a restoration-grade streaming codec" and "carries cross-chunk states". To my understanding, propagating temporal states across chunks is already a common design in causal video autoencoders, including TAEHV.

W2. I agree that the proposed modules substantially accelerate the Wan2.2-5B backbone. However, I find the claim that the "gain does not come from the backbone's latent compression" still somewhat overstated. SwiftVR and FlashVSR follow similar acceleration strategies: efficient attention and lightweight decoding. The current experiments do not sufficiently disentangle the contribution of differences in latent compression to the approximately 3× speedup over FlashVSR. This also underlies my concern about the incremental nature of the contribution, as SwiftVR largely follows the same acceleration pipeline as FlashVSR, with refinements to existing components rather than fundamentally new designs.

Q1. My concern has been addressed.

Q2. A comparison with actual causal streaming is still missing. Overlap may smooth transitions across chunk boundaries (e.g., the central yellow pattern in the provided SPMCS video appears inconsistent across chunks, gradually changing near chunk boundaries). In contrast, causal streaming can condition subsequent chunks on previously generated content to maintain consistency. Moreover, KV caching reuses previous computations, potentially introducing less overhead than overlap. Could the authors clarify the actual efficiency and quality trade-offs of causal streaming with KV caching?

If I have misunderstood any aspects of the method or experiments, I welcome further clarification and discussion from the authors.

We thank the reviewer for the continued discussion and for acknowledging the novelty and effectiveness of MFSWA.

W1 (ReAE). We do not claim ReAE as a new architecture, and cross-chunk state propagation is inherited from TAEHV. ReAE adapts an existing tiny autoencoder to the two requirements of this task: latency within the real-time budget, and reconstruction quality sufficient for restoration. Neither available option meets both (Table R2). Wan2.2-VAE reaches 35.48 dB but decodes in 2714 ms, and the released TAEHV decodes in 40 ms but reaches only 30.89 dB. ReAE reaches 32.74 dB at 99 ms: +0.46 dB comes from our recipe, which fine-tunes ReAE jointly with the DiT along the one-step inference path, and a further +1.39 dB comes from the wider decoder.

W2 (attribution and relation to FlashVSR). We revise the sentence to "the gain does not come only from the backbone's latent compression"; the word "only" was missing in our previous response. Compression alone is far from real time. The unmodified Wan2.2-TI2V-5B, with the same latent, runs at about 2.9 FPS at 1080p on an H100 (Fig. 2), and MFSWA and ReAE bring the same backbone to 54 FPS. The speedup over FlashVSR compares complete systems. Directly adopting FlashVSR's acceleration would not reach this result, for two reasons:

  1. Portability. FlashVSR relies on a Block-Sparse-Attention kernel that has no supported acceleration path on Blackwell consumer GPUs. MFSWA uses one dense SDPA call per layer, and the same weights run unchanged on five attention backends (Table 8) and on the RTX 5090.
  2. Effectiveness under stronger compression. In a more compressed latent space, each token carries more image content and the sequence is shorter. This leaves less redundancy for restricted attention to exploit, so it is not given in advance that windowed attention stays accurate. MFSWA does: it stays within 0.28 dB of the full-attention teacher.

We do not claim fundamentally new designs. Our contribution is to extend the deployment boundary of generative VR. SwiftVR is the first generative VR method to run in real time at 1080p on a consumer GPU (26 FPS on an RTX 5090). It is also the only diffusion-based method in our evaluation that runs at 4K on a single H100.

Q2 (causal streaming with KV caching). We do not claim that our chunk-wise processing is better than actual causal streaming. The paper provides a simple and effective base pipeline. Overlap is not part of our method: the paper splits the stream into non-overlapping chunks (Sec. 3.3), and all reported SwiftVR results, including the comparisons in Table 1, are obtained without overlap (App. B.1). Our core contribution is real-time high-resolution generative VR on consumer hardware through MFSWA and ReAE, so we did not discuss cross-chunk DiT modeling in depth. Following the suggestion, we implemented causal KV caching. Each DiT layer caches the keys and values of the last 1 or 2 latent frames of the previous chunk, and current tokens attend within their MFSWA window to the cached and current frames. The variant is training-free and uses the same weights. Results at 2K (L=24, one H100; tLPIPS ×10⁻²):

Cross-chunk DiT contexttLPIPS boundarytLPIPS allPSNRLPIPSFPS
None (default)3.612.3728.450.186631.35
1-latent overlap2.902.3428.450.186327.51
2-latent overlap2.872.3328.450.186124.57
1-latent KV cache2.842.3328.460.186130.35
2-latent KV cache2.772.3128.470.185829.73

As the reviewer anticipated, KV caching is both cheaper and more effective than overlap. With the same context, it matches or improves on overlap in every quality metric at a much smaller throughput cost: 3.2% versus 12.2% for one latent, and 5.2% versus 21.6% for two. Overlap re-denoises the shared frames in every chunk, whereas the cache reuses their keys and values. The effect concentrates at the boundaries: two cached latents lower boundary tLPIPS by 23% (3.61→2.77). For the video as a whole the gain is small, at most 0.06×10⁻² tLPIPS, 0.02 dB PSNR, and 0.0008 LPIPS, because boundaries are only 12 of 296 frame pairs. With the ReAE states, the base pipeline already provides most of the temporal consistency at the highest throughput.

Both mechanisms are inference-time options that need no retraining, and a deployment can enable KV caching when boundary consistency matters more than the last few percent of throughput. We will report this comparison in the final version.

We hope these results resolve the remaining concerns. If so, we would be grateful if the reviewer would consider raising the score.

Reviewer_Lst9OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SwiftVR is a one-step video restoration model built on Wan2.2-TI2V-5B. Through a number of engineering and algorithmic optimizations, it achieves real-time streaming video restoration. The optimizations include Mask-Free Shifted-Window Attention (MFSWA), which allows the shifted-window attention computation to use the standard fused SDPA kernel, and a lightweight Restoration-aware Autoencoder (ReAE) that speeds up decoding. Training has three stages: first, a full-attention flow-matching teacher is trained on the straight LQ-to-HQ path; it is then distilled into a windowed student; finally, the model is fine-tuned in pixel space with L1, LPIPS, temporal, and GAN losses. The final model runs at 54/31/14 FPS at 1080p/1440p/4K respectively on an H100, and at 26 FPS at 1080p on an RTX 5090.

Critical strengths

This work addresses two key problems in real-time high-resolution video restoration, and the solutions are validated as effective. To reach real time, the work targets two key bottlenecks, the computational cost of attention and the decoding cost of the VAE, and optimizes each of them. It greatly increases generation speed while keeping competitive restoration quality, and the ablations show where the speedup comes from. Table 4 compares MFSWA with global attention and a Swin-style implementation under matched window size and training settings: MFSWA reaches 31.32 FPS, SW-MSA 27.47 FPS, and the global-attention teacher 19.36 FPS. Table 5 separately shows why replacing the autoencoder matters: ReAE takes 99 ms to decode, versus 2,714 ms for Wan2.2-VAE. It is worth noting that this is the first work to reach 26 FPS at 1080p on a single 32 GB consumer GPU without custom kernels, which gives it real deployment value.

Critical weaknesses
  1. Most of the speedup does not come from the paper's core innovations, and the contribution on the attention side is relatively weak. Aiming at real-time video restoration, this work combines a highly compressed latent, a lightweight autoencoder, one-step training, chunked processing, and efficient attention, and arrives at a usable speed–quality trade-off. MFSWA is only one effective local improvement within this system, and compared with the Swin variant that uses masking and cyclic shifts, its speedup is only 1.14×. SeedVR already avoids cyclic shifts and masking by packing variable-size boundary windows into varlen FlashAttention, so the descriptions of prior work on pages 2 and 4 overstate the gap. Following from this, making the window shapes uniform so that plain SDPA can run without a varlen interface is, in my view, an implementation-level choice. Likewise, ReAE plays the same role as FlashVSR's tiny decoder and is not strictly a novel contribution either.

  2. The work lacks an evaluation of continuity across temporal windows, and on most metrics its performance is not the best. The DiT sees nothing across the boundaries between temporal windows, yet the paper reports neither warping error nor tLPIPS, nor any boundary analysis, to show how it avoids abrupt changes in texture, color, sharpness, or object detail when outputs are produced chunk by chunk. The advantages in Table 1 fall mainly on MUSIQ and CLIP-IQA, two metrics that tend to reward sharpening, which happens to favor a GAN-fine-tuned model. LPIPS is never the best (0.2508 on UDM10, versus 0.2172 for DOVE). On VideoLQ, the real-world dataset closest to the target scenario, RealBasicVSR beats SwiftVR on CLIP-IQA, MANIQA, and NIQE, and SwiftVR ranks fourth on DOVER (0.5305, versus 0.5610 for FlashVSR-Tiny, Table 7).

Questions for authors
  1. Could you explain the design choice behind ReAE? Why not simply train a lightweight decoder instead? That would keep the latent distribution unchanged, so the Wan weights could be reused more effectively, and it would also remove the need for the joint training in Stage 3, lowering the training cost.
  2. Could you compare MFSWA, on the same base model and under the same budget, against non-shifted windows, SeedVR-style varlen windows, and Neighborhood Attention or Sliding Tile Attention (ICML 2025)? If MFSWA clearly wins on speed/quality, that would change my assessment of its novelty. This is the experiment I care about most.
  3. Could you show, at different resolutions, whether abrupt changes in texture, color, sharpness, or object detail appear at temporal window boundaries? Ideally this would be a quantitative evaluation on a set of clips with fast motion and shot cuts.
  4. What are the end-to-end latency and throughput at L = 8 and L = 12, and is the quality maintained?
5 comments

Response to Reviewer Lst9

We thank the reviewer for the precise comments, and in particular for naming the comparison that would change the assessment (Q2). We ran it and report it first. New results are given in Tables R5–R6 of this response (Tables R1–R4 are in our response to rBep). Paper table numbers refer to the revised PDF, whose wording changes are marked in blue.

(Part 1/3)

Q2. MFSWA versus non-shifted windows, SeedVR-style varlen windows, and NA/STA (same backbone, same budget).

We trained every windowed variant on the same backbone with the same recipe, the same training budget, and (16,16)(16,16) windows. All are evaluated on UDM10, and speed is measured at 2560×14402560{\times}1440 with L=24L{=}24 on one H100. The teacher, SW-MSA, and MFSWA rows are from Table 4.

Table R6. Attention variants on the same backbone and training budget, all with (16,16)(16,16) windows.

VariantKernel dependencyPSNR$\uparrow$LPIPS$\downarrow$DiT (ms)↓\downarrowFPS$\uparrow$
Full attention (teacher)dense SDPA25.860.24171039.1119.36
Non-shifted windowsdense SDPA25.520.2618548.5732.01
SW-MSA (Swin, cyclic shift + mask)SDPA w/ mask25.340.2637674.4927.47
SeedVR-style varlen windowsflash_attn varlen25.400.2621542.6432.22
MFSWA (ours)dense SDPA25.580.2508566.3131.32

Quality. MFSWA clearly wins on quality: it is the best windowed variant on both metrics. Its LPIPS is lower by 0.0110, 0.0113, and 0.0129 than that of the non-shifted, varlen, and Swin variants. Each of these gaps is larger than the gap between MFSWA and the full-attention teacher (0.0091). MFSWA thus recovers more than half of the LPIPS that windowing costs, while the other windowed variants stay close to one another.

Speed. MFSWA is 14% faster end to end than Swin SW-MSA and within 3% of the fastest variant (31.32 vs. 32.22 FPS). The remaining 3.1% DiT-time gap to non-shifted windows is the cost of the extra windows created by the half-window shift. Both variants run the same dense SDPA call and differ only in their gather indices.

Why each alternative loses quality.

  • Non-shifted windows. No information ever crosses the fixed window borders, so the output shows seams along the 512-pixel window grid. These seams are clearly visible in playback because they stay at the same position in every frame, yet full-frame PSNR barely registers them: at 2K the ±2\pm2-pixel bands around the borders contain only 1.2% of the pixels. The LPIPS gap (0.011) reflects them clearly.

  • SeedVR-style varlen windows. Border windows are cut rather than clamped. At 2K (latent 45×8045{\times}80), the shifted layers split the 45 rows into windows of 8, 16, 16, and 5 rows, and the 80 columns into 8, 16, 16, 16, 16, and 8 columns. Corner tokens therefore attend within as few as 5×85{\times}8 positions per frame instead of 16×1616{\times}16. MFSWA’s boundary clamping instead gives every token a full 16×1616{\times}16 neighborhood of valid tokens, without padding or masks.

  • Swin SW-MSA. After the cyclic shift, border windows contain tokens from opposite sides of the frame. The mask removes these pairs, so border tokens again attend to fewer valid neighbors. The mask also forces SDPA off its fastest path (674.49 ms).

Kernel dependency. The varlen variant’s 2.9% speed edge comes from the external flash_attn varlen interface. MFSWA reaches nearly the same speed with standard dense SDPA, and the same weights run unchanged on five attention backends (Table 8) and on the RTX 5090. This combination is the core of MFSWA: every attention call has the same shape at every resolution, so quality, speed, and portability are obtained together.

NA and STA. Neighborhood Attention and Sliding Tile Attention obtain their wall-clock speedups from dedicated kernels (NATTEN and ThunderKittens), so their speed is tied to the hardware these kernels support. MFSWA obtains its speed from standard dense SDPA and therefore runs at full speed on any GPU with a standard SDPA backend. Table R6 already includes the strongest kernel-assisted window baseline, SeedVR-style varlen, and MFSWA comes within 3% of its speed with better quality. We will add the NA/STA discussion to Sec. 2.3 of the final version.

(Part 2/3)

W1. Contribution of MFSWA/ReAE; description of SeedVR.

SeedVR. We thank the reviewer for this correction. Cyclic shifts and masks are used by Swin-style designs such as SwinIR and Uformer, whereas SeedVR and SeedVR2 pack variable-sized boundary windows into varlen FlashAttention. We have corrected Sec. 1 and Sec. 2.3 accordingly (marked in blue). MFSWA differs from SeedVR in that it keeps every window the same size through boundary clamping, so every attention call is a uniformly shaped dense SDPA call with no varlen interface. The controlled comparison above shows that this design improves quality as well as portability.

Contribution of MFSWA and ReAE. MFSWA and ReAE remove the two bottlenecks of Fig. 2 (attention and the VAE) using only standard operators, and together they make the 5B generative backbone real-time. Each has a measured, independent effect. MFSWA gives the best quality of all windowed variants (Table R6). ReAE widens TAEHV’s decoder, is trained jointly with the DiT, and carries cross-chunk states. Its wider decoder gives +1.39+1.39 of the +1.85+1.85 dB gain over the tiny autoencoder under a recipe-matched control (Table R2 in our response to rBep), and its cross-chunk states reduce boundary tLPIPS 4.7×4.7\times (Table R5).

W2. Continuity across temporal windows; metrics.

Continuity. We now report tLPIPS over all frame pairs and separately over chunk-boundary and interior pairs (Table R5, abridged; see our response to rBep for the full table).

Table R5 (abridged). tLPIPS (×10−2\times10^{-2}) on 20 videos at 2K, L=24L{=}24.

VariantallboundaryinteriorFPS
w/o ReAE cross-chunk states3.4116.852.8431.35
SwiftVR (default, no overlap)2.373.612.3231.35
SwiftVR, 1-latent overlap2.342.902.3127.51

The ReAE states reduce boundary tLPIPS by 4.7×4.7\times. Boundaries are only 12 of 296 pairs, so overall tLPIPS (2.37) is close to the interior value (2.32), and an optional one-latent overlap lowers boundary tLPIPS by a further 20% (3.61$\to$2.90).

Metrics. The full-reference metrics, which sharpening cannot inflate, confirm SwiftVR’s quality. First, against FlashVSR-Tiny, the closest streaming baseline, SwiftVR is better in PSNR, SSIM, and DISTS on all three synthetic benchmarks, with PSNR gains of 0.77, 1.56, and 0.43 dB. Sharpening does not raise PSNR and SSIM. Second, SwiftVR has the best DISTS on YouHQ40 (0.1195 vs. 0.1265), and its LPIPS on YouHQ40 is within 0.0023 of the best method. On UDM10, DOVE has a lower LPIPS but runs 36×36\times slower at 2K (0.87 vs. 31.32 FPS).

Q1. Why ReAE instead of only training a lightweight decoder?

We considered decoder-only replacements that keep the Wan2.2 latent space, such as Turbo-VAED (Zou et al., 2026) and Flash-VAED (Zhu et al., 2026), which distill or prune the original VAE decoder while keeping its encoder. Keeping the encoder decides the question. The Wan2.2-VAE encoder alone takes 818 ms per 25-frame 1080p clip, versus 34 ms for ReAE (Table 5). Adding the 328 ms 1080p DiT time gives 1.15 s per 24-frame chunk even with a free decoder. This exceeds the 1 s budget for 24 FPS, so a decoder-only design cannot be real-time at 1080p on the H100, let alone on the RTX 5090.

Stage 3 would also still be needed with a lightweight decoder. Stages 1–2 supervise only latent velocities. Stage 3 trains the DiT along the actual one-step, pixel-space inference path with pixel, perceptual, temporal, and adversarial losses, and this need does not depend on the choice of decoder.

Q3. Boundary artifacts under fast motion and shot cuts, at different resolutions.

Table R5 measures continuity at chunk boundaries separately from the interior on the 2K test set. Chunks have a fixed length, so boundaries fall at fixed frame indices regardless of content and include moving scenes. The table shows that the ReAE states carry continuity across chunks. For content with very fast motion at a boundary, the one-latent overlap described in App. C lowers boundary tLPIPS by a further 20% at a 12% throughput cost, and SwiftVR remains real-time with it (27.51 FPS at 2K). The three side-by-side videos in comparison_visual.zip (Supplementary Material; VideoLQ, SPMCS, and YouHQ40) run across multiple chunks, so the chunk transitions can be inspected directly.

References

  • Y. Zou, J. Yao, S. Yu, et al. Turbo-VAED: Fast and Stable Transfer of Video-VAEs to Mobile Devices. AAAI, 2026.

  • L. Zhu, Y. Huang, X. Ge, et al. Flash-VAED: Plug-and-Play VAE Decoders for Efficient Video Generation. ICML, 2026.

(Part 3/3)

Q4. Latency and quality at L=8L{=}8 and L=12L{=}12.

Quality is maintained (Table R1 in our response to rBep):

  • L=8L{=}8: 31.05 FPS, first-frame latency 0.76 s, PSNR 28.41, LPIPS 0.1899, tLPIPS 2.49.

  • L=12L{=}12: 32.02 FPS, first-frame latency 1.04 s, PSNR 28.42, LPIPS 0.1887, tLPIPS 2.42.

  • L=24L{=}24 (default): 31.32 FPS, first-frame latency 1.93 s, PSNR 28.45, LPIPS 0.1866, tLPIPS 2.37.

Compared with L=24L{=}24, L=8L{=}8 costs only 0.04 dB PSNR and 0.003 LPIPS, cuts first-frame latency by 2.5×2.5\times, and lowers peak memory from 38.0 to 26.5 GB.

Table R6 is the comparison the reviewer named in Q2. On the same backbone and budget, MFSWA clearly wins on quality at near-identical speed, using standard SDPA alone. We are happy to provide further analysis during the discussion.

Reviewer_rBepOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SwiftVR accelerates one-step generative video restoration through mask-free shifted-window attention and a lightweight restoration-aware autoencoder. It processes videos in causal chunks and achieves sustained throughput of 31.32 FPS at 2K on an H100 and 26 FPS at 1080p on an RTX 5090.

Critical strengths

The acceleration is substantial: at 2K on an H100, throughput is approximately 3.3 times that of FlashVSR-Tiny. Using standard SDPA interfaces reduces reliance on specialized sparse-attention kernels and offers practical deployment value.

Critical weaknesses
  1. The real-time claim is inadequately supported. The introduction emphasizes strict per-frame latency, whereas the experiments primarily report throughput. Appendix C acknowledges that collecting a 24-frame chunk at 24 FPS takes approximately one second, before inference. A throughput of 26 FPS does not demonstrate suitability for low-latency live streaming. The paper lacks measured end-to-end latency and a quality-latency trade-off across chunk lengths.
  2. The independent technical contribution is not sufficiently established. Window partitioning and shifting have well-established precedents, and MFSWA improves end-to-end throughput over Swin by 14%. ReAE is initialized from an existing autoencoder, yet its parameter count increases from 11.42M to 40.95M without sufficient architectural detail or a capacity-matched control. The results do not disentangle the contributions of architectural design, increased capacity, and the training recipe.
  3. The evaluation omits recent open-source diffusion-based video super-resolution methods, including SparkVSR[1] and SATB-VR[2], both released with inference code and weights before the submission deadline. SATB-VR's auxiliary predictor must be included in runtime measurements. Consumer-GPU deployment is a central claim, yet no baseline comparisons are provided on the RTX 5090. The efficiency evaluation also lacks non-diffusion baselines under supported mixed-precision deployment settings, leaving the practical speed-quality trade-off incompletely established.
  4. The claim of cross-chunk continuity lacks direct evidence. The authors infer that ReAE boundary states suffice from aggregate DOVER scores, without ablations of state removal, chunk length, or overlapping processing, or an evaluation targeting chunk boundaries. On VideoLQ, DOVER is 0.5305 versus 0.5610 for FlashVSR-Tiny, which does not support an across-the-board temporal quality advantage.
  5. The experimental protocol and key ablations are incomplete. Figure 1 shows 4x super-resolution, but input dimensions, scale factors, and whether pre-upsampling is included in timing are not consistently specified across quality and runtime experiments. Teacher distillation, third-stage joint training, and the boundary-output selection rule lack independent ablations, leaving the necessity of these designs unverified.

[1] Yu J, Gao X, Verlani P, et al. Sparkvsr: Interactive video super-resolution via sparse keyframe propagation[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2026: 397-415.

[2] Bai H, Chen X, Liu X, et al. SATB-VR: Training Few-Step Video Restoration Diffusion Model using SNR-Aware Trajectory Blending[J]. arXiv preprint arXiv:2606.28677, 2026.

Questions for authors
  1. What are the measured first-frame latency, steady-state output latency, and restoration quality at different chunk lengths?
  2. Which architectural changes account for ReAE's approximately 3.6-fold parameter increase? Does its advantage persist against a capacity-matched autoencoder trained with the same recipe?
  3. Do the quality and efficiency advantages hold after comparisons with SparkVSR in reference-free mode and SATB-VR in its one-step setting, using the same hardware and upscaling factor and including SATB-VR's auxiliary predictor in runtime measurements?
  4. How does temporal error at chunk boundaries change when ReAE's cross-chunk state is removed or short overlaps are introduced?
3 comments

Response to Reviewer rBep

We thank the reviewer for the detailed questions. Each point is addressed below with new measurements. New results are given in Tables R1–R5 of this response (Table R6 is in our response to Lst9). Paper table numbers refer to the revised PDF, whose wording changes are marked in blue.

(Part 1/3)

W1 / Q1. Real-time claim; latency and quality versus chunk length.

We define real-time as sustained output throughput no lower than the source frame rate, and the revised abstract and Sec. 1 state this definition explicitly. Latency is set by the chunk length LL, and Table R1 measures it. The test set has 20 videos of 300 frames each (297 evaluated after alignment), restored from 640×360640{\times}360 to 2560×14402560{\times}1440 on one H100. With a 24 FPS live input, the first chunk holds L+4L{+}4 frames because of the ReAE temporal stride, so first-frame latency is (L+4)/24+T(L{+}4)/24+T, where TT is the processing time of one chunk. Throughput stays above 24 FPS for every LL, so no queue builds up in the steady state, and per-frame latency ranges from TT to L/24+TL/24+T.

Table R1. Effect of chunk length LL (20 videos, 640×360→2560×1440640{\times}360\to2560{\times}1440, one H100, bf16).

LLTT (ms)FPSFirst-frame lat. (s)Mem. (GB)PSNRLPIPSMUSIQtLPIPS (×10−2\times10^{-2})
8257.631.050.7626.5328.410.189959.292.49
12374.832.021.0429.6028.420.188759.612.42
24766.031.321.9338.0128.450.186659.942.37
481623.229.573.7953.0928.500.185260.052.34

Going from L=24L{=}24 to L=8L{=}8 cuts first-frame latency by 2.5×2.5\times (1.93 s$\to$0.76 s) and peak memory by 30% (38.0$\to$26.5 GB), while throughput stays at 31 FPS. Quality is preserved: PSNR changes by −0.04-0.04 dB, LPIPS by +0.003+0.003, and tLPIPS by +0.12×10−2+0.12\times10^{-2}. Every tested LL is real-time, so a deployment can choose LL for its latency or memory budget without giving up real-time throughput or quality.

W2 / Q2. Independent contribution; source of ReAE’s parameters; capacity-matched control.

Parameter increase (Q2). ReAE keeps the encoder (1.49M parameters) and the memory-block topology of TAEHV and widens the decoder: the channels go from [256,128,64,64][256,128,64,64] to [512,256,128,64][512,256,128,64]. This grows the decoder from 9.94M to 39.45M parameters and accounts for the entire 11.42M$\to$40.95M increase. It also explains why encoding time is unchanged (33 vs. 34 ms) while decoding time rises from 40 to 99 ms.

Capacity versus recipe. To separate the two factors, we trained TAEHV at its original width with our recipe (Table R2, same setting as Table 5).

Table R2. ReAE parameter breakdown and recipe-matched control on 25×1088×192025{\times}1088{\times}1920 clips.

VariantDec. channelsEnc. (M)Dec. (M)Total (M)PSNRLPIPSEnc. (ms)Dec. (ms)
Wan2.2-VAE–––704.6935.480.05138182714
TAEHV (released)[256,128,64,64]1.499.9411.4230.890.11833340
TAEHV + our recipe[256,128,64,64]1.499.9411.4231.350.10623340
ReAE (ours)[512,256,128,64]1.4939.4540.9532.740.07773499

The recipe alone gives +0.46+0.46 dB. The wider decoder adds a further +1.39+1.39 dB and lowers LPIPS by 27% (0.1062$\to$0.0777). ReAE’s advantage therefore holds against an autoencoder trained with the same recipe. It costs only 59 ms per 25-frame 1080p clip, and ReAE still decodes 27×27\times faster than Wan2.2-VAE.

MFSWA versus Swin (14%). The 14% figure is end-to-end, so it includes ReAE encoding and decoding. These non-DiT steps are identical for both variants and take 200 of the 766 ms chunk time. For the DiT alone, MFSWA is 16.0% faster (674.49$\to$566.31 ms) and also more accurate (PSNR/LPIPS 25.34/0.2637$\to$25.58/0.2508, Table 4). We have also run a controlled comparison with non-shifted and SeedVR-style varlen windows on the same backbone and budget (Table R6 in our response to Lst9). MFSWA gives the best quality of all windowed variants and runs within 3% of the fastest, on standard dense SDPA only. This is also why the same weights run unchanged on five attention backends (Table 8) and on the RTX 5090. MFSWA is therefore an independent contribution with a measured effect on both quality and speed.

(Part 2/3)

W3 / Q3. SparkVSR, SATB-VR, RTX 5090 baselines, and mixed precision.

SparkVSR and SATB-VR (Q3). We evaluated both under our protocol on the same hardware (one H100) and with the same ×4\times4 upscaling. SparkVSR runs in reference-free mode. SATB-VR runs in its one-step setting, with its auxiliary predictor included in the runtime. Both require tiled inference at 2K.

Table R3. Comparison on UDM10; efficiency at 2560×14402560{\times}1440 over 24 output frames on one H100.

MethodPSNRLPIPSDISTSMUSIQFPS (2K)Mem. (GB)
DOVE26.970.21720.129163.330.8759.24
FlashVSR-Tiny24.020.24830.133465.959.6134.35
SparkVSR (ref-free)26.930.24160.155960.080.7420.21
SATB-VR (1-step, w/ aux.)23.020.35280.179570.170.1832.80
SwiftVR (ours)25.580.25080.131167.3431.3238.01

SwiftVR has the best DISTS of the three new comparisons and runs 42×42\times faster than SparkVSR and 174×174\times faster than SATB-VR. SparkVSR is higher in PSNR/LPIPS and SATB-VR in MUSIQ, but SATB-VR trails SwiftVR by 2.56 dB PSNR and 0.10 LPIPS. Neither method comes close to real-time at this resolution, and both are slower than DOVE. SwiftVR is the only one of these methods that delivers generative restoration in real time.

RTX 5090. FlashVSR’s speedup relies on its Block-Sparse-Attention kernel, which is built on FlashAttention-2. In the released implementation, this kernel has no supported acceleration path on Blackwell consumer GPUs (sm_120), so FlashVSR cannot run on the 5090 with its intended kernel. The other diffusion baselines need 35–59 GB at 2K on the H100, more than the 32 GB of the 5090. None of them reaches real-time even on the H100 at 2K (at most 9.61 FPS, Table 2), and the 5090 is slower than the H100 (SwiftVR: 54 vs. 26 FPS at 1080p). SwiftVR is therefore the only real-time generative option on this card. It uses only standard torch SDPA and runs on the 5090 unchanged, at 26 FPS for 1080p.

Mixed precision. Our original numbers used each method’s released default precision, which is fp32 for the non-diffusion methods (App. B.1). We re-measured these baselines in fp16 (Table R4).

Table R4. Non-diffusion baselines at 2560×14402560{\times}1440 on one H100, 24-frame chunks.

MethodFPS (fp32)Mem. (fp32)FPS (fp16)Mem. (fp16)
Real-ESRGAN4.163.155.582.03
RealBasicVSR18.127.1228.093.69
RealViFormer25.535.8238.233.36
SwiftVR (bf16)––31.3238.01

SwiftVR runs faster than Real-ESRGAN and RealBasicVSR even after both are moved to fp16. RealViFormer reaches 38 FPS in fp16, but at clearly lower perceptual quality (UDM10 MUSIQ 58.85 vs. 67.34 for SwiftVR, Table 1). Among generative methods, SwiftVR is 3.3×3.3\times faster than the fastest baseline.

(Part 3/3)

W4 / Q4. Cross-chunk continuity: state removal and short overlaps.

We now evaluate continuity directly at chunk boundaries (Table R5). tLPIPS is computed separately for boundary pairs (the last frame of chunk kk and the first frame of chunk k+1k{+}1; 12 per video) and interior pairs (284 per video). The setup is L=24L{=}24 on the same 20 videos as Table R1.

Table R5. Temporal consistency at chunk boundaries (tLPIPS ×10−2\times10^{-2}; overlap counted in latent frames).

VarianttLPIPS allboundaryinteriorPSNRLPIPSFPS
w/o ReAE cross-chunk states3.4116.852.8427.890.193631.35
SwiftVR, overlap 0 (default)2.373.612.3228.450.186631.35
SwiftVR, overlap 12.342.902.3128.450.186327.51
SwiftVR, overlap 22.332.872.3128.450.186124.57

State removal. Resetting the ReAE states at every chunk leaves the runtime unchanged. However, it raises boundary tLPIPS 4.7×4.7\times (3.61$\to$16.85), raises overall tLPIPS from 2.37 to 3.41, and costs 0.56 dB PSNR. The cross-chunk states are therefore what keep the stream continuous, at zero runtime cost.

Short overlaps. With the states in place, a one-latent overlap lowers boundary tLPIPS by a further 20% at a 12% throughput cost, and a second latent adds little. Because boundary pairs are only 12 of 296 pairs, overall tLPIPS moves only from 2.37 to 2.33. We therefore keep overlap 0 as the real-time default and provide overlap 1 as an option. The chunk-length study is given in Table R1.

DOVER. SwiftVR ranks first on SPMCS and YouHQ40 (Table 7). DOVER scores whole clips and does not isolate chunk boundaries, so Table R5 measures them directly.

W5. Protocol and ablations.

Protocol. All experiments use ×4\times4 upscaling. For runtime, the inputs are 480×270480{\times}270, 640×360640{\times}360, and 960×540960{\times}540 for the 1920×10801920{\times}1080, 2560×14402560{\times}1440, and 3840×21603840{\times}2160 outputs. The bilinear pre-upsampling runs on the GPU inside the timed region, so all reported times include it. The supplementary code reproduces every setting, and Sec. 4.1 of the final version will state this explicitly.

Ablations. Each training stage addresses a separate gap, and the main effects are isolated:

  • Teacher distillation (Stage 2). The window-restricted student is initialized from the full-attention teacher and distilled from it. Table 4 shows that it comes within 0.28 dB PSNR of the teacher while running 1.62×1.62\times faster. All windowed variants in Table R6 are trained with the same recipe and budget, so the comparison among them is controlled.

  • Joint fine-tuning (Stage 3). Stages 1–2 supervise only latent velocities, which constrain the decoded video only indirectly. Stage 3 is the only stage that trains the model along the one-step, pixel-space inference path used at deployment, and the only one that adapts ReAE together with the DiT.

  • Boundary-output selection. Where clamped windows overlap, the scatter step writes each token exactly once, using a fixed window priority. This keeps the output deterministic, avoids an extra accumulate-and-normalize pass, and is what allows every window to keep the same size under half-window shifting. Its effect is ablated directly by the non-shifted variant in Table R6, which loses 0.011 LPIPS.

Each point above is answered with new measurements, and we are happy to provide further details during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.