- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces LightVAE, an approach to making existing video VAEs more computationally efficient without training a new architecture from scratch. The main idea is to reduce unnecessary computation in pretrained models while limiting the degradation in video reconstruction quality.
The proposed method consists of four key components: 1) Temporal Low-Rank Compression (TLR) preserves spatial processing while reducing temporal computation through low-rank approximation. 2) Weight-Folded Execution reuses cached features and combines operations to accelerate inference. 3) Width-Preserving Block Pruning removes redundant blocks without reducing channel dimensions. 4) Probe-Guided Feature Alignment uses teacher feedback to improve student reconstruction quality.
Critical strengths
- The proposed framework can directly compress pretrained video VAEs while preserving their latent interfaces, making it practical to integrate into existing video generation pipelines without retraining the entire model.
- The paper provides a solid analysis of the different compression sensitivities of spatial and temporal components, leading to a principled compression strategy. Furthermore, the authors connect the mathematical formulation with practical engineering optimizations, such as weight folding and caching, to achieve actual inference speedups rather than merely reducing theoretical computational costs.
Critical weaknesses
- Since temporal reconstruction quality is also important for video VAEs, additional evaluation of temporal consistency would strengthen the experimental validation. While the paper reports frame-level reconstruction metrics and VBench-2.0 scores, including complementary video-level metrics, such as FVD, could help assess whether temporal information is preserved after compression.
- The paper motivates TLR by temporal continuity and shows that temporal-only compression yields lower approximation errors than spatial or joint compression. However, it remains unclear whether this observation consistently holds across individual videos with different levels of temporal variation. In particular, it would be valuable to examine whether the proposed compression strategy remains effective under large inter-frame changes, and whether the stage-wise temporal sensitivity observed in Figure 5 still holds in such cases.
Questions for authors
- The paper presents an interesting motivation and overall approach. However, I would like to see additional evaluation of temporal consistency to better assess video reconstruction quality. I am also curious whether the underlying assumption of temporal redundancy holds consistently for both highly similar consecutive frames and frames with substantial temporal changes.
5 comments
Thank you for the additional video-level results. I found that LightVAE-Lite has a higher rFVD than Turbo-VAED on UCF101 (9.32 vs. 6.13), but a lower rFVD on DAVIS (25.96 vs. 37.55) using Wan2.2. Could the authors provide some explanation for this difference?
Response to Reviewer KYmL
Thank you for pointing out this difference. This is because LightVAE-Lite adopts a more aggressive compression strategy to target real-time decoding. It reduces the number of residual blocks in the Wan2.2 decoder from 14 to 5 and achieves 2.24× the decoding throughput of Turbo-VAED (Table 1). The trade-off is that LightVAE-Lite’s reconstruction performance is less consistent across datasets than that of LightVAE-Pro. Its rFVD is higher than Turbo-VAED’s on UCF101 and lower on DAVIS.
LightVAE-Pro offers a more direct comparison with Turbo-VAED because the two decoders are closer in depth. LightVAE-Pro retains all 14 residual branches of the original Wan2.2 decoder. It achieves higher decoding throughput than Turbo-VAED (38.84 vs. 29.24 FPS, Table 1) and lower rFVD on both UCF101 (4.49 vs. 6.13) and DAVIS (18.06 vs. 37.55).
These results show that LightVAE can accelerate decoding at different parameter budgets, with different trade-offs between throughput and reconstruction quality to suit deployment needs.
Response to Reviewer KYmL
Response to Weaknesses 1–2 and Question 1
Thank you for suggesting additional video-level evaluation and analysis of inter-frame variation. We followed your suggestions by adding reconstruction FVD evaluations on UCF101 and DAVIS and per-video comparisons of compression strategies under low and high inter-frame variation. The experiments show that LightVAE-Pro achieves the lowest reconstruction FVD among the evaluated compressed decoders within each backbone, while temporal-only compression retains its error advantage on every tested video, including those with large inter-frame changes.
1. Video-level reconstruction quality
We added reconstruction FVD (rFVD) evaluations on UCF101 and DAVIS, using features from an I3D model pretrained on Kinetics-400 [1] to compare the distributions of source and reconstructed videos.
Combined with Table 1, Wan2.2 Pro and Wan2.1 Lite offer higher H100/5090 throughput and lower rFVD on both datasets than Turbo-VAED and Flash-VAED, respectively. These results complement the frame-level reconstruction metrics with a video-level distributional assessment.
2. Robustness to large inter-frame changes
We quantify inter-frame variation as the mean absolute RGB difference between consecutive frames, with pixel values normalized to [0, 1]. Higher scores indicate larger changes in appearance. Before examining compression errors, we selected 128 low-variation and 128 high-variation videos from the same Wan2.1-generated corpus using this measure. Their median scores were 0.0004 and 0.1153, respectively. The table reports mean relative L₂ error against the original decoder’s output for the same latent input.
Across both groups, temporal-only compression produces lower error than spatial-only and joint compression for every tested video at each of the five decoder stages. Here, “up” is short for “upsampling” and labels stages along the decoder’s upsampling path. The stage-wise trend in Figure 5 also holds: in both groups, mean temporal-only error decreases from up1 to up3, toward later stages at higher spatial resolutions.
We have included these results, evaluation settings, and per-video comparisons in the revised manuscript.
References
[1] João Carreira and Andrew Zisserman. Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6299–6308, 2017.