- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents FastFrames, a low-latency autoregressive video generation framework designed for interactive applications. The authors profile the inference pipeline of existing video diffusion models based on Wan, identify key computational bottlenecks beyond the diffusion sampling process, and propose lightweight, trainable alternatives to expensive components, particularly VAE decoding and KV-cache updates. Combined with system-level optimizations, these modifications substantially reduce end-to-end generation latency while maintaining comparable visual quality.
Critical strengths
The paper presents compelling quantitative results, achieving substantial reductions in generation latency while maintaining VBench scores comparable to, or slightly better than, those of the corresponding autoregressive baselines. The authors further demonstrate the applicability of their approach to camera-conditioned (LingBot) and physics-conditioned (RealWonder) video generation, achieving significant speedups, although these applications only approximately reach the targeted interactive latency.
The main strength of the paper lies in its systematic analysis of the inference pipeline to identify computational bottlenecks beyond diffusion sampling. By replacing expensive components, particularly the VAE decoder and KV-cache refinement, with lightweight, specially trained networks, and combining these replacements with system-level optimizations, the authors demonstrate substantial improvements in the speed–quality trade-off. This provides a practical approach to accelerating autoregressive video generation without substantially compromising visual quality.
Critical weaknesses
My main reservation concerns the novelty and methodological contribution of the paper. FastFrames is primarily an efficiency-oriented work that optimizes the inference pipeline of autoregressive video diffusion models, rather than introducing a fundamentally new video generation approach. The computational overhead associated with KV-cache management is a known limitation of such models. While the proposed learned KV-cache refinement network is an interesting contribution, the replacement of the VAE decoder with a lightweight, separately trained architecture, together with the hardware-specific system optimizations, appears to be largely engineering-driven. Although these modifications yield practical improvements, their methodological novelty and broader significance for the ICLR community remain somewhat limited, IMO.
A related concern is the lack of systematic quantitative ablations isolating the contributions of the individual learned modules. While Figures 3 and 6 provide qualitative comparisons, the paper does not sufficiently quantify how each replacement affects generation quality, particularly temporal consistency and long-term autoregressive stability.
Finally, the proposed architectures and training procedures appear to be largely tailored to the Wan-based models. Although the authors demonstrate their applicability to LingBot and RealWonder, these experiments do not clearly establish generalization to fundamentally different video diffusion architectures. In particular, the requirement to train a separate pixel decoder for each model raises questions about the generality and transferability of the approach.
Questions for authors
See above