acceptodds
Withdrawn from ICLR 2027

FastFrames: Video Generation at Interactive Latency

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Autoregressive video diffusion models generate frames based on new observations and inputs, making them a promising medium for interactive visual experiences. However, existing models are at least an order of magnitude too slow for fluid interaction. This paper introduces FastFrames, a family of fast autoregressive video diffusion models that achieves interactive latency without compromising quality. We profile the inference pipeline of state-of-the-art models and find that operations often treated as fixed implementation details, such as VAE decoding and KV cache updates, in fact account for over half of end-to-end inference latency. We challenge the assumption that these components are indispensable and replace them with lightweight alternatives. Our 1.3B variant runs at 36 ms per frame (28 non-amortized FPS) on a single B200 GPU while matching the generation quality of a state-of-the-art model that is nearly 8x slower. Our 14B variant runs at 45 ms per frame (22 non-amortized FPS) on an 8x B200 GPU node and yields significantly better visual quality and realism. We show that FastFrames enables real-time interaction in applications such as camera-conditioned and physics-conditioned video generation.

Primary area:
generative models
Keywords:
real-time video generation, interactive world models, video diffusion models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2yACOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents FastFrames, a low-latency autoregressive video generation framework designed for interactive applications. The authors profile the inference pipeline of existing video diffusion models based on Wan, identify key computational bottlenecks beyond the diffusion sampling process, and propose lightweight, trainable alternatives to expensive components, particularly VAE decoding and KV-cache updates. Combined with system-level optimizations, these modifications substantially reduce end-to-end generation latency while maintaining comparable visual quality.

Critical strengths

The paper presents compelling quantitative results, achieving substantial reductions in generation latency while maintaining VBench scores comparable to, or slightly better than, those of the corresponding autoregressive baselines. The authors further demonstrate the applicability of their approach to camera-conditioned (LingBot) and physics-conditioned (RealWonder) video generation, achieving significant speedups, although these applications only approximately reach the targeted interactive latency.

The main strength of the paper lies in its systematic analysis of the inference pipeline to identify computational bottlenecks beyond diffusion sampling. By replacing expensive components, particularly the VAE decoder and KV-cache refinement, with lightweight, specially trained networks, and combining these replacements with system-level optimizations, the authors demonstrate substantial improvements in the speed–quality trade-off. This provides a practical approach to accelerating autoregressive video generation without substantially compromising visual quality.

Critical weaknesses

My main reservation concerns the novelty and methodological contribution of the paper. FastFrames is primarily an efficiency-oriented work that optimizes the inference pipeline of autoregressive video diffusion models, rather than introducing a fundamentally new video generation approach. The computational overhead associated with KV-cache management is a known limitation of such models. While the proposed learned KV-cache refinement network is an interesting contribution, the replacement of the VAE decoder with a lightweight, separately trained architecture, together with the hardware-specific system optimizations, appears to be largely engineering-driven. Although these modifications yield practical improvements, their methodological novelty and broader significance for the ICLR community remain somewhat limited, IMO.

A related concern is the lack of systematic quantitative ablations isolating the contributions of the individual learned modules. While Figures 3 and 6 provide qualitative comparisons, the paper does not sufficiently quantify how each replacement affects generation quality, particularly temporal consistency and long-term autoregressive stability.

Finally, the proposed architectures and training procedures appear to be largely tailored to the Wan-based models. Although the authors demonstrate their applicability to LingBot and RealWonder, these experiments do not clearly establish generalization to fundamentally different video diffusion architectures. In particular, the requirement to train a separate pixel decoder for each model raises questions about the generality and transferability of the approach.

Questions for authors

See above

Reviewer_xrZ1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies low-latency inference in autoregressive video generation. To let users see the effect of their input in a timely manner, it optimizes the latency of the first response frame. First, the authors replace the VAE with a lightweight pixel decoder that directly reads DiT internal features. In addition, the noisy KV from the last denoising step and the predicted latent are used to approximate the cache that would otherwise require an additional full DiT forward pass. This appears to be a very effective auxiliary computation approach.

Critical strengths

1.The pixel decoder exploits the rich features already computed by a fixed DiT, turning generic latent reconstruction into a model-specific pixel readout; the cache repair network reuses information already available from the denoising stage to approximate expensive cache updates. Both ideas are of practical design value.

2.The overall speedup is substantial and is supported across multiple models and applications.

Critical weaknesses

1.The experimental results do not separately compare the quality of replacing only the pixel decoder, replacing only the cache update network, and replacing both. This is not yet sufficient to determine whether errors accumulate over successive generations.

2.The main text attributes the quality improvement to adversarial training, but there is no corresponding controlled comparison with and without adversarial training.

3.The pixel decoder is mainly compared with the original VAE and a single-layer linear projection, and lacks representative fast-decoder baselines. For example, FlashDecoder [1] studies streaming decoding acceleration that preserves reconstruction quality in the Wan latent space.

[1] Kang, Minguk, and Suha Kwak. FlashDecoder: Real-Time Latent-to-Pixel Streaming Decoder with Transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5294–5305, 2026.

  1. The paper does not adequately explain how input arriving midway through a generation is handled, nor whether input waiting, output buffering, and display scheduling are included in the timing.

  2. The VBench composite score of the final system is close to the baseline, which can support that the overall quality under the test setting is roughly comparable, but it is not sufficient to prove separately that the two replacement modules incur no quality cost.

  3. In addition, there is a lack of a clear correspondence between the input-to-response latency defined in the paper and the steady-state generation time measured directly in the main experiments.

Questions for authors

1.The paper has not yet sufficiently validated the quality impact of each of the two core replacements, or their advantage over existing fast decoding schemes. The authors should add quality validation of the core modules, comparisons with representative decoders, and measurements of actual interactive latency. This would help assess the value of the work.

2.The authors should report the final reconstruction metrics on the existing validation set, and add a controlled comparison with and without adversarial training. Could the authors choose a compatible fast decoder and compare quality and latency under the same setting, so as to highlight the advantage of directly reading DiT features?

3.Does the latency described in the results start from the arrival of the external control input and end at the display of the first RGB frame affected by it?

Reviewer_nJNBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In this paper, the authors introduce FastFrames. It is a method of autoregressive video diffusion models capable of low-latency interactive streaming video generation. The paper identifies that the problems of VAE decoding and KV-cache updates contribute significantly to end-to-end inference latency. The authors propose to address these bottlenecks by 1. replaces the original VAE decoder with a lightweight ConvNeXt-based decoder, 2. use a learned KV refinement network instead. Experiments demonstrate that FastFrames-S achieves 36 ms/frame on a single B200 GPU and FastFrames-L achieves 45 ms/frame on eight B200 GPUs. The qualitative results remain similar to the base methods. The paper also demonstrates its optimizations to camera-conditioned and physics-conditioned video generation as a further extension.

Critical strengths
  1. FastFrame's finding is quite interesting. While most baseline works try to optimize at the step level or model level, it finds that the heaviest part might be the VAE, as well as the one forward pass brought by KV-Cache. The latency breakdown in Figure 2 is very informative.

  2. The experiments indicate that it achieves impressive speedups while maintaining good video quality (similar to the base model). In particular, FastFrames-S achieves 36 ms.

  3. The paper also shows the potential to extend its optimizations to LingBot and RealWonder rather than purely text-to-video generation.

Critical weaknesses
  1. Although the paper achieves good inference speedups, the evaluation of generation quality mainly relies on VBench scores. However, VBench scores are known to be perceptually inaccurate in some areas. I would encourage the authors to include some additional experiments on 1. VAE decoder v.s. propose decoder reconstruction; 2. a user study between proposed method and baseline methods.

  2. The paper emphasizes interactive video generation, but the results only show latency and is tested on B200s. I believe the authors should also include Throughput as a metric, as both Throughput and Latency are important metrics to evaluate speed of video generation. And the baselines should also be tested on the same GPUs.

Questions for authors

Please see the weakness section and address accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.