- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes an efficient video generation model. The main goal is to improve the token compression efficiency of existing video generation models, which consist of an autoencoder and a diffusion transformer (DiT). Specifically, it aims to make the latent more compact than that of the pretrained model, while preserving as much knowledge of the pretrained autoencoder and DiT as possible. The key idea is to use the pretrained VAE encoder to produce a base latent from a low-resolution version of the input video, and a residual encoder to produce a residual latent from the full-resolution input, so that reconstruction quality is preserved with making the latent distribution easier for the DiT to learn. To make the two latents work as designed, a generation-aware alignment loss and asymmetric denoising are introduced. The effect of each key component is verified through experiments, and the full model is compared with the pretrained anchor (Wan2.1), a single-latent baseline at the same compression ratio, and other high-compression autoencoder-based methods, showing comparable VBench scores to Wan2.1 with 8× fewer tokens and higher scores than the other methods.
Critical strengths
Below are the two main reasons I recommend acceptance.
S1: The dual-latent design is simple yet effective. Instead of retraining a new high-compression autoencoder, the paper reuses the pretrained VAE encoder on a low-resolution input to produce the base latent, which stays close to the latent distribution the pretrained DiT already knows, and adds a residual encoder on the full-resolution input to recover the missing details. This asymmetric design directly addresses the key trade-off the paper identifies, where reducing tokens degrades reconstruction and increasing channels degrades generation, and its effectiveness is clearly demonstrated in the experiments (Tables 2 and 3). I therefore consider the paper to propose a novel architecture (and training strategies for it).
S2: Compared with Wan2.1 (its pretrained anchor), the full model maintains the generation quality of the anchor with 8× fewer tokens, while speeding up generation by 11.1×. It also achieves better generation quality and lower latency than both the single-latent baseline and DC-VideoGen, the closest prior work (Table 2). The human evaluation, presented with a detailed setup, is consistent with the quantitative results (Appendix D.2). I believe this result is a clear step forward in efficient video generation.
Critical weaknesses
None
Questions for authors
There are no questions whose answers could materially affect my assessment.
Response to Reviewer NtG1
We sincerely thank the reviewer for the positive assessment and for recognizing the simplicity and effectiveness of our design and the efficiency gains. In response to the other reviewers, we have further strengthened the paper with a fine-tuned uncompressed baseline, two additional backbones (Wan2.2-A14B and Cosmos-Predict2.5-14B), a higher compression setting (f16t16), a comparison with an additional alignment target, and analyses of the training recipe, as summarized in the General Response.