- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes VideoRAE, a representation autoencoder built on a frozen video foundation model. A lightweight 1D self-attention projector compresses multi-scale VFM features into compact latents: continuous KL-free latents for diffusion Transformers and discrete tokens for autoregressive models via Multi-Codebook SimVQ. During decoding, a local–global Representation Alignment objective preserves the semantic structure of the frozen teacher, replacing the KL regularizer. Experiments cover reconstruction on UCF-101/TokenBench, class-conditional generation, a 4B-scale from-scratch text-to-video study, and adaptation to an 11B OpenSora2 backbone.
Critical strengths
-
The paper addresses a paradigm-level question: whether frozen VFMs can serve as the tokenizer foundation for video generation. It is an active and fast-moving direction.
-
Strong compression results: discrete rFVD of 13 and continuous rFVD of 14 on UCF-101; class-conditional gFVD of 40 (AR) and 94 (DiT), with the AR variant using fewer tokens and a smaller generator than SweetTok.
-
The empirical design is solid. The paper provides learning curves showing an approximately 5× convergence speedup over the respective baselines for both AR and DiT generators, and in a controlled 4B-scale from-scratch T2V study, VideoRAE wins at all 11 checkpoints under matched settings.
-
A unified framework for continuous KL-free and discrete multi-codebook latents.
Critical weaknesses
-
The core recipe (frozen encoder + semantic alignment + no KL) is inherited from image RAEs; the video-side contribution consists of engineering adaptation and large-scale validation rather than new concepts.
-
The main results are on UCF-101 at 256×256, with no matched comparison against concurrent works such as DeRA. DeRA adopts a trainable encoder with frozen FMs as alignment teachers—a different paradigm—so a direct comparison would clarify the trade-offs between the two routes.
-
The introduction is not problem-driven: the "nontrivial" claim is stated only through three abstract descriptors (temporal dynamics, redundancy, compression requirements), without an explicit obstacle–design correspondence, and the supporting ablations appear only later in the paper.
Questions for authors
Among the four video-side designs (multi-layer aggregation, 1D/3D projectors, multi-codebook SimVQ, and local–global REPA), which are necessary specifically for video? Can the "obstacle → design" correspondence be made explicit in the introduction?
2 comments
Author Response to Reviewer dRoF (2/2)
Q3: Comparison with DeRA
Thank you for highlighting DeRA as an important comparison. Since an official implementation was not available to us, we compare against the results reported in the DeRA paper. To align the input resolution and sequence budget, we adapt VideoRAE to 128 × 128 resolution and use 512 1D latent tokens, with the generator size also set to 343M parameters. The DeRA reconstruction and generation results are taken from the 512-token setting in Figure 4 of its ECCV version.
| Method | Result Source | Resolution | 1D Latent Tokens | Generator Parameters | rFVD ↓ | gFVD ↓ |
|---|---|---|---|---|---|---|
| DeRA | Reported | 128 × 128 | 512 | 343M | 43 | 155 |
| VideoRAE | Our evaluation | 128 × 128 | 512 | 343M | 38 | 131 |
The two methods differ in how foundation-model representations enter the tokenizer. DeRA trains a video encoder and uses frozen foundation models as alignment teachers; VideoRAE uses a frozen video foundation model as the feature extractor itself and learns only the compression and reconstruction modules. The trade-off is as follows. VideoRAE inherits the pretrained representation directly and obtains better reconstruction and generation in the comparison above, but its tokenizer is tied to the chosen backbone, including its native downsampling ratio. DeRA is free to choose encoder architectures and downsampling ratios while still acquiring semantics through alignment. This comparison is included in Appendix L.
Summary
We sincerely thank you for your constructive feedback. Your suggestions have helped us clarify the contributions beyond image RAEs, provide more direct evidence for the roles of video feature compression and multi-scale fusion, and better explain the relationship between VideoRAE and DeRA. We have uploaded a new version of the PDF, which focuses on articulating the correspondence between video-specific obstacles and our design choices and incorporates the methodological comparison with DeRA(Appendix L), to improve the clarity and completeness of the paper. We hope these responses address your main concerns and that you will consider reassessing your rating in light of the new evidence. We would be happy to provide any further clarification.
Author Response to Reviewer dRoF (1/2)
Thank you for recognizing the importance of exploring frozen video foundation models for generation and the effectiveness of our framework across continuous and discrete representations. We appreciate your suggestions regarding the contributions beyond image RAEs, the necessity of the video-side designs, and the comparison with DeRA. We address these points below.
Q1: Contributions beyond image RAEs
We acknowledge that the underlying concept is not new: VideoRAE builds on the representation-autoencoder paradigm introduced in the image domain. Our contribution lies in adapting and validating this paradigm for video generation, with three main advances.
First, we develop a video generation framework built on frozen video foundation models and validate it across class-to-video and text-to-video tasks. This includes large-scale adaptation and evaluation, extending the use of frozen video representations from understanding to reconstruction and generation. In addition to UCF-101 class-conditional generation, we conduct a 4B-scale T2V training study and adapt an 11B pretrained video generation model.
Second, we support both continuous diffusion-based generation and discrete autoregressive generation within the same representation-autoencoding framework. Our experiments demonstrate that frozen video foundation representations support strong reconstruction and generation in both settings. On UCF-101, the discrete and continuous variants achieve gFVD scores of 40 and 93, respectively, under their corresponding evaluation protocols. This extends the investigation beyond a single latent formulation or generative paradigm.
Third, we explicitly address the stronger compression requirements of video. Compared with images, videos contain substantial temporal redundancy and many more spatiotemporal feature positions, making direct modeling of native foundation-model features costly. We introduce 1D and 3D projectors to convert these features into compact generative interfaces: the 1D projector provides a flexible output token budget through global aggregation, while the 3D projector preserves an explicit spatiotemporal grid.
Q2: Which designs are particularly important for video?
We make the obstacle → design correspondence explicit below, and have rewritten the third paragraph of the introduction accordingly.
Compression projector. We compare variants with and without feature-dimension compression in the continuous setting. Both use V-JEPA 2.1 features under the 3D configuration; in the variant without the projector, features are only temporally downsampled and keep their 1024-d dimensionality, and the generator follows the RAE recipe for high-dimensional latents.
The projector yields a 16× smaller latent with comparable reconstruction and no loss in generation quality. Our claim is that compression makes frozen video features a practical generative interface, not that generation from uncompressed features is impossible.
Multi-layer aggregation. We compare the optical-flow endpoint error (EPE) of reconstructions from a final-layer-only variant and the multi-scale variant.
Multi-scale fusion reduces EPE by about 25%, i.e., intermediate layers preserve motion information that the final layer discards. This complements Table 14, where adding shallower layers raises PSNR from 26.59 to 29.39.
Quantization and REPA. High-dimensional multi-codebook quantization is needed to carry VFM semantics through discretization (Table 12: gFVD 95 for VQ vs. 40), and REPA improves generation in both latent types (Table 13). We regard both as general mechanisms for VFM-based tokenizers rather than video-specific designs.