acceptodds
Under review as a conference paper at ICLR 2027

VideoRAE: Taming Video Foundation Models for Generative Modeling via Representation Autoencoders

Abstract

Video generation models typically rely on 3D-VAEs trained for pixel-level reconstruction, whose latent spaces may underrepresent semantic structure. We introduce VideoRAE, a representation autoencoder that converts features from a frozen video foundation model into compact, reconstruction-capable latents for video generation. Lightweight projectors compress multi-scale hierarchical features into compact 1D sequences or structured 3D grids, supporting continuous latents for Diffusion Transformers and discrete tokens for autoregressive models through multi-codebook high-dimensional quantization. During decoding, a local–global representation alignment objective transfers semantic structure from the frozen encoder and removes the need for KL regularization. Comprehensive experiments show that VideoRAE achieves strong reconstruction in both continuous and discrete regimes. On UCF-101, autoregressive and diffusion generators built on VideoRAE achieve class-conditional gFVD scores of 40 and 93, respectively, while converging approximately 5x faster than autoencoder baselines. In a 4B-scale text-to-video study with randomly initialized diffusion backbones, frozen VideoRAE-3D achieves higher VBench total scores than Wan2.1-VAE at all 11 evaluated checkpoints, reaching 69.07 versus 68.24 at 200k training steps. These results establish frozen video foundation representations as compact, versatile, and generation-friendly video latents. Code and models will be released.

Primary area:
generative models
Keywords:
representation learning, video generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_dRoFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes VideoRAE, a representation autoencoder built on a frozen video foundation model. A lightweight 1D self-attention projector compresses multi-scale VFM features into compact latents: continuous KL-free latents for diffusion Transformers and discrete tokens for autoregressive models via Multi-Codebook SimVQ. During decoding, a local–global Representation Alignment objective preserves the semantic structure of the frozen teacher, replacing the KL regularizer. Experiments cover reconstruction on UCF-101/TokenBench, class-conditional generation, a 4B-scale from-scratch text-to-video study, and adaptation to an 11B OpenSora2 backbone.

Critical strengths
  1. The paper addresses a paradigm-level question: whether frozen VFMs can serve as the tokenizer foundation for video generation. It is an active and fast-moving direction.

  2. Strong compression results: discrete rFVD of 13 and continuous rFVD of 14 on UCF-101; class-conditional gFVD of 40 (AR) and 94 (DiT), with the AR variant using fewer tokens and a smaller generator than SweetTok.

  3. The empirical design is solid. The paper provides learning curves showing an approximately 5× convergence speedup over the respective baselines for both AR and DiT generators, and in a controlled 4B-scale from-scratch T2V study, VideoRAE wins at all 11 checkpoints under matched settings.

  4. A unified framework for continuous KL-free and discrete multi-codebook latents.

Critical weaknesses
  1. The core recipe (frozen encoder + semantic alignment + no KL) is inherited from image RAEs; the video-side contribution consists of engineering adaptation and large-scale validation rather than new concepts.

  2. The main results are on UCF-101 at 256×256, with no matched comparison against concurrent works such as DeRA. DeRA adopts a trainable encoder with frozen FMs as alignment teachers—a different paradigm—so a direct comparison would clarify the trade-offs between the two routes.

  3. The introduction is not problem-driven: the "nontrivial" claim is stated only through three abstract descriptors (temporal dynamics, redundancy, compression requirements), without an explicit obstacle–design correspondence, and the supporting ablations appear only later in the paper.

Questions for authors

Among the four video-side designs (multi-layer aggregation, 1D/3D projectors, multi-codebook SimVQ, and local–global REPA), which are necessary specifically for video? Can the "obstacle → design" correspondence be made explicit in the introduction?

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer dRoF (1/2)

Thank you for recognizing the importance of exploring frozen video foundation models for generation and the effectiveness of our framework across continuous and discrete representations. We appreciate your suggestions regarding the contributions beyond image RAEs, the necessity of the video-side designs, and the comparison with DeRA. We address these points below.

Q1: Contributions beyond image RAEs

We acknowledge that the underlying concept is not new: VideoRAE builds on the representation-autoencoder paradigm introduced in the image domain. Our contribution lies in adapting and validating this paradigm for video generation, with three main advances.

First, we develop a video generation framework built on frozen video foundation models and validate it across class-to-video and text-to-video tasks. This includes large-scale adaptation and evaluation, extending the use of frozen video representations from understanding to reconstruction and generation. In addition to UCF-101 class-conditional generation, we conduct a 4B-scale T2V training study and adapt an 11B pretrained video generation model.

Second, we support both continuous diffusion-based generation and discrete autoregressive generation within the same representation-autoencoding framework. Our experiments demonstrate that frozen video foundation representations support strong reconstruction and generation in both settings. On UCF-101, the discrete and continuous variants achieve gFVD scores of 40 and 93, respectively, under their corresponding evaluation protocols. This extends the investigation beyond a single latent formulation or generative paradigm.

Third, we explicitly address the stronger compression requirements of video. Compared with images, videos contain substantial temporal redundancy and many more spatiotemporal feature positions, making direct modeling of native foundation-model features costly. We introduce 1D and 3D projectors to convert these features into compact generative interfaces: the 1D projector provides a flexible output token budget through global aggregation, while the 3D projector preserves an explicit spatiotemporal grid.

Q2: Which designs are particularly important for video?

We make the obstacle → design correspondence explicit below, and have rewritten the third paragraph of the introduction accordingly.

Obstacle in videoDesignSpecific to video?Evidence
Many spatiotemporal positions, strong redundancy1D/3D compression projectorYesTable below; Appendix K
Fine-grained motion lost in final-layer featuresMulti-layer aggregationPartly (multi-layer features are also used for images; the motion benefit is video-specific)EPE below; Table 14; Appendix M
High-dimensional VFM semantics under quantizationMulti-codebook SimVQNo (addresses high-dimensional VFM features in general)Table 12
Generation qualityLocal–global REPANo (general mechanism)Tables 13, 16

Compression projector. We compare variants with and without feature-dimension compression in the continuous setting. Both use V-JEPA 2.1 features under the 3D configuration; in the variant without the projector, features are only temporally downsampled and keep their 1024-d dimensionality, and the generator follows the RAE recipe for high-dimensional latents.

SettingRepresentationLatent ConfigurationPSNR ↑LPIPS ↓rFVD ↓gFVD ↓
ContinuousWithout projector16 × 16 × 4 × 102432.110.07699
ContinuousWith projector16 × 16 × 4 × 6431.880.07793

The projector yields a 16× smaller latent with comparable reconstruction and no loss in generation quality. Our claim is that compression makes frozen video features a practical generative interface, not that generation from uncompressed features is impossible.

Multi-layer aggregation. We compare the optical-flow endpoint error (EPE) of reconstructions from a final-layer-only variant and the multi-scale variant.

Feature ConfigurationEPE ↓
Final-layer features only1.633
Multi-scale feature fusion1.225

Multi-scale fusion reduces EPE by about 25%, i.e., intermediate layers preserve motion information that the final layer discards. This complements Table 14, where adding shallower layers raises PSNR from 26.59 to 29.39.

Quantization and REPA. High-dimensional multi-codebook quantization is needed to carry VFM semantics through discretization (Table 12: gFVD 95 for VQ vs. 40), and REPA improves generation in both latent types (Table 13). We regard both as general mechanisms for VFM-based tokenizers rather than video-specific designs.

AuthorsOpenReview ↗

Author Response to Reviewer dRoF (2/2)

Q3: Comparison with DeRA

Thank you for highlighting DeRA as an important comparison. Since an official implementation was not available to us, we compare against the results reported in the DeRA paper. To align the input resolution and sequence budget, we adapt VideoRAE to 128 × 128 resolution and use 512 1D latent tokens, with the generator size also set to 343M parameters. The DeRA reconstruction and generation results are taken from the 512-token setting in Figure 4 of its ECCV version.

MethodResult SourceResolution1D Latent TokensGenerator ParametersrFVD ↓gFVD ↓
DeRAReported128 × 128512343M43155
VideoRAEOur evaluation128 × 128512343M38131

The two methods differ in how foundation-model representations enter the tokenizer. DeRA trains a video encoder and uses frozen foundation models as alignment teachers; VideoRAE uses a frozen video foundation model as the feature extractor itself and learns only the compression and reconstruction modules. The trade-off is as follows. VideoRAE inherits the pretrained representation directly and obtains better reconstruction and generation in the comparison above, but its tokenizer is tied to the chosen backbone, including its native downsampling ratio. DeRA is free to choose encoder architectures and downsampling ratios while still acquiring semantics through alignment. This comparison is included in Appendix L.

Summary

We sincerely thank you for your constructive feedback. Your suggestions have helped us clarify the contributions beyond image RAEs, provide more direct evidence for the roles of video feature compression and multi-scale fusion, and better explain the relationship between VideoRAE and DeRA. We have uploaded a new version of the PDF, which focuses on articulating the correspondence between video-specific obstacles and our design choices and incorporates the methodological comparison with DeRA(Appendix L), to improve the clarity and completeness of the paper. We hope these responses address your main concerns and that you will consider reassessing your rating in light of the new evidence. We would be happy to provide any further clarification.

Reviewer_WhBgOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how frozen video foundation model features can be converted into compact representations for video generation. The proposed VideoRAE aggregates features from multiple layers, compresses them through trainable projectors, and produces either continuous latents for diffusion models or discrete representations for autoregressive models. A decoder is trained with reconstruction objectives and local–global feature alignment. The experiments show that this framework supports both video reconstruction and downstream generation, suggesting that pretrained video representations can serve as useful foundations for generative modeling.

Critical strengths
  • Broad evaluation. Experiments cover continuous and discrete representations, class-conditional generation, and larger-scale text-to-video generation.
  • Useful reconstruction–generation distinction. Better generation despite weaker reconstruction than Wan2.1-VAE shows that reconstruction metrics alone are insufficient for evaluating generative representations.
Critical weaknesses
  • Missing key baselines. The paper does not compare against the original RAE approach of directly using frozen features as latents, nor does it evaluate directly aligning the compressed latents, leaving the necessity of the projector and decoder-feature alignment insufficiently justified.
  • Insufficient justification for the 1D design. It performs similarly to the 3D variant without an efficiency comparison, while the larger-scale study uses 3D.
Questions for authors

Please refer to the weaknesses section.

7 comments
AuthorsOpenReview ↗

Author Response to Reviewer WhBg (1/2)

Thank you for recognizing the breadth of our experimental evaluation and our investigation of the trade-off between reconstruction fidelity and generation quality. We also appreciate your constructive feedback. Below, we address your concerns regarding the missing key baselines and the insufficient justification for the 1D design.

Q1: Missing key baselines

Thank you for pointing out this omission. Directly using high-dimensional features for reconstruction and generation is an important transfer baseline that should be included in the comparison. We therefore add the following three comparisons:

  1. Frame-wise RAE, which applies an image RAE(DINOv3) independently to each video frame;
  2. High-dimensional video representations without feature-dimension compression, which use the frozen video encoder's output with temporal compression only, retaining the original feature dimensionality;
  3. A latent-alignment variant, which applies REPA directly to the compressed latents.

These comparisons examine the roles of feature compression and representation alignment. Note that the ablations in this set are conducted under the VideoRAE-3D configuration, which differs from the 1D setting used in Tables 6/13 of the main paper.

MethodLatent Configuration (H × W × T × D)REPA Alignment LocationPSNR ↑LPIPS ↓rFVD ↓gFVD ↓
Frame-wise RAE16 × 16 × 16 × 768—21.160.26154—
Video representations without feature-dimension compression16 × 16 × 4 × 1024—32.110.07699
VideoRAE (without REPA)16 × 16 × 4 × 64—32.050.078106
VideoRAE (latent alignment)16 × 16 × 4 × 64Latent31.950.07892
VideoRAE (decoder alignment)16 × 16 × 4 × 64Decoder31.880.07793

Frame-wise RAE and High-dimensional video autoencoder. The results show that directly applying frame-wise RAE to video leads to a clear reduction in reconstruction fidelity, making this straightforward transfer from images to video a suboptimal choice in the evaluated setting. For the high-dimensional video representation, retaining 1024-dimensional features brings only a slight reconstruction gain, and its gFVD does not outperform the compressed VideoRAE variant; in addition, designing a generator for such high-dimensional latents requires considerable specialized effort. Thus, increasing the latent feature dimensionality substantially enlarges the representation and downstream modeling burden, without a corresponding improvement in reconstruction or generation quality. These results support our design choice: using a learned projector to convert frozen video features into a more compact representation while retaining reconstruction and generation capabilities.

Alignment location. Both REPA variants improve generation over the variant without REPA (92/93 vs. 106), and they perform comparably to each other. Our choice of decoder alignment is motivated by feature correspondence: the 1D interface produces globally aggregated tokens with no spatiotemporal grid and no one-to-one correspondence with teacher positions, whereas shallow decoder features have a 3D structure that can be matched to the teacher. Decoder alignment therefore applies uniformly to both interfaces at no measurable cost relative to latent alignment.

AuthorsOpenReview ↗

Author Response to Reviewer WhBg (2/2)

Q2: Insufficient justification for the 1D design

Thank you for pointing this out. The original manuscript did not sufficiently explain the intended use of the two interfaces and lacked a direct comparison. We address efficiency, quality, and the choice for the large-scale study in turn.

Here, we first conduct an efficiency comparison and find that the two variants have similar efficiency.

Tokenization pipelineActual Latency (ms) ↓Peak GPU Memory (GB) ↓
1D57.741.426
3D51.481.424

We would like to clarify that our paper provides both 1D and 3D projector interfaces; the 3D variant is already included. Our use of the 1D variant is primarily intended to align with the representation format used by prior discrete autoregressive methods. The 1D projector uses learnable query tokens to aggregate information globally from the complete video features, allowing the output token count to be specified independently of a regular spatiotemporal grid. This design provides a flexible compression budget and is particularly suitable for exploring compact discrete video representations with shorter sequences.

For continuous generation, we also investigate the 1D variant to explore its potential for achieving better performance with compact representations. The results in Tables 8, 10, and 11 further show that, with the same V-JEPA 2 backbone and a matched latent budget of 1024 × 64, the 1D variant achieves slightly better reconstruction and generation performance than the 3D variant. We further conduct 1D versus 3D reconstruction experiments at a matched budget of 16,384 latent scalars, where the reconstruction advantage of the 1D variant becomes more evident under the more compact compression setting. Moreover, the 1D interface supports shorter sequences: the 512 × 32 configuration achieves a gFVD of 93 on UCF-101 while using only half as many latent tokens as the 1024-token 3D configuration, substantially reducing the computational requirements of global attention when other architectural settings are held fixed.

MethodLatent ConfigurationPSNR ↑LPIPS ↓rFVD ↓
VideoRAE-1D (V-JEPA 2)512 × 3230.400.0914
VideoRAE-3D (V-JEPA 2)16 × 16 × 4 × 1629.870.1015

Furthermore, we also validate the feasibility of the 1D variant in large-scale T2V experiments, with the results shown below. The 1D variant achieves strong T2V performance comparable to the 3D variant, confirming that both choices are viable.

RepresentationTotal ScoreQuality ScoreSemantic ScoreDynamic DegreeHuman ActionColorSpatial RelationScene
VideoRAE-1D (Ours)69.3277.1737.9162.7335.8075.387.8811.48
VideoRAE-3D (Ours)69.0777.1436.7879.1735.0078.4612.5810.67

In summary, 1D emphasizes compact representations with a flexible token budget, while 3D emphasizes preserving spatiotemporal structure, extension to arbitrary resolutions, and compatibility with conventional video diffusion models. We have included direct comparisons of quality and efficiency in the revised manuscript and clarified the conditions under which each design is applicable.

Summary

We believe that the additional experiments and analyses strengthen the empirical support for our design choices. We have uploaded a revised PDF, with additional analyses of the points discussed above in Sections G, I, J, and K of the appendix to improve the clarity and completeness of the paper. We hope these responses address your main concerns and that you will consider reassessing your rating in light of the new evidence. We would be happy to provide any further clarification.

Thanks for the response. Most of my concerns are addressed. I have several remaining questions about "Video representations without feature-dimension compression":

  • Which frozen video encoder is used, and what is the output shape of this frozen video encoder?
  • How is the temporal compression implemented to get the latents? Is it trainable?
  • The table indicates that this baseline does not use REPA. If the temporal compression operation is trainable, have you tried applying REPA to the latents?
Reviewer_S1aAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper extends the representation-autoencoder idea (RAE, VFMTok) to video. A frozen video foundation model (V-JEPA 2 or VideoMAEv2) encodes the input videos. Multi-layer features are combined and compressed by a trainable module (with 1D learnable-query self-attention, or 3D conv+attention layers). For continuous tokens, the compressed tokens are linearly projected to a KL-free bottleneck and used for DiTs. For discrete tokens, the tokens are quantized with Multi-Codebook SimVQ and used for LLaMA-style AR models. A Transformer decoder reconstructs the videos with a combination of L1, LPIPS, GAN, and feature alignment losses. The feature alignment loss aligns shallow decoder features with the VFM's final-layer features. Experiments in the paper demonstrate competitive reconstruction on UCF-101 and TokenBench, class-to-video generation on UCF-101, and text-to-video generation with a 4B MMDiT model.

Critical strengths
  • The paper demonstrates the important finding that frozen, highly abstract Video Foundation Models (VFMs) like V-JEPA 2 can be leveraged directly as autoencoder backbones.
  • The resulting latent enable the downstream generator training to achieve a 5x convergence speed up. The convergence gains is important to improve the cost of training video generators.
  • The paper provides a useful empirical insight: The consistent reconstruction-vs-generation inversion between VideoMAEv2 (better PSNR/rFVD) and V-JEPA 2 (better gFVD) suggests latent-prediction SSL features are more generation-friendly than pixel-MAE features.
Critical weaknesses
  • One of the main claim that the method "regularizes the latent manifold" and enables "semantically structured latent space" is implied rather than shown. REPA is applied to the decoder features, not to the latent itself. Any effect on latent geometry is therefore indirect. The paper never measures latent properties (linear probing or latent statistics). So "semantically structured latent space" is inferred from downstream gFVD alone. The KL comparison consists of a single configuration with an unspecified KL weight. A KL weight sweep and REPA-on-latent vs. REPA-on-decoder would be needed to support the claim.
  • VFMs have fixed native resolutions and clip lengths, so it is unclear how the approach handles higher resolution, longer clips, or variable aspect ratio, which are standard requirements for modern video VAEs (Wan2.1-VAE, LTX-VAE).
  • Extracting multi-scale intermediate features from heavy frozen VFMs could introduce computational and memory overhead during data preprocessing and encoding. The paper lacks an analysis on the running time and GPU memory consumption comparing total end-to-end processing (VFM feature extraction + projector + generator step) against standard 3DVAEs.
  • No video results were provided. It is difficult to assess the temporal quality of the reconstructed/generated from frame sequences.
Questions for authors
  • Does REPA change the latent itself? Linear-probe or retrieval results on the latent with and without REPA (and vs. a pixel VAE), plus a KL-weight sweep, could help test the core claim on the latent.
  • What is the wall-clock latency and peak GPU memory required for VFM feature extraction (V-JEPA 2) during video tokenization compared to standard 3D-VAE encoders?
  • How does the proposed method perform with videos of different length and resolution?
4 comments
AuthorsOpenReview ↗

Author Response to Reviewer S1aA (1/2)

Thank you for recognizing the value of using frozen video foundation models for generative modeling, the improvements in training convergence, and the distinction between reconstruction fidelity and generation quality. We appreciate your constructive suggestions and address your concerns below.

Q1: Evidence for the effect of REPA on latent representations and comparison with KL regularization

Thank you for pointing out that downstream generation results alone do not directly characterize the latent representations. REPA can influence the latents themselves: although the loss is applied to decoder features, its gradients propagate through the decoder to the trainable projector. We agree, however, that this optimization path alone does not establish a particular latent structure. We therefore measure the latents directly.

We compare VideoRAE-3D variants: REPA on the latent, REPA on decoder features, a variant without REPA or KL, and three KL-regularized variants with different weights. All other architecture and training settings are identical. These ablations use the VideoRAE-3D (V-JEPA 2.1) configuration (16 × 16 × 4 × 64 latents; the decoder-REPA row corresponds to the VideoRAE-3D (V-JEPA 2.1) entries in Tables 8 and 11), which differs from the 1D V-JEPA 2 setting of Tables 6 and 13.

RegularizationKL WeightPSNR ↑LPIPS ↓rFVD ↓gFVD ↓
Without REPA or KL032.050.078106
REPA on decoder features031.880.07793
REPA on latent031.950.07892
KL regularization2e-730.310.0911102
KL regularization1e-629.010.1118104
KL regularization5e-626.350.1436113

KL-weight sweep. The results show that removing REPA degrades generation performance, while the tested KL configurations introduce a trade-off between reconstruction and generation quality. A small KL weight of 2×10−72\times10^{-7} modestly improves gFVD from 106 to 102 relative to the unregularized baseline, but reduces PSNR by approximately 1.7 dB. Increasing the weight further degrades reconstruction fidelity; at 5×10−65\times10^{-6}, PSNR drops to 26.35 and rFVD rises to 36, while generation also becomes worse than the unregularized baseline, with a gFVD of 113. In contrast, applying REPA to either the latent or decoder features preserves reconstruction fidelity close to the unregularized baseline, with PSNR values of 31.88–31.95 versus 32.05, while achieving the best generation results among the evaluated configurations, reducing gFVD from 106 to 92–93.

Linear probing. To further assess whether REPA helps preserve semantically informative latent representations, we conduct linear probing on UCF-101. We freeze each representation model and train a linear classifier using a consistent feature-pooling and probe-training protocol. In addition to the configurations above, we include the pixel-reconstruction-based Wan2.1-VAE as a reference.

RepresentationRegularizationUCF-101 Linear-Probe Top-1 Accuracy (%) ↑
VideoRAE-3DWithout REPA or KL63.9
VideoRAE-3DREPA on decoder features72.3
VideoRAE-3DREPA on latent71.5
VideoRAE-3DKL, β = 2e-760.5
VideoRAE-3DKL, β = 1e-664.3
VideoRAE-3DKL, β = 5e-663.5
Wan2.1-VAEOriginal pretrained model22.4

Wan2.1-VAE achieves the lowest probing accuracy among the evaluated representations. VideoRAE-3D without REPA and its KL-regularized variants already retain useful semantic information, while applying REPA further improves linear-probe accuracy to 72.3% and 71.5% for decoder and latent alignment, respectively. These results provide direct evidence that REPA improves the linear separability of semantic information in the compressed latents, supporting our description of a more semantically structured representation.

Regarding alignment location, direct latent alignment and decoder-feature alignment achieve comparable performance. We would like to clarify that our choice of decoder alignment is primarily motivated by feature correspondence, particularly for the 1D version of VideoRAE. Its compressed 1D tokens do not have an explicit spatiotemporal grid, whereas shallow decoder features provide a 3D structure that can be matched to the teacher features. This makes shallow decoder features a natural location for applying REPA, while achieving performance comparable to direct latent alignment.

AuthorsOpenReview ↗

Author Response to Reviewer S1aA (2/2)

Q2: Computational overhead and peak GPU memory

Thank you for raising this concern. To examine the potential overhead of extracting multi-scale intermediate features from the VFM, we measure the actual latency and peak GPU memory consumption of the complete tokenization pipeline, including VFM feature extraction and the projector, and compare it with a standard 3D-VAE encoder. Wan2.1-VAE uses an input of 256 × 256 with 17 frames, while VideoRAE uses 256 × 256 with 16 frames. All other measurement conditions are kept identical.

RepresentationTokenization Latency (ms/clip) ↓Peak GPU Memory (GB) ↓
Wan2.1-VAE62.491.253
VideoRAE51.481.424

The measured latency of VideoRAE is lower than that of the standard 3D-VAE baseline, with slightly higher peak GPU memory consumption. Under these input settings, the complete VideoRAE tokenization pipeline is therefore not slower despite including VFM feature extraction. We have included these latency and memory measurements in the revised manuscript to make the computational cost explicit.

Q3: Performance at different video lengths and resolutions

Thank you for asking us to clarify the flexibility of the two interfaces. The 1D and 3D variants accommodate changes in input configuration in different ways.

For the 1D variant, adapting to a different resolution requires fine-tuning. Its learnable query tokens aggregate information from the input feature sequence, and fine-tuning allows this aggregation to adapt to the spatial organization and feature distribution at the new resolution. We have clarified this adaptation requirement in the revised manuscript.

The 3D projector preserves an explicit spatiotemporal grid and is not constrained by a fixed number of output query tokens. This structure naturally accommodates different spatial grid sizes. We evaluate the complete 3D model at additional resolutions and video lengths, including 20-frame videos at 384 × 384 and 32-frame videos at 360p, and compare its reconstruction performance with LTX-VAE (25/33 frames) and Wan2.1-VAE (21/33 frames).

Input SettingMethodPSNR ↑LPIPS ↓rFVD ↓
384 × 384, 25 framesLTX-VAE35.580.0916
384 × 384, 21 framesWan2.1-VAE38.140.041
384 × 384, 20 framesVideoRAE-3D34.880.074
360p, 33 framesLTX-VAE35.420.0915
360p, 33 framesWan2.1-VAE38.540.041
360p, 32 framesVideoRAE-3D34.670.064

These results show that VideoRAE-3D maintains reconstruction quality beyond the original 256 × 256, 16-frame setting, supporting its applicability to different resolutions and longer video clips.

Q4: Video results for assessing temporal quality

We agree that frame sequences alone are insufficient to assess the temporal quality of video reconstruction and generation. We have therefore included reconstructed and generated videos in the supplementary material so that temporal consistency and motion can be assessed directly. We hope these examples address your concern.

Summary

We sincerely thank you for your constructive feedback. Your suggestions have helped us strengthen the experimental evidence and clarify the design choices and scope of VideoRAE. We have uploaded a revised PDF and added the experimental results and related analyses discussed above to Sections G, H, and I of the supplementary material to improve the clarity and completeness of the paper. We hope these responses address your main concerns and that you will consider reassessing your rating in light of the new evidence. We would be happy to provide any further clarification.

I would like to thank the authors for their efforts responding to my review. My concerns have been thoroughly addressed in the authors' responses. I intend to keep my recommendation for acceptance.

Dear Reviewer S1aA,

Thank you for your positive follow-up and continued support for acceptance. We are glad that our responses have thoroughly addressed your concerns. Your constructive feedback helped us strengthen the experimental evidence and clarify the presentation of our work. We sincerely appreciate your time and careful review.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

Dear Area Chair,

We thank you and the reviewers for your careful evaluation and constructive feedback. Reviewer S1aA has confirmed that their concerns have been thoroughly addressed and maintained their recommendation for acceptance; Reviewer WhBg has also confirmed that their concerns have been addressed following our further clarifications and additional experiments, raising their rating from 2 to 3 (Weak acceptance). We summarize the main experimental additions and revisions below:

  1. Semantic structure of the latent representations. We added linear probing, a KL-weight sweep, and alignment-location ablations. REPA improves probe accuracy from 63.9% to 72.3% and gFVD from 106 to 93, while largely preserving reconstruction quality, providing direct evidence of improved semantic separability in the compressed latents.

  2. High-dimensional baselines and compact representations. We added frame-wise RAE and high-dimensional video baselines, and completed a from-scratch experiment with REPA applied to the high-dimensional latents. The high-dimensional baseline achieves gFVD of 99 without REPA and 98 with REPA, while VideoRAE achieves 93 using 1/16 as many latent scalars and retaining comparable reconstruction quality. These results support our choice of compact representations.

  3. Use cases for the 1D and 3D interfaces. Additional comparisons show that 1D offers a flexible token budget, better reconstruction under compact latent budgets, and shorter sequences for generation. The 3D interface preserves explicit spatiotemporal structure and readily accommodates existing video diffusion models. Large-scale T2V experiments further demonstrate the viability of both interfaces.

  4. Extension to higher resolutions and longer videos. We added reconstruction results at 384 × 384 with 20 frames and 360p with 32 frames, demonstrating that the 3D interface extends to different input sizes without fine-tuning. We also clarified that the 1D interface requires fine-tuning for such extensions.

  5. Video-specific design rationale and comparison with related methods. We revised the introduction to explicitly connect video challenges with our design choices. Additional experiments show that multi-scale fusion reduces optical-flow EPE by approximately 25%. We also added a comparison with DeRA: at matched resolution, token count, and generator size, VideoRAE achieves rFVD/gFVD of 38/131, compared with DeRA's reported 43/155.

The main additional experiments and analyses have been incorporated into the revised manuscript. We sincerely thank the reviewers and the Area Chair for their time and valuable feedback.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.