acceptodds
Under review as a conference paper at ICLR 2027

V-RAE: Rethinking Video Latent Spaces for Generation

Abstract

Latent video generation relies on autoencoders to define a compact space in which generative models operate. Although video autoencoders have evolved substantially, their latent spaces are still optimized primarily for pixel-level reconstruction and capture limited high-level semantics. A reconstruction-optimal latent space, however, need not be well suited to generative modeling. We propose V-RAE, a video representation autoencoder that builds temporally compact generative latents on top of frozen vision foundation models (VFMs). A lightweight temporal pooling module reduces temporal redundancy while preserving semantic structure, and a video decoder reconstructs videos from the compressed features. We evaluate V-RAE with four representative VFMs on video reconstruction, semantic probing, and class-conditional generation. V-RAE achieves 2.13 rFVD on K600, outperforming all evaluated large-scale pretrained video VAEs. Its latents retain substantially more semantic information than conventional video tokenizer latents. V-RAE achieves gFVD scores of 117.86 on UCF101 and 19.16 on K600, and matches Wan2.2 VAE's converged gFVD with up to 6× faster convergence. We further show that reconstruction quality alone is insufficient to characterize generative utility and introduce tFVD, a temporal-coherence diagnostic that correlates more strongly with downstream generation quality than rFVD. Beyond video generation, V-RAE also improves future video prediction on Cityscapes over the Wan 2.2 VAE latent space under matched prediction settings. Taken together, these results show that frozen semantic representations provide an effective basis for video reconstruction, generation, and predictive modeling.

Primary area:
generative models
Keywords:
Video Generation, Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zu9TOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes V-RAE, a video autoencoder built on frozen visual foundation models. It combines features from multiple encoder layers, compresses them with local temporal attention, and trains a video decoder to reconstruct the input. A class-conditional DiT is then trained in the resulting latent space. The authors test four representation encoders and evaluate reconstruction, semantic probing, and generation on UCF101 and K600. They also propose tFVD, which measures videos decoded from temporally interpolated latents, and include a future-prediction experiment on Cityscapes.

Critical strengths
  1. The temporal pooler addresses a real trade-off. In the controlled pooling comparison, attention pooling reaches 6.12 rFVD and 89.13% UCF101 probing accuracy. Convolution gives slightly lower rFVD, 5.91, but much lower probing accuracy, 78.26%. This is useful evidence that the proposed pooler preserves more of the frozen encoder’s representation while remaining reconstructable.

  2. Several V-RAE variants show substantial gains in the reported class-conditional gFVD results. The main table uses different output heads, but Appendix C checks this issue on UCF101. With neither the wide DDT head nor auxiliary supervision, V-RAE obtains 138.79 gFVD against 154.64 for Wan2.2 VAE. That comparison gives me more confidence that the latent space itself contributes to the gain.

Critical weaknesses
  1. The reported convergence advantage is not yet an end-to-end efficiency result. V-RAE and Wan2.2 VAE have the same temporal and spatial latent grid in the reconstruction comparison, but V-RAE uses 1024 channels where Wan2.2 uses 48. This is about 21 times as many latent values. The matched token count therefore does not imply matched latent storage or total computation. The main generation table also uses a wide DDT head and auxiliary supervision for V-RAE, although the UCF101 ablation partly addresses that difference. The “up to 6× faster” result is measured in optimizer updates; without GPU time, memory, or throughput, its practical size remains unclear.

  2. The evidence that tFVD predicts generative utility is preliminary. For VAEs with a separately encoded first frame, the first interpolated latent averages a one-frame code with a four-frame code. V-RAE interpolates between codes that each represent four frames. That difference may affect the cross-tokenizer tFVD ranking. Figure 8 then correlates tFVD with gFVD across only about ten tokenizer configurations, including four closely related V-RAE variants, without uncertainty estimates or a check on unseen tokenizer families. I would treat tFVD as a promising diagnostic rather than an established predictor on the present evidence.

Questions for authors
  1. Can you report GPU hours and peak memory to reach the same gFVD threshold, along with encoding, decoding, and sampling throughput? This would clarify how the reduction in training updates relates to the cost of the higher-dimensional latents.

  2. Can you recompute tFVD without the interpolation involving the VAE’s separately encoded first frame, and report whether its correlation with gFVD remains stronger than rFVD under uncertainty or leave-one-family-out analysis? This would help separate interpolation quality from a difference in tokenizer layout.

2 comments
AuthorsOpenReview ↗

Response Part-1/2

We thank the reviewer for the constructive feedback. We provide additional experiments addressing computational efficiency, evaluation fairness, and the robustness of tFVD.


W1 / Q1: The reported convergence advantage is not yet an end-to-end efficiency result. V-RAE and Wan2.2 VAE have the same temporal and spatial latent grid in the reconstruction comparison, but V-RAE uses 1024 channels where Wan2.2 uses 48. This is about 21 times as many latent values. The matched token count therefore does not imply matched latent storage or total computation. The main generation table also uses a wide DDT head and auxiliary supervision for V-RAE, although the UCF101 ablation partly addresses that difference. The “up to 6× faster” result is measured in optimizer updates; without GPU time, memory, or throughput, its practical size remains unclear. Can you report GPU hours and peak memory to reach the same gFVD threshold, along with encoding, decoding, and sampling throughput? This would clarify how the reduction in training updates relates to the cost of the higher-dimensional latents.

A: We appreciate the reviewer highlighting that matched token counts do not necessarily imply matched computational costs. We therefore provide additional FLOPs, peak-memory, and actual GPU-hour measurements.

DiT computational cost. Although V-RAE uses 1,024 latent channels versus 48 for Wan2.2, both share the same primary DiT backbone and 1,280-token budget. The additional channel-dependent computation primarily affects the input/output projections.

ConfigurationLatent ChannelsDiT GFLOPs/Forward↓Peak Memory↓
Wan2.2 (DiT-XL)482,767.955.75 GiB
V-RAE (DiT-XL + DDT head)1,0242,785.868.38 GiB

DiT FLOPs exclude the additional DDT head and tokenizer. Peak training memory includes the DDT head and is measured at 256×256, 20 frames, with local batch size 16.

Despite 21.3× more latent channels, core DiT forward FLOPs increase by only 0.65%. Peak training memory increases by 22.7%, primarily due to the additional DDT head.

Tokenizer computational cost. We additionally measure encoding and decoding FLOPs at 256×256 resolution, using each tokenizer's native 17-frame or 20-frame configuration. (also shows in Figure 20 in Appendix H.2)

TokenizerEncoding GFLOPs↓Decoding GFLOPs↓Total GFLOPs↓
EUPE-B V-RAE956.5871,195.1252,151.712
AToken-So/C1,272.3521,265.3852,537.737
V-JEPA 2.1-L V-RAE2,211.9191,280.4373,492.356
SigLIP2-L V-RAE3,253.4591,195.8804,449.339
DINOv3-L V-RAE3,318.9391,195.8804,514.819
LTX-2.5 VAE2,776.8082,870.0915,646.898
Wan2.1 VAE5,690.1929,567.54615,257.738
MiniMax H3 VAE8,954.9376,712.17515,667.112
Wan2.2 VAE3,965.51820,781.68324,747.201
Hunyuan VAE10,368.31319,538.48029,906.793
CogVideoX VAE10,020.00820,860.72330,880.731

Practical training efficiency. More importantly, we measure the NVIDIA B300 GPU-hours required to reach matched gFVD thresholds, using FlashAttention-4.

TokenizerK600 GPU-hours↓Speedup↑UCF101 GPU-hours↓Speedup↑
V-RAE (V-JEPA 2.1-L)665.86×325.03×
V-RAE (EUPE-B)636.14×413.93×
V-RAE (DINOv3-L)814.78×542.98×
Wan2.2 VAE3871.00×1611.00×
Wan2.1 VAE4510.86×1850.87×

Target gFVD: 55 on K600 and 155 on UCF101. Speedups are relative to Wan2.2 VAE.

V-RAE achieves 4.78–6.14× GPU-hour efficiency gains on K600 and 2.98–5.03× on UCF101. These measurements demonstrate that our faster convergence translates into substantial practical generation-training efficiency, rather than merely fewer training steps. We do not equate these gains with end-to-end inference throughput.

AuthorsOpenReview ↗

Response Part-2/2

W2 / Q2: The evidence that tFVD predicts generative utility is preliminary. For VAEs with a separately encoded first frame, the first interpolated latent averages a one-frame code with a four-frame code. V-RAE interpolates between codes that each represent four frames. That difference may affect the cross-tokenizer tFVD ranking. Figure 8 then correlates tFVD with gFVD across only about ten tokenizer configurations, including four closely related V-RAE variants, without uncertainty estimates or a check on unseen tokenizer families. I would treat tFVD as a promising diagnostic rather than an established predictor on the present evidence. Can you recompute tFVD without the interpolation involving the VAE’s separately encoded first frame, and report whether its correlation with gFVD remains stronger than rFVD under uncertainty or leave-one-family-out analysis? This would help separate interpolation quality from a difference in tokenizer layout.

A: We thank the reviewer for identifying the potential confound introduced by the separately encoded first frame in causal VAEs. To address this concern, we revise the tFVD protocol to align VAE and V-RAE temporal chunks exactly.

Specifically, we prepend a duplicate of the first video frame for causal VAEs, allowing the special first-frame code to encode this duplicate while the original video frames are grouped into regular four-frame chunks, exactly aligned with V-RAE. Interpolation is performed exclusively between regular chunk latents, excluding the special first-frame code from interpolation endpoints. This removes the direct first-frame interpolation confound and ensures matched temporal chunk boundaries.

For direct comparison with the original results, all V-RAE evaluations below use the same checkpoints as the original main-text experiments, rather than the newly trained decoder variants presented in our responses to Reviewers 1DBz and Aw4x.

Updated tFVD results:

TokenizerUCF101tFVD↓K600tFVD↓
Wan2.1 VAE384.65247.2294
Wan2.2 VAE354.07244.1541
HunyuanVideo326.60205.8400
CogVideoX299.89236.7335
Cosmos247.15144.9133
AToken255.57172.6155
V-RAE (DINOv3-L)148.2684.5785
V-RAE (SigLIP2-L)165.91100.9793
V-RAE (EUPE-B)145.9587.1770
V-RAE (V-JEPA 2.1-L)164.73102.7848

All four V-RAE variants retain substantially lower tFVD than the evaluated conventional tokenizers under the revised protocol.

We further recompute correlations with downstream gFVD across all ten tokenizer configurations, including 95% confidence intervals.

DatasetMetricPearsonr95%CISpearmanρ
UCF101Revised tFVD0.630[0.000, 0.902]0.782
rFVD0.200[−0.491, 0.737]0.430
K600Revised tFVD0.918[0.684, 0.981]0.879
rFVD0.473[−0.223, 0.850]0.576

Pearson 95% confidence intervals are estimated using Fisher's z-transformation (n=10).

Crucially, the revised protocol leaves the pooled tFVD–gFVD correlations nearly unchanged: 0.621 → 0.630 on UCF101 and 0.919 → 0.918 on K600. Both remain numerically stronger than the corresponding rFVD correlations.

These results demonstrate that the observed tFVD–gFVD association is not primarily driven by the special first-frame interpolation.

Reviewer_Aw4xOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

V-RAE combines frozen foundation-model features, temporal attention pooling, and a spatiotemporal decoder into video latents with 4× temporal compression. It improves class-conditional generation FVD across four encoders and introduces tFVD to evaluate decoded temporal interpolations and diagnose generative utility.

Critical strengths
  1. Generation gains across encoders and benchmarks. The experiments cover four frozen encoders, two generation benchmarks, and three probing benchmarks. All four V-RAE variants improve K600 gFVD over the evaluated conventional tokenizers; the best result is 19.16 versus 41.66 (Table 3).

  2. Controls support a contribution from the representation. Appendix C/Figure 12 evaluates both latent spaces under all four combinations of standard/wide-head DiT and auxiliary supervision. With neither addition, V-RAE achieves 138.79 gFVD versus 154.64 for Wan2.2 on UCF101.

Critical weaknesses
  1. FVD gains do not establish detail and temporal fidelity, which are critical in practical video generation (complicated actions, image/video conditioning/reference generation). On K600, V-JEPA2.1 V-RAE achieves 28.48 dB PSNR and 0.110 LPIPS, versus 35.73 dB and 0.049 or Wan2.2 (Table 1). Fine foliage textures also appear unstable in the K600 demo's boy-with-foliage clip (second row, fourth column). TRED doesn't help as it misses equal-amplitude oscillations around a constant target: absolute error stays constant and TRED is zero despite flicker (Appendix E, Eqs. 18–19). The two selected texture regions in Figure 3 do not validate sensitivity to this failure. Input appearance preservation is also untested in image/keyframe/reference-conditioned generation. Figure 11's rFVD/probing ablation does not isolate the frozen encoder/pooler's effect on these fidelity measures.

  2. tFVD's temporal sensitivity and predictive value need separate validation. The paper uses tFVD to assess coherent temporal interpolation and diagnose generative utility (§4.3). No controlled test separates sensitivity to flicker and motion discontinuities from appearance quality. Figure 8's pooled correlation also leaves within-family predictive value unresolved: on K600, V-JEPA2.1 has the highest tFVD among the four V-RAE variants but the lowest gFVD. The pooled result may be driven mainly by differences between tokenizer families. Since tFVD and gFVD share I3D features, their correlation alone does not establish sensitivity to temporal defects.

I have tried almost every metric in the literature that claims to be "predictive" of generation performance. None of them provides any useful information for large-scale T2V (unless the compared system is badly trained and has miserably and unacceptably bad performance). Therefore, I extremely doubt the "predictive value" (if any) of tFVD, unless a thorough systematic analysis is given that actually distinguishes useful aspects of well-trained large-scale systems.

Questions for authors
  1. Compare foliage, grass, and fine-texture flicker against a VAE baseline, and provide paired reconstruction comparisons and generation results for gymnastics and fighting. How does frozen encoder or attention pooling affect these results relative to traditional VAE encoder or convolutional pooling under matched settings?
  2. How well does V-RAE preserve the input in I2V/IT2V, first/last-frame conditioning, and reference image/video generation? Report PSNR/SSIM/LPIPS on constrained frames and identity/appearance consistency in generated frames against a VAE baseline.
  3. Does tFVD detect temporal defects and predict generation quality beyond differences between tokenizer families? Test flicker and motion discontinuities at matched per-frame appearance quality, alongside appearance-only degradations, using temporal judgments independent of I3D. Also report within-family rank correlations and uncertainty, and explain the reversed tFVD/gFVD ordering for V-JEPA2.1.
12 comments
AuthorsOpenReview ↗

Response Part-1/4

We thank the reviewer for the detailed feedback. We have added quantitative fidelity evaluations, controlled architectural comparisons, and extensive qualitative video comparisons.


W1-1: FVD gains do not establish detail and temporal fidelity, which are critical in practical video generation (complicated actions, image/video conditioning/reference generation).

A: We thank the reviewer for this comment. FVD is a widely used metric for video generation, but I3D-based FVD does not fully capture fine details or temporal artifacts. We therefore complement gFVD with reconstruction metrics (LPIPS, PSNR, SSIM) and additional qualitative comparisons of fine textures, temporal stability, and complex actions, as detailed below.


W1-2 / Q1-1: On K600, V-JEPA2.1 V-RAE achieves 28.48 dB PSNR and 0.110 LPIPS, versus 35.73 dB and 0.049 or Wan2.2 (Table 1). Fine foliage textures also appear unstable in the K600 demo's boy-with-foliage clip (second row, fourth column). Compare foliage, grass, and fine-texture flicker against a VAE baseline, and provide paired reconstruction comparisons and generation results for gymnastics and fighting.

A: As discussed in our response to Reviewer 1DBz (W2), V-RAE has lower pixel-level reconstruction fidelity than large-scale VAEs, but achieves better generation quality on both UCF101 and K600, with up to 6× faster convergence. This suggests that reconstruction fidelity alone does not determine generative utility.

To improve reconstruction, we introduce a parameter-matched (~400M) Diffusion Decoder while keeping the frozen encoder and temporal pooler unchanged. Table 1 shows substantial improvements across both datasets. (Details in Appendix G)

Table 1. Reconstruction fidelity (ViT → Diff Decoder).

EncoderDatasetPSNR↑LPIPS↓SSIM↑
V-JEPA 2.1-LUCF10127.82 → 31.680.121 → 0.0860.856 → 0.924
V-JEPA 2.1-LK60028.48 → 31.840.110 → 0.0730.858 → 0.919
EUPE-BUCF10126.76 → 30.660.122 → 0.0950.829 → 0.906
EUPE-BK60027.30 → 30.860.114 → 0.0890.829 → 0.892

The Diffusion Decoder improves PSNR by 3.36–3.90 dB using the same semantic latents. It also improves final gFVD by 0.3–4.6 points while maintaining comparable convergence speed (Fig. 11). These results indicate that the original reconstruction gap is partly due to decoder limitations rather than the latent representations alone.

We have also added the following comparisons in our anonymous repository:

  • Boy-with-foliage: We revisit the reviewer's example and compare the ViT and Diffusion Decoders using identical generated latents to examine fine textures and temporal stability.
  • Foliage, grass, and fine textures: We provide paired reconstructions against Wan2.2 VAE, including temporal artifacts. Fig. 10 further shows improved text and glyph reconstruction with the Diffusion Decoder.
  • Fighting and gymnastics: We add generation examples to examine complex actions, body movements, and temporal coherence.

Please check these out in our anonymous repository: https://anonymous.4open.science/r/v-rae

These results show that the same V-RAE latents can achieve substantially better reconstruction with a stronger decoder, while retaining their generation and convergence advantages.


W1-3: TRED doesn't help as it misses equal-amplitude oscillations around a constant target: absolute error stays constant and TRED is zero despite flicker (Appendix E, Eqs. 18–19). The two selected texture regions in Figure 3 do not validate sensitivity to this failure.

A: We thank the reviewer for pointing out this limitation. Indeed, TRED cannot detect equal-amplitude oscillations, as it measures fluctuations in absolute reconstruction error. We therefore use it as a diagnostic rather than a comprehensive flickering metric.

We also investigated a range of existing temporal consistency metrics, but found that none consistently aligned with human visual judgments. For example, tLP favors frame-wise RAEv2 over Wan2.2 VAE on both UCF101 (0.0077 vs. 0.0215) and K600 (0.0086 vs. 0.0102), despite substantially more visible flickering in the examined RAEv2 reconstructions.

Given the limitations of these metrics, we selected representative video examples for the reviewer to assess flickering directly. These comparisons are available in our anonymous repository and include paired comparisons of 720p videos featuring foliage, branches, grass, and fine text.

These comparisons show the Diffusion Decoder substantially reduces visible flickering in both reconstruction and generation. In the examined cases, its temporal stability is comparable to Wan2.2 VAE.

While TRED cannot capture all forms of flickering, these additional comparisons provide direct visual evidence of the improvements in temporal stability.

AuthorsOpenReview ↗

Response Part-2/4

W1-4 / Q2: Input appearance preservation is also untested in image/keyframe/reference-conditioned generation. How well does V-RAE preserve the input in I2V/IT2V, first/last-frame conditioning, and reference image/video generation? Report PSNR/SSIM/LPIPS on constrained frames and identity/appearance consistency in generated frames against a VAE baseline.

A: We appreciate this constructive suggestion. While dedicated I2V/TI2V and reference-conditioned generation have not yet been evaluated, our Cityscapes future-prediction experiments provide evidence in a conditional generation setting that requires preserving scene structure and temporal dynamics.

Under matched prediction architectures and training settings (§4.4; Appendix B.4), we additionally evaluate future-frame fidelity using PSNR, LPIPS, and SSIM, alongside gFVD.

Table 2. Cityscapes future video prediction.

MethodPSNR↑LPIPS↓SSIM↑gFVD↓
V-RAE (ViT Decoder)21.260.3060.603111.36
V-RAE (Diffusion Decoder)22.820.2890.739103.85
Wan2.2 VAE22.030.3000.676144.47

V-RAE with the Diffusion Decoder outperforms Wan2.2 VAE on all four metrics, achieving better future-frame fidelity and generation quality. Qualitative comparisons (Figs. 9 and 18) also show better subject consistency, object contours, and spatial relationships across predicted frames.

These results demonstrate that V-RAE can support fidelity-sensitive conditional video generation. We will further investigate larger-scale I2V/TI2V, first/last-frame, and reference-conditioned generation in future work.


W1-5: Figure 11's rFVD/probing ablation does not isolate the frozen encoder/pooler's effect on these fidelity measures. How does frozen encoder or attention pooling affect these results relative to traditional VAE encoder or convolutional pooling under matched settings?

A: We thank the reviewer for this suggestion. We conducted additional experiments to examine the effects of encoder choice, temporal compression, and pooling strategy on reconstruction fidelity.

Encoder choice and temporal compression. Under matched settings (Reviewer 1DBz, W1-1, Table 1), DINOv3-L outperforms SigLIP2-L on K600 in PSNR (28.03 vs. 27.05) and LPIPS (0.099 vs. 0.134). Increasing DINOv3-L's temporal compression from 4× to 8× also reduces PSNR (28.03 → 27.11) and increases LPIPS (0.099 → 0.137). These results show that both encoder choice and temporal compression affect reconstruction fidelity.

Attention vs. convolutional pooling. We further compare the two pooling strategies using the same frozen DINOv3-L encoder, RAEv2-initialized chunk-causal decoder, 4× temporal compression, and reconstruction training settings.

Table 3. Reconstruction fidelity with different pooling strategies.

DatasetPoolingLPIPS↓PSNR↑SSIM↑rFVD↓
UCF101Temporal Attn0.11127.760.8586.12
Convolution0.10528.010.8675.91
K600Temporal Attn0.10427.960.8502.76
Convolution0.10127.860.8472.45

Convolutional pooling achieves slightly better reconstruction on UCF101, while the results are mixed on K600. However, temporal attention preserves substantially more semantic information, achieving 89.13% vs. 78.26% probing accuracy (+10.87 points; Appendix B.1, Fig. 13).

These results show that encoder choice and temporal compression affect reconstruction, while temporal attention better preserves semantic information with a small reconstruction trade-off. This supports our use of attention pooling for semantically rich video latents.

AuthorsOpenReview ↗

Response Part-3/4

W2-1: tFVD's temporal sensitivity and predictive value need separate validation. The paper uses tFVD to assess coherent temporal interpolation and diagnose generative utility (§4.3). No controlled test separates sensitivity to flicker and motion discontinuities from appearance quality. Since tFVD and gFVD share I3D features, their correlation alone does not establish sensitivity to temporal defects.

A: We welcome the opportunity to clarify this point. Unlike reconstruction metrics, which evaluate decoded videos at observed latent states, tFVD evaluates the quality of decoded temporal interpolations. Inspired by iFID (Xu et al., 2026) [1], we interpolate between temporally adjacent latent states within the same video, rather than using nearest-neighbor matching across samples. This allows us to assess how well the decoder handles intermediate latent states that are not directly observed during encoding.

Although tFVD and gFVD share I3D features, rFVD uses the same features but shows much weaker correlations with gFVD: 0.200 vs. 0.621 on UCF101 and 0.473 vs. 0.919 on K600 (Fig. 8). This suggests that evaluating temporal interpolations provides useful information beyond reconstruction fidelity.

However, tFVD is not designed to isolate specific temporal artifacts such as flickering or motion discontinuities. Its within-family predictive value also depends on decoder configurations, as further examined in W2-2. We therefore position tFVD as a complementary metric for evaluating temporal interpolation quality, rather than a standalone measure of temporal fidelity or generation performance. We will further investigate its sensitivity to temporal artifacts and its applicability to large-scale T2V generation in future work.


W2-2: Figure 8's pooled correlation also leaves within-family predictive value unresolved: on K600, V-JEPA2.1 has the highest tFVD among the four V-RAE variants but the lowest gFVD. The pooled result may be driven mainly by differences between tokenizer families.

A: We appreciate the reviewer’s careful reading and attention to this detail. To examine the predictive value of tFVD within tokenizer families, we computed Pearson, Spearman, and Kendall correlations between tFVD and gFVD.

Table 4. Within-family correlations between tFVD and gFVD.

FamilyDatasetPearson rSpearman ρKendall τ
V-RAE (n=4)UCF1010.1180.4000.333
K600−0.315−0.400−0.333
Non-V-RAE baselines (n=6)UCF1010.1230.1430.067
K6000.9250.8290.733

The correlations vary across families and datasets. However, we identified a key configuration difference: V-JEPA 2.1 originally uses full spatiotemporal decoder attention, while the other V-RAE variants use chunk-causal attention. We therefore trained V-JEPA 2.1 with a chunk-causal decoder under matched reconstruction settings.

Table 5. Effects of decoder attention on V-JEPA 2.1.

DatasetMetric↓Full AttentionChunk-Causal
UCF101tFVD167.58123.491
gFVD117.86118.77
rFVD6.656.84
K600tFVD104.775583.93
rFVD2.132.67
gFVD19.16—

Chunk-causal attention reduces tFVD by 26.3% on UCF101 and 19.9% on K600, while maintaining similar UCF101 gFVD (117.86 vs. 118.77). Interestingly, rFVD becomes slightly worse on UCF101 (6.65 → 6.84) and K600 (2.13 → 2.67). The K600 gFVD for the chunk-causal variant is not yet available, as it requires an additional full DiT training run (~2 days), which could not be completed within the rebuttal period.

A possible explanation is how decoder attention handles interpolated latents. These latents may deviate from the training distribution. Full attention allows such perturbations to affect frames in both temporal directions, potentially causing temporal inconsistencies. In contrast, chunk-causal attention prevents future latent chunks from affecting earlier frames, limiting the propagation of interpolation errors.

After matching decoder attention, we observe:

  1. UCF101: Within-family Pearson correlation increases from 0.118 to 0.980, Spearman from 0.400 to 1.000, and Kendall from 0.333 to 1.000. The tFVD ranking now exactly matches the gFVD ranking across all four V-RAE variants.
  2. K600: V-JEPA 2.1's tFVD improves from the worst (104.78) to the best (83.93) among the four V-RAE variants, consistent with its original leading gFVD. The chunk-causal gFVD has not yet been evaluated.

These results show that decoder attention contributes to the original ranking inconsistency and that tFVD depends on the complete encoder–decoder system. The improved within-family alignment under matched settings supports tFVD's potential for comparing latent representations, although further validation with more encoders and larger-scale generation is needed.

Reviewer_1DBzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

V-RAE extends representation autoencoders to video: a frozen vision foundation model (DINOv3-L, SigLIP2-L, EUPE-B, or V-JEPA 2.1-L), a small attention pooler for 4× temporal compression, and a trained decoder. With DiT-XL, all four variants achieve lower class-conditional gFVD than six pretrained video VAEs on UCF101 and K600 and converge up to 6× faster than Wan2.2 VAE. The paper also proposes tFVD (FVD of videos decoded from temporally interpolated latents) as a generator-free proxy for generation quality.

Critical strengths
  • Simple design with consistent generation gains across four encoders and two datasets.
  • Broad evaluation (reconstruction, probing, generation, future prediction), useful ablations, and candid limitations.
Critical weaknesses
  • W1. Encoder inductive biases are invoked but not analyzed. The paper attributes dataset-dependent reconstruction rankings to each encoder's "spatiotemporal inductive bias" (L234–237) without analysis, and the variants also differ in model size, tubelet/pooling ratio, decoder attention and initialization, and training length (Table 5, App. B.2). Which semantics help generation is also unclear: the variants' UCF101 linear-probe ranking is the exact reverse of their UCF101 gFVD ranking (SigLIP2-L: best probe, 90.92%, but worst gFVD, 142.60; V-JEPA 2.1-L: the opposite, 86.65% and 117.86), and V-JEPA 2.1-L leads only on the motion-centric SSv2 probe (72.91%). The kind of semantics may matter more than probe accuracy, which qualifies FINDING 2.

  • W2. Reconstruction fidelity is far below the baselines. Every variant is worse than all six large-scale tokenizers on LPIPS, PSNR, and SSIM (Table 1). PSNR is 6.3–8.7 dB below Wan2.2 VAE, near the benchmark-scale tokenizers (Open-MAGVIT2: 27.74 dB), although the latent has ~10–17× more values than those of Wan2.1/2.2. On Cityscapes, rFVD is ~4× worse than Wan2.2 (29.29 vs. 7.03; Table 4). The text understates this ("trails some" VAEs, L238) and does not discuss fidelity-critical uses such as image-to-video or editing.

  • W3. Faster convergence may partly reflect an easier, lower-fidelity target; the quality ceiling is untested. Because fine detail is discarded (W2), part of the faster convergence and lower gFVD (Sec. 4.2) may come from an easier target rather than from semantics. Among the six VAEs, lower PSNR goes with better gFVD (PSNR–gFVD Pearson r = 0.88 on K600, 0.73 on UCF101; n = 6), although the trend reverses within V-RAE. gFVD uses high-level I3D features and is not designed to capture fine detail, and all experiments use one scale (DiT-XL, 256×256, 17 frames), which App. G acknowledges. The VAE curves plateau or worsen within the budget (Figs. 1 and 5), so a lower ceiling is not shown, but it is not ruled out either.

Questions for authors

Please refer to the Weakness Section and address concerns

6 comments
AuthorsOpenReview ↗

Response Part-1/4

We thank the reviewer for the detailed feedback. Our central finding is that reconstruction fidelity alone does not determine generative utility. We address the concerns through controlled encoder comparisons, motion-sensitive probing, improved decoding, and checkpoint-level convergence analysis.


W1-1: Encoder inductive biases are invoked but not analyzed. The paper attributes dataset-dependent reconstruction rankings to each encoder's "spatiotemporal inductive bias" (L234–237) without analysis, and the variants also differ in model size, tubelet/pooling ratio, decoder attention and initialization, and training length (Table 5, App. B.2).

A: The original configurations were optimized for each encoder's reconstruction performance rather than strictly matched across all settings. The main differences are:

  • Model size (Three encoders use comparable ViT-L/16 backbones; EUPE-B uses ViT-B/16. All use a ~420M-parameter decoder.)
  • Temporal compression (4× overall temporal compression)
  • Decoder attention (Frame-wise encoders use chunk-causal attention; V-JEPA 2.1-L uses full attention.)
  • Decoder initialization & Training length (DINOv3-L and EUPE-B decoders use RAEv2 initialization; others train from scratch, with 40–50 epochs depending on initialization and convergence.)

To isolate these factors, we conducted matched experiments in Table 9 (rows 2, 4, 5, 7, and 9), using scratch-trained chunk-causal decoders with identical 4× temporal compression, training data (UCF101 + K600), batch size (128), training budget (50 epochs), and GAN schedule. These controls remove decoder configuration and training differences, enabling a more direct comparison of encoder effects.

Table 1. Reconstruction under the original and matched settings.

No.EncoderCausalInit.Comp.UCF101LPIPS↓PSNR↑SSIM↑rFVD↓K600LPIPS↓PSNR↑SSIM↑rFVD↓
1DINOv3-L✓RAEv24×0.11127.760.8586.120.10427.960.8502.76
2DINOv3-L✓Scratch4×0.11927.540.8606.310.09928.030.8622.72
3DINOv3-L✓Scratch8×0.14527.030.8279.120.13727.110.8133.25
4DINOv3-S✓Scratch4×0.12127.980.8327.210.11827.980.8712.03
5SigLIP2-L✓Scratch4×0.14226.410.8159.830.13427.050.8163.38
6EUPE-B✓RAEv24×0.12226.760.8298.050.11427.300.8293.36
7EUPE-B✓Scratch4×0.11327.030.8137.790.11227.310.8213.21
8V-JEPA 2.1-L✗Scratch4×0.12127.820.8566.650.11028.480.8582.13
9V-JEPA 2.1-L✓Scratch4×0.12027.350.8446.840.11228.100.8402.67

The results show three main findings:

  1. Encoder choice matters. Under matched settings, DINOv3-L outperforms SigLIP2-L on both datasets (rows 2, 5), despite their similar frame-wise ViT-L architectures. This suggests that pretrained representations affect reconstruction quality.

  2. Temporal compression also matters. Increasing compression from 4× to 8× degrades DINOv3-L reconstruction on both datasets (rows 2–3), showing the importance of preserving temporal information.

  3. Other factors have limited effects on the overall trends. Chunk-causal attention increases V-JEPA 2.1-L's K600 rFVD from 2.13 to 2.67, but it still slightly outperforms DINOv3-L (2.72) under matched settings (rows 8–9). Changes in initialization and training budget yield relatively small differences (rows 1–2, 6–7). Encoder scaling also shows no consistent benefit: DINOv3-S outperforms DINOv3-L on K600 (2.03 vs. 2.72) but trails on UCF101 (7.21 vs. 6.31), consistent with Image RAE [1].

The results shows that encoder choice and temporal compression are key factors in reconstruction quality, while other training configurations have relatively minor effects.

AuthorsOpenReview ↗

Response Part-2/4

W1-2: Which semantics help generation is also unclear: the variants' UCF101 linear-probe ranking is the exact reverse of their UCF101 gFVD ranking (SigLIP2-L: best probe, 90.92%, but worst gFVD, 142.60; V-JEPA 2.1-L: the opposite, 86.65% and 117.86), and V-JEPA 2.1-L leads only on the motion-centric SSv2 probe (72.91%). The kind of semantics may matter more than probe accuracy, which qualifies FINDING 2.

A: We appreciate this constructive suggestion and we agree that the type of semantics may matter more than classification accuracy alone, particularly for temporal understanding.

Prior studies have shown that UCF101 and K400 contain substantial static appearance biases [2], while SSv2 and Diving48 rely more on temporal dynamics [2, 3]. TimeSformer [4] also shows that space-time attention brings a much larger improvement on SSv2 (36.6% → 59.5%) than on K400 (76.9% → 78.0%).

Motivated by this distinction, we added Diving48 probing and compared all four benchmarks with generation performance on UCF101 and K600.

Table 2. Semantic probing and generation performance across V-RAE encoders.

V-RAE EncoderUCF101↑K400↑SSv2↑Diving48↑UCF101 gFVD↓K600 gFVD↓
DINOv3-L89.13 / 88.9483.12 / 84.3666.55 / 67.1982.84 / 83.50131.4030.09
SigLIP2-L90.92 / 90.8882.56 / 83.9165.39 / 66.0881.22 / 81.88142.6034.48
EUPE-B90.16 / 89.9382.21 / 83.3165.67 / 66.2183.86 / 84.82125.9824.77
V-JEPA 2.1-L86.65 / 86.6580.70 / 81.8772.91 / 73.5085.38 / 86.40117.8619.16

Probing accuracy is reported as Single / TTA Top-1 (%).

Across the four encoders, Spearman's ρ between probing accuracy and gFVD is +0.8 (UCF101), +0.8 (K400), −0.8 (SSv2), and −1.0 (Diving48) on both generation datasets.

Motion-sensitive probing shows a negative correlation with gFVD, while appearance-biased probing shows the opposite trend. In particular, V-JEPA 2.1-L achieves the best SSv2 and Diving48 probing accuracy and the lowest gFVD on both datasets, despite having the lowest UCF101 probing accuracy.

These results suggest that temporal semantics are more relevant to video generation than generic classification accuracy. We have therefore revised Finding 2 as follows:

Finding 2: Video generation may benefit more from temporally informative latents than from static semantics alone.

This observation is based on four encoders and does not establish causality.

AuthorsOpenReview ↗

Response Part-3/4

W2: Reconstruction fidelity is far below the baselines. Every variant is worse than all six large-scale tokenizers on LPIPS, PSNR, and SSIM (Table 1). PSNR is 6.3–8.7 dB below Wan2.2 VAE, near the benchmark-scale tokenizers (Open-MAGVIT2: 27.74 dB), although the latent has ~10–17× more values than those of Wan2.1/2.2. On Cityscapes, rFVD is ~4× worse than Wan2.2 (29.29 vs. 7.03; Table 4). The text understates this ("trails some" VAEs, L238) and does not discuss fidelity-critical uses such as image-to-video or editing.

A: Thanks for highlighting the reconstruction gap. We acknowledge that V-RAE has lower pixel-level reconstruction fidelity than large-scale VAEs. However, our key finding is that better reconstruction does not necessarily lead to better generation.

For example, HunyuanVideo VAE achieves better LPIPS, PSNR, and SSIM than CogVideoX VAE, but worse UCF101 gFVD (211.53 vs. 159.20; Tables 1 and 3). Similarly, V-RAE achieves the best generation performance among the evaluated tokenizers despite its lower reconstruction fidelity. We have also added detailed frame-level comparisons of UCF101 generations using Wan2.2 VAE, HunyuanVideo VAE, EUPE V-RAE, and V-JEPA V-RAE in Appendix Fig. 17. Despite its strong reconstruction performance, HunyuanVideo exhibits severe object deformation and geometric distortions in generated videos, further illustrating the gap between reconstruction fidelity and generation quality.

This is also evident in Cityscapes future prediction. Although V-RAE has a much higher reconstruction rFVD than Wan2.2 VAE (29.29 vs. 7.03), it achieves better gFID (11.52 vs. 15.02) and gFVD (111.36 vs. 144.47). Fig. 9 further shows better subject consistency and spatial relationships.

But we still agree that reconstruction fidelity remains important, especially for i2v generation and video editing. One possible reason for V-RAE's lower fidelity is the limited decoding capability of the original ViT decoder. To investigate this, we introduce a parameter-matched (~400M) Diffusion Decoder while keeping the frozen encoder and temporal pooler unchanged. (Please refer to Appendix G for more information about architecture of diffusion decoder). V-RAE's semantically rich latents are well suited for conditioning a diffusion decoder.

Table 3. Reconstruction fidelity (ViT → Diffusion Decoder).

EncoderDatasetPSNR↑LPIPS↓SSIM↑
V-JEPA 2.1-LUCF10127.82 → 31.680.121 → 0.0860.856 → 0.924
V-JEPA 2.1-LK60028.48 → 31.840.110 → 0.0730.858 → 0.919
EUPE-BUCF10126.76 → 30.660.122 → 0.0950.829 → 0.906
EUPE-BK60027.30 → 30.860.114 → 0.0890.829 → 0.892

Table 3 shows that the Diffusion Decoder improves reconstruction PSNR by 3.36–3.90 dB, with improvements in LPIPS and SSIM across both datasets. We further evaluate its effect on Cityscapes future prediction in Table 4.

Table 4. Cityscapes future prediction.

MethodPSNR↑LPIPS↓SSIM↑gFVD↓
V-RAE (ViT Decoder)21.260.3060.603111.36
V-RAE (Diffusion Decoder)22.820.2890.739103.85
Wan2.2 VAE22.030.3000.676144.47

With the Diffusion Decoder, V-RAE surpasses Wan2.2 VAE on all four Cityscapes metrics, improving both future-frame prediction fidelity and generation quality without changing the latent representation.

We also provide paired reconstruction comparisons of the ViT Decoder, Diffusion Decoder, and Wan2.2 VAE on challenging scenes with dense foliage, grass, and fine text.

Check it out in our anonymous repository: https://anonymous.4open.science/r/v-rae

The revised manuscript includes the Diffusion Decoder architecture.

These results suggest that the original ViT decoder does not fully exploit the fine-grained information preserved in V-RAE's semantically rich latents. More importantly, a well-structured latent space provides a strong foundation for generation, while a more capable decoder can further improve both reconstruction fidelity and generation quality. Conversely, high reconstruction fidelity alone does not guarantee strong generation when the underlying latent space is less suitable for generative modeling.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

Dear reviewers,

We thank all reviewers for their thoughtful and constructive feedback. We have conducted additional experiments and revised the manuscript accordingly. We would also like to reiterate the main contributions of our work:

  1. V-RAE: We introduce a video representation autoencoder combining frozen VFMs, lightweight temporal pooling, and spatiotemporal decoding to produce compact, semantically rich video latents.
  2. tFVD: We introduce a temporal interpolation diagnostic that correlates more strongly with generation quality than reconstruction fidelity.
  3. Empirical findings: V-RAE achieves competitive reconstruction, superior generation with up to 6× faster convergence, and improved future video prediction.

Our experiments also highlight two key findings:

(1) Better video reconstruction does not guarantee better video generation.

(2) Temporally informative semantics matter more for video generation than static semantics alone.

We would also like to address several common concerns raised by the reviewers:

Concern-1: Reconstruction fidelity and temporal quality.

  • A: We introduce a Diffusion Decoder that improves PSNR by 3.36–3.90 dB and further reduces gFVD by 0.3–4.6 points, while maintaining similar convergence speed. We also provide additional visual comparisons covering fine textures, temporal flickering, and complex motion. On Cityscapes future prediction, V-RAE with diffusion decoding outperforms Wan2.2 VAE across all four reported metrics. We have also added many reconstruction/generation comparisons in our anonymous repo, please check these out in our anonymous repository: https://anonymous.4open.science/r/v-rae

Concern-2: Encoder representations and architectural factors.

  • A: We conduct additional controlled experiments on encoder choice, temporal compression, decoder configurations, and pooling strategies. Further SSv2 and Diving48 probing results suggest that motion-related semantics are more relevant to generation quality than generic classification accuracy.

Concern-3: tFVD evaluation protocol and reliability.

  • A: We correct the evaluation protocol to align temporal chunks across tokenizers and eliminate first-frame-specific effects in causal VAEs. The tFVD–gFVD correlations remain stable (0.621 → 0.630 on UCF101; 0.919 → 0.918 on K600). We also identify decoder attention as a key factor affecting V-JEPA 2.1's tFVD, with matched chunk-causal attention raising the UCF101 within-family Pearson correlation from 0.118 to 0.980. We acknowledge that further validation at larger generation scales remains necessary.

Concern-4: Practical training efficiency.

  • A: We provide additional FLOPs, memory, tokenizer cost, and GPU-hour measurements. At matched gFVD thresholds, V-RAE achieves up to 6.14× GPU-hour speedup over Wan2.2 VAE, confirming that faster convergence translates into practical training savings.

We have incorporated these additional experiments, analyses, and clarifications into the revised manuscript. Our main codebase, including training and evaluation pipelines, is already available in the anonymized repository. We will also release the Diffusion Decoder and remaining model checkpoints to support reproducibility.

Beyond the current study, we are extending V-RAE toward larger-scale T2V, I2V, TI2V, and omni-modal video generation.

We appreciate the reviewers' time and feedback, and welcome further discussion. If any questions or concerns remain, we would be happy to address them with additional evidence or clarification.

AuthorsOpenReview ↗

Update on Three Missing Reconstruction Examples

Dear Reviewers,

Due to an upload issue, three reconstruction examples were missing from our anonymous repository. They have now been added, and all examples are accessible. You can now review the complete set for a more comprehensive visual assessment of reconstruction quality. We apologize for the inconvenience.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.