- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes V-RAE, a video autoencoder built on frozen visual foundation models. It combines features from multiple encoder layers, compresses them with local temporal attention, and trains a video decoder to reconstruct the input. A class-conditional DiT is then trained in the resulting latent space. The authors test four representation encoders and evaluate reconstruction, semantic probing, and generation on UCF101 and K600. They also propose tFVD, which measures videos decoded from temporally interpolated latents, and include a future-prediction experiment on Cityscapes.
Critical strengths
-
The temporal pooler addresses a real trade-off. In the controlled pooling comparison, attention pooling reaches 6.12 rFVD and 89.13% UCF101 probing accuracy. Convolution gives slightly lower rFVD, 5.91, but much lower probing accuracy, 78.26%. This is useful evidence that the proposed pooler preserves more of the frozen encoder’s representation while remaining reconstructable.
-
Several V-RAE variants show substantial gains in the reported class-conditional gFVD results. The main table uses different output heads, but Appendix C checks this issue on UCF101. With neither the wide DDT head nor auxiliary supervision, V-RAE obtains 138.79 gFVD against 154.64 for Wan2.2 VAE. That comparison gives me more confidence that the latent space itself contributes to the gain.
Critical weaknesses
-
The reported convergence advantage is not yet an end-to-end efficiency result. V-RAE and Wan2.2 VAE have the same temporal and spatial latent grid in the reconstruction comparison, but V-RAE uses 1024 channels where Wan2.2 uses 48. This is about 21 times as many latent values. The matched token count therefore does not imply matched latent storage or total computation. The main generation table also uses a wide DDT head and auxiliary supervision for V-RAE, although the UCF101 ablation partly addresses that difference. The “up to 6× faster” result is measured in optimizer updates; without GPU time, memory, or throughput, its practical size remains unclear.
-
The evidence that tFVD predicts generative utility is preliminary. For VAEs with a separately encoded first frame, the first interpolated latent averages a one-frame code with a four-frame code. V-RAE interpolates between codes that each represent four frames. That difference may affect the cross-tokenizer tFVD ranking. Figure 8 then correlates tFVD with gFVD across only about ten tokenizer configurations, including four closely related V-RAE variants, without uncertainty estimates or a check on unseen tokenizer families. I would treat tFVD as a promising diagnostic rather than an established predictor on the present evidence.
Questions for authors
-
Can you report GPU hours and peak memory to reach the same gFVD threshold, along with encoding, decoding, and sampling throughput? This would clarify how the reduction in training updates relates to the cost of the higher-dimensional latents.
-
Can you recompute tFVD without the interpolation involving the VAE’s separately encoded first frame, and report whether its correlation with gFVD remains stronger than rFVD under uncertainty or leave-one-family-out analysis? This would help separate interpolation quality from a difference in tokenizer layout.
2 comments
Response Part-2/2
W2 / Q2: The evidence that tFVD predicts generative utility is preliminary. For VAEs with a separately encoded first frame, the first interpolated latent averages a one-frame code with a four-frame code. V-RAE interpolates between codes that each represent four frames. That difference may affect the cross-tokenizer tFVD ranking. Figure 8 then correlates tFVD with gFVD across only about ten tokenizer configurations, including four closely related V-RAE variants, without uncertainty estimates or a check on unseen tokenizer families. I would treat tFVD as a promising diagnostic rather than an established predictor on the present evidence. Can you recompute tFVD without the interpolation involving the VAE’s separately encoded first frame, and report whether its correlation with gFVD remains stronger than rFVD under uncertainty or leave-one-family-out analysis? This would help separate interpolation quality from a difference in tokenizer layout.
A: We thank the reviewer for identifying the potential confound introduced by the separately encoded first frame in causal VAEs. To address this concern, we revise the tFVD protocol to align VAE and V-RAE temporal chunks exactly.
Specifically, we prepend a duplicate of the first video frame for causal VAEs, allowing the special first-frame code to encode this duplicate while the original video frames are grouped into regular four-frame chunks, exactly aligned with V-RAE. Interpolation is performed exclusively between regular chunk latents, excluding the special first-frame code from interpolation endpoints. This removes the direct first-frame interpolation confound and ensures matched temporal chunk boundaries.
For direct comparison with the original results, all V-RAE evaluations below use the same checkpoints as the original main-text experiments, rather than the newly trained decoder variants presented in our responses to Reviewers 1DBz and Aw4x.
Updated tFVD results:
| Tokenizer | UCF101tFVD↓ | K600tFVD↓ |
| Wan2.1 VAE | 384.65 | 247.2294 |
| Wan2.2 VAE | 354.07 | 244.1541 |
| HunyuanVideo | 326.60 | 205.8400 |
| CogVideoX | 299.89 | 236.7335 |
| Cosmos | 247.15 | 144.9133 |
| AToken | 255.57 | 172.6155 |
| V-RAE (DINOv3-L) | 148.26 | 84.5785 |
| V-RAE (SigLIP2-L) | 165.91 | 100.9793 |
| V-RAE (EUPE-B) | 145.95 | 87.1770 |
| V-RAE (V-JEPA 2.1-L) | 164.73 | 102.7848 |
All four V-RAE variants retain substantially lower tFVD than the evaluated conventional tokenizers under the revised protocol.
We further recompute correlations with downstream gFVD across all ten tokenizer configurations, including 95% confidence intervals.
| Dataset | Metric | Pearsonr | 95%CI | Spearmanρ |
| UCF101 | Revised tFVD | 0.630 | [0.000, 0.902] | 0.782 |
| rFVD | 0.200 | [−0.491, 0.737] | 0.430 | |
| K600 | Revised tFVD | 0.918 | [0.684, 0.981] | 0.879 |
| rFVD | 0.473 | [−0.223, 0.850] | 0.576 |
Pearson 95% confidence intervals are estimated using Fisher's z-transformation (n=10).
Crucially, the revised protocol leaves the pooled tFVD–gFVD correlations nearly unchanged: 0.621 → 0.630 on UCF101 and 0.919 → 0.918 on K600. Both remain numerically stronger than the corresponding rFVD correlations.
These results demonstrate that the observed tFVD–gFVD association is not primarily driven by the special first-frame interpolation.
Response Part-1/2
We thank the reviewer for the constructive feedback. We provide additional experiments addressing computational efficiency, evaluation fairness, and the robustness of tFVD.
W1 / Q1: The reported convergence advantage is not yet an end-to-end efficiency result. V-RAE and Wan2.2 VAE have the same temporal and spatial latent grid in the reconstruction comparison, but V-RAE uses 1024 channels where Wan2.2 uses 48. This is about 21 times as many latent values. The matched token count therefore does not imply matched latent storage or total computation. The main generation table also uses a wide DDT head and auxiliary supervision for V-RAE, although the UCF101 ablation partly addresses that difference. The “up to 6× faster” result is measured in optimizer updates; without GPU time, memory, or throughput, its practical size remains unclear. Can you report GPU hours and peak memory to reach the same gFVD threshold, along with encoding, decoding, and sampling throughput? This would clarify how the reduction in training updates relates to the cost of the higher-dimensional latents.A: We appreciate the reviewer highlighting that matched token counts do not necessarily imply matched computational costs. We therefore provide additional FLOPs, peak-memory, and actual GPU-hour measurements.
DiT computational cost. Although V-RAE uses 1,024 latent channels versus 48 for Wan2.2, both share the same primary DiT backbone and 1,280-token budget. The additional channel-dependent computation primarily affects the input/output projections.
DiT FLOPs exclude the additional DDT head and tokenizer. Peak training memory includes the DDT head and is measured at 256×256, 20 frames, with local batch size 16.
Despite 21.3× more latent channels, core DiT forward FLOPs increase by only 0.65%. Peak training memory increases by 22.7%, primarily due to the additional DDT head.
Tokenizer computational cost. We additionally measure encoding and decoding FLOPs at 256×256 resolution, using each tokenizer's native 17-frame or 20-frame configuration. (also shows in Figure 20 in Appendix H.2)
Practical training efficiency. More importantly, we measure the NVIDIA B300 GPU-hours required to reach matched gFVD thresholds, using FlashAttention-4.
Target gFVD: 55 on K600 and 155 on UCF101. Speedups are relative to Wan2.2 VAE.
V-RAE achieves 4.78–6.14× GPU-hour efficiency gains on K600 and 2.98–5.03× on UCF101. These measurements demonstrate that our faster convergence translates into substantial practical generation-training efficiency, rather than merely fewer training steps. We do not equate these gains with end-to-end inference throughput.