- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes VGGA, a geometry-grounded video autoencoder designed to produce more generation-friendly latent representations. In addition to RGB reconstruction, VGGA introduces depth, optical-flow, and projective-warping objectives through a partially shared decoder. Training uses pseudo depth and flow labels on general videos and camera-based warping supervision on RealEstate10K. Experiments on RealEstate10K, UCF101, and Taichi-HD show substantially improved downstream generation while largely preserving reconstruction quality.
Critical strengths
- Well-motivated geometric perspective. Viewing videos as projections of dynamic 3D scenes provides a clear and meaningful motivation for incorporating geometry into video autoencoding.
- Simple and effective design. The shared decoder integrates depth, motion, and projective-warping supervision naturally while largely preserving reconstruction quality.
- Strong empirical validation. VGGA consistently improves generation across multiple datasets, and the controlled baseline and ablation studies help demonstrate the contribution of each geometric objective.
Critical weaknesses
- Limited reproducibility of the training setup. The curated 10M-video dataset is not described in sufficient detail, making it difficult to reproduce the results or assess the influence of data composition.
- Potentially non-independent geometric evaluation. The latent probes are evaluated using VGGT and RAFT pseudo-labels that are also used during training, so this evidence may partly reflect alignment with the teacher models.
- Limited long-range geometric modeling. The proposed objectives mainly capture frame-level or short-range consistency and do not explicitly address long-term dynamics or complete 3D scene structure.
Questions for authors
- Can the authors provide more information about the 10M-video dataset and clarify whether it, or a reproducible substitute, will be available?
- Are all VAE comparisons in Table 1 conducted with identical downstream generator architectures, training schedules, and data?
- Can geometric representation quality be evaluated using ground-truth geometry or teachers different from those used to construct the training targets?
2 comments
Response to Reviewer 8U6F (2/2)
W2&Q3. Potentially non-independent geometric evaluation. The latent probes are evaluated using VGGT and RAFT pseudo-labels that are also used during training, so this evidence may partly reflect alignment with the teacher models.
We address the points raised below.
(1) Geometric probing under joint reconstruction and grounding. The existing probes in Figure 6(a) aim to evaluate whether geometric supervision makes structural and motion information more accessible from the learned VAE latents. Specifically, we freeze the autoencoders, train linear depth and optical-flow probes, and evaluate them on the RealEstate10K test split using VGGT and RAFT pseudo-labels as reference targets. Compared with VGGA w/o Geo, VGGA improves depth from 0.355 to 0.813 and reduces the flow outlier rate from 56.4% to 23.8%. These results show that geometric grounding makes the supervised geometric information more linearly accessible from the VAE latents.
(2) Ground-truth depth and optical-flow probing on KITTI. To evaluate the learned geometric representations independently of the pseudo-label teachers used during training, we conduct additional depth and optical-flow probing experiments on KITTI [1] using ground-truth annotations. Specifically, we freeze the encoders of VGGA and VGGA w/o Geo, and train identical linear probes on their respective latent representations using the corresponding training splits. Both methods follow the same probing architecture, training data, and optimization settings, and are evaluated on the corresponding test splits.
Table r2. Latent depth probing on KITTI.
| Method | KITTI ( ) |
|---|---|
| VGGA w/o Geo | 0.11 |
| VGGA | 0.62 |
Table r3. Latent optical-flow probing on KITTI.
| Method | KITTI (F1-all (%) ) |
|---|---|
| VGGA w/o Geo | 57.3 |
| VGGA | 29.5 |
As shown in Tables r2 and r3, VGGA substantially outperforms the reconstruction-only baseline on both depth and optical-flow probing tasks. These improvements against ground-truth annotations demonstrate that VGGA learns more geometrically informative latent representations, making scene structure and motion correspondence more linearly accessible beyond simply matching the pseudo-label teachers (VGGT and RAFT) used during training.
We have reported the above experimental results in Figure 2.
W3 Limited long-range geometric modeling. The proposed objectives mainly capture frame-level or short-range consistency and do not explicitly address long-term dynamics or complete 3D scene structure.
We agree with the reviewer that VGGA primarily focuses on frame-level and short-range geometric consistency. We have explicitly acknowledged this limitation in Sec. 5 (Limitations): "VGGA focuses on frame-level and short-range geometric consistency, without explicitly modeling long-horizon dynamics or full 3D scene structure". Our work aims to demonstrate that incorporating geometric supervision improves video latent representations and downstream generation while preserving reconstruction fidelity. Extending geometric grounding to long-range temporal dynamics and more comprehensive 3D scene modeling remains an important direction for future work, as also discussed in Sec. 5.
Q2. Are all VAE comparisons in Table 1 conducted with identical downstream generator architectures, training schedules, and data?
Yes. All VAE comparisons in Table 1 use the same downstream generator backbone, training schedule, optimization strategy, and benchmark-specific training data. For each VAE, we freeze its parameters and independently train a Latte-XL/1 [2] generator on its latents. Table r4 summarizes the downstream configuration. We have included this configuration table in Appendix A.4 for clarity.
Table r4. Downstream generation settings used across VAE comparisons.
| Setting | Configuration |
|---|---|
| Architecture | |
| Generator backbone | Latte-XL/1 [2] |
| Num. layers | 28 |
| Hidden dim. | 1,152 |
| Num. heads | 16 |
| Patch size | |
| Optimization | |
| Training iterations | 250K |
| Optimizer | AdamW |
| lr | |
| GPUs | 16 H100 GPUs |
| Diffusion training | |
| Training objective | v-prediction |
| Path type | Linear |
| Sampler | Euler |
| Sampling steps | 250 |
| Data and evaluation | |
| Training datasets | RealEstate10K, UCF101, and Taichi-HD, respectively |
| Video length and resolution |
[1] Vision meets robotics: The kitti dataset, 2013.
[2] Latte: Latent Diffusion Transformer for Video Generation, TMLR 2025
Response to Reviewer 8U6F (1/2)
We sincerely thank the reviewer for the valuable comments. We address each concern below and have revised the manuscript accordingly, with all newly added content highlighted in blue.
W1&Q1 Reproducibility of the training setup. The curated 10M-video dataset is not described in sufficient detail, making it difficult to reproduce the results or assess the influence of data composition.
(1) Dataset collection and curation. Our training corpus consists of 10M publicly accessible online videos collected under appropriate licenses, primarily from YouTube and TikTok. We adopt a multi-stage data curation pipeline to ensure video quality, temporal continuity, and data diversity:
(2) Data availability. Due to organizational confidentiality restrictions, we are currently unable to release our curated 10M-video corpus. We expect these restrictions to remain in place for approximately six months and will make the dataset publicly available once permission is granted. To facilitate reproducibility, we will release the VGGA model weights and complete training and inference code upon acceptance.
(3) Reproducibility with a public dataset (reproducible substitute). To further support reproducibility and investigate the influence of training data composition, we conduct additional experiments using the publicly available OpenVid-1M dataset as a substitute for our curated 10M-video corpus. Specifically, we repeat the ablation study on the proportion of warp-supervised samples (Figure 5(c)) using OpenVid-1M and RealEstate10K, while keeping all other training settings fixed. The results are reported below.
Table r1. Ablation study of warp-supervised sample proportions using OpenVid-1M.
The experiments on OpenVid-1M achieve comparable generation performance to our original 10M-video corpus (67.42 of Table r1 vs. 64.31 of Figure 6(c)) and exhibit a similar trend across different warp-supervision ratios. In particular, both datasets achieve their best generation performance at approximately 20% warp-supervised samples, supporting the effectiveness of our default sampling strategy. These results further demonstrate that VGGA generalizes well across different training datasets, with a consistent benefit from geometric grounding. We hope these experiments provide a publicly reproducible reference for evaluating VGGA and investigating the influence of training data composition.
We have included dataset details in Appendix A.5 and reported the experimental results on OpenVid-1M in Appendix C.6.
[1] Vision meets robotics: The kitti dataset, 2013.
[2] Latte: Latent Diffusion Transformer for Video Generation, TMLR 2025