acceptodds
Under review as a conference paper at ICLR 2027

VGGA: Geometry-Grounded Autoencoder for Video Generation

Abstract

Autoencoders are the core component of latent video generative models, compressing high-dimensional videos into compact latent spaces for downstream generation. However, existing video autoencoders are primarily optimized for pixel-level reconstruction, providing limited supervision for learning structured latent representations. We argue that videos should not be treated merely as signals to be reconstructed, but rather as 2D projections of dynamic 3D worlds that exhibit explicit and well-structured geometric properties. Building on this insight, we propose Video Geometry Grounded Autoencoder (VGGA), a geometry-aware video autoencoder that explicitly incorporates geometric constraints into the reconstruction process. Specifically, VGGA grounds the learned representations along three complementary geometric dimensions: Structural Depth, Motion Correspondence, and Projective Warping. These objectives are unified within a Video Geometry Decoder, which employs a shared spatio-temporal upsampling trunk to jointly optimize geometric learning and RGB reconstruction. Through these geometry-grounded objectives, VGGA encourages the latent representations to capture the underlying 3D structure and motion of dynamic scenes. VGGA consistently improves video generation across both geometry-centric and human-motion domains, reducing gFVD by 28.1% on RealEstate10K and 37.7% on Taichi-HD. Notably, these substantial generation improvements are achieved while maintaining comparable reconstruction quality, demonstrating that geometry-grounded representation learning yields a more effective latent space for video generation.

Primary area:
generative models
Keywords:
Video Autoencoder, Video Generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8U6FOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VGGA, a geometry-grounded video autoencoder designed to produce more generation-friendly latent representations. In addition to RGB reconstruction, VGGA introduces depth, optical-flow, and projective-warping objectives through a partially shared decoder. Training uses pseudo depth and flow labels on general videos and camera-based warping supervision on RealEstate10K. Experiments on RealEstate10K, UCF101, and Taichi-HD show substantially improved downstream generation while largely preserving reconstruction quality.

Critical strengths
  1. Well-motivated geometric perspective. Viewing videos as projections of dynamic 3D scenes provides a clear and meaningful motivation for incorporating geometry into video autoencoding.
  2. Simple and effective design. The shared decoder integrates depth, motion, and projective-warping supervision naturally while largely preserving reconstruction quality.
  3. Strong empirical validation. VGGA consistently improves generation across multiple datasets, and the controlled baseline and ablation studies help demonstrate the contribution of each geometric objective.
Critical weaknesses
  1. Limited reproducibility of the training setup. The curated 10M-video dataset is not described in sufficient detail, making it difficult to reproduce the results or assess the influence of data composition.
  2. Potentially non-independent geometric evaluation. The latent probes are evaluated using VGGT and RAFT pseudo-labels that are also used during training, so this evidence may partly reflect alignment with the teacher models.
  3. Limited long-range geometric modeling. The proposed objectives mainly capture frame-level or short-range consistency and do not explicitly address long-term dynamics or complete 3D scene structure.
Questions for authors
  1. Can the authors provide more information about the 10M-video dataset and clarify whether it, or a reproducible substitute, will be available?
  2. Are all VAE comparisons in Table 1 conducted with identical downstream generator architectures, training schedules, and data?
  3. Can geometric representation quality be evaluated using ground-truth geometry or teachers different from those used to construct the training targets?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 8U6F (1/2)

We sincerely thank the reviewer for the valuable comments. We address each concern below and have revised the manuscript accordingly, with all newly added content highlighted in blue.

W1&Q1 Reproducibility of the training setup. The curated 10M-video dataset is not described in sufficient detail, making it difficult to reproduce the results or assess the influence of data composition.

(1) Dataset collection and curation. Our training corpus consists of 10M publicly accessible online videos collected under appropriate licenses, primarily from YouTube and TikTok. We adopt a multi-stage data curation pipeline to ensure video quality, temporal continuity, and data diversity:

  • Video validation. We verify video integrity and metadata, including duration, frame count, frame rate, and spatial resolution, and discard corrupted videos or those that cannot provide valid training clips.
  • Clip extraction. For multi-shot videos, we detect scene boundaries and extract temporally contiguous clips within individual shots, avoiding abrupt scene transitions that could disrupt temporal consistency during training.
  • Quality screening. We apply automatic quality checks to identify clips with severe blur, compression artifacts, abnormal exposure, excessive overlays, or corrupted/frozen frames. This is followed by manual inspection to further exclude clips with poor visual quality or temporal discontinuities.
  • Deduplication and benchmark-overlap checks. We remove exact duplicates using file hashes and identify near-duplicate content based on feature similarity between sampled video frames. To avoid evaluation leakage, we further compare source-video identifiers and visual similarity against the evaluation test splits, excluding overlapping videos and segments, including re-encoded copies and partially overlapping excerpts.

(2) Data availability. Due to organizational confidentiality restrictions, we are currently unable to release our curated 10M-video corpus. We expect these restrictions to remain in place for approximately six months and will make the dataset publicly available once permission is granted. To facilitate reproducibility, we will release the VGGA model weights and complete training and inference code upon acceptance.

(3) Reproducibility with a public dataset (reproducible substitute). To further support reproducibility and investigate the influence of training data composition, we conduct additional experiments using the publicly available OpenVid-1M dataset as a substitute for our curated 10M-video corpus. Specifically, we repeat the ablation study on the proportion of warp-supervised samples (Figure 5(c)) using OpenVid-1M and RealEstate10K, while keeping all other training settings fixed. The results are reported below.

Table r1. Ablation study of warp-supervised sample proportions using OpenVid-1M.

Warp-Supervised ProportionRealEstate10K FVD ↓\downarrowUCF101 FVD ↓\downarrow
0%80.13168.24
10%78.57158.74
20% (default)67.42145.27
30%70.35153.91
40%76.37160.38

The experiments on OpenVid-1M achieve comparable generation performance to our original 10M-video corpus (67.42 of Table r1 vs. 64.31 of Figure 6(c)) and exhibit a similar trend across different warp-supervision ratios. In particular, both datasets achieve their best generation performance at approximately 20% warp-supervised samples, supporting the effectiveness of our default sampling strategy. These results further demonstrate that VGGA generalizes well across different training datasets, with a consistent benefit from geometric grounding. We hope these experiments provide a publicly reproducible reference for evaluating VGGA and investigating the influence of training data composition.

We have included dataset details in Appendix A.5 and reported the experimental results on OpenVid-1M in Appendix C.6.

[1] Vision meets robotics: The kitti dataset, 2013.

[2] Latte: Latent Diffusion Transformer for Video Generation, TMLR 2025

AuthorsOpenReview ↗

Response to Reviewer 8U6F (2/2)

W2&Q3. Potentially non-independent geometric evaluation. The latent probes are evaluated using VGGT and RAFT pseudo-labels that are also used during training, so this evidence may partly reflect alignment with the teacher models.

We address the points raised below.

(1) Geometric probing under joint reconstruction and grounding. The existing probes in Figure 6(a) aim to evaluate whether geometric supervision makes structural and motion information more accessible from the learned VAE latents. Specifically, we freeze the autoencoders, train linear depth and optical-flow probes, and evaluate them on the RealEstate10K test split using VGGT and RAFT pseudo-labels as reference targets. Compared with VGGA w/o Geo, VGGA improves depth δ1\delta_1 from 0.355 to 0.813 and reduces the flow outlier rate from 56.4% to 23.8%. These results show that geometric grounding makes the supervised geometric information more linearly accessible from the VAE latents.

(2) Ground-truth depth and optical-flow probing on KITTI. To evaluate the learned geometric representations independently of the pseudo-label teachers used during training, we conduct additional depth and optical-flow probing experiments on KITTI [1] using ground-truth annotations. Specifically, we freeze the encoders of VGGA and VGGA w/o Geo, and train identical linear probes on their respective latent representations using the corresponding training splits. Both methods follow the same probing architecture, training data, and optimization settings, and are evaluated on the corresponding test splits.

Table r2. Latent depth probing on KITTI.

MethodKITTI (δ1\delta_1 ↑\uparrow)
VGGA w/o Geo0.11
VGGA0.62

Table r3. Latent optical-flow probing on KITTI.

MethodKITTI (F1-all (%) ↓\downarrow)
VGGA w/o Geo57.3
VGGA29.5

As shown in Tables r2 and r3, VGGA substantially outperforms the reconstruction-only baseline on both depth and optical-flow probing tasks. These improvements against ground-truth annotations demonstrate that VGGA learns more geometrically informative latent representations, making scene structure and motion correspondence more linearly accessible beyond simply matching the pseudo-label teachers (VGGT and RAFT) used during training.

We have reported the above experimental results in Figure 2.

W3 Limited long-range geometric modeling. The proposed objectives mainly capture frame-level or short-range consistency and do not explicitly address long-term dynamics or complete 3D scene structure.

We agree with the reviewer that VGGA primarily focuses on frame-level and short-range geometric consistency. We have explicitly acknowledged this limitation in Sec. 5 (Limitations): "VGGA focuses on frame-level and short-range geometric consistency, without explicitly modeling long-horizon dynamics or full 3D scene structure". Our work aims to demonstrate that incorporating geometric supervision improves video latent representations and downstream generation while preserving reconstruction fidelity. Extending geometric grounding to long-range temporal dynamics and more comprehensive 3D scene modeling remains an important direction for future work, as also discussed in Sec. 5.

Q2. Are all VAE comparisons in Table 1 conducted with identical downstream generator architectures, training schedules, and data?

Yes. All VAE comparisons in Table 1 use the same downstream generator backbone, training schedule, optimization strategy, and benchmark-specific training data. For each VAE, we freeze its parameters and independently train a Latte-XL/1 [2] generator on its latents. Table r4 summarizes the downstream configuration. We have included this configuration table in Appendix A.4 for clarity.

Table r4. Downstream generation settings used across VAE comparisons.

SettingConfiguration
Architecture
Generator backboneLatte-XL/1 [2]
Num. layers28
Hidden dim.1,152
Num. heads16
Patch size1×1×11\times1\times1
Optimization
Training iterations250K
OptimizerAdamW
lr1×10−41\times10^{-4}
GPUs16 H100 GPUs
Diffusion training
Training objectivev-prediction
Path typeLinear
SamplerEuler
Sampling steps250
Data and evaluation
Training datasetsRealEstate10K, UCF101, and Taichi-HD, respectively
Video length and resolution17×256×25617\times256\times256

[1] Vision meets robotics: The kitti dataset, 2013.

[2] Latte: Latent Diffusion Transformer for Video Generation, TMLR 2025

Reviewer_NoFfOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Video Geometry Grounded Autoencoder (VGGA), a geometry-aware video autoencoder designed to improve latent representations for downstream video generation. Unlike conventional video autoencoders that primarily focus on pixel-level reconstruction, VGGA incorporates three complementary geometric objectives-structural depth, motion correspondence, and projective warping-to encourage the latent space to capture the underlying 3D structure and motion of dynamic scenes. These objectives are jointly optimized with RGB reconstruction through a shared spatio-temporal decoder. Experimental results demonstrate substantial improvements in video generation.

Critical strengths

1.The paper addresses an important limitation of reconstruction-oriented video autoencoders: good reconstruction quality does not necessarily imply a latent representation suitable for generative modeling. Incorporating video-specific geometric priors is a meaningful direction.
2.The authors ensure fair experimental comparisons by training a reconstruction-only baseline with the same data and training settings, effectively isolating the contribution of geometry grounding.
3.Compared with the reconstruction-only counterpart, VGGA reduces FVD from 89.49 to 64.31 on RealEstate10K and from 195.44 to 121.76 on Taichi-HD under the same latent configuration. These improvements are substantial.
4.VGGA achieves significant improvements in generation quality while largely preserving reconstruction fidelity, outperforming representation-enhanced autoencoders that compromise reconstruction quality.

Critical weaknesses

1.The reconstruction-only baseline demonstrates the effectiveness of additional geometry-related objectives, but it remains unclear whether comparable improvements could be obtained using other dense auxiliary targets or pretrained visual feature distillation under matched computational budgets.
2.The proposed objective enforces consistency between feature-space warping followed by RGB decoding and directly warped RGB images. This primarily encourages decoder equivariance under projective transformations, rather than directly enforcing consistency with an independently observed target view. Furthermore, the projection-validity mask does not appear to explicitly handle occlusions or visibility changes. These limitations should be discussed and ideally evaluated.
3.The paper provides insufficient details about the 10M-video dataset, including where the videos come from, how they are selected and deduplicated, whether they overlap with evaluation datasets, and how much computation is required to generate pseudo-labels.
4.Although the Met3R results are promising, the evaluation mainly focuses on overall video generation quality. More direct tests are needed to show whether VGGA improves consistency across viewpoints, handles occlusions, and captures 3D scene geometry and motion.
5.The method introduces multiple loss terms with several balancing hyperparameters. Although Appendix A.3 describes these losses in detail, the rationale for selecting their weights remains unclear, and the sensitivity to these choices is not sufficiently evaluated.The method introduces multiple loss terms with several balancing hyperparameters. Although Appendix A.3 describes these losses in detail, the rationale for selecting their weights remains unclear, and the sensitivity to these choices is not sufficiently evaluated.

Questions for authors
  1. How does VGGA compare with an autoencoder trained using generic dense feature distillation, such as DINOv2, under matched model capacity and training budget?
  2. Does the projective warping objective explicitly account for occluded pixels and source-view visibility? If not, how sensitive is the method to occlusion boundaries and depth estimation errors?
  3. Can the authors evaluate geometric probing using ground-truth depth and optical flow rather than the same pseudo-label teachers used during training?
  4. Could the authors provide a full factorial ablation of the three geometric objectives? Table 3 shows the impact of removing each component, but does not evaluate their individual effectiveness when used alone or clarify how they interact.
  5. What is the complete training overhead introduced by geometric supervision, including offline pseudo-label generation? How much does the method affect inference-time autoencoder complexity?
  6. Could the authors better describe how the 10M-video training dataset was collected and processed, and explain whether potential overlap with the evaluation benchmarks was checked?
5 comments
AuthorsOpenReview ↗

Response to Reviewer NoFf (1/4)

We sincerely thank the reviewer for the valuable comments. We address each concern below and have revised the manuscript accordingly, with all newly added content highlighted in blue.

W1&Q1 Comparison with autoencoder trained using other dense auxiliary targets or pretrained visual feature distillation.

To directly compare geometric grounding with other dense auxiliary targets, we introduce an additional baseline, VAE with DINOv2-based Representation Alignment (REPA), following VA-VAE [1]. Specifically, we align the video VAE latents with features extracted from a frozen DINOv2 [2] encoder during training, while retaining the original reconstruction objectives.

Table r1. Generation comparison.

AutoencoderRealEstate10KUCF101Taichi-HD
FVD ↓Met3R ↓FVD ↓FVD ↓
VGGA w/o Geo89.490.0514163.78195.44
VGGA w/o Geo + REPA (DINOv2)70.420.0471139.83134.91
VGGA64.310.0421141.27121.76

Table r2. Reconstruction on Kinetics-400 at 17×256×25617\times256\times256.

AutoencoderrFVD ↓PSNR ↑LPIPS ↓
VGGA w/o Geo3.5633.610.019
VGGA w/o Geo + REPA (DINOv2)5.1832.100.031
VGGA3.4233.520.018

The results in Tables r1&r2 highlight the distinct benefits of DINOv2-based representation alignment and explicit geometric grounding.

  • Generation quality. Both REPA and VGGA substantially improve downstream generation over the vanilla VAE. REPA achieves better FVD on UCF101 (139.83 vs. 141.27), while VGGA performs better on RealEstate10K (64.31 vs. 70.42) and Taichi-HD (121.76 vs. 134.91). VGGA also achieves a lower Met3R on RealEstate10K (0.0421 vs. 0.0471), suggesting that explicit geometric grounding better preserves geometric consistency.

  • Reconstruction quality. Despite its improvements in generation, REPA noticeably degrades reconstruction fidelity (rFVD 3.56 to 5.18 and PSNR 33.61 to 32.10). In contrast, VGGA largely preserves reconstruction quality. This suggests that generic semantic feature alignment introduces a trade-off between generation and reconstruction, whereas geometry-aware supervision provides more compatible structural guidance for video representations.

W2&Q2 Projective warping objective (warped RGB images vs. independently observed target views) and the projection-validity mask.

Thank you for your questions regarding the projective warping objective and projection-validity mask. We clarify these two aspects below.

(1) Projective warping objective. Our objective uses directly warped RGB images rather than independently observed target views as supervision targets. This encourages geometric equivariance between feature-space warping and RGB-space warping, allowing the model to learn camera-aware geometric representations. We also experimented with using independently observed target views for supervision, but observed slightly worse performance. This is because corresponding 3D points may have different RGB appearances across views or timestamps, even when their geometric correspondences are correct. Such differences can result from changes in illumination, exposure, or view-dependent effects such as specular reflections. As a result, supervision from independently observed target views may introduce appearance differences unrelated to geometry, making it harder to learn geometric consistency. Therefore, we adopt directly warped RGB images as supervision targets to avoid interference from cross-view appearance variations.

(2) Projection-validity mask and visibility handling. We clarify that our implementation already incorporates a projection-validity mask MM (Lines 761–767 of Appendix A.3) to handle invalid correspondences arising from occlusions and visibility changes. Specifically, we adopt standard target-driven backward warping, where each target pixel is independently mapped to a source coordinate through camera geometry. The validity mask MM identifies whether the projected 3D point has positive depth in the source camera coordinate system and falls within the source image boundaries. Only valid correspondences contribute to the projective warping loss, thereby excluding out-of-view regions and invalid projections. This formulation also avoids ambiguous pixel assignments, as each target pixel is associated with a single source sampling coordinate. For clarity, we have moved the formulation of the projective warping loss with projection-validity mask MM to the main paper (Lines 321–330).

[1] Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models, CVPR 2025.

[2] DINOv2: Learning Robust Visual Features without Supervision, TMLR 2024.

AuthorsOpenReview ↗

Response to Reviewer NoFf (2/4)

W3&Q6 The paper provides insufficient details about the 10M-video dataset, including where the videos come from, how they are selected and deduplicated, whether they overlap with evaluation datasets, and how much computation is required to generate pseudo-labels.

We provide further clarification on the data collection, curation, benchmark-overlap checks, and computational cost of pseudo-label generation. For clarify, we have added these details to Appendix A.5.

(1) Dataset collection and curation. Our training corpus consists of 10M publicly accessible online videos collected under appropriate licenses, primarily from YouTube and TikTok. We adopt a multi-stage data curation pipeline to ensure video quality, temporal continuity, and data diversity:

  • Video validation. We verify video integrity and metadata, including duration, frame count, frame rate, and spatial resolution, and discard corrupted videos or those that cannot provide valid training clips.
  • Clip extraction. For multi-shot videos, we detect scene boundaries and extract temporally contiguous clips within individual shots, avoiding abrupt scene transitions that could disrupt temporal consistency during training.
  • Quality screening. We apply automatic quality checks to identify clips with severe blur, compression artifacts, abnormal exposure, excessive overlays, or corrupted/frozen frames. This is followed by manual inspection to further exclude clips with poor visual quality or temporal discontinuities.
  • Deduplication and benchmark-overlap checks. We remove exact duplicates using file hashes and identify near-duplicate content based on feature similarity between sampled video frames. To avoid evaluation leakage, we further compare source-video identifiers and visual similarity against the evaluation test splits, excluding overlapping videos and segments, including re-encoded copies and partially overlapping excerpts.

To facilitate reproducibility, we will release the VGGA model weights and complete training and inference code upon acceptance.

(2) Computation of pseudo-labels generation. We report the cost detail of pseudo-label generation in Table r11 (see Q5).

W4 Although the Met3R results are promising, the evaluation mainly focuses on overall video generation quality. More direct tests are needed to show whether VGGA improves consistency across viewpoints, handles occlusions, and captures 3D scene geometry and motion.

(1) Additional geometric metrics To provide a more direct evaluation, we introduce three additional metrics following GeCo [3]:

  • Motion Consistency (Motion ↓\downarrow) measures discrepancies between estimated optical flow and geometry-induced rigid motion, evaluating whether generated videos exhibit inconsistent geometric deformations.
  • Structure Consistency (Structure ↓\downarrow) measures depth reprojection discrepancies across frames, capturing inconsistencies in 3D scene structure, including occlusion-related geometric artifacts.
  • Overall Geometric Consistency (Fused ↓\downarrow) combines motion and structure inconsistencies to provide a comprehensive assessment of geometric consistency.

Table r3. Geometric evaluation on RealEstate10K.

AutoencoderMEt3R ↓\downarrowGeCo Motion ↓\downarrowGeCo Structure ↓\downarrowGeCo Fused ↓\downarrow
VGGA w/o Geo0.05140.0440.1420.149
VGGA0.04210.0210.0890.091

As shown in Table r3, VGGA consistently improves all GeCo metrics over the baseline VAE. Together with the improved MEt3R score, these results demonstrate that VGGA enhances multi-view consistency while reducing geometric deformations and structural inconsistencies.

(2) Video-format visualization. We also provide \href{https://anonymous.4open.science/r/VGGA-16617}{\textcolor{green!50!black}{video-format visualizations}} for better comparison. These visualizations clearly demonstrate the geometric fidelity and generation-friendly advantages of our method.

[3] GeCo: Evaluating Geometric Consistency for Video Generation via Motion and Structure, ECCV 2026.

AuthorsOpenReview ↗

Response to Reviewer NoFf (3/4)

W5. Selection and Sensitivity of Loss Balancing Hyperparameters

(1) Balancing hyperparameters for the reconstruction objectives. We mainly follow previous works [4, 5] in setting the loss weights for VAE reconstruction. To improve training stability, we reduce the adversarial loss weight λgan\lambda_{\mathrm{gan}} to 0.1. For KL regularization, we adopt a relatively small weight of β=1×10−8\beta = 1 \times 10^{-8}, as recent studies on VAE–DiT pipelines [6, 7] suggest that strong KL regularization may compromise reconstruction fidelity and the suitability of learned latents for downstream diffusion modeling. To further examine the sensitivity to KL regularization, we conduct an ablation study with different values of β\beta, as shown in Table r4.

Table r4. Ablation study of KL regularization weight β\beta.

KL Weight (β\beta)RealEstate10K (FVD ↓\downarrow)PSNR ↑\uparrow
069.3233.15
1×10−81 \times 10^{-8} (Ours)64.3133.52
1×10−71 \times 10^{-7}69.0132.98
1×10−61 \times 10^{-6}71.4631.66

(2) Geometric loss hyperparameter sensitivity. We first clarify that all reconstruction and geometric losses are normalized to comparable numerical scales of approximately O(1)O(1) to ensure stable training, as described in Lines 769–775 of Appendix A.3. Based on this normalization, we adopt equal weights (λd=λf=λw=1.0\lambda_d=\lambda_f=\lambda_w=1.0) as a simple default setting for VGGA.

To further evaluate the sensitivity to geometric loss weights, we conduct additional ablation studies by varying each of λd\lambda_d, λf\lambda_f, and λw\lambda_w over {0.2,0.5,1.0,2.0,3.0}\{0.2, 0.5, 1.0, 2.0, 3.0\}, while keeping the other two weights fixed at 1.0. Reconstruction performance is evaluated on Kinetics-400 at 17×256×25617\times256\times256. For generation, we freeze each VAE and train a separate Latte-XL/1 generator on its latent representations for 250K steps using 16 H100 GPUs. The results are reported below.

Table r5. Sensitivity to depth-loss weight λd\lambda_d (λf=λw=1.0\lambda_f=\lambda_w=1.0).

λd\lambda_dRealEstate10K FVD ↓\downarrowK400 PSNR ↑\uparrow
0.270.8233.55
0.566.4533.52
1.0 (default)64.3133.52
2.065.1033.36
3.070.5332.89

Table r6. Sensitivity to flow-loss weight λf\lambda_f (λd=λw=1.0\lambda_d=\lambda_w=1.0).

λf\lambda_fRealEstate10K FVD ↓\downarrowK400 PSNR ↑\uparrow
0.268.7633.58
0.563.4633.61
1.0 (default)64.3133.52
2.067.7432.93
3.073.6232.01

Table r7. Sensitivity to warping-loss weight λw\lambda_w (λd=λf=1.0\lambda_d=\lambda_f=1.0).

λw\lambda_wRealEstate10K FVD ↓\downarrowK400 PSNR ↑\uparrow
0.270.2633.57
0.565.0233.54
1.0 (default)64.3133.52
2.066.3132.35
3.072.8432.13

The results show that VGGA is robust to moderate variations in geometric loss weights. Overall, the default equal-weight setting achieves competitive generation and reconstruction performance, demonstrating the effectiveness and robustness of our approach. We have reported these results in Appendix E.2.

Q3. Can the authors evaluate geometric probing using ground-truth depth and optical flow rather than the same pseudo-label teachers used during training?

To provide an evaluation independent of the pseudo-label teachers used during training, we additionally evaluate depth and optical-flow probing on KITTI [8] using ground-truth annotations. The results are reported below.

Table r8. Latent depth probing on KITTI.

MethodKITTI (δ1\delta_1 ↑\uparrow)
VGGA w/o Geo0.11
VGGA0.62

Table r9. Latent optical-flow probing on KITTI.

MethodKITTI (F1-all (%) ↓\downarrow)
VGGA w/o Geo57.3
VGGA29.5

As shown in Tables r8 and r9, VGGA substantially improves both depth and optical-flow probing performance over the reconstruction-only baseline (VGGA w/o Geo). These improvements on ground-truth benchmarks demonstrate that VGGA learns more geometrically informative latent representations, making scene structure and motion correspondence more linearly accessible, rather than merely matching the pseudo-label teachers (VGGT and RAFT) used during training. We have reported the above experimental results in Figure 2.

[4] Wan: Open and Advanced Large-Scale Video Generative Models, 2025.

[5] Delving into Latent Spectral Biasing of Video VAEs for Superior Diffusability, ECCV 2026.

[6] Qwen-Image-VAE-2.0 Technical Report, 2026.

[7] Diffusion Transformers with Representation Autoencoders, ICLR 2026.

[8] Vision meets robotics: The kitti dataset, 2013.

Reviewer_5xdbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VGGA trains a Wan2.2-architecture video VAE and adds geometry supervision on the decoder's shared spatio-temporal upsampling trunk. Depth and flow heads regress VGGT and RAFT pseudo-labels on all videos. On RealEstate10K only, a projective warping loss requires a source-frame feature, warped with the target frame's VGGT depth and the camera pose, to decode to the source frame warped by the same map. With frozen latents and a Latte-XL/1 generator, VGGA lowers FVD relative to a reconstruction-only baseline (w/o Geo) on RealEstate10K, UCF101, and Taichi-HD, by 28.1% on RealEstate10K and 37.7% on Taichi-HD. Reconstruction on Kinetics-400 remains comparable.

Critical strengths

Shaping video latents with the geometry of video formation is an intuitive idea, and the recipe is simple. The geometry heads branch off the trunk and are not needed for RGB decoding. The w/o Geo baseline, trained from scratch on identical data, makes the comparison controlled. Gains are consistent across three benchmarks and two latent configurations, and they persist throughout generator training. Reconstruction is largely preserved, and the ablations are informative.

Critical weaknesses

W1. Scalability (main concern). The paper claims VGGA "supports large-scale training without explicit geometric annotations" (Sec. 1) and is "scaled to large-scale unlabelled videos" (Sec. 5), but this holds only for the depth and flow terms. The warp loss, which the paper credits for the camera-aware priors, "requires reliable camera poses and assumes static scene geometry." It is therefore applied only to RealEstate10K (20% of samples), and Fig. 5(c) shows FVD worsening once that share exceeds 20%. The tested scale is also small: one 100K-step autoencoder schedule and Latte-XL/1 at 17×256².

W2. Geometric conventions, and what the warp loss tests. Correct warps need VGGT depth and the camera pose to share a scale and convention, but the paper describes no scale alignment or pose convention. If the RealEstate10K trajectories (from SLAM and bundle adjustment) are used, their scale is set independently of VGGT's normalized depth. Likewise, nothing requires the RAFT flow targets to agree with depth and pose. Such mismatches would go unnoticed, because Eqs. 9, 10, and 14 compare the decoded warped feature with the source frame warped by the same map, never with the real target frame I_j. The loss is thus self-consistent for any warp, so its gains alone do not show that correct geometry matters.

W3. Novelty and related work. The method is a standard video VAE plus auxiliary heads that distill off-the-shelf teachers. This is analogous to VA-VAE (Yao et al., 2025b), with geometric instead of semantic targets. The warp loss is an EQ-VAE-style regularizer (Kouzelis et al., 2025), which the paper cites only as generic latent regularization. Two lines of closely related work are not discussed. The first is geometry and motion supervision of video generators: Geometry Forcing (Wu et al., 2025) aligns diffusion features with VGGT, VideoJAM (Chefer et al., 2025) predicts motion jointly with appearance, and Track4Gen (Jeong et al., CVPR 2025) adds point-tracking supervision. The second is geometry-aware autoencoders, such as Lai et al. (Video Autoencoder, ICCV 2021) and Dupont et al. (Equivariant Neural Rendering, ICML 2020). A comparison with applying the same teachers on the generator side would show whether the autoencoder is the right place for this supervision.

W4. Motivation is asserted, not measured. Fig. 1 labels the vanilla VAE as "lacking explicit 3D geometry and view consistency," and Sec. 1 says generators "must infer depth, motion, and viewpoint consistency implicitly from compressed RGB latents." Yet neither claim is measured on existing VAEs. Depth and flow can be re-estimated from a faithful reconstruction, so the issue is accessibility rather than lost information. The paper itself puts it this way ("more linearly accessible," Sec. 4.3), but the Fig. 5(a) probes cover only w/o Geo and VGGA. Similarly, being "predominantly reconstruction-centric" (Sec. 2) is not a problem in itself, since VGGA also keeps reconstruction and only adds auxiliary losses. Probes on public VAEs such as Wan2.2 and Hunyuan-VAE would ground the motivation.

Questions for authors

Please refer to the Weakness Section to addres concerns.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 5xdb (1/5)

We sincerely thank the reviewer for the valuable comments. We address each concern below and have revised the manuscript accordingly, with all newly added content highlighted in blue.

W1. Scalability (main concern). The paper claims VGGA "supports large-scale training without explicit geometric annotations" (Sec. 1) and is "scaled to large-scale unlabelled videos" (Sec. 5), but this holds only for the depth and flow terms. The warp loss, which the paper credits for the camera-aware priors, "requires reliable camera poses and assumes static scene geometry." It is therefore applied only to RealEstate10K (20% of samples), and Fig. 5(c) shows FVD worsening once that share exceeds 20%. The tested scale is also small: one 100K-step autoencoder schedule and Latte-XL/1 at 17×256².

Thank you for raising this important concern. We address the points raised below.

(1) Large-scale training without explicit geometric annotations. We first clarify that our claims of "supporting large-scale training without explicit geometric annotations" (Sec. 1) and "scaling to large-scale unlabeled videos" (Sec. 5) refer to the ability of our hybrid geometric supervision strategy to support training on large-scale unlabeled videos, rather than the scalability of all three geometric objectives to unlabeled videos. To avoid this misunderstanding, we have revised these statements to "supports training on large-scale unlabeled videos through our proposed hybrid geometric supervision strategy" and "supports training on large-scale unlabeled videos", respectively.

In fact, our hybrid geometric supervision strategy is specifically designed to accommodate the requirement of reliable camera poses and static scene geometry for Projective Warping Grounding. Specifically, given a large-scale unlabeled video corpus (our curated 10M-video dataset), we apply Structural Depth Grounding and Motion Correspondence Grounding using pseudo-depth and optical-flow annotations generated by VGGT and RAFT. To further incorporate camera-aware geometric priors into the representations learned from these unlabeled videos, we introduce a smaller auxiliary dataset with available camera poses (RealEstate10K) to provide Projective Warping Grounding.

We have explained this hybrid geometric supervision strategy in Lines 115–119:

"Through joint training, the projective geometry learned from these static scenes is absorbed into the shared latent representation and complements the depth and motion supervision from general videos, allowing camera-aware geometric priors to transfer beyond the statically supervised data and improve generation on dynamic videos."

Through this hybrid strategy, the curated 10M-video corpus receives explicit depth and motion supervision while implicitly benefiting from camera-aware geometric priors learned from the auxiliary pose-annotated static-scene dataset. The effectiveness of this strategy is further demonstrated in Figure 6(b).

Importantly, this design enables VGGA to learn geometry-aware representations from large-scale unlabeled videos without requiring camera poses or manually annotated geometric labels for the 10M-video corpus.

(2) Why does FVD worsen when the proportion of RealEstate10K samples exceeds 20%? RealEstate10K serves as an auxiliary dataset that provides camera-aware geometric supervision during training. We explain the FVD trend observed in Figure 6(c) from two perspectives.

  • Improvement from 0% to 20%. Increasing the proportion of RealEstate10K samples from 0% to 20% improves the generation FVD from 76.48 to 64.31. This demonstrates the benefit of incorporating camera-aware geometric constraints through our hybrid supervision strategy, allowing the shared VAE to learn more geometrically friendly representations.

  • Performance degradation beyond 20%. Learning camera-aware geometric priors through Projective Warping Grounding requires reliable camera poses and assumes static scene geometry. We therefore introduce the static-scene dataset RealEstate10K as an auxiliary supervision source, which naturally introduces a trade-off between camera-aware geometric supervision and training data diversity. Specifically, increasing the proportion of RealEstate10K samples strengthens camera-aware geometric supervision but also reduces the relative contribution of diverse general videos from our curated 10M-video corpus. This may bias the learned VAE representations toward static-scene content, reducing their ability to model diverse video distributions and consequently degrading downstream generation performance when the proportion exceeds 20%.

Overall, Figure 6(c) highlights the importance of balancing camera-aware geometric supervision from auxiliary static-scene videos with the diversity of large-scale unlabeled video data, with the default 20% sampling ratio achieving an effective empirical balance.

AuthorsOpenReview ↗

Response to Reviewer 5xdb (2/5)

W1. Scalability (main concern). The paper claims VGGA "supports large-scale training without explicit geometric annotations" (Sec. 1) and is "scaled to large-scale unlabelled videos" (Sec. 5), but this holds only for the depth and flow terms. The warp loss, which the paper credits for the camera-aware priors, "requires reliable camera poses and assumes static scene geometry." It is therefore applied only to RealEstate10K (20% of samples), and Fig. 5(c) shows FVD worsening once that share exceeds 20%. The tested scale is also small: one 100K-step autoencoder schedule and Latte-XL/1 at 17×256².

(3) The tested scale is also small: one 100K-step autoencoder schedule and Latte-XL/1 at 17×256². To further clarify our training settings and evaluate VGGA, we conduct additional experiments with longer autoencoder and generator training schedules, as well as higher-resolution video generation.

(3.1) Longer autoencoder training schedules. We investigate the effect of autoencoder training steps on reconstruction and downstream generation performance, as reported in Table r1.

Table r1. Effect of VAE training steps on reconstruction and downstream video generation.

MetricAutoencoder20K50K80K100K (default)200K800K
Reconstruction (K400 PSNR ↑\uparrow)VGGA w/o Geo27.4631.3232.9133.6133.7833.89
VGGA29.2132.2633.1333.5233.6333.77
Generation (RealEstate10K FVD ↓\downarrow)VGGA w/o Geo110.4792.3190.7889.4990.3689.35
VGGA98.4784.4569.8964.3162.5762.59

As shown in Table r1, both VGGA and the baseline VAE approach convergence around 100K training steps, with only modest improvements from further training. Importantly, VGGA consistently achieves better downstream generation performance while maintaining comparable reconstruction quality, even with extended training. Given the marginal gains beyond 100K steps, we adopt 100K as a computationally efficient default autoencoder training schedule.

(3.2) Longer generator training schedules. To investigate whether the benefits of VGGA persist with more extensive downstream generator training, we extend Latte-XL/1 training from the default 250K steps to 2M steps. The results are reported in Table r2.

Table r2. Effect of Latte-XL/1 training steps on video generation at 17×256×25617\times256\times256 on RealEstate10K.

Autoencoder / Training Steps50K100K250K (default)500K1M2M
VGGA w/o Geo FVD ↓\downarrow138.15119.2189.4987.4586.9186.32
VGGA FVD ↓\downarrow153.3491.7664.3161.4359.4858.60

As shown in Table r2, extending generator training further improves VGGA's FVD from 64.31 at 250K steps to 58.60 at 2M steps. VGGA also consistently outperforms the baseline VAE from 100K steps onward, further demonstrating the effectiveness of geometry-grounded priors in improving downstream video generation.

(3.3) Higher-resolution video generation. We further evaluate VGGA on video generation at 17×512×51217\times512\times512 using Latte-XL/1. The results are reported in Table r3.

Table r3. Video generation performance at 17×512×51217\times512\times512 resolution on RealEstate10K.

AutoencoderFVD ↓\downarrowKVD ↓\downarrowMEt3R ↓\downarrow
VGGA w/o Geo83.484.010.0513
VGGA59.233.210.0422

As shown in Table r3, VGGA consistently outperforms the baseline model across all three metrics at higher resolution, demonstrating that its improvements in generation quality and geometric consistency on 17×512×51217\times512\times512 video generation. We have reported above results in Appendix C.1 and C.2 and C.3.

(3.4) Extension to Large-Scale Text-to-Video Generation. We are currently extending VGGA to large-scale text-to-video generation to further investigate its effectiveness and scalability in general video synthesis. These experiments are ongoing, and the results will be reported in the near future.

AuthorsOpenReview ↗

Response to Reviewer 5xdb (3/5)

W2. Geometric conventions, and what the warp loss tests.

We agree that consistent depth and camera-pose scales are important for accurate projective warping. During our investigation, we have explored several warping strategies, including depth–pose scale alignment. We clarify their implementations and provide these experimental comparisons below.

  • (a) Equivariance under spatial transformations. We initially explored equivariance regularization following EQ-VAE [1], using spatial transformations such as scaling and rotation.

  • (b) Projective warping with VGGT depth and camera pose pseudo-labels. We then extended equivariance regularization to camera-aware projective transformations. Specifically, we used both depth maps and camera poses predicted by VGGT to construct the projective warping maps. Since VGGT jointly predicts depth and camera poses under a shared geometric scale, this setting avoids the scale mismatch.

  • (c) Projective warping with scale-aligned VGGT depth pseudo-labels and RealEstate10K camera poses. To reduce potential errors in VGGT-predicted camera poses, we further explored using the camera trajectories provided by RealEstate10K. Since these trajectories are estimated independently of VGGT depth, we explicitly account for their geometric conventions and scale differences. As OpenReview does not support rendering complex equations, please refer to Appendix B.1 for details on the scale-aligned VGGT pseudo-depth labels.

  • (d) Projective warping with original VGGT depth pseudo-labels and RealEstate10K camera poses. Interestingly, during scale alignment, we observed that the estimated scale factors were generally close to 1.0 for the examined videos. We therefore additionally evaluated projective warping using the original VGGT depth maps and RealEstate10K camera poses without explicit scale alignment.

We summarize the downstream generation and geometric consistency results below.

Table r4. Comparison of different warping strategies for VGGA w/o Geo on dataset RealEstate10K. The "+" symbol indicates that the corresponding strategy is applied individually to VGGA w/o Geo.

Warping StrategyDepthPoseFVD ↓\downarrowMet3R ↓\downarrow
VGGA w/o Geo——89.490.0514
+ EQ-VAE-style Spatial Transformations——77.270.0482
+ Projective Warping (VGGT)VGGTVGGT68.110.0445
+ Projective Warping (Scale-Aligned)VGGT (Aligned)RE10K67.130.0431
+ Projective Warping (Original)VGGT (Original)RE10K67.460.0436

(1) Scale mismatch between VGGT depth pseudo-labels and RealEstate10K camera poses. As shown in Table r4, projective warping with scale-aligned VGGT depth achieves comparable performance to that using the original VGGT depth, as the estimated scale factors are generally close to 1.0. Therefore, we use the original VGGT depth pseudo-labels in our experiments for computational efficiency. Nevertheless, we agree with the reviewer on the importance of consistent depth–pose scales and geometric conventions. We have accordingly revised the projective warping formulation to explicitly incorporate depth–pose scale alignment, ensuring more geometrically consistent transformations.

(2) Correct geometry matters in our projective warping. We further compare our scale-aligned projective warping with geometry-independent spatial transformations following EQ-VAE. As shown in Table r4, our projective warping achieves substantially better downstream generation performance (FVD of 67.13 vs. 77.27 and MEt3R of 0.0431 vs. 0.0482). These results highlight the importance of incorporating camera-aware geometric information into projective warping, rather than relying solely on generic spatial equivariance.

(3) Nothing requires the RAFT flow targets to agree with depth and pose. We would like to clarify that VGGA incorporates two designs to accommodate differences among geometric supervision signals and facilitate stable joint optimization.

  • Loss normalization. Since depth, optical flow, and projective warping involve different physical units and numerical scales, we normalize all dense objectives to comparable magnitudes of approximately O(1)O(1). This prevents objectives with larger numerical magnitudes from dominating joint optimization and improves training stability, as detailed in Loss Scaling and Joint Optimization in Appendix A.3.

  • Task-specific heads. We employ lightweight, task-specific heads for depth, optical flow, and projective warping, allowing each geometric objective to supervise the shared VAE representations through its own prediction pathway. This design accommodates differences among geometric supervision signals without requiring strong alignment between RAFT optical flow and depth–pose geometry.

[1] EQ-VAE: Equivariance Regularized Latent Space for Improved Generative Image Modeling, ICML 2025.

Comments on the paper

1 comment

Thank you for the additional experiments and clarifications. They strengthen the paper, but the following concerns remain.

My main remaining concern is scalability. Although the revised procedure accommodates depth-pose scale differences, the evidence remains limited to the tested settings, and the projective objective continues to depend on pose-available static scenes. It remains unclear how effectively this hybrid supervision strategy transfers to large-scale, diverse T2V generation.

Regarding W2, the comparison with spatial scaling/rotation supports the usefulness of projective transformations, but does not isolate whether scene-consistent geometry drives the gains. Since both loss branches use the same warp, the objective enforces equivariance without independently verifying the geometric correspondence. Controls with perturbed or shuffled geometry, while keeping transformation magnitude and validity coverage comparable, would help distinguish the benefit of correct geometry from that of projective-transformation regularization.

Since the large-scale T2V experiments are still ongoing, I would recommend reassessing the paper with those results included. Longer training and higher resolution strengthen the current evidence, but broader T2V validation would be particularly important for resolving my original scalability concern.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.