- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes GAE, a method to make a geometry-native latent space for world and video generation. Instead of using a regular image VAE and adding camera poses on top, this work takes the 4 feature levels from a frozen Depth Anything 3 (DA3) model and compresses them down into a 64 or 128 channel latent. Then a decoder reconstructs the features so the frozen DA3 head can read out depth and point clouds, while a separate head outputs RGB. The authors train a DiT flow model on this latent and compare against pixel and semantic VAEs on RealEstate10K and DL3DV.
Critical strengths
- Good latent idea: Compressing multi-level geometry features into one compact latent while keeping the perception head frozen is a good idea. It avoids making a cascade of flow models and makes geometry natively readable.
- Solid controlled experiments: The paper compares different latents using the exact same flow backbone and training setup. The gains in camera accuracy and 3D consistency are clear.
Critical weaknesses
- Only tested on static scenes: The paper only evaluates on RealEstate10K and DL3DV, which are just static indoor and outdoor scenes. Real videos often have moving objects. Since DA3 assumes static geometry, it is not clear at all what happens when objects move, or if dynamic scenes will just create weird floaters in the point clouds.
- Fine details look a bit blurry: In the rollout videos on the project website, the 3D consistency and camera motion look good, but small details and textures look blurry compared to normal video models.
- Relies heavily on extra teacher models: Training the codec needs multiple frozen teacher models like C-RADIO and DINOv2. Without the structure loss, Table 1 shows that spatial structure drops a lot, so training the latent seems pretty tricky without all these teachers.
Questions for authors
The paper is well written and the idea makes sense, but I have a few questions:
- How does the method handle dynamic scenes with moving objects? Will moving things break the geometry latent?
- Why do fine textures in the rollout videos look a bit blurry? Is this limited by the DA3 patch size or the compression bottleneck?
- How much does inference time increase when decoding both RGB and the full feature hierarchy compared to a normal video VAE?
Overall the method is sound and the experiments are clean, so I lean towards weak accept.
I thank the authors for the additional experiments, which substantially address my original concerns. Notably, the extended ablations show that GAE gains are not only due to alignment. The controlled GLD comparison and quantitative evaluation at higher resolution and longer sequence lengths also address the missing comparisons.
Overall, the new evidence improves my assessment, and I am raising my rating.