acceptodds
Under review as a conference paper at ICLR 2027

GAE: Learning a Geometry-Native Latent Space for 3D-Consistent World Generation

12 views

Abstract

Visual generators can produce photorealistic frames without preserving the scene they depict. We argue that this is not only a modeling problem but a representation problem: generators evolve latents organized around appearance, while perception systems recover geometry in a different space. Rather than add geometry as another output, we make a compact reparameterization of a geometry foundation model's feature space the generator's native state. We realize this shift with the geometry-native autoencoder (GAE), whose latent is jointly decodable to appearance, depth, cameras, and point maps. With this state, a standard conditional flow supports diverse generation tasks. In controlled comparisons that hold the generator and training protocol fixed, replacing the latent with GAE improves both visual quality and independently measured 3D coherence: FVD falls by and on RealEstate10K and DL3DV, and camera-trajectory error is halved on RealEstate10K. Together, these results show that the latent space is central to geometry-consistent generation and can serve as a shared interface between perception and generation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
World model, Novel view synthesis, Video generation, 3D reconstruction

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_9vhuOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes GAE, a method to make a geometry-native latent space for world and video generation. Instead of using a regular image VAE and adding camera poses on top, this work takes the 4 feature levels from a frozen Depth Anything 3 (DA3) model and compresses them down into a 64 or 128 channel latent. Then a decoder reconstructs the features so the frozen DA3 head can read out depth and point clouds, while a separate head outputs RGB. The authors train a DiT flow model on this latent and compare against pixel and semantic VAEs on RealEstate10K and DL3DV.

Critical strengths
  • Good latent idea: Compressing multi-level geometry features into one compact latent while keeping the perception head frozen is a good idea. It avoids making a cascade of flow models and makes geometry natively readable.
  • Solid controlled experiments: The paper compares different latents using the exact same flow backbone and training setup. The gains in camera accuracy and 3D consistency are clear.
Critical weaknesses
  • Only tested on static scenes: The paper only evaluates on RealEstate10K and DL3DV, which are just static indoor and outdoor scenes. Real videos often have moving objects. Since DA3 assumes static geometry, it is not clear at all what happens when objects move, or if dynamic scenes will just create weird floaters in the point clouds.
  • Fine details look a bit blurry: In the rollout videos on the project website, the 3D consistency and camera motion look good, but small details and textures look blurry compared to normal video models.
  • Relies heavily on extra teacher models: Training the codec needs multiple frozen teacher models like C-RADIO and DINOv2. Without the structure loss, Table 1 shows that spatial structure drops a lot, so training the latent seems pretty tricky without all these teachers.
Questions for authors

The paper is well written and the idea makes sense, but I have a few questions:

  • How does the method handle dynamic scenes with moving objects? Will moving things break the geometry latent?
  • Why do fine textures in the rollout videos look a bit blurry? Is this limited by the DA3 patch size or the compression bottleneck?
  • How much does inference time increase when decoding both RGB and the full feature hierarchy compared to a normal video VAE?

Overall the method is sound and the experiments are clean, so I lean towards weak accept.

Reviewer_xjFsOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors present GAE, which embeds a compact latent space for 3D-consistent generation based on the geometry foundation model DA3. To achieve this, the method has two stages.

(i) A codec, placed between DA3's frozen encoder and frozen geometry head, compresses DA3's four feature levels into a single latent and reconstructs them, so that depth, cameras and point maps can still be decoded by the frozen head. A learned RGB head also decodes appearance from this latent. Additionally, the latent is regularized for generation by aligning it with C-RADIO and DINOv2 features.

(ii) A DiT-style conditional flow model is trained in this latent space to jointly generate RGB and geometry. It is conditioned on reference views, camera rays, and text.

Using the same flow model and training protocol, GAE is compared against pixel, semantic and raw DA3 latents on RealEstate10K and DL3DV, where it reduces FVD and roughly halves camera-trajectory error on RealEstate10K.

Critical strengths
  • (s.1) The paper is well motivated. It frames 3D inconsistency in generation as a representation problem and supports this with a clear diagnosis of why raw DA3 features are not well suited for generation.
  • (s.2) The idea of reparameterizing a geometry foundation model's feature hierarchy into a single compact latent, readable by both the frozen geometry head and an RGB head, is simple and interesting.
  • (s.3) The comparison is clear. All latents are evaluated with the same flow model, training budget, and sampling protocol, which isolates the effect of the latent space.
  • (s.4) The results show improvements in both visual quality and 3D consistency. Additionally, a single model supports text-to-image, camera-controlled, and reference-conditioned generation.
Critical weaknesses
  • (w.1) The appearance gains seem to depend on the representation-alignment loss rather than on the geometry-native latent itself.
    • Without Lrepr\mathcal{L}_{\text{repr}}, GAE-128 is worse than SD-VAE on RealEstate10K in FVD (266.4 vs 258.6) and PSNR (19.19 vs 19.32) (Table 7 vs Table 5), while still being better in 3D consistency.
    • Since the pixel VAE baselines receive no comparable alignment to C-RADIO/DINOv2, the comparison does not fully isolate the contribution of the geometry-native representation. The observed appearance improvements may primarily result from the additional representation-alignment supervision rather than the geometric inductive bias itself.
    • A fairer comparison would for instance to apply the same alignment objectives to the pixel VAE baselines.
  • (w.2) GLD, which the paper presents as the closest prior work, is only reported as an external pipeline and is not retrained under the paper's protocol.
    • Since a central motivation of GAE is to replace hierarchical feature generation with a single compact latent, the absence of a controlled comparison with GLD makes it difficult to realize whether the improvements come from the proposed representation or from other differences such as the architecture, training, or computational budget.
  • (w.3) The quantitative evaluation is limited to short, low-resolution clips (nine 252×252 views, one reference view, 64 scenes per dataset). The claim that larger-scale experiments "confirm that these gains persist with more data, higher resolution, and longer generation sequences" is only supported by qualitative results from a separately trained model with no baselines.
Questions for authors
  • See weaknesses.
  • (q.1) Could the authors report results for the pixel VAE baselines trained with the same C-RADIO/DINOv2 alignment objectives?
  • (q.2) Could the authors provide the Lrepr\mathcal{L}_{\text{repr}} ablation for GAE-64 and on DL3DV?
  • (q.3) What training data, resolution, and compute budget does the reported GLD pipeline use?
  • (q.4) Could the authors provide a quantitative comparison with at least one baseline at the larger scale (81 views at 672×378) reported in the paper?
  • (q.5) How do appearance quality and 3D consistency change as the number of generated views increases?
1 comment

I thank the authors for the additional experiments, which substantially address my original concerns. Notably, the extended ablations show that GAE gains are not only due to alignment. The controlled GLD comparison and quantitative evaluation at higher resolution and longer sequence lengths also address the missing comparisons.

Overall, the new evidence improves my assessment, and I am raising my rating.

Reviewer_jrF5OpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper argues that 3D inconsistency in video generation is partly a latent-space problem. It proposes GAE, which compresses the multi-level features of a frozen DA3 encoder into a compact latent that decodes to both RGB and geometry, and shapes it for generation via alignment to C-RADIO and DINOv2 features. A standard conditional flow model on this latent supports text-to-image, camera-controlled video, and novel-view synthesis. With the generator and training protocol fixed, GAE outperforms pixel, semantic, and raw-geometry latents, reducing FVD by 12.7% and 23.1% on RealEstate10K and DL3DV and roughly halving camera trajectory error on RealEstate10K.

Critical strengths
  • Clear motivation. Placing the 3D inductive bias in the latent itself is a clear idea, and the diagnosis of the DA3 hierarchy (3,072 channels with an effective rank of about 11) concretely motivates the compact reparameterization.

  • Careful design. The frozen DPT head, reference-only encoding of observed views, and clean reference tokens are well reasoned.

  • Convincing qualitative results. The supplementary videos show long rollouts whose decoded geometry and camera trajectories remain consistent with the generated RGB.

  • Clear presentation. The paper is easy to follow.

Critical weaknesses
  • Attribution of the gains and relation to VA-VAE. Without L_repr, the GAE codec does not yet improve FVD over SD-VAE (66.4 in Tab. 7 vs. 258.6 on RealEstate10K in Tab. 5), although its camera accuracy is already better. Part of the appearance gain possibly comes from the representation alignment, which the pixel-VAE baselines do not receive. Moreover, L_repr appears closely related to the VF loss of VA-VAE (Yao et al., CVPR 2025), which also combines a per-location cosine alignment and a pairwise similarity-matrix alignment on the tokenizer latent; this work is neither cited nor discussed.

  • Baseline configuration. All latents share one DiT configuration (width 768), which may not suit high-dimensional latents such as RAEv2 and raw DA3 features. The handling of WAN2.1's temporal compression is also unclear, and the strongest competitor is the single-image SD-VAE.

  • Some statements could be clarified. The conclusion mentions that larger-scale experiments "further confirm" the gains, whereas the final model appears to be evaluated only qualitatively; slightly softer wording may better match the presented evidence. It would also be helpful to specify the source of the metric pose scale.

Questions for authors
  • Could the authors clarify how L_repr differs from the VF loss of VA-VAE, and, if feasible, compare with a pixel VAE trained with a comparable alignment?

  • Was the DiT configuration (e.g., width, timestep shift) adjusted for high-dimensional latents, and how are WAN2.1's temporal compression and differing token counts handled?

  • The GLD numbers on RealEstate10K appear nearly identical in Tab. 3 and Tab. 6. Could the authors clarify whether this is expected?

  • Could the authors add uncompressed DA3 on real frames to Tab. 3 as an upper bound?

  • Could the authors report the variance of the generation metrics (Tabs. 5–6) across a few sampling seeds, given that the ranking between GAE-64 and GAE-128 changes between the 64- and 256-scene pools?

  • Could the authors discuss limitations briefly?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.