acceptodds
Desk rejected at ICLR 2027

Aligned Geometry Latent Space for Improved World Consistency in Joint Video-Geometry Modeling

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

While jointly modeling video with compressed geometry latents has been shown to effectively improve geometric consistency in video generation, such methods typically suffer from slow convergence or training instability. Our analysis suggests that this issue is closely related to the misalignment between compressed geometry latents and pretrained video models: higher reconstruction fidelity does not necessarily make the latents more effective for video models to understand and utilize. Motivated by this, we propose geometry latent alignment (GeoLA), a post-training framework that directly leverages the video generator to supervise geometry compression and produce a well-aligned geometry latent. Specifically, we use geometry latents compressed by a geometry adapter as conditions for video denoising, and optimize the adapter with the diffusion loss. This allows the video model to implicitly guide the adapter to retain the geometric information most useful for denoising. We then freeze the aligned adapter for joint video-geometry modeling, followed by dual-stream distribution matching distillation for low-latency joint generation. To support training and evaluation, we further introduce GeoVid-80K, a dataset containing over 80,000 video clips with rich geometric variations. Experiments show that GeoLA converges faster during joint training and generates videos with improved geometric consistency.

Primary area:
generative models
Keywords:
video generation, joint modeling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YvxAOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes GeoLA, a three-stage framework for improving geometric consistency in video generation by aligning compressed geometry latents with the needs of a pretrained video diffusion model. The key observation is that geometry latents with better reconstruction fidelity are not necessarily easier for a video generator to use; in the authors’ analysis, reconstruction-oriented compression can produce weaker geometric control and slower convergence in joint video–geometry modeling. GeoLA therefore trains a lightweight geometry adapter directly with the video flow-matching objective so that the compressed latent retains the geometric information most useful for denoising. The aligned adapter is then frozen and used for joint video–geometry flow matching, after which the joint distribution is distilled into a few-step generator for efficient inference. The paper also introduces GeoVid-80K, a curated dataset of 84,877 videos with geometry and motion annotations. Experiments on both text-to-video and image-to-video generation show improved 3D reconstruction and epipolar consistency while maintaining competitive visual quality, and controlled ablations indicate benefits from the proposed alignment objective, joint geometry–video modeling, and joint distribution distillation.

Critical strengths
  1. A well-motivated geometry-latent alignment objective. The paper identifies an important mismatch between reconstruction-oriented geometry compression and the needs of a pretrained video generator: higher geometry reconstruction fidelity does not necessarily lead to better geometric control or faster joint video–geometry training. GeoLA addresses this by directly optimizing the geometry adapter using the video flow-matching objective, allowing the downstream generator itself to determine which geometric information should be retained. I find this formulation intuitive and practically meaningful.

  2. A complete and empirically supported pipeline. The paper evaluates all three stages—geometry alignment, joint video–geometry modeling, and joint distribution distillation—with dedicated ablations. In particular, the controlled comparisons in Tables 4–6 provide evidence that the proposed alignment objective, joint modeling, and geometry-aware distillation each contribute to improved 3D consistency. The resulting model also achieves strong geometric metrics with only five inference steps, making the overall system practically compelling.

Critical weaknesses
  1. The headline comparisons are not fully apples-to-apples. The main T2V and I2V tables compare GeoLA against released checkpoints from different methods, while GeoLA uses Wan2.1-1.3B, DepthAnything3 geometry features, and an additional Wan2.1-14B teacher for distillation. The compared systems therefore differ in backbone, model scale, training data, geometry representation, training budget, and inference pipeline. As a result, Tables 2–3 demonstrate that the complete GeoLA system is strong, but they do not cleanly isolate the advantage of the proposed geometry-latent alignment itself. The controlled ablations in Tables 4–6 partially address this issue, but only for selected baselines and components.

  2. The behavior of the distilled model on genuinely dynamic scenes is insufficiently characterized. GeoVid-80K includes both static-scene camera motion and dynamic subject/object motion, but many qualitative examples emphasize static environments with strong camera movement. The current 3D consistency metrics therefore do not make it clear whether the distilled model preserves strong geometry while also maintaining complex object/human dynamics, or whether some of the gains are dominated by easier static-scene camera-motion cases. This is particularly important because prior work such as VideoGPA shows that enforcing 3D consistency can also benefit dynamic scenes.

Questions for authors
  1. Could the authors provide a more apples-to-apples comparison with strong baselines? Ideally, I would like to see at least one strong baseline reimplemented with the same Wan backbone, the same DepthAnything3 geometry encoder, the same GeoVid-80K training data, and a comparable training/distillation budget, changing only the geometry representation/compression strategy. This would make it much clearer how much of the gain comes specifically from GeoLA not from differences in backbone, data, or distillation setup.

  2. How does the final distilled model perform specifically on dynamic-subject or dynamic-object scenes? VideoGPA suggests that regulating 3D consistency can also improve generation in dynamic scenes. I am therefore particularly interested in the behavior of GeoLA after joint distribution distillation on such cases. Could the authors report results separately for static-subject/camera-motion scenes and genuinely dynamic scenes, or provide additional quantitative and qualitative examples involving human/object motion and interactions? This would help establish that the improved 3D consistency is preserved without sacrificing dynamic scene generation.

I am happy to say it's above borderline paper due to its solid engineering and novel idea. I am more than happy to raise the score if my concerns can be resolve during the rebuttal.

5 comments

1. Apples-to-apples comparisons

Thanks for the constructive feedback. We would like to clarify that all methods in Table 4 (DreamWorld, GeometryVAE, Gen3R, and Ours) use exactly the same experimental settings as Ours in Table 3, including the Wan2.1-1.3B architecture, training data, geometry encoder, and training budget, with the only difference being that distillation is not applied in Table 4 (as already clarified in the appendix of our submitted manuscript). This controlled comparison demonstrates that our performance gains primarily stem from our proposed geometry representation and compression strategy, rather than differences in experimental settings.

2. More Evaluation on Static and Dynamic Scenes

2.1. Qualitative Results

Thank you for this valuable suggestion. Our method preserves 3D consistency without sacrificing dynamic scene generation, as demonstrated by four experimental settings: (1) static scenes with standard camera motion, (2) static scenes with large camera motion, (3) dynamic scenes with standard camera motion, and (4) dynamic scenes with large camera motion.

Table (1). Static Scenes with Standard Camera Motion

MethodBackground ↑Subject ↑PSNR ↑SSIM ↑LPIPS ↓Epipolar Error ↓Inlier Rate ↑
GeoVideo0.93960.909320.39290.73670.276522.07060.4129
VideoGPA0.92770.875418.51430.60730.300012.76660.5164
GeoLA-T2V0.93830.954823.77340.78000.16218.03550.6726

Table (2). Static Scenes with Large Camera Movements

MethodBackground ↑Subject ↑PSNR ↑SSIM ↑LPIPS ↓Epipolar Error ↓Inlier Rate ↑
GeoVideo0.93090.898718.93440.65160.328531.73400.3163
VideoGPA0.92200.863716.04470.50960.34068.96320.6564
GeoLA-T2V0.92990.958521.81860.74330.19196.27880.7411

Table (3). Dynamic Scenes with Standard Motion

MethodBackground ↑Subject ↑PSNR ↑SSIM ↑LPIPS ↓Epipolar Error ↓Inlier Rate ↑
GeoVideo0.89930.868318.76410.63620.365341.58070.1949
VideoGPA0.89300.799017.15670.53800.371211.58490.5981
GeoLA-T2V0.92670.954220.68710.74600.24008.59110.6232

Table (4). Dynamic Scenes with Large Motion

MethodBackground ↑Subject ↑PSNR ↑SSIM ↑LPIPS ↓Epipolar Error ↓Inlier Rate ↑
GeoVideo0.90010.844817.46960.58770.429947.89330.1326
VideoGPA0.89190.795614.66550.49440.445211.75710.5750
GeoLA-T2V0.92680.947818.35040.66710.358410.41980.5713

Across all four settings, the better reconstruction quality and lower epipolar error demonstrate that GeoLA-T2V maintains superior 3D geometric consistency in both static and dynamic scenes, even under large camera and object motions. Moreover, the higher subject consistency across all settings and improved background consistency in dynamic scenes further indicate that our method preserves high visual quality under challenging motion conditions.

2.2. Quantitative Results

Thank you for the suggestion. We have added qualitative examples involving human and object motions and interactions, with the new video results included in the ZIP archive of the Supplementary Material. Unlike static scenes, these examples involve changing subject poses and spatial relationships between objects over time. The generated videos illustrate that our method can accommodate such dynamic changes, providing further evidence that joint RGB–geometry modeling does not restrict dynamic scene generation.

Dear Reviewer YvxA,

Thank you once again for your valuable comments and suggestions. We hope that our responses have effectively addressed your concerns and provided sufficient clarification on the points you raised. We would sincerely welcome any remaining questions or additional suggestions, as your insights would be extremely helpful in further refining and strengthening our paper.

As the rebuttal period is approaching its end, we would greatly appreciate any further feedback you may have. Thank you again for your time and thoughtful consideration.

Thank you for your response. Great work! It would be great if you can benchmark on different VideoGPA backbones. GeoVideo has very bad reward hacking problems.it van only generate static, noise, yet large motion scenes. As far as I know it only released a very bad version on static scene. VideoGPA has ckpt on WAN and Cogvideo. It would be very helpful and informative to the community if you can benchmarking on both Cogvideo and Wan backbone.

Reviewer_Ba2tOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GeoLA, a method for improving geometric consistency in video generation. The method first learns a compact geometry adapter from a pretrained geometry model (DA3) using the diffusion denoising objective by jointly optimizing the parameters of the adapter and the model. It then freezes the geometry adapter and trains the model to jointly denoise video and geometry latents. The authors also introduce a joint distribution matching distillation for few-step generation. In addition, the paper introduces GeoVid-80K and evaluates the method on text-to-video and image-to-video generation.

Critical strengths

The paper is well-written and easy to follow. The method of using learned adapter to encode geometry feature is simple and reasonable, and the gains on geometric consistency metrics are noticeable.

Critical weaknesses

First, the choice of using features from a reconstruction model as the geometry representation seems rather heuristic to me. These features are used by the original reconstruction model to predict geometry, such as depth maps or point clouds, and are highly compressed into only four channels to be fed into the diffusion model. It is unclear what additional geometric information they contain compared with a simple depth map, which is the end product of the reconstruction model and naturally compatible with a VAE. Aligning these features with the video generator does not directly answer this question. My feeling is that jointly generating RGBD could achieve similar results.

Second, the idea of using a learned adapter that is jointly trained with the generation objective is very straightforward. Similar ideas have been explored in many previous works. It is a common way to learn additional information encoders. The introduced dual-stream distribution-matching distillation is a straightforward extension of the DMD framework and is loosely related to the main purpose of the paper.

A follow-up question is whether this post-training affects the general generation ability of the base model. For example, can it still generate stylized videos or complex motions? The examples also do not sufficiently cover large object motion or deformation, where geometry features from the reconstruction model become less reliable. I think the paper should illustrate these cases and include their visual results too, and discuss whether improving geometric consistency comes at the price of losing general generation capability.

Questions for authors

Please refer to the weakness section.

5 comments

1 Motivation for Using Geometric Features Instead of Depth Maps

Thank you for your valuable comment. We choose intermediate geometry features rather than depth maps for two main reasons:

  1. Intermediate features contain more geometric information beyond depth. Specifically, large-scale reconstruction models jointly learn depth, point maps, camera parameters, and cross-view correspondences through a shared backbone with multiple prediction heads. As a result, their intermediate backbone features capture diverse geometric cues, including camera pose and cross-view correspondences, whereas depth maps represent only one aspect of scene geometry. This allows our joint modeling framework to leverage geometric information beyond depth alone.

  2. Implicit geometry latents provide structural guidance in a softer, non-pixel-aligned form. Recent studies [1, 2, 3] suggest that diffusion models can leverage their strong priors to accommodate small geometric inconsistencies in implicit geometry latents. This allows these latents to provide informative and robust structural guidance while reducing sensitivity to depth noise, further motivating our choice of intermediate geometry features over explicit depth maps.

[1] Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors. NeurIPS 2025.

[2] GSFixer: Improving 3D Gaussian Splatting with Reference-Guided Video Diffusion Priors. ICML 2026.

[3] LaVR: Scene Latent Conditioned Generative Video Trajectory Re-rendering Using Large 4D Reconstruction Models. CVPR 2026.

To further validate this design choice through experiments, we compare joint RGB–depth and RGB–feature modeling on both I2V and T2V tasks.

  1. For I2V, we implement a joint RGB–depth modeling baseline (DepthJoint) using the Wan 1.3B backbone. Both DepthJoint and our GeoLA-I2V are trained on the fully public RealEstate10K dataset. The results are shown below.
MethodBackground Consistency ↑Subject Consistency ↑PSNR ↑SSIM ↑LPIPS ↓Epipolar Error ↓Inlier Rate ↑
DepthJoint0.93180.939920.46540.70890.26386.45750.7223
GeoLA-I2V0.95730.968421.18260.72750.31095.35820.7788

The results on RealEstate10K show that both RGB–depth and RGB–feature joint modeling achieve competitive performance. However, GeoLA-I2V outperforms the depth-based baseline on six out of seven metrics, achieving better visual quality and 3D geometric consistency. These improvements demonstrate the effectiveness of intermediate geometric features for joint modeling, supporting our first motivation to leverage richer geometric information beyond depth maps.

  1. For T2V, we retrain GeoLA-T2V exclusively on the fully public DL3DV-10K dataset and compare it with GeoVideo, which adopts joint RGB–depth prediction. The results are shown below.
MethodBackground Consistency ↑Subject Consistency ↑PSNR ↑SSIM ↑LPIPS ↓Epipolar Error ↓Inlier Rate ↑
GeoVideo0.93300.900219.21800.67640.312020.86910.3254
GeoLA-T2V0.94760.933421.02400.67250.235415.82580.4280

On the more challenging DL3DV-10K dataset with rapid and irregular viewpoint changes, GeoLA-T2V outperforms the depth-based baseline on six out of seven metrics. This demonstrates that the advantages of feature-based joint modeling persist even under challenging camera motions, providing further support for our second motivation to use intermediate features for robust structural guidance.

2. Novelty

Thank you for your comment. Our method is straightforward yet effective, and its novelty lies not in introducing new architectural components, but in providing new insights into geometry compression and joint modeling, and developing a novel and effective joint modeling framework based on these insights.

We would like to clarify our novelty and contributions in two aspects.

First, we first reveal a mismatch between reconstruction-oriented geometry learning and geometry-aware video generation. As shown in Fig. 2 and Table 4, our experiments show that higher geometry reconstruction fidelity does not necessarily lead to better geometric consistency in generated videos, challenging the conventional assumption that better reconstruction naturally benefits downstream generation.

Second, we propose a generation-oriented geometry modeling framework to address this mismatch. Instead of optimizing geometry compression for reconstruction fidelity, we leverage supervision from the video generation model to learn representations better suited for generation, shifting from reconstruction-driven to generation-driven geometry compression. The diffusion-loss-trained adapter is a practical implementation of this paradigm, rather than the core novelty itself.

Third, we address the limitations of conventional distillation for real-time joint RGB–geometry generation. Existing single-stream DMD methods are not designed for joint RGB–geometry modeling and may fail to preserve the geometric consistency gains achieved through joint modeling. To address this, we propose a joint modeling-aware dual-stream DMD strategy to preserve these gains while enabling real-time video generation.

3. Additional Qualitative Results on Complex Motion Generation

Thank you for your insightful suggestion. Following your suggestion, we conduct additional experiments to further evaluate the effectiveness of our method under complex motion scenarios, including (1) static scenes with complex camera motion and (2) dynamic scenes with both complex object motion.

  1. Static scenes with complex camera motion. We evaluate our method on static scenes with complex camera trajectories. The results are shown below.
MethodBackground ↑Subject ↑PSNR ↑SSIM ↑LPIPS ↓Epipolar Error ↓Inlier Rate ↑
GeoVideo0.93090.898718.93440.65160.328531.73400.3163
VideoGPA0.92200.863716.04470.50960.34068.96320.6564
GeoLA-T2V0.92990.958521.81860.74330.19196.27880.7411
  1. Dynamic scenes with complex motion. We further evaluate our method on dynamic scenes with complex subject motion. The results are shown below.
MethodBackground ↑Subject ↑PSNR ↑SSIM ↑LPIPS ↓Epipolar Error ↓Inlier Rate ↑
GeoVideo0.90010.844817.46960.58770.429947.89330.1326
VideoGPA0.89190.795614.66550.49440.445211.75710.5750
GeoLA-T2V0.92680.947818.35040.66710.358410.41980.5713

The better reconstruction quality and lower epipolar error across both challenging scenarios demonstrate that GeoLA maintains superior 3D geometric consistency even under complex motion. Moreover, the higher subject consistency and comparable background consistency indicate that GeoLA preserves visual quality despite large camera and subject motions. Together, these results further demonstrate the effectiveness of joint RGB–geometry modeling under complex motion.

Qualitative Results. We have also added qualitative examples involving complex camera and subject motions, with the new video results included in the ZIP archive of the Supplementary Material. These examples provide visual evidence that GeoLA can generate videos with large viewpoint changes and dynamic subject motions, further illustrating that improved geometric consistency does not come at the expense of motion generation capabilities.


Reviewer_NDxTOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The authors point out that higher reconstruction fidelity of geometry features does not necessarily correspond to better generative learnability. Therefore, they propose GeoLA: first, a geometry adapter and a video DiT are jointly trained with a video flow-matching loss, so that the compressed geometry representation is suitable for video denoising; then, the adapter is frozen, and video and geometry are jointly generated in a fixed representation space; finally, joint distribution matching distillation is performed. The authors also construct GeoVid-80K, a dataset containing 84,877 video clips. Results show that the method outperforms compared methods on multiple reconstruction and epipolar consistency metrics, while maintaining competitive video quality with fewer sampling steps.

Critical strengths

1) GeoLA is somewhat innovative;

2) On the authors' curated GeoVid-80K dataset, both the quantitative comparison results and the visual results are favorable, further demonstrating the reasonableness of GeoLA's design.

Critical weaknesses

Critical Weakness: Training and evaluation data are not reproducible, and fair comparison is opaque.

GeoVid-80K contains approximately 22% undisclosed Private collection data, and the public sources (RealEstate10K, SpatialVID, MiraData) were also filtered by the authors. However, the paper does not provide the complete filtering pipeline, thresholds, or the final clip list. This leads to the following issues: (1) external parties cannot reproduce the training set; (2) it is impossible to determine whether other baselines were trained or tuned under identical data conditions, making fair comparison opaque; and (3) it cannot be confirmed whether the method's advantages depend on private data or filtered data.

Questions for authors

GeoVid-80K contains an undisclosed private data collection (approximately 22%), and the public sources (RealEstate10K, SpatialVID, MiraData) were also filtered by the authors. However, the paper does not provide the complete filtering pipeline or the final clip list. This limits the reproducibility of the training set; meanwhile, the transparency of the test set is also insufficient. This makes fair comparison with other methods under identical data conditions difficult. We recommend that the authors additionally supplement results on standard, fully public evaluation sets to demonstrate that the method's generalization does not depend on private data or filtered data.

3 comments

1. Reproducibility and Transparency

Thank you for raising these concerns. For the training set, we will release all code, datasets, and filtering pipelines with full implementation details after the anonymity period. For the test set, we have already added the complete T2V prompt list and I2V image list to the revised supplementary material.

2. Evaluation on fully public evaluation sets

Thank you for your insightful suggestion. Following your suggestion, we conduct additional experiments to further evaluate the generalization ability of our method on fully public datasets.

  1. For T2V, we retrain and evaluate GeoLA-T2V exclusively on DL3DV-10K. The results are shown below.
MethodBackground Consistency ↑Subject Consistency ↑PSNR ↑SSIM ↑LPIPS ↓Epipolar Error ↓Inlier Rate ↑
GeoVideo0.92290.884719.21800.67640.312020.86910.3254
VideoGPA0.91330.840215.25850.53880.352816.21780.4920
GeoLA-T2V0.94130.920021.02400.67250.235415.82580.4280
  1. For I2V, the results reported in Table 3 of the main manuscript are already obtained using only the fully public RealEstate10K dataset for both training and evaluation. We copy the tables here for your reference.
MethodBackground Consistency ↑Subject Consistency ↑PSNR ↑SSIM ↑LPIPS ↓Epipolar Error ↓Inlier Rate ↑
ViewCrafter0.97170.962118.75810.66510.389621.98720.3301
Gen3C0.96840.969519.93750.67550.36305.71020.7578
Gen3R0.98210.967519.39770.68640.41045.54250.7708
GeoLA-I2V0.95730.968421.18260.72750.31095.35820.7788

The better reconstruction quality and lower epipolar error on fully public datasets demonstrate that GeoLA maintains its advantages in 3D geometric consistency, consistent with the findings in Tables 2 and 3 of the main manuscript. Together, these results demonstrate the generalization ability of our method without relying on additional private or specially curated training data.


Dear Reviewer NDxT,

Thank you once again for your constructive comments. We would like to know if our responses have adequately addressed your concerns. We are also eager to hear any additional concerns or suggestions from you that could help us further strengthen our paper. Your feedback would be invaluable in making this work even more compelling.

As the rebuttal window is closing soon, we genuinely appreciate your feedback. We hope our responses have addressed your concerns.

Dear Reviewer NDxT,

As the rebuttal period ends today, we sincerely thank you for your valuable feedback and constructive suggestions. We hope that our responses have adequately addressed your concerns.

Thank you again for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary for the Area Chair

Dear Area Chair,

We thank you and the reviewers for the constructive discussion. Below we briefly summarize our contributions, the main additions from the rebuttal, and the current status of the reviews.

Contributions.

  • We first reveal a mismatch between reconstruction-oriented geometry learning and geometry-aware video generation. (Fig. 2 and Table 4 in manuscript).
  • Second, we propose a generation-oriented geometry modeling framework to address this mismatch.
  • Third, we address the limitations of conventional distillation through real-time joint RGB–geometry generation.

Main additions from the rebuttal.

  • Fairness (NDxT). When trained on public data alone, GeoLA retains its gains on most metrics for both I2V on RealEstate10K and T2V on DL3DV-10K (21.02 dB reconstruction PSNR vs. 19.22 for GeoVideo).
  • Transparency (NDxT). We will release the complete training data, code, and filtering pipeline after the anonymity period. The revised supplementary material already includes the full test set.
  • Motivation (Ba2t). We clarified why we chose geometry features over depth maps and validated this choice experimentally (epipolar error 5.36 vs. 6.46 for I2V and 15.83 vs. 20.87 for T2V).
  • Novelty (Ba2t). We clarified that GeoLA’s novelty lies in experimentally identifying the reconstruction–generation mismatch and proposing a generation-oriented geometry modeling framework to address it.
  • Performance on Dynamic Scenes (YvxA, Ba2t). GeoLA outperforms VideoGPA-CogVideo and VideoGPA-Wan2.2 even with large subject motion (subject consistency 0.9478 vs. 0.7956/0.9370, epipolar error 10.42 vs. 11.76/12.64).

Revisions (supplementary material). We added the complete text-to-video prompt and image-to-video input lists and provided qualitative results for dynamic scenes.

Status of the reviews. Two of the three reviewers recommend weak acceptance before rebuttal.

  • Reviewer YvxA (3, confidence 4) cited the paper’s “solid engineering and novel idea” and said they would consider raising their score if their concerns were addressed. In follow-up, they wrote “Great work!” and requested the two-backbone VideoGPA comparison, which we provided.
  • Reviewer NDxT (3) recognized our contributions. Following their suggestion, we evaluated GeoLA on fully public datasets and added the complete test input lists and results to the supplement.
  • Reviewer Ba2t (2). We responded to all of the reviewer’s concerns and have received no follow-up comments from them to date.

Thank you for considering our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.