acceptodds
Withdrawn from ICLR 2027

Latent Bridge: From Video Latents to 3D Geometry via Feature Alignment

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video generation and feed-forward 3D reconstruction typically use separate representations, requiring RGB decoding and visual re-encoding to recover geometry from generated latents. We propose Latent Bridge, which connects video generative latent spaces with geometric feature spaces to directly predict 3D geometry from video latents. We develop representation-specific bridges for Wan2.1 and V-RAE, enabling feed-forward prediction of cameras, depth, and 3D point clouds without the intermediate RGB pathway. Each bridge is trained on RGB videos through feature distillation, while the video encoder and geometry model remain frozen. Training requires neither ground-truth geometry nor execution of the downstream geometric aggregator or prediction heads. At inference, the same video latents support both RGB decoding and geometry prediction. The same trained bridge can be reused for image-to-video and text-to-video generation within the same video latent space, without bridge retraining. Experiments on Co3Dv2 and WildRGB-D demonstrate competitive geometric reconstruction quality with improved inference efficiency, highlighting latent alignment as an efficient and reusable connection between video generation and 3D reconstruction.

Primary area:
generative models
Keywords:
Video Generation, 3D Reconstruction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_s2E5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Latent Bridge, which maps generated video latents to the feature space of a frozen VGGT-Ω model. It thereby predicts cameras, depth, and point clouds without decoding the latents to RGB and re-encoding the frames for geometry. The authors train separate Wan2.1 and V-RAE bridges through feature distillation, without ground-truth geometry or executing the downstream geometry heads during bridge training. On CO3Dv2 and WildRGB-D, the bridge pathway gives point-cloud results close to an RGB-to-Geometry pathway and reduces latency. The Wan2.1 variant also has the lowest reported end-to-end latency among the compared systems.

Critical strengths
  1. The frozen feature interface is a practical way to reuse a pretrained geometry model without training its aggregator or prediction heads. The supervision and trainable modules comparison in Table 6 shows a substantial reduction in per-step training time and memory when the bridge is trained at the feature level.

  2. The paper evaluates two distinct video latent representations. With the generated latents and VGGT-Ω backend held fixed, the proposed bridge maintains broadly comparable reconstruction quality while reducing inference time. These controlled comparisons provide convincing evidence for the effectiveness of the proposed interface. Furthermore, Table 1 demonstrates a clear system-level latency advantage for the Wan2.1 variant under the reported experimental setup.

Critical weaknesses
  1. The technical novelty is limited. Gen3R already aligns VGGT-derived geometry latents with video latents, while VIST3A stitches video latents to a pretrained 3D model. This paper’s main distinction is feature-only alignment to a frozen VGGT-Ω interface. Table 4 shows that the full Bridge outperforms a simple variant in matching teacher features and depth, but does not establish a matched parameter budget. It is therefore unclear whether the gain comes from the proposed design or greater model capacity. The ablation also does not show whether this gain improves point-cloud reconstruction from generated videos.

  2. Limited evidence for generalization under challenging conditions. The paper claims that the Bridge can be reused across generators sharing the same video latent space, but provides only qualitative results for this claim. Its reconstruction performance on videos generated with more challenging camera trajectories and on longer sequences also remains unclear. Quantitative point-cloud results in these settings, compared with the RGB→VGGT-Ω route on the same generated videos, would strengthen the evaluation. For longer sequences, the authors should also clarify how they are processed and assess geometric consistency.

Questions for authors
  1. Novelty. Gen3R and VIST3A have already connected video latents with 3D reconstruction representations. Beyond mapping to a fixed VGGT-Ω feature interface while keeping the geometry model frozen, what is the key technical novelty of the proposed Bridge relative to these methods?

  2. Architecture versus capacity. Are the simple Bridge baseline and the full Bridge in Table 4 matched in parameter count and computational cost? Could you provide a capacity-matched comparison, including geometry results on generated videos, to distinguish gains from the architecture from gains due to model capacity?

  3. Effect of the geometry backbone. Gen3R uses VGGT, whereas your method uses VGGT-Ω. Could you provide a controlled quantitative comparison using the same geometry backbone? For example, evaluating Gen3R with VGGT-Ω or applying your method to VGGT.

  4. Generalization. Could you report quantitative point-cloud reconstruction results for more challenging camera trajectories, longer sequences, and other compatible generators? How does the method handle longer sequences and maintain geometric consistency? In these settings, how does the Bridge compare with the RGB→VGGT-Ω route on the same generated videos?

1 comment
AuthorsOpenReview ↗

Thank you for your review. We have conducted additional experiments to address your concerns and provided detailed responses. We look forward to your further feedback!

W1&Q1: Key technical novelty of the proposed Bridge relative to these methods

Our core innovation lies in aligning the generative latent space with the reconstruction (DINO) latent space, rather than jointly generating RGB and geometry or training an entirely new geometry decoder, as illustrated in Figure 1 of the main paper.

  1. Difference from GEN3R: Our method follows a fundamentally different paradigm. GEN3R jointly generates RGB and geometry tokens, as illustrated in Figure 1 (a) of the main paper, whereas our method trains a model to bridge the two representations.

  2. Difference from VIST3A: VIST3A selects a stitching layer for each model pairing and adapts the downstream geometry network via LoRA. In contrast, our method focuses on learning a mapping from video latents to the geometry feature space. This offers two advantages: (1) training requires neither the geometry aggregator nor the prediction heads, reducing computation and GPU memory usage; and (2) the pretrained geometry model’s parameters remain unchanged, avoiding adaptation-induced changes to its pretrained capabilities while enabling reuse of the aligned features for other downstream tasks with compatible DINO feature interfaces.

W1&Q2: Architecture versus capacity.

Thank you for raising this concern. We have extended Table 4 to report parameter counts and downstream 3D reconstruction results.

The interpolation and Simple variants have approximately 8.83% fewer parameters, primarily because replacing the learned temporal upsampling module with parameter-free interpolation necessarily reduces the parameter count unless additional capacity is introduced elsewhere.

In terms of reconstruction quality, the interpolation variant and Simple baseline each perform better on one dataset but worse on the other, whereas Full offers a more balanced trade-off across the two datasets. Moreover, Table 4 shows that Full achieves the best depth-fitting metrics while maintaining competitive feature alignment.

AblationChanged Module Params (M)Total Params (M)Parameter ChangeCO3D CD ↓WildRGB-D CD ↓
Full—35.652—0.2705940.069990
w/o PixelUnshuffle → Average Pooling0.0333 → 0.008735.627-0.069%0.2717890.070188
w/o Causal Conv3D + Phase Rearrangement → Interpolation3.1478 → 032.504-8.829%0.2679530.070764
w/o Spatiotemporal Mixing Blocks → Token-wise MLPs31.6723 → 31.694035.673+0.061%0.2846230.070128
Simple: All Three Replacements34.8534 → 31.702732.501-8.837%0.2998410.067987
w/o Cosine LossUnchanged35.65200.2763950.070484
w/o Delta LossUnchanged35.65200.2750200.070473
w/o Relation LossUnchanged35.65200.2705970.070150

Q3: Effect of the geometry backbone.

We use VGGT-Ω because its visual encoder and V-RAE are both based on DINOv3, whereas the original VGGT uses DINOv2. This choice provides a more compatible feature space for alignment. We additionally train a V-RAE Bridge to align with the original VGGT, with the results shown below:

Geometry Backbone / BridgeCO3D Acc↓Comp↓CD↓WildRGB-D Acc↓Comp↓CD↓
VRAE VGGT-Ω 50K0.323100.209270.266180.023710.107620.06566
VRAE VGGT 50K0.314830.173780.244310.016170.125840.07101

As shown in the table, the VGGT-based variant performs better on CO3D but slightly worse on WildRGB-D, suggesting that VGGT and VGGT-Ω achieve broadly comparable performance within our framework. Adjusting the training-data mixture may enable the VGGT-based variant to achieve more balanced performance across both datasets, but time constraints prevented us from conducting additional training experiments.

W2&Q4: Generalization.

Since our original model was trained on fixed-length sequences of 49 frames, inference on longer, more complex sequences requires further fine-tuning on longer sequences. We conducted experiments by fine-tuning on sequences of 100 and 150 frames, with the results shown in the table below.

Checkpoint / Pipeline100-frame CD ↓150-frame CD ↓
Original Bridge94K0.104470.12932
Mixed-length fine-tuning, 4,437 steps0.104910.07948
RGB→Ω0.094670.07599

The results show that, after fine-tuning on longer sequences, our method still achieves geometric reconstruction quality comparable to that of the RGB decoding pathway, indicating that Bridge can scale to longer sequences with fine-tuning. In fact, other methods, such as GEN3R, also struggle to support longer sequences and require more costly fine-tuning. Since our method only requires fine-tuning the Bridge, the training cost is lower, further highlighting its advantage.

Reviewer_soF9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Latent Bridge, a feature-alignment framework that maps compressed video-generation latents from Wan2.1 or V-RAE directly into the patch-token space of a pretrained VGGT-Ω model, thereby bypassing RGB decoding and visual re-encoding for 3D reconstruction. The Bridge is trained by distilling features from the frozen VGGT-Ω encoder, while the video encoder, geometry aggregator, and depth/camera prediction heads remain frozen. Consequently, the method requires neither ground-truth geometry nor execution of the downstream geometry network during training. Experiments on Co3Dv2 and WildRGB-D demonstrate competitive reconstruction performance and reduced inference latency compared with the conventional RGB-decoding pipeline.

Critical strengths
  1. The paper presents a conceptually simple and practically relevant approach to connecting video-generation models with pretrained 3D reconstruction systems.

  2. The encoder-specific Bridge designs appropriately handle the spatial, temporal, and channel mismatches between video latents and reconstruction features.

  3. The method also largely preserves the reconstruction quality of the RGB-decoding pipeline

Critical weaknesses
  1. According to Table 3, the RGB-decoding pipeline and the proposed Bridge achieve largely comparable reconstruction performance. However, the improvement in inference speed appears relatively modest, especially for V-RAE, where the latency is reduced by only about one second—a difference that may be acceptable in many practical settings. In contrast, the RGB-decoding pipeline provides greater architectural flexibility: its RGB output can be readily consumed by reconstruction models with different architectures, such as Pi3,WorldMirror etc. and it can also accommodate different video generation models like LTX-Video or MiniMax. The proposed Bridge, however, seems tightly coupled to the specific video latent representation, encoder configuration, and model checkpoint on which it is trained. Given this trade-off, what is the key advantage of the proposed paradigm over the more general RGB-decoding pipeline, and in what practical scenarios would its relatively modest efficiency gain justify the reduced model compatibility and the additional Bridge training cost?

  2. The paper lacks a direct comparison with VIST3A, which is arguably one of the most relevant baselines. Please discuss different advantages and limitations. [VIST3A: Text-to-3D by Stitching a Multi-view Reconstruction Network to a Video Generator]

  3. The Bridge is trained on latents obtained by encoding real RGB videos, whereas at inference time it operates on latents generated by a diffusion model. These two latent distributions may differ substantially. The paper does not quantify this train–test distribution gap or evaluate the robustness of the Bridge to different samplers, guidance scales, denoising budgets, or generator checkpoints. It would be important to compare performance on encoded latents and generated latents, and to analyze how geometry quality degrades as the generated latent moves away from the VAE-encoded data manifold.

  4. The comparison in Table 1 conflates video-generation quality with geometry-decoding quality, since different methods appear to use different generators and system configurations. A lower Chamfer Distance may result from a stronger or more geometrically consistent video generator rather than from a better latent-to-geometry interface.

  5. Although the method claims to recover depth, camera intrinsics, camera extrinsics, and point clouds, the main evaluation focuses almost exclusively on point-cloud distances. The paper should additionally report standard depth metrics and camera-pose errors, including rotation, translation direction, and possibly focal-length error.

Questions for authors

Please refer to the weaknesses.

2 comments
AuthorsOpenReview ↗

Thank you for your review. We will address each of your questions in turn and look forward to your response!

W1: The Core Advantages of Latent Bridge over the RGB-Based Pipeline

The key advantage of our proposed paradigm is that it decouples tasks such as 3D point cloud reconstruction and foreground segmentation (see the appendix) from RGB decoding. On the one hand, when downstream tasks do not require RGB output, we can bypass time-consuming VAE decoding and predict geometric or semantic information directly from the latents. On the other hand, when downstream tasks require both RGB and geometry, our method still provides some speedup because RGB and geometry decoding can run in parallel.

As shown in the table below, when RGB output is not required, the time and computation for latent-to-RGB decoding can be eliminated entirely. When both RGB and 3D outputs are required, latent-to-RGB and latent-to-3D decoding can run in parallel. In our tests, this reduces decoding latency by 50% for WAN and 20% for VRAE.

WAN (RTX 5090):

PipelineLatent→RGBLatent→3dOutput SynchronizationTotal Latency ↓Latency Reduction
Latent→RGB→Ω1.947 s2.692 s0.13 ms4.640 s—
RGB+Bridge (Sequential)1.947 s2.333 s0.12 ms4.281 s7.7%
RGB+Bridge (Single-GPU Parallel)4.239 s3.139 s0.12 ms4.239 s8.6%
RGB+Bridge (Dual-GPU Parallel)1.927 s2.316 s4.55 ms2.321 s50.0%

V-RAE:

PipelineLatent→RGBLatent→3dOutput SynchronizationTotal Latency ↓Latency Reduction
Latent→RGB decoder→ Ω0.238 s2.705 s0.13 ms2.943 s—
RGB+Bridge (Sequential)0.238 s2.314 s0.13 ms2.553 s13.3%
RGB+Bridge (Single-GPU Parallel)2.335 s2.472 s0.13 ms2.472 s16.0%
RGB+Bridge (Dual-GPU Parallel)0.235 s2.307 s4.56 ms2.312 s21.4%

Our Latent Bridge requires only feature-distillation-based alignment, keeping its training cost low, and can be reused across different generative models sharing the same VAE latent space without retraining. Below, we outline several application scenarios where decoupling RGB decoding from downstream geometry reconstruction can offer substantial efficiency benefits:

Application ScenarioRequired OutputsBenefit of Latent Bridge
Geometry-based screening of generated scenesDepth or point cloudsEvaluate candidate geometry first and decode RGB only for selected candidates.
Imagined-world planningGeometry for occupancy and collision reasoningEvaluate hypothetical futures without rendering every candidate into RGB.
Semantic 3D grouping and spatial queriesSemantic features, masks, or labeled point cloudsSupport task-specific predictions through auxiliary heads without RGB decoding.
Interactive video-and-3D previewsBoth RGB and geometryExecute independent decoding branches concurrently to reduce joint-output latency.

W2: Adding a Comparison with VIST3A

We added a comparison with VIST3A. Since VIST3A primarily focuses on T2V, while our work focuses on I2V, we use our I2V model paired with VIST3A’s 3D decoder as the baseline for this comparison.

MethodCo3Dv2 Accuracy ↓Co3Dv2 Completeness ↓Co3Dv2 CD ↓WildRGB-D Accuracy ↓WildRGB-D Completeness ↓WildRGB-D CD ↓Latency (s) ↓
Aether0.69820.54370.62090.05910.13400.0965181.33
FantasyWorld0.59390.36980.48180.04400.12580.0849448.63
Gen3R0.38200.17960.28080.02470.11290.0688189.57
VIST3A decoder*0.38240.18620.28430.01800.12630.072270.64
Ours (Wan2.1)0.35360.18330.26850.01890.12090.069970.09
Ours (V-RAE)0.32310.20930.26620.02370.10760.0657165.57

Distinction: VIST3A selects a stitching layer for each model pairing and adapts the downstream geometry network via LoRA. In contrast, our method focuses on learning a mapping from video latents to the geometry feature space. This offers two advantages: (1) training requires neither the geometry aggregator nor the prediction heads, reducing computation and GPU memory usage; and (2) the pretrained geometry model’s parameters remain unchanged, avoiding adaptation-induced changes to its pretrained capabilities while enabling reuse of the aligned features for other downstream tasks with compatible DINO feature interfaces.

W3: train–test distribution gap

We compared Generated Latents and Real-Encoded Latents across two reconstruction pathways. The results reveal a clear dataset dependence in the reconstruction gap between the two latent types: the gap is larger on CO3Dv2 and relatively smaller on WildRGB-D. Spatial misalignment between the generated content and the ground-truth masks may be one contributing factor. For both latent types, the Bridge pathway achieves reconstruction results comparable to those of the RGB decoding pathway, indicating that Bridge can transfer from latents encoded from real videos to generated latents. However, this does not imply that the distribution gap between training and inference has been eliminated.

ModelDecoding RouteCO3D: Generated LatentsCO3D: Real-Encoded LatentsWildRGB-D: Generated LatentsWildRGB-D: Real-Encoded Latents
WANLatent → RGB → Ω0.26570.1421810.07000.070843
WAN Bridge94KLatent → Bridge → Ω0.26850.1493570.06990.071347
V-RAELatent → RGB → Ω0.25830.1130260.06610.071810
V-RAE Bridge50KLatent → Bridge → Ω0.26620.1234260.06570.071027

We additionally conducted a fine-tuning experiment using generated latents. We added 10,000 generated latent samples to the Bridge training data and continued training from 94k to 114k steps. The results show that incorporating generated latents reduces the discrepancy relative to the RGB → VGGT-Ω pipeline. For example, the CD on CO3Dv2 decreased from 0.26849 to 0.26617, suggesting that training with generated latents helps narrow the distribution gap. However, the improvement is modest, making this approach suitable for post-training fine-tuning of the Bridge.

MethodCO3Dv2 Accuracy ↓CO3Dv2 Completeness ↓CO3Dv2 CD ↓WildRGB-D Accuracy ↓WildRGB-D Completeness ↓WildRGB-D CD ↓
Latent → RGB → VGGT-Ω (Wan2.1)0.35590.17540.26570.01740.12270.0700
Original Bridge (94K)0.353650.183330.268490.018860.120890.06987
Bridge fine-tuned on generated latents (114K)0.353720.178620.266170.018250.122890.07057
Bridge fine-tuned on real-video latents (114K)0.353980.184720.269350.017970.122140.07006

W4: Different methods appear to use different generators and system configurations

Our task is to evaluate which system produces better 3D results given an initial image and a camera trajectory, rather than to improve latent-to-geometry decoding quality in isolation. In practice, we cannot ensure that all systems use the same generator for two reasons: (1) Aether and FantasyWorld use generative backbones with different parameter counts from the other methods, making it infeasible to match them; (2) FantasyWorld and Gen3R incorporate geometry tokens during generation, which differs fundamentally from our design and prevents full alignment. Nevertheless, we additionally compare our method with VIST3A using the same generative model. Under this controlled setting, our method achieves better latent-to-3D decoding quality, as shown in Table in W2. This also demonstrates that our latent-to-geometry interface is simple and delivers high-quality results.

W5: Depth metrics and camera-pose errors

We have added camera-pose and depth evaluations across different methods. Our method can predict reasonably accurate camera poses and depth maps.

CO3Dv2:

MethodRRE ↓ (°)AUC@20 ↑ (%)AbsRel ↓δ1 ↑ (%)
Aether22.12104.28931.1851364.87
FantasyWorld23.17785.25880.1881074.14
Gen3R4.839442.45750.0877693.41
VIST3A3.402154.87010.0920993.09
Ours (Wan2.1)2.582553.63600.0951093.77
Ours (V-RAE)8.253942.28760.0840193.48

WildRGB-D:

MethodRRE ↓ (°)AUC@20 ↑ (%)AbsRel ↓δ1 ↑ (%)
Aether8.544117.31310.1877584.01
FantasyWorld19.679216.72350.1736780.28
Gen3R1.954967.53460.0500697.39
VIST3A1.829382.22700.0314298.69
Ours (Wan2.1)0.747586.36020.0292698.73
Ours (V-RAE)0.886484.70080.0325298.39

RRE ↓ (°) measures the angular error between predicted and ground-truth relative camera rotations, averaged over camera pairs. Lower values indicate more accurate relative orientations.

AUC@20 ↑ (%) is the normalized area under the relative-pose accuracy curve over angular thresholds from 0° to 20°. For each camera pair, the pose error is the maximum of the rotation error and translation-direction error. Higher values indicate better pose accuracy.

Reviewer_VRAKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The manuscript proposes Latent Bridge, which connects video latents to a pretrained 3D geometry model, allowing direct prediction of cameras, depth, and point clouds without decoding RGB frames and re-encoding them. It uses separate bridge networks for Wan2.1 and V-RAE to match VGGT-Omega's visual features. Only the bridges are trained, using RGB videos and feature-matching losses, while the video encoders and geometry model remain frozen.

Critical strengths
  1. Matching patch tokens makes training much cheaper. Unlike methods that jointly denoise geometry tokens or train through large geometry networks, Latent Bridge aligns features before the frozen geometry model’s aggregator. This requires no 3D labels and skips the aggregator and prediction heads during training. Table 6 reports significantly faster computation and much less memory.

  2. The comparison reveals a useful trade-off between pixel quality and geometry quality. V-RAE compresses space more strongly and has lower pixel scores than Wan2.1, but its DINOv3-based latents produce better point-cloud reconstruction. It also uses about 1/3 of the training budget. This suggests that features useful for visual understanding can also help recover geometry from video latents.

  3. The bridge is easy to reuse when models share the same latent or feature space. Without changing the video or geometry models, the Wan2.1 Bridge works with both a Waymo video generator and Wan2.1-T2V-1.3B. The appendix also connects the bridge to a frozen VGGT-S head for cross-frame mask propagation and 3D object segmentation, showing reuse beyond reconstruction.

Critical weaknesses
  1. The claimed speedup may not hold when both RGB video and geometry are needed. VAE decoding is still required, so Bridge only replaces DINOv3 encoding and may even be slower. Table 3 saves just 1.12–3.00 s. Table 1's much larger gains also depend on generator sizes, sampling steps, and CFG settings that are not reported.

  2. The gap between training and inference inputs is not properly tested. Bridge trains on latents encoded from real videos but receives diffusion-generated latents at inference. Skipping RGB decoding exposes Bridge directly to this gap. Tests on real-video latents and accuracy results with geometry supervision are needed to separate the sources of error.

  3. Geometry tests and point-cloud alignment details are incomplete. The manuscript reports no camera-pose accuracy or geometry results on DL3DV and RealEstate10K. It does not explain how predicted point clouds are scaled and aligned before measuring errors in meters. Using ground-truth pixel masks on generated frames that may not match their positions also needs explanation.

Questions for authors
  1. When both RGB video and 3D geometry are required, VAE decoding must still run. How long does DINOv3 encoding of the decoded frames take compared with Bridge, and does Bridge still save time? Please also break down Table 1's runtime into video generation and 3D reconstruction, reporting DiT sizes and denoising steps.

  2. On the 64 validation videos or CO3Dv2 test set, how do real-video latents and diffusion-generated latents differ in evaluation metrics? Can noise augmentation or training on generated latents reduce Table 3's completeness and accuracy gaps?

2 comments
AuthorsOpenReview ↗

Thank you for your review. We provide additional latency benchmarks, showing that parallel RGB and geometry decoding preserves speedups when both outputs are required. We also evaluate the training–inference distribution gap, test fine-tuning with generated latents, and add camera-pose comparisons while clarifying the geometric evaluation protocol and its limitations. We look forward to your response!

W1&Q1: Does the speedup still hold when both RGB video and 3D geometry are required?

RE: Yes. When both RGB and geometry are required, our method still provides a speedup for two reasons:

  1. Since our geometry decoding does not depend on RGB decoding, the two can run in parallel. As shown in the table below, our pipeline supports parallel inference across two GPUs, reducing latency by 50% for WAN and 21% for VRAE(49 frames at a resolution of 560 × 560).
  2. Bridge inference is itself faster than DINOv3 encoding. In our tests, DINOv3 encoding takes approximately 0.561 seconds, while Bridge inference takes approximately 0.065 seconds.

WAN:

PipelineLatent→RGBGeometry BranchTotal Latency ↓Latency Reduction
Latent→RGB→Ω1.947 s2.692 s4.640 s—
RGB+Bridge (Sequential)1.947 s2.333 s4.281 s7.7%
RGB+Bridge (Single-GPU Parallel)4.239 s3.139 s4.239 s8.6%
RGB+Bridge (Dual-GPU Parallel)1.927 s2.316 s2.321 s50.0%

V-RAE:

PipelineLatent→RGBGeometry BranchTotal Latency ↓Latency Reduction
Latent→RGB decoder→ Ω0.238 s2.705 s2.943 s—
RGB+Bridge (Sequential)0.238 s2.314 s2.553 s13.3%
RGB+Bridge (Single-GPU Parallel)2.335 s2.472 s2.472 s16.0%
RGB+Bridge (Dual-GPU Parallel)0.235 s2.307 s2.312 s21.4%

We provide a breakdown of the end-to-end generation time for each model in the table below. With the same number of sampling steps, our method achieves higher accuracy than larger models such as Aether and FantasyWorld. Moreover, compared with GEN3R, which has the same parameter count and uses the same number of sampling steps, our method achieves faster generation because it does not require additional geometry tokens for joint generation.

MethodDiT ParametersDenoising StepsVideo/Joint GenerationGeometry PredictionOther OverheadTotal Latency
Aether5.57B30 + 4153.9523.004.38181.33
FantasyWorld14B30445.331.102.20448.63
Gen3R1.62B30186.221.981.69189.89
Ours (Wan2.1)1.62B3064.745.35N/A70.09
Ours (V-RAE)0.85B30154.445.795.35165.57

W2&Q2: The gap between training and inference inputs is not properly tested.

RE:We compared Generated Latents and Real-Encoded Latents across two reconstruction pathways. The results reveal a clear dataset dependence in the reconstruction gap between the two latent types: the gap is larger on CO3Dv2 and relatively smaller on WildRGB-D. Spatial misalignment between the generated content and the ground-truth masks may be one contributing factor. For both latent types, the Bridge pathway achieves reconstruction results comparable to those of the RGB decoding pathway, indicating that Bridge can transfer from latents encoded from real videos to generated latents. However, this does not imply that the distribution gap between training and inference has been eliminated.

ModelDecoding RouteCO3D: Generated LatentsCO3D: Real-Encoded LatentsWildRGB-D: Generated LatentsWildRGB-D: Real-Encoded Latents
WANLatent → RGB → Ω0.26570.1421810.07000.070843
WAN BridgeLatent → Bridge → Ω0.26850.1493570.06990.071347
V-RAELatent → RGB → Ω0.25830.1130260.06610.071810
V-RAE BridgeLatent → Bridge → Ω0.26620.1234260.06570.071027

We added 10,000 generated latent samples to the Bridge training data and continued training from 94k to 114k steps. The results show that incorporating generated latents reduces the discrepancy relative to the RGB → VGGT-Ω pipeline. For example, the CD on CO3Dv2 decreased from 0.26849 to 0.26617, suggesting that training with generated latents helps narrow the distribution gap. However, the improvement is modest, making this approach suitable for post-training fine-tuning of the Bridge.

MethodCO3Dv2 Accuracy ↓CO3Dv2 Completeness ↓CO3Dv2 CD ↓WildRGB-D Accuracy ↓WildRGB-D Completeness ↓WildRGB-D CD ↓
Latent → RGB → VGGT-Ω (Wan2.1)0.35590.17540.26570.01740.12270.0700
Original Bridge (94K)0.353650.183330.268490.018860.120890.06987
Bridge fine-tuned on generated latents (114K)0.353720.178620.266170.018250.122890.07057
Bridge fine-tuned on real-video latents (114K)0.353980.184720.269350.017970.122140.07006

W3: Geometry tests and point cloud alignment details are incomplete.

We added an evaluation of the camera poses predicted by different models.

CO3Dv2:

MethodRRE ↓ (°)RTE ↓ (°)AUC@20 ↑ (%)nATE ↓
Aether22.121051.23784.28930.3936
FantasyWorld†23.177854.19905.25880.4322
Gen3R4.839415.904142.45750.1701
Ours (Wan2.1)2.582514.054553.63600.1421
Ours (V-RAE)8.253921.097742.28760.2537

WildRGB-D:

MethodRRE ↓ (°)RTE ↓ (°)AUC@20 ↑ (%)nATE ↓
Aether8.544143.553217.31310.3819
FantasyWorld†19.679230.141816.72350.2998
Gen3R1.95496.675367.53460.0653
Ours (Wan2.1)0.74752.919586.36020.0234
Ours (V-RAE)0.88643.516884.70080.0351

We cannot report geometric evaluation results on DL3DV and RealEstate10K because these datasets do not provide 3D ground truth. We follow the same GT foreground-mask protocol as the Gen3R baseline in our evaluation. Since generated videos may not exactly match the reference content and ground-truth point clouds are available only for foreground objects, we apply the corresponding GT masks to all generated point clouds for evaluation. We acknowledge that content misalignment may introduce errors, nevertheless, better scores under this shared protocol generally indicate greater geometric fidelity and spatial consistency with the reference scene.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.