- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Latent Bundle Adjustment (Latent BA), a test-time optimization framework for jointly refining camera poses and dense geometry predicted by existing 3D reconstruction models, such as VGGT-Ω and π³. The main idea is to optimize the intermediate latent representations of a pretrained reconstruction network using a reprojection loss. Specifically, the method first constructs a co-visibility graph from the initial reconstruction, extracts 2D correspondences using RoMa v2, and then minimizes the reprojection error by updating the latent tokens through gradient descent. Experiments show that applying Latent BA as a test-time optimization step improves the performance of existing reconstruction models across multiple benchmarks.
Critical strengths
- The paper proposes an intuitive approach that incorporates correspondence-based bundle adjustment by optimizing the intermediate representations of pretrained reconstruction models. The pretrained model provides a good initialization, while the additional correspondences provide strong geometric constraints for refining camera poses and depth estimates.
- The method demonstrates good improvements in camera pose estimation and dense depth reconstruction across multiple datasets. Figure 5 further shows that the method can refine camera poses even when the initial reconstruction is inaccurate.
Critical weaknesses
-
Limited novelty and unclear benefits of latent optimization. Joint optimization of camera poses and dense geometry through differentiable geometric objectives has already been explored in prior works. The main novelty of this paper seems to be optimizing intermediate latent representations of pretrained reconstruction models rather than explicit geometry or network weights. While this is an interesting idea, I am not fully convinced that the paper clearly demonstrates why this parameterization is more effective. In particular, it is unclear how much of the improvement comes from the learned geometric prior, the choice of optimization variables, or simply having a strong feed-forward initialization.
-
Computational overhead and runtime. As shown in Figure 9, the default configuration (k=14) can only handle around 80 frames on a 96 GB GPU, raising concerns about scalability to longer sequences. Moreover, the runtime analysis only covers the latent optimization stage, without including RoMa v2 correspondence extraction, co-visibility graph construction, or geometric filtering. A complete end-to-end runtime and memory comparison with conventional BA would help clarify whether the additional computational cost is justified.
-
Limited robustness and failure case analysis. The method relies on a reasonable initialization and reliable correspondences, both of which are important for successful bundle adjustment. However, the current experiments do not sufficiently explore how the method behaves when these assumptions are violated. For example, it is unclear whether the method can still achieve meaningful improvements when the initial poses or depth estimates are inaccurate, or when the correspondences contain significant noise and outliers.
Questions for authors
1. Benefits of latent optimization. Could the authors clarify what makes optimizing latent representations particularly advantageous compared to other parameterizations, such as fine-tuning a subset of the model weights? I am also curious how much the method relies on pretrained weights and initialization. For example, what happens if the model is randomly initialized, or if the initial latent representations are significantly perturbed?
2. Runtime and memory usage. Could the authors provide the end-to-end runtime and peak GPU memory usage of the full pipeline, including RoMa v2 correspondence extraction, co-visibility graph construction, geometric filtering, and latent optimization? A comparison with conventional BA would help clarify the computational trade-offs.
3. Robustness to noisy initialization and correspondences. How sensitive is the method to the quality of the initial camera poses, depth estimates, and extracted correspondences? Could the authors provide results under different levels of pose/depth initialization errors or correspondence noise (e.g. using classical SIFT + nearest neighbor) ? This would help assess whether latent optimization is more robust than conventional BA under challenging conditions. Also can the authors provide some failure cases?
4. Scalability to long sequences. Is there a practical way to extend the method to longer video sequences, given the memory limitations shown in Figure 9?
3 comments
Failure cases
Failure cases are typically caused by wrongly matched pairs such as doppelgangers, or missing pair matches that can create informative loop closures for large scale corrections. Orphan or wrongly matched frames can lead to individual frames not getting optimised or converging to a wrong solution. The typical consequences of bad or incorrect matches still affect our method. An example can be found in the teaser of the paper Fig. 1, the couple of frames with long red lines are examples of frames that had matching issues. We will add more qualitative failure examples in the final manuscript.
Scalability to long sequences
A pragmatic approach to enable scaling to larger sequences is to chunk the scene into submaps and optimise them individually, which is a direction we plan to explore in future work. While fundamentally this feed forward models are memory bounded. We inherit this limitation from feed forward reconstruction models, which are getting more efficient so we expect our method to get more efficient as they improve. More importantly this method can be viewed as an offline data generator for dense annotations for training future foundation feed forward models.
Thank you for the detailed response and additional experiments. I appreciate the authors' efforts in addressing my concerns and encourage them to include these additional results, analyses, and failure cases in the revised paper. I would like to maintain my current rating of Weak Accept.
We thank the reviewer for their valuable feedback.
Benefits of latent optimisation
Our analysis in section 4.3 attempts to answer some of the questions that you are rightfully asking. We attribute the effectiveness of our parametrisation to the following to the following:
Perturbing the initial latent
Thanks for the experiment suggestion, we ran an additional experiment adding gaussian noise to the initial predicted latent. We ran this on a subset of the ETH3D MVS. For sigma<=1 the estimation quality doesn’t drop. Pose estimates seem to be more robust than the depth refinement, but at sufficiently high perturbed noise within the same optimisation budget of 100 iterations, the recovered geometry degrades.
Runtime and memory usage
Please see our general comment on runtime and memory analysis above. There we include a breakdown and full comparison with classic BA as well as the other baselines.
Robustness to noisy initialization and correspondences
Experiments on Oxford Spires and SMERF show the initialisation from VGGT-Ω and π3 were very noisy as witnessed by the low AUC@1/5/20 results, our method manages to recover/optimise these scenes despite the bad initialisation. As shown in the tables below. Fig.1 and Fig. 5 provide visualisations on how bad the initialization looks like.
Robustness to inaccurate geometry predictions
Thanks for the valid concern. The experiment above about perturbing the latents demonstrates how much noise our optimisation can deal with given the fixed optimisation budget. While it is challenging to induce noisy outputs naturally without creating unrealistic artificial setups, luckily in our evaluations the front ends that we evaluated failed on a couple of the scenes or performed poorly. Qualitatively, this can be seen in Fig. 5 along with our method successfully recovering them. Below we show a snippet from Table. 1, where the feed forward models initial estimates are very low, scoring zeros in some tight tolerances, while Latent BA is able to significantly boost the accuracy of these models on these datasets.
Pose estimation results on SMERF. Extracted from Table. 1
Pose estimation and geometry results on Oxford Spires. Extracted from Table. 1 and Table. 2.
Performance under weaker matches
We provide additional results with SIFT + nearest neighbor instead of RoMa v2 on ETH3D MVS. Our method still performs well in comparison to classical BA.