acceptodds
Under review as a conference paper at ICLR 2027

Latent Bundle Adjustment

19 views

Abstract

We present a method for dense bundle adjustment of 3D scenes by optimising the latent representations of feed-forward reconstruction models, such as VGGT-Omega and . These learned models can be more robust than traditional Structure-from-Motion (SfM) pipelines, particularly on challenging scenes. However, combining their robustness with the accuracy of classical bundle adjustment remains challenging and is typically limited to poses and sparse points or patches.We propose a novel parametrisation of the bundle adjustment problem by optimising a set of tokens within a feed-forward reconstruction model. Specifically, we solve for latent updates that minimise reprojection error given a set of sparse correspondences between the input frames. Latents jointly encode the dense scene structure and camera poses, and the change in representation provides implicit regularisation during optimisation, allowing for refinement of dense geometry from sparse matches. We evaluate our approach on 3D reconstruction and camera pose estimation tasks across different datasets, demonstrating state-of-the-art performance. We further analyse parametrisations, both qualitatively and quantitatively.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Computer vision, Bundle adjustment, SFM

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_u9rsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Latent Bundle Adjustment (Latent BA), a test-time optimization framework for jointly refining camera poses and dense geometry predicted by existing 3D reconstruction models, such as VGGT-Ω and π³. The main idea is to optimize the intermediate latent representations of a pretrained reconstruction network using a reprojection loss. Specifically, the method first constructs a co-visibility graph from the initial reconstruction, extracts 2D correspondences using RoMa v2, and then minimizes the reprojection error by updating the latent tokens through gradient descent. Experiments show that applying Latent BA as a test-time optimization step improves the performance of existing reconstruction models across multiple benchmarks.

Critical strengths
  • The paper proposes an intuitive approach that incorporates correspondence-based bundle adjustment by optimizing the intermediate representations of pretrained reconstruction models. The pretrained model provides a good initialization, while the additional correspondences provide strong geometric constraints for refining camera poses and depth estimates.
  • The method demonstrates good improvements in camera pose estimation and dense depth reconstruction across multiple datasets. Figure 5 further shows that the method can refine camera poses even when the initial reconstruction is inaccurate.
Critical weaknesses
  1. Limited novelty and unclear benefits of latent optimization. Joint optimization of camera poses and dense geometry through differentiable geometric objectives has already been explored in prior works. The main novelty of this paper seems to be optimizing intermediate latent representations of pretrained reconstruction models rather than explicit geometry or network weights. While this is an interesting idea, I am not fully convinced that the paper clearly demonstrates why this parameterization is more effective. In particular, it is unclear how much of the improvement comes from the learned geometric prior, the choice of optimization variables, or simply having a strong feed-forward initialization.

  2. Computational overhead and runtime. As shown in Figure 9, the default configuration (k=14) can only handle around 80 frames on a 96 GB GPU, raising concerns about scalability to longer sequences. Moreover, the runtime analysis only covers the latent optimization stage, without including RoMa v2 correspondence extraction, co-visibility graph construction, or geometric filtering. A complete end-to-end runtime and memory comparison with conventional BA would help clarify whether the additional computational cost is justified.

  3. Limited robustness and failure case analysis. The method relies on a reasonable initialization and reliable correspondences, both of which are important for successful bundle adjustment. However, the current experiments do not sufficiently explore how the method behaves when these assumptions are violated. For example, it is unclear whether the method can still achieve meaningful improvements when the initial poses or depth estimates are inaccurate, or when the correspondences contain significant noise and outliers.

Questions for authors

1. Benefits of latent optimization. Could the authors clarify what makes optimizing latent representations particularly advantageous compared to other parameterizations, such as fine-tuning a subset of the model weights? I am also curious how much the method relies on pretrained weights and initialization. For example, what happens if the model is randomly initialized, or if the initial latent representations are significantly perturbed?

2. Runtime and memory usage. Could the authors provide the end-to-end runtime and peak GPU memory usage of the full pipeline, including RoMa v2 correspondence extraction, co-visibility graph construction, geometric filtering, and latent optimization? A comparison with conventional BA would help clarify the computational trade-offs.

3. Robustness to noisy initialization and correspondences. How sensitive is the method to the quality of the initial camera poses, depth estimates, and extracted correspondences? Could the authors provide results under different levels of pose/depth initialization errors or correspondence noise (e.g. using classical SIFT + nearest neighbor) ? This would help assess whether latent optimization is more robust than conventional BA under challenging conditions. Also can the authors provide some failure cases?

4. Scalability to long sequences. Is there a practical way to extend the method to longer video sequences, given the memory limitations shown in Figure 9?

3 comments

We thank the reviewer for their valuable feedback.

Benefits of latent optimisation

Our analysis in section 4.3 attempts to answer some of the questions that you are rightfully asking. We attribute the effectiveness of our parametrisation to the following to the following:

  • Choice of variables. In Fig. 7, we show that our latent parametrisation achieves more accurate local corrections close to the matches compared to optimising the network weights. Our interpretation of this is that modifying the network weights modifies the prior that provides the gradient signal, since this is where the learned prior about 3D scenes is stored. Fig. 11 provides visual examples of coupling effects on depth under different parametrisations.
  • Pretrained vs randomly initialised. We fundamentally rely on having a structured latent space that a decoder can transform to meaningful outputs. NeRFs are an example of networks that are optimised per scene from scratch. However, they do require a source of dense supervision to get a dense output. In our case we are able to achieve dense refinement from sparse measurements thanks to the learned prior providing correlations between the scene elements.

Perturbing the initial latent

Thanks for the experiment suggestion, we ran an additional experiment adding gaussian noise to the initial predicted latent. We ran this on a subset of the ETH3D MVS. For sigma<=1 the estimation quality doesn’t drop. Pose estimates seem to be more robust than the depth refinement, but at sufficiently high perturbed noise within the same optimisation budget of 100 iterations, the recovered geometry degrades.

sigmaStart AUC@1/5/20Final AUC@1/5/20Start AbsRelFinal AbsRel
017.3/62.2/86.080.7/95.7/98.90.01990.0075
0.515.7/57.5/84.180.8/95.8/98.90.02140.0077
13.4/36.1/72.682.4/96.2/99.10.03230.0087
20.0/1.7/18.874.9/94.1/98.50.08110.0204
30.0/0.2/4.759.6/84.7/94.60.13830.0463

Runtime and memory usage

Please see our general comment on runtime and memory analysis above. There we include a breakdown and full comparison with classic BA as well as the other baselines.

Robustness to noisy initialization and correspondences

Experiments on Oxford Spires and SMERF show the initialisation from VGGT-Ω\Omega and π3\pi^3 were very noisy as witnessed by the low AUC@1/5/20 results, our method manages to recover/optimise these scenes despite the bad initialisation. As shown in the tables below. Fig.1 and Fig. 5 provide visualisations on how bad the initialization looks like.

Robustness to inaccurate geometry predictions

Thanks for the valid concern. The experiment above about perturbing the latents demonstrates how much noise our optimisation can deal with given the fixed optimisation budget. While it is challenging to induce noisy outputs naturally without creating unrealistic artificial setups, luckily in our evaluations the front ends that we evaluated failed on a couple of the scenes or performed poorly. Qualitatively, this can be seen in Fig. 5 along with our method successfully recovering them. Below we show a snippet from Table. 1, where the feed forward models initial estimates are very low, scoring zeros in some tight tolerances, while Latent BA is able to significantly boost the accuracy of these models on these datasets.

Pose estimation results on SMERF. Extracted from Table. 1

ModelMethodAUC@1 ↑AUC@5 ↑AUC@20 ↑
π3\pi^3Init0.214.149.3
Latent BA46.376.287.1
VGGT-Ω\OmegaInit1.336.678.5
Latent BA31.874.092.8

Pose estimation and geometry results on Oxford Spires. Extracted from Table. 1 and Table. 2.

ModelMethodAUC@1 ↑AUC@5 ↑AUC@20 ↑Depth AbsRel ↓
π3\pi^3Init0.08.753.40.0986
Latent BA55.682.791.30.0831
VGGT-Ω\OmegaInit0.07.254.30.1101
Latent BA30.560.480.70.0954

Performance under weaker matches

We provide additional results with SIFT + nearest neighbor instead of RoMa v2 on ETH3D MVS. Our method still performs well in comparison to classical BA.

MethodAUC@1 meanAUC@5 meanAUC@20 mean
π3\pi^3 init11.850.479.5
Classic BA56.381.693.5
Latent BA58.885.794.2

Failure cases

Failure cases are typically caused by wrongly matched pairs such as doppelgangers, or missing pair matches that can create informative loop closures for large scale corrections. Orphan or wrongly matched frames can lead to individual frames not getting optimised or converging to a wrong solution. The typical consequences of bad or incorrect matches still affect our method. An example can be found in the teaser of the paper Fig. 1, the couple of frames with long red lines are examples of frames that had matching issues. We will add more qualitative failure examples in the final manuscript.

Scalability to long sequences

A pragmatic approach to enable scaling to larger sequences is to chunk the scene into submaps and optimise them individually, which is a direction we plan to explore in future work. While fundamentally this feed forward models are memory bounded. We inherit this limitation from feed forward reconstruction models, which are getting more efficient so we expect our method to get more efficient as they improve. More importantly this method can be viewed as an offline data generator for dense annotations for training future foundation feed forward models.

Thank you for the detailed response and additional experiments. I appreciate the authors' efforts in addressing my concerns and encourage them to include these additional results, analyses, and failure cases in the revised paper. I would like to maintain my current rating of Weak Accept.

Reviewer_G8vyOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Existing bundle adjustment (BA) methods lack correlation between point clouds. They also do not optimize pose and depth jointly, even though depth should move consistently whenever the pose changes. In contrast, recent feed-forward networks constrain their gradients to move along the geometry manifold, and their latents jointly encode pose and depth. This paper leverages these properties to perform BA.

Critical strengths
  • The motivation is good. Existing BA methods optimize points and poses separately and then require additional optimization to obtain dense results. This paper addresses this problem through latent optimization.
  • The experimental results show that the method achieves consistently strong performance across various datasets.
  • The analysis is also interesting. It is notable that a larger k captures more global context of the sequence, and that the optimization converges well instead of getting stuck in local minima.
Critical weaknesses
  1. Limited contribution. The contribution is quite limited. Simply put, the paper shows that BA can be achieved by adapting the intermediate weights of a feed-forward model. For such a simple idea, proofs or deeper analysis become more important. The analysis in Fig. 2 is good, but I would like to understand why this phenomenon occurs, that is, why the optimization converges to minima with good metrics rather than to poor ones. An analysis of how VGGT’s weights make the points move when the tokens change would also strengthen the paper.

  2. Missing baselines. In the experiment tables, it would be good to also report the other baselines on VGGT-Omega. In addition, to my knowledge, VGGT and VGGT-Omega come with their own very fast BA. A comparison with these would be helpful.

  3. Computational cost. My biggest concern is the computational cost. Since the computational graph extends deep into the network, the memory consumption and runtime are likely to be very high.

  4. Dependence on the foundation model. The reliance on a foundation model is also a minor concern. Since VGGT already predicts point correspondences, could these be used to perform BA in a self-contained manner?

Questions for authors

My major concern is the contribution of this work. Could you clarify it? Or I just think that this paper needs more analysis of why it works effectively. I am also concerned about the trade-off between efficiency and performance.

3 comments

We thank the reviewer for their valuable feedback.

Contribution and analysis clarification

To our knowledge dense BA has not been shown with feed forward foundation models, especially our latent parametrisation using them has not been demonstrated and analysed before. Our method shows strong performance in pose estimation showing its strong performance compared to other baselines with a simple and elegant formulation. Please check our general note around contributions for more detailed information.

Analysis on how points move during optimisation

Fig. 11 provides visual examples of coupling effects via the tangent kernel in depth under different parametrisations. In essence, the tangent kernel shows if we need a change in one point, how the rest of the points move. Additionally in Fig. 7, we show that our latent parametrisation achieves more accurate local corrections close to the matches compared to optimising the network weights.

Clarification around Fig. 2

The wrong minimas shown in Fig. 2 are reached by taking the optimised solution then modifying it via adversarial objectives, they are not solutions that our optimisation reaches on its own.

Additional baselines

VGGT-Ω\Omega based results

Here are the results on Classic BA, EPO* with VGGT-Ω\Omega for completeness. We also add Latent Ba optimisation via π3\pi^3 using VGGT-Ω\Omega’s matches.

MethodETH3D MVSNRGBDOxford SpiresSMERF
EPO with matches79.8 / 95.1 / 98.756.6 / 87.9 / 96.832.7 / 55.4 / 73.320.6 / 62.0 / 87.4
Classic BA83.3 / 95.3 / 98.765.0 / 89.2 / 96.043.4 / 65.2 / 80.346.9 / 79.2 / 92.9
Latent BA (VGGT-Ω\Omega)85.1 / 96.6 / 99.272.3 / 92.6 / 98.030.7 / 60.7 / 78.331.8 / 73.9 / 92.8
Latent BA (π3\pi^3, VGGT-Ω\Omega matches)85.1 / 96.4 / 99.172.4 / 92.8 / 98.153.6 / 79.9 / 89.854.0 / 85.6 / 94.4

We observe that VGGT-Ω\Omega performs well compared to classical BA except on the tighter tolerances on Oxford Spires and SMERF. When running Latent BA with π3\pi^3 on the VGGT-Ω\Omega’s co-visiability graph, we outperform classical BA across the board. Due to architectural differences, optimisation via VGGT-Ω\Omega takes half the time compared to π3\pi^3 as shown in our run time analysis, so under different hyperparameters for VGGT-Ω\Omega, similar performance could be achieved.

Clarifications for other baselines

  • VGGT BA performs sparse BA which is equivalent to our classical BA experiment which we already show with 2 different features (RoMa v2 & SIFT) on both π3\pi^3 and VGGT-Ω\Omega.
  • To the best of our knowledge VGGT-Ω\Omega doesn’t provide any particular BA recipe in the paper nor in their code base.

Computational cost

Please see our general comment on runtime and memory analysis above. Gradient checkpointing can be used to lower the memory requirements.

Dependence on the foundation model.

  • Our method, and our baselines for this matter, are agnostic to the source of feature matches. For fairness across our experiments we use the same matcher in this case RoMa v2. To the best of our knowledge, MAST3R and VGGT were the only models that predicted point tracks between the input frames, while the more recent ones, such as the front-ends we tested, do not predict point tracks. In principle, point tracks/feature matches predicted by the front end can be swapped in directly.
  • We provide additional results of using SIFT features and nearest neighbour matches and our method still shows a strong performance.
MethodAUC@1 meanAUC@5 meanAUC@20 mean
π3\pi^3 init11.850.479.5
Classic BA56.381.693.5
Latent BA58.885.794.2

Trade-off between efficiency and performance

  • Fig. 9 and Fig. 6 can be used together to provide a lens into trade-offs between a larger K performance and its time and memory implications with different numbers of frames.
  • We hope that our additional experiments on runtime and memory analysis provide more insights as well.

I thank the authors for their response. I have read the other reviews and the rebuttal, but the answers stay close to what I originally asked about without addressing it, so my concerns remain unresolved.

  1. Lack of analysis. I understand that the performance is strong. However, strong performance alone is different from a paper being useful to the community. The paper’s core claim is that optimizing the weights of a model such as VGGT is more beneficial than optimizing classical low-level representations. But the idea that VGGT weights can act as a constraint (prior) during optimization is one that anyone could come up with, and the method itself is very simple. For this reason, I believe an analysis of why this helps is necessary. My question about Fig. 2 was what causes the phenomenon and why it does not occur with your model. I was not asking for a description of the phenomenon itself.

  2. Contribution and limitations. The same applies to the contribution. I accept that the performance is higher, but the limitations remain unchanged: the method is extremely slow and memory-intensive, which restricts the number of input images. I am also concerned about how the method would handle a larger number of images.

  3. Relation to prior work. I am aware that Test3R, which the paper cites, performs adaptation in a similar spirit. Apart from being built on DUSt3R and using a different objective, the underlying concept, that optimizing the network weights to some degree is beneficial, seems essentially the same. I therefore do not understand the claim that no such analysis exists in prior work.

We thank the reviewer you for the additional feedback.

Parametrisation

We optimise latent tokens not network weights, the network weights are frozen. In our analysis we include a comparison with optimising the network weights rather than latent tokens. We provide an analysis comparing them (Sec. 4.3, App. A.4 - qualitative comparison).

Simplicity and positioning

We view the simplicity and elegance of our approach as a strength which is a valuable contribution to the community while demonstrating the effectiveness of our approach. Our method is able to compete with more complex dense SFM systems such as MP-SFM while being faster (please refer to our general comment on runtime analysis). We do not propose this method as a replacement to classical BA, if sparse points and poses are what is required, then classic BA can be suitable if speed is a constraint, although we show that Latent BA is generally more robust. But if dense geometry and poses are needed, this is what our method is designed for. We believe that downstream application is what dictates what is needed, so for example if dense annotation is required for training multiview reconstruction models from videos in the wild, our method can be very suitable.

Additional analysis

Why does it work / Fig. 2? In appendix A.4, we provide additional analysis of the coupling between the scene points under different parametrisations using the tangent kernel which is well-studied (Jacot et al) and provides analysis of the optimisation dynamics using it. We compare network weight optimisation via LoRA, and Test3r style prompt tuning with our latent parametrisation. We show this visually and it is clear that our parametrisation creates correlations that capture local geometrical relationships more accurately compared to the other parametrisations. This tangent kernel analysis is one of several analyses, which provide insights into why our method provides the correct solutions that it manages to reach.

Scalability

We agree that scaling to large scenes can be challenging. As mentioned in future work, our analysis of architectures and parallels of the tangent kernel to explicit uncertainty indicates that structured uncertainty could be predicted by these networks and used in optimisation similar to BLast3R, CodeSLAM, etc, which would enable scalability and avoid backpropagation through the network. Furthermore, existing alternatives would be to optimise subgraphs, linearise at the minimum, and perform global synchronisation across all subgraphs with these overlapping linearised subgraph constraints. In addition, multi-view reconstruction methods are continuing to improve in efficiency (e.g. VGGT → VGGT-Ω\Omega) and are able to handle larger sequences. Architectures are also evolving, for instance Solv3R (Kwak et al) can be explored for use in Latent BA optimising large sets of frames in an interleaved fashion, this is a research direction that we are excited about.

Related work

Our research builds on a rich line of work that we elaborate on in our related works section. The novelty of our method is in jointly refining dense geometry, pose and intrinsic using multi-view feed forward reconstruction models. We do acknowledge methods such as Test3r and compare against it (please see section 4.3 where we explore alternative parameterisations).

Reviewer_deoSOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Latent Bundle Adjustment, which refines the poses and dense depth of a frozen feed-forward reconstruction model (π³, VGGT-Ω) by optimising its intermediate tokens against a reprojection error over sparse RoMa v2 matches. The frozen remainder of the network decodes the tokens into poses and depth, which is argued to act as an implicit prior. Experiments on ETH3D, NRGBD, Oxford Spires and SMERF show large gains over the feed-forward outputs and improvements over classical BA.

Critical strengths

The method is simple and improves pretrained models without changing their weights, across several backbones. With the same initialisation and matches, it outperforms classical BA.

Critical weaknesses

My main concern is novelty. Optimising learned latents through a frozen pretrained network under multi-view constraints is well explored: NICE-SLAM jointly optimises latent feature grids and camera poses through frozen pretrained geometry decoders, although with a per-scene grid and RGB-D supervision, and Test3R optimises visual prompts of a frozen feed-forward reconstruction model at test time. The main change here is the choice of variable, intermediate tokens. I find this an incremental extension rather than a new BA framework, and NICE-SLAM is not cited.

Unfair use of ground-truth intrinsics. All main results use fixed ground-truth intrinsics, which the feed-forward models do not have. Without them, π³ with Latent BA drops from 84.5 to about 38 AUC@1° on ETH3D (Figure 10), while classical BA with the same intrinsics already reaches 82.9. The intrinsics are also not optimised, so it is unclear whether the method can correct calibration.

Coupling ablation. Table 3 only decouples depth, as free maps that change only at matched pixels, which is weak by construction. The more natural variant is the reverse, explicit poses with depth decoded from the latent, since poses have few degrees of freedom. Without it, the benefit of coupling is not isolated from that of the learned depth prior.

Limited scope. The method is not shown on real indoor scenes or large image collections, and memory limits the default setting to about 80 frames, so it may lack generalization capability.

Computational cost. Each iteration backpropagates through 14 transformer blocks, and RoMa v2 runs on every co-visible pair, yet no end-to-end runtime is reported. The average number of co-visible images per image on a few datasets, and an accuracy versus runtime comparison with classical BA, would help.

Depth evaluation. Table 2 uses a single scene-median scale. Since Figure 4 shows that a change in one pixel's depth spreads over the whole frame, much of the gain may come from correcting per-frame scale rather than local geometry; AbsRel after per-frame alignment would separate the two.

Minor. Several typos, e.g. "dense dense depth", "for for", "steams around".

I am open to changing my assessment based on the other reviews and the authors' response.

Questions for authors

Please see the weaknesses above.

2 comments

We thank the reviewer for their valuable feedback.

Novelty clarifications

  • Optimising latents has indeed been explored before, as we describe in our related work section, and this work is a continuation in this line of research. To our knowledge feed forward reconstruction models have not been investigated in this setting, and given our strong results we think it is an important direction to consider.
  • BA is about jointly refining poses, geometry and intrinsics. The novelty of our method is in the coupled representation that encodes poses, dense geometry and intrinsics via the latents of feed forward reconstruction models, and we show that just a simple reprojection objective, simply works well with our parametrisation, performing strongly against dense baselines such as MP-SFM, a COLMAP derivative with a more involved pipeline including incremental reconstruction, propagation and offline calibration of uncertainties, and alternating optimisation objectives to handle dense geometry.
  • As part of our study in Section 4.3 and alternatives mentioned in related work, we provided an analysis of alternative latent parameterisation for solving BA and we evaluate the proposed idea from Test3r, optimising visual prompts, and show how it compares along with other parameterisation to our proposed latent one, which our analysis demonstrates to yield better results.
  • Please see our general comment for a clear statement on our contribution.
  • Thanks for the pointer, we will add the missing citation to Nice-SLAM in our final manuscript.

Calibration questions

Intrinsics optimisation as a function of the latent.

We provide additional results on ETH3D MVS showing intrinsics optimised along with the dense geometry and poses all as a function of the latents, yielding significant improvements in pose and dense geometry over using the estimated intrinsics. In this setup we assume a single shared intrinsic across all the frames.

ModelCameraAUC@1 ↑AUC@5 ↑AUC@20 ↑ATE (cm) ↓Depth AbsRel ↓
π3\pi^3Feed-forward K37.772.890.07.590.0147
Optimised K78.694.798.61.890.0068
GT K84.596.299.00.930.0065
VGGT-Ω\OmegaFeed-forward K74.893.898.41.810.0062
Optimised K83.696.199.01.010.0060
GT K85.296.799.20.920.0059

Use of GT intrinsics

Thanks for raising this concern. For clarification, all our results so far used GT intrinsics including our baselines (classic BA, EPO, Blast3r and MP-SFM), except the feed forward models. While this does not affect the pose estimation nor depth estimation results of feed forward models, since we can not condition them on intrinsics, our pose estimation and depth results and conclusions about them thus hold.

π3\pi^3 uncalibrated results in Fig.10

Across the board Latent BA improves the initial reconstruction and this is our main objective. The quality of the predicted intrinsics vary from model to model. This explains the discrepancy between π3\pi^3’s relatively modest improvement at AUC@1 compared to the others in the uncalibrated fixed intrinsics setting .

Coupling ablation

Thanks for the constructive feedback on that experiment. We add the requested free pose variant, which strengthens our ablation. Our results still show that the fully coupled representation is still superior as shown below:

π3\pi^3 initDecoupled: free depth mapsDecoupled: free posesCoupled: latent BA
AUC@1/5/20 ↑16.3/62.0/87.576.2/94.7/98.744.1/82.4/95.184.1/96.6/99.1
ATE (cm) ↓9.024.034.390.97
Depth AbsRel ↓0.01960.01870.01150.0066

Computational cost and scope clarifications.

  • We evaluate on SMERF which is composed of an indoor scene with up to 110 frames.
  • We provide matching metrics in the runtime analysis comment.

Depth evaluation clarification

Thanks for raising this concern, we ran an additional evaluation of the dense geometry on ETH3D MVS with per frame scaling. Scale correction is part of our overall geometry gain but per frame scaling evaluation shows improvement over the initial estimate demonstrating that within frame corrections account for roughly half of the total correction shown in Table. 2. This indicates that the optimisation not only improves global consistency, but also local geometry as well. The results are shown below:

MethodInitAbsRel ↓, scene scaleAbsRel ↓, per-frame scaleδ<1.01 ↑, scene scaleδ<1.01 ↑, per-frame scale
Feed-forwardπ3\pi^30.02220.01400.4070.619
Latent BA (ours)π3\pi^30.00650.00590.8830.905
Feed-forwardVGGT-Ω\Omega0.01120.00890.6600.777
Latent BA (ours)VGGT-Ω\Omega0.00590.00580.8990.901

We will fix the typos in the final version!

Thank you for the detailed response and additional experiments. I find the results with unknown intrinsics and the per-frame depth evaluation particularly strong. I also appreciate the novelty of optimizing latents in feed-forward reconstruction networks. These results address my main concerns, and I will raise my rating.

I encourage the authors to discuss NICE-SLAM in more detail as potentially the most closely related precedent, while clarifying the differences in representation and setting.

Reviewer_Ux6sOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Latent Bundle Adjustment (Latent BA), a test-time refinement method that optimizes intermediate latent tokens ZkZ_k of frozen feed-forward 3D reconstruction models (such as π3\pi^3 and VGGT-Ω\Omega) using a robust sparse reprojection loss over RoMa v2 correspondences. Because the frozen suffix layers decode ZkZ_k jointly into camera poses and dense depth maps, sparse 2D measurements simultaneously refine camera poses and dense 3D geometry. Experiments on ETH3D, NRGBD, Oxford Spires, and SMERF, alongside ablations on coupling, layer depth kk, and test-time parameterizations, demonstrate consistent gains over feed-forward initializations and explicit BA baselines.

Critical strengths
  1. Effective joint refinement of poses and dense geometry from sparse matches. Optimizing intermediate latents through frozen decoder layers enables sparse 2D correspondences to improve dense depth as well as camera poses (e.g., reducing ETH3D AbsRel from 0.0221 to 0.0065 with π3\pi^3 and from 0.0112 to 0.0059 with VGGT-Ω\Omega in Table 2), and generalizes across four feed-forward backbones (Figure 6).
  2. Well-controlled experimental validation. Table 1 provides Classic BA (GTSAM) and EPO* with the exact same initial reconstruction and RoMa v2 matches as Latent BA, and Table 3 directly isolates the benefit of coupled latent optimization over decoupled depth maps (improving ETH3D AUC@1 from 76.2% to 84.1% and AbsRel from 0.0187 to 0.0066).
Critical weaknesses
  1. High memory/compute overhead and incomplete end-to-end runtime comparison. Figure 9 shows that on a 96-GB GH200 GPU, the default π3\pi^3 (k=14k=14) setting is capped at ~80 frames and takes ~1.5–2 s per iteration (100 iterations total), whereas Section 4 evaluates Oxford Spires sequences up to 110 frames. The paper does not report how sequences exceeding 80 frames are handled, nor does it report end-to-end wall-clock runtime and peak memory alongside Classic BA, EPO*, and MP-SfM in Table 1.
  2. Missing optimization hyperparameters and unspecified details in the Figure 2 basin analysis. While the paper attributes implicit regularization to small gradient steps from Zk(0)Z_k^{(0)}, it omits the numerical Adam learning rate, Cauchy scale, co-visibility thresholds, and Umeyama RANSAC inlier thresholds. In addition, Equation (6) writes an explicit prior P(Zk)P(Z_k) that is absent from the actual objective in Equation (7), and Figure 2 does not explain how the “Other minimum” (which shows lower Camera ATE yet worse Depth AbsRel and distorted geometry at k=3k=3) and the two latent-space directions were constructed.
Questions for authors
  1. What are the end-to-end runtime (matching + 100 optimization iterations) and peak GPU memory of Latent BA compared with Classic BA, EPO*, and MP-SfM, and how were 110-frame Oxford Spires sequences processed under the k=14k=14 setting given the 80-frame limit in Figure 9?
  2. What exact numerical values were used for the Adam learning rate, Cauchy loss scale, and co-visibility/RANSAC pruning thresholds, and are they fixed across all datasets and backbones?
  3. In Figure 2 (k=3k=3), how were the “Other minimum” (purple triangle) and the two latent directions obtained, and why does the purple solution exhibit a lower Camera ATE than the green “Optimised” solution despite visibly distorted dense geometry and worse Depth AbsRel?
2 comments

We thank the reviewer for their valuable feedback.

End-to-end runtime and peak memory

Please see our general comment on runtime and memory analysis above. We are able to process 110 frames by using gradient checkpointing, the memory analysis above elaborates on that.

Dropping regularisation term in Eq. 6

Thanks for raising this point. The prior term is meant to show the general case, we will clarify the text to make this more clear, in practice we found it had no effect hence we dropped it. To back this claim, here is an additional ablation on a subset of ETH3D MVS confirming that a regularisation term doesn’t have an impact on the final solution, here we show the results from introducing a weighted L2 regularisation term over the latent:

No regularisationL2, w=10L2, w=100
AUC@1/5/20 ↑81.4/95.9/99.081.8/95.9/99.082.3/96.0/99.0
ATE (cm) ↓1.171.191.19
Depth AbsRel ↓0.00730.00730.0073

Hyperparameters details

All results use one configuration across every scene, dataset and backbone, unless stated otherwise in the paper. Latent BA runs for 100 iterations of Adam with learning rate 3e-3 and cosine decay to zero. The reprojection loss uses a Cauchy kernel with a scale of 5 px. A frame pair is kept only if its co-visibility 3D to 2D projection is at least 0.05, and matches with certainty below 0.7 are discarded. Each pair is then checked with Umeyama RANSAC with an inlier tolerance of 0.15 × the median point spread of the pair. Pairs with an inlier ratio below 0.25 are dropped. We will make sure to include all these details in the final manuscript.

Fig. 2 clarifications

The other solution is found by an adversarial objective initialised from the optimised solution. We optimise a loss consisting of keeping the reprojection error no greater than the optimised solution while pushing the depth away from the GT depth. The main purpose of this plot was to provide intuition that while we are optimising a very high-dimensional space that contains other low reprojection error solutions, in practice, we find solutions that correlate with accurate camera poses and dense, coherent geometry.

Figure 2 shows such a case where a solution from “Token Proj” contains lower reprojection error, but this solution produces more accurate poses but much worse depth. This was to highlight the analogy to explicit parameterizations as well. For example, in low parallax scenarios, and in the extreme case of pure rotation, it may be possible to achieve accurate poses while points are only constrained along a ray due to lack of observability. More generally, noise in feature detection and matching under low parallax scenarios may thus result in inaccurate 3D points, while camera poses are typically constrained by many matches across frames and less sensitive. Bundle adjustment is the case of explicit parameterization, camera poses and 3D points, while Figure 2 also shows that larger k (backpropagating further) provides more coupling and regularization to these issues.

Reviewer_Ux6sOpenReview ↗

Follow-up after the author response

Thank you for the detailed response and additional experiments. The runtime and memory breakdown, checkpointing explanation, and numerical hyperparameters substantially address my reproducibility concerns. The clarification that the purple solution in Figure 2 was constructed adversarially is also helpful. My original 80-frame timing estimate was too low.

I maintain my rating of 3 (Weak acceptance). The empirical evidence supports useful joint refinement of camera poses and dense geometry. The latent directions and metric alignment in Figure 2 still need specification, and the general MAP motivation should be distinguished from the implemented objective and its implicit regularization. These remaining concerns limit the interpretation of the analysis but do not overturn the empirical contribution.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General remarks

We thank the reviewer for their constructive feedback. We are excited to see that all the reviewers acknowledge the effectiveness of our proposed approach and its strong performance across various datasets.

Our objective is to refine the dense geometry, poses and intrinsics of feedforward models. We present this in a form of strong performant SFM pipeline.

Our contributions are:

  • We propose a novel dense BA method that jointly refines poses, intrinsics and dense geometry predicted by feed-forward foundation models, by optimising a latent parametrisation using only sparse 2D–2D correspondences.
  • We demonstrate the effectiveness of our approach on multiple frontends, showing significant improvements across all of them on multiple datasets. In addition we show strong results against other baselines on both pose estimation and dense geometry.
  • We provide analysis and insights around effects of optimisation at different layers of these models, we investigate alternative latent parametrisations and we measure the benefits of our coupled representation. We also show how our method scales computational with different configurations.

General remarks:

  • Classical BA is a strong reference point of our performance on pose estimation, and our method outperforms it. Please note that Classic BA cannot refine dense geometry on its own, while our method does.
  • We view the simplicity of Latent BA as a strength rather than a limitation. Competing approaches rely on multi-stage pipelines, such as explicit BA followed by dense refinement (MP-SfM), or hand-designed priors and decoupled sparse/MLP pose parametrisations (EPO, BLASt3R). Latent BA instead uses a single, standard reprojection objective: the geometric prior comes entirely from the pretrained network, no retraining nor per scene hyperparameters. Despite this, it outperforms these more complex pipelines on most benchmarks, and it applies directly to different feed-forward backbone Fig. 6. We believe a simple, general mechanism that achieves this is a meaningful contribution in its own right.
AuthorsOpenReview ↗

Runtime and memory analysis

Runtime analysis

Based on the reviewers request, here we provide the end to end run time of Latent BA along with the baselines:

MethodRefined geometryFeed-forward initPair selectionMatchingFilteringMono priorsOptimisationTotalGPU peak (GiB)RAM peak (GiB)
Classic BA (π3\pi^3)sparse6.9*0.6*33.5*6.3*–166.0213.310.97.8
Classic BA (VGGT-Ω\Omega)sparse6.0*0.6*34.2*6.8*–188.9236.619.79.1
EPO with matches (π3\pi^3)sparse6.9*0.6*33.5*6.3*–6.854.210.88.4
EPO with matches (VGGT-Ω\Omega)sparse6.0*0.6*34.2*6.8*–6.153.724.19.8
MP-SfM + RoMaV2dense–7.3272.217.1291.4471.51059.410.37.5
Latent BA (π3\pi^3)dense6.90.633.56.3–823.0870.457.67.9
Latent BA (VGGT-Ω\Omega)dense6.00.634.26.8–446.1493.837.19.2

These experiments were done on a machine with H200 GPU and 16 core CPU. * indicate shared timing from latent BA with coresponding frontend. This was measured on the SMERF dataset with 20 sequences each containing 75-110 frames. Gradient checkpointing was used for Latent BA.

Takeaway: The best performing alternative that is also able to dense geometry refinement, MP-SfM, is considerably slower than our proposed method. Our pipeline is simpler with less moving parts while outperforming the COLMAP based baseline on 3 out of 4 our eval datasets while being very competitive on Oxford Spires. That being said, our dense BA is complementary to existing SFM methods.

Matching statistics

For the end to end runtime analysis above we provide extra statistics about covisibility and matches.

MethodMean matched pairsMean co-visible frames per frameMean 2D matches per pair
π3\pi^3 init49710.1560
VGGT-Ω\Omega init55211.2560

Memory analysis

We clarify that our results were obtained with gradient checkpointing, we will update the plot in Fig. 9 to include gradient checkpointing numbers.

Below we show the frame capacity and time need for a single backpropagation pass on π3\pi^3 at K=14 on a H200 GPU, showing the frame capacity and speed achieved given a memory budget.

GPU memory budgetk=14k=14 with Gradient checkpointing
25 GiB18 (0.4 s)40 (1.9 s)
50 GiB40 (1.5 s)86 (7.4 s)
75 GiB62 (3.3 s)132 (16.5 s)
100 GiB84 (5.7 s)178 (29.1 s)
125 GiB106 (8.7 s)224 (45.4 s)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.