acceptodds
Under review as a conference paper at ICLR 2027

On Self-Improving Feed-Forward SfM: A Generalizable Multi-View Pose Estimator

Abstract

Classical Structure-from-Motion (SfM) is an optimization method: it jointly recovers camera poses, verified image correspondences, and depth (i.e., a point cloud) from an image collection without labels. Modern feed-forward SfM models instead regress these quantities in a single pass. Yet they must first be trained on labeled data, part of which is itself produced by classical SfM pipelines. Like other learned models, they fail on inputs beyond the training distribution. We close this gap by making feed-forward SfM produce its own labels: an optional bundle adjustment (BA) stage refines its predictions, and the refined output fine-tunes the network, forming a self-improving loop. We realize this loop for camera intrinsic and extrinsic estimation, formulating SfM as three foundation models: a frozen depth and correspondence estimator, and a tunable feed-forward pose estimator (the SfM pose estimator). We propose a global BA that depends entirely on model outputs: initialized with the feed-forward camera estimates, it optimizes for projective consistency between the predicted depth and correspondence maps, then extracts a subset of accurately registered images whose refined poses serve as pseudo ground truth. The pipeline scales end-to-end: from feed-forward initialization to BA, and from small scenes to 15,000-image unconstrained internet collections. Continual retraining on this self-generated supervision adapts the SfM pose estimator to the new domain. Across small-scene (IMC2021, ETH3D), large-scene (FastMap, T&T), and unconstrained internet (GLDv2, 1DSfM) datasets, our BA-refined poses approach the accuracy of SoTA SfM, making them reliable pseudo labels: fine-tuning on them substantially improves the feed-forward pose estimator. These results point to an emerging self-improving learning paradigm for SfM that generalizes across unseen domains.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Structure-from-Motion, Bundle Adjustment, Camera Pose Estimation, Self-improving Models, Pseudo-labeling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DWWiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper addresses generalization of camera-pose networks to new domains by using unlabeled image collections to generate additional supervision. To achieve this, the paper relies on a set of strong tool -- such as a depth foundation model (MoGe-2), a pretrained correspondence matcher (RoMa?) with Doppelgangers++ verification -- which are combined with a trainable pose estimator.

The estimator follows a π³-like architecture with an image encoder initialized from DINOv2 and a multi-view decoder initialized from VGGT, made permutation-equivariant by removing the anchor-frame encoding. It takes as input RGB images and depth predicted by MoGe-2 and outputs camera rotations, translations, intrinsics, and per-view depth scales.

The model is first trained with in a supervised manner using annotated data from MegaDepth, ScanNet, Map-Free Relocalization, BlendedMVS, TartanAir, DL3DV, and ScanNet++ with ground truth data. Then unlabeled collections are added, for which predicted poses are refined with marginalized bundle adjustment while views registered unreliable are filtered out. The new data with the retained refined cameras poses are mixed with annotated data to supervise the final network fine-tuning. During this stage most of the backbone is frozen and primarily the camera head is trained.

The main adapted model (Ours†) is transductive because its fine-tuning pool includes the evaluation images, without their ground-truth poses. At inference the model takes the target RGB images and MoGe-2 depth predictions to estimate the camera parameters. Optionally, a Bundle Adjustment is applied to further refine the camera intrinsics, extrinsics and depth scale.

Critical strengths

The strongest argument for acceptance is the extensively evaluated pipeline showing that one can turn unconstrained, unlabeled image collections into useful self-supervision for improving camera-pose estimation, while supporting joint processing of thousands of views.

More detailed:

  1. The paper makes effective use of unlabeled images by refining its camera predictions through geometric consistency checks and using the refined predictions to improve the network. In addition, the pipeline handles challenging image collections by rejecting misleading matches and filtering out unreliable views, including unrelated photos and images of similar-looking but different surfaces. This allows it to process unconstrained image collections gathered from the web.

  2. To improve scalability, the paper proposes to distribute global attention across GPUs in a manner to preserve the full attention computation. This has the advantage to reduce memory requirements without approximating the attention or splitting the collection into independent groups and hence enables joint camera estimation over very large image collections. Indeed the paper shows results for scenes with over 6K images (in Eyeful Tower dataset), claiming to be able to handle 15K images.

  3. The paper provides extensive experimental comparisons including small scenes, large-scale collections, and unconstrained internet images and large set of qualitative illustrations in the Appendix. The paper further provides several ablations illustrating the benefits of fine-tuning on unlabeled images, of the bundle adjustment after adaptation, and the contribution of free-point refinement. Leave-one-domain-out experiments also assess generalization beyond the domains included in fine-tuning.

Critical weaknesses

The main argument against acceptance is that the strongest results rely on computationally expensive transductive adaptation and selective image registration, leaving generalization and efficiency gains insufficiently established, while the methodological novelty beyond integrating existing components remains limited.

Detailed critical weaknesses:

  1. One of my main concern is that the strongest results obtained in the paper combine transductive adaptation where all evaluation data is included in the self-supervised training phase, which requires substantial computation and selective image registration, making generalization and efficiency hard to assess. Indeed for Ours† the fine-tuning pool includes the evaluation images, supervised by pseudo labels generated through Ours◦ and then refined with bundle adjustment. This means that Ours† learned from the specific collections on which it is evaluated and therefore the comparisons with the other models is somewhat unfair, especially what concern runtime comparison in Table B.1 where the reported inference time for Ours† excludes the bundle adjustment used to generate its pseudo labels and the subsequent fine-tuning. Furthermore it is difficult to compare wall-clock runtime when two different hardwares are used.

  2. From a practical point of view the images used to obtain Ours† could be considered as already labeled/posed images. Therefore, while it is interesting to show the results as an independent transductive adaptation scenario, in practice it would be more interesting to focus on the evaluation of Ours◦ and even more on the leave-one-domain-out models Ours⊖ which provides the most relevant evidence for performance on previously unseen collections and the gain obtained with training on unlabelled collection.

  3. The performance of Ours◦ without BA is not consistently strong across benchmarks. Comparisons of Ours◦+BA should therefore also include competing feed-forward methods with BA refinement, as in Table 2, while reporting results without BA separately. Ours⊖ performs better than Ours◦, could be interesting to add this model with and without in the tables .

  4. Another interesting scenario could be to use the proposed pipeline where the model is refined purely on the unlabelled target domain (test time adaptation).

  5. In section 4 (lines 354-358) the authors then state that +BA configurations are scored on the subset retained after view filtering, and explicitly acknowledge that this favors the filtered setting, which according to Table C.1 (Reg) can be quite important in some cases. This makes pose error comparison rather difficult as not all methods are evaluated on the exact same set. The accuracy reported in the main should be therefore "all" or "retained together with the percentage of registered images).

  6. My second main concern is the technical novelty. The method combines several existing, powerful, and computationally demanding models, as outlined in the paper summary. While I acknowledge that their integration represents a substantial engineering effort, there is limited methodological contribution in the paper.

  7. Finally, I find the paper being very dense and quit difficult to follow, requiring frequent back-and-forth between the main text and the appendix.

Questions for authors

Clarification issues:

  1. There are inconsistencies in the paper as in section Section 3.3 the authors mention Doppelgangers++ as a MASt3R-based matcher, while in the reproducibility statement they talk about RoMa correspondences with Doppelgangers++ verification. Can you clarify what was used at the end.

  2. The paper claims to be able to handle 15K images. Was it tested on such scene and if yes which collection contains approximately 15,000 images? What are the inference time, memory consumption, registration rate for this collection?

4 comments

We thank the reviewer for the thoughtful review and detailed summary. We appreciate the reviewer's recognition that our pipeline turns unconstrained, unlabeled collections into useful supervision. We are also grateful for the recognition of its ability to process thousands of views jointly. The reviewer's critique of evaluation fairness, computational cost, novelty, and presentation helps us clarify our contributions and their limitations.

A1: Performance and evaluation fairness (W1).

We thank the reviewer for highlighting the need to distinguish transductive adaptation from generalization to unseen domains. Following the reviewer's suggestion, we have added Ours⊖ to the main benchmark tables in the revised manuscript. Ours⊖ excludes the entire evaluation domain from pseudo-label fine-tuning and is evaluated on all frames. It remains competitive with recent feed-forward methods, outperforming AMB3R (2026) on ETH3D, π³ (2026) on Tanks & Temples, and Depth Anything 3 (2026) on IMC2021. We could not report the other feed-forward baselines on FastMap because they encountered out-of-memory (OOM) errors at its scale.

A2: Adaptation cost and runtime (W1).

We agree that the cost of pseudo-label generation and fine-tuning should be considered when assessing Ours†. Generating pseudo-labels for the full pool takes approximately 4 days on 8 A100 nodes, compared with 8 days for full training on the same resources. Pseudo-label generation adds substantial cost, but we consider it acceptable relative to the computational resources required for full training. We do not claim runtime efficiency as a contribution. We provide the reported runtimes for reference. Following the FastMap convention, these runtimes exclude preprocessing, including correspondence estimation and depth generation. The Ours† timing also excludes offline pseudo-label generation and fine-tuning. The revised supplementary runtime section now clarifies these exclusions and the different hardware used by the baselines.

During fine-tuning, we update all parameters of the pose estimator and train on the original annotated data together with the new pseudo-labels. This is more like extended training on a larger pool of supervision than a separate targeted fine-tuning run for each dataset. Our aim is to produce a single model that can generalize to unseen domains. Ours† is the main model trained on the combined pool. We apologize for the error in the original manuscript that may have caused this misunderstanding.

A3: Test-time adaptation (W2, W4).

We thank the reviewer for suggesting test-time adaptation as a separate evaluation scenario. In addition to the unseen-domain evaluation of Ours⊖ discussed in A1, we have included test-time pseudo-label fine-tuning results starting from Ours° in supplementary Table B.2 of the revised manuscript. Test-time fine-tuning achieves comparable performance to Ours†, although it performs slightly worse across these benchmarks. This may reflect the smaller pseudo-labeled training pool available for test-time adaptation. We summarize the results below.

Test-time fine-tuning results (revised Table B.2). All results use feed-forward poses. Metrics: RTA@5 for ETH3D and T&T, AUC@3 for IMC2021, and average RTA@3 for FastMap (all ↑).

ConfigurationETH3DIMC2021T&TFastMap
Ours°60.430.974.073.1
Ours° + TTO87.248.677.486.2
Ours†88.549.678.089.0

A4: Comparisons with and without optimization (W3).

We thank the reviewer for this valuable comment. We agree that feed-forward predictions and optimization-based results should be reported separately. For a fairer comparison, we have added Ours⊖ to the main tables of the revised manuscript. Supplementary Table B.2 reports our models with and without BA, including all-frame and retained-view scores for BA. Following the reviewer's suggestion, all four main benchmark tables also report the frame registration rates for Ours°+BA.

Tables 1–3 also include methods that combine learned geometry with optimization. On ETH3D (Table 1), these include MASt3R-SfM and Glob3R. On IMC2021 (Table 2), we report π³ and VGGT both with and without BA. On Tanks & Temples (Table 3), we include SAIL and SAIL-OPT, as well as MASt3R-SfM and VGGT-SLAM. The latter methods use different forms of refinement:

  • MASt3R-SfM (Duisterhof et al., 2025) optimizes MASt3R's predicted geometry and matches.
  • Glob3R (Deng et al., 2026a) applies BA to π³X-based geometry and tracks.
  • SAIL-OPT (Deng et al., 2026b) refines poses predicted by the trained SAIL model using a rendering loss.
  • VGGT-SLAM (Maggio et al., 2025) aligns VGGT submaps with pose-graph optimization.

On FastMap (Table 4), we could not include the other feed-forward baselines or their BA-refined results because their inference scripts encountered out-of-memory errors at this scale.

(Continued from last post)

A5: Selective registration (W5).

We thank the reviewer for highlighting the difficulty of comparing results on different sets of images. Following the reviewer's suggestion, the revised main tables report registration rates for Ours°+BA. Supplementary Table B.2 reports BA accuracy separately on all frames and on the retained views, together with the registration rate. We do not claim that our BA generally outperforms other SfM methods. Its primary role in our pipeline is to produce reliable pose pseudo-labels for fine-tuning.

A6: Novelty (W6).

We thank the reviewer for this critique. We acknowledge that our pipeline integrates existing models and optimization components. Our research goal is to study a generalizable and scalable learning paradigm for 3D foundation models that can use unlabeled data.

Our core methodology chooses a relatively “boring” approach: offline pseudo-labeling. We believe that validating this simple strategy is itself part of our contribution. Indeed, other works explore more sophisticated self-improvement strategies. Self-Geometry uses geometric-consistency losses (Youn et al., 2026), Fin3R distills a monocular teacher (Ren et al., 2025), Selfi uses feature alignment (Deng et al., 2026c), and Depth Anything 3 also uses a monocular teacher (Lin et al., 2026).

These methods apply supervision through losses evaluated during model updates. In contrast, we believe our simple offline pseudo-labeling strategy has the potential to provide higher-quality supervision at a larger scale. Our BA pipeline seeks poses that are geometrically consistent with foundation-model predictions (see A2 in our response to R2, Kw7X, for more details). It jointly optimizes poses across an image collection, potentially spanning thousands of views rather than only the views in a training batch. It can take many optimization steps before producing pseudo-labels. View filtering then provides an additional quality gate by excluding unreliable views. Indeed, this “simple” strategy brings substantial improvements in feed-forward pose accuracy. For example, RTA@3 rises from 23.1 to 86.6 on Mill 19 building, and RTA@5 rises from 60.4 to 88.5 on ETH3D (revised Table B.2). We hope this clarifies the novelty of our work.

A7: Presentation (W7).

We thank the reviewer for highlighting the difficulty of following the paper. We agree that the presentation should require less back-and-forth between the main text and the supplement. In the revised manuscript, we have consolidated references to supplementary method details at the end of Sec. 3. We have also added a brief implementation summary in Sec. 4. The main benchmark tables now include inductive results, so readers no longer need to consult the appendix for these comparisons.

A8: Correspondence models (Q1).

We apologize for the inconsistency. We use features from RoMa to identify covisible image pairs and construct the covisibility graph (App. D.2). Doppelgangers++ provides the correspondences used by BA (Sec. 3.3). We have corrected the reproducibility statement to distinguish their roles.

A9: The 15K-image collection (Q2).

The approximately 15K-image collection is Trafalgar in 1DSfM, which contains 15,685 images. We will include this detail in the revised manuscript.

References

Lin et al. (2026). Depth Anything 3: Recovering the Visual Space from Any Views. ICLR.

Deng et al. (2026c). Selfi: Self-improving Reconstruction Engine via 3D Geometric Feature Alignment. CVPR.

Ren et al. (2025). Fin3R: Fine-tuning Feed-forward 3D Reconstruction Models via Monocular Knowledge Distillation. NeurIPS.

Youn et al. (2026). Self-Geometry: GT-Free and Plug-and-Play Test-Time Adaptation for Geometrically Consistent 3D Vision Foundation Models. arXiv:2608.10708.

Deng et al. (2026a). Glob3R: Global Structure-from-Motion with 3D Foundation Models. arXiv:2607.09225.

Deng et al. (2026b). SAIL-Recon: Large SfM by Augmenting Scene Regression with Localization. 3DV.

Duisterhof et al. (2025). MASt3R-SfM: A Fully-Integrated Solution for Unconstrained Structure-from-Motion. 3DV.

Maggio et al. (2025). VGGT-SLAM: Dense RGB SLAM Optimized on the SL(4) Manifold. NeurIPS.

I appreciate the authors’ considerable effort in addressing the reviewers' concerns through additional experiments and clarifications, particularly the expanded implementation details and ablation studies. I also thank them for further comparisons between inductive transfer and transductive adaptation, adding test-time pseudo-label fine-tuning results, and reporting the percentage of retained frames alongside complementary scores computed over all frames. I acknowledge that with these extra experiments and clarification the authors substantially strengthened the paper, and would be happy to raise the score to a borderline, but no score is available for that.

Despite the above improvements, several concerns remain unresolved, preventing me from raising my score to weak accept.

  1. For a collection used in adaptation, BA has already produced refined camera poses before fine-tuning begins, so the benefit of subsequently training the network and predicting those poses again remains unclear.
  2. The authors acknowledge that main contribution lies in validating scalable offline pseudo-labeling through global BA and view filtering, rather than introducing fundamentally new components. However, the reported gains do not establish that this strategy produces better supervision than alternative pseudo-labeling pipelines under comparable data and compute budgets. They mention Self-Geometry, Fin3R and Selfi as "sophisticated self-improvement strategies", however there is no numerical or cost comparison to support the superiority of the "simple" approach proposed in this paper.
  3. Table B.2 provides more convincing evidence of usefulness through substantial inductive gains over the model before fine-tuning. However, inductive performance generally remains below that achieved with target-specific BA or TTO, suggesting that training on other domains transfers only part of their benefit. Furthermore, the relatively modest gains of transductive fine-tuning over TTO on several benchmarks leave the additional value of the broader fine-tuning pool unclear. The practical trade-off between generalization, accuracy, and adaptation cost therefore remains insufficiently established.

We thank you again for your thoughtful follow-up comments.

Follow-up concern 1: Why fine-tune when BA has already recovered the poses? We answer this from a practical application perspective. Feed-forward SfM models are being actively integrated into classical SfM pipelines, with improved camera-pose initialization as a key practical benefit. Commercial efforts at DroneDeploy and Niantic Spatial are pursuing this direction. Research efforts such as GLUEMAP confirm a similar trend. We observe the same benefit: improved initialization raises all-frame BA RTA@3 from 51.8 to 99.8 on Mill 19 building (revised Table B.2). These results show that model adaptation and BA are complementary: BA refines the predicted poses, while improving the initializer can also improve the final reconstruction. By transferring BA-refined poses back into the feed-forward model, our method alleviates its target-domain generalization gap, supporting its adaptation in SfM applications. We emphasize that fine-tuning on BA-refined poses addresses a concrete practical need, reflected in ongoing efforts across both industry and academia to integrate feed-forward models into SfM systems.

Follow-up concern 3: What is the value of pooled fine-tuning if gains over TTO are modest? A key research interest in TTO is retaining the benefits of adaptation across multiple target domains in a single model. For example, Niantic's ACE and ACE-G research how learning across scenes can improve a shared representation that supports scene-specific optimization. Our method naturally addresses the broader goal explored by ACE and ACE-G by consolidating adaptation across scenes into one shared model. A single Ours† model improves over Ours° + TTO across the summarized benchmarks (revised Table B.2).

Follow-up concern 2: Comparison with alternative supervision strategies. We thank the reviewer for highlighting this limitation. We emphasize two factual differences over alternative self-improvement methods. Our pipeline could: (1) leverage unconstrained image collections and (2) define supervision over pools of 1K+ images. We also believe that a direct pose loss against BA-refined & filtered pseudo-labels provides a more robust supervision signal. We acknowledge that we have not conducted a direct experimental comparison with these alternatives. We will continue working toward such comparisons under matched data and compute budgets.

Reviewer_Kw7XOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a dense bundle adjustment system, which optimises camera poses and affine parameters for predicted depth maps.

Given a set of images and its inferred monocular depths, it provides a learned initialisation (aka feed-forward prediction) for the camera poses, intrinsics, and affine depth displacements.

The second component is an optimisation pipeline, that refines these camera poses, intrinsics and affine depth parameters via optimising a reprojection cost of matches, obtained by an off-the-shelve matcher.

For the self-improvement component, the authors run their bundle adjustment system on additional scenes and add them to the training set for the learned initialisation.

Critical strengths

Bundle adjustment performance seems to be OK.

The most interesting result in my opinion is buried in the supplementary. The fact that there's some generalisation (Table C.1) i.e that Ours_{\ominus}, i.e model trained on a mixure of high-quality and the outputs of the author's bundle adjustment is an indicator that its a good performing system.

Critical weaknesses

My biggest concern about the paper: Is it really a self improving system? For it to be a self-improving system, I'd want to see finetuned + BA being better than vanilla + BA. Then it would show some promise to hill-climb on lots of unsupervised data. Currently, it feels like that the quality is just bounded by the quality of the matcher + monocular depth.

Say if your correspondence estimator/matcher fails then what happens? Its not being optimised too in conjunction with the weights of the network, so it does not have a chance to improve.

“We close the generalization gap of feed-forward SfM identified in Sec. 1 with a self-improving loop that adapts the SfM pose estimator to unconstrained target domains.” — this is simply not true. If the RoMa matcher will fail on some really weird domain, no self-improvement will be gained, will it?

Overall, i get the high-lever motivation — it would be nice to get some capabilities along the lines of improved system performance after trying to solve a new task (BA on a new scene in this case), but never seeing the GT answer. However, I struggle to see the “loop” here.

Ultimately, if we want the best reconstruction on a particular scene, the BA versions of the base and improved models have almost the same performance, right? Why should we bother with collecting additional pseudo-labelled data first, fine-tune the model and then get worse performance anyway?

In a way, the results are somewhat disappointing for what the paper tries to advertise. Bundle adjustment results on its own are pretty good from what it seems. As for the “self-improvement”. Yes, training on the target domain with more data helps, but isn’t it how all of these feed-forward models trained? To train VGGT on additional real data samples we would have to obtain camera poses and depth predictions on it, in a similar fashion. We would have to do bundle adjustment to get the camera poses out and depth measurements out of e.g a dense MVS system or some other source. All additional real data will be “pseudo-labelled” in one way or another.

We do get new information from RoMA correspondences. So basically the paper's takeway, more data leads to better performance? Or am I missing something.

As a dense bundle adjustment system, which might be a contribution on its own, but it would probably requite a significant re-write of the paper and focus experiments more on the components of the BA. Optimising affine parameters for depth and cameras is a known formulation on its own, e.g MegaSAM. So for it to be a meaningful contribution I’d like to know what made it good specifically.

Questions for authors
  • Why do we have to consume monocular depth? Arguably, mono depth predictions are even less well constrained for metric depth estimation. Is the depth provided by pointmaps from pi3 or a separate monodepth estimator? What’s the motivation of that. Ideally, it would be good to see an ablation of using the depth maps from pi3 directly.

  • Also with an additional refinement stage (lines 297-306) it would probably destroy points that are on non-photometrically rich surfaces? Matching quality there would still probably be subpar. Are all points unfrozen there? Is it even track-table with that amount of images?

  • Was the cost robustification from eq (2) necessary? Why just not optimise reprojection error?

  • For some reason the authors insist that the rotation group is SE(3), i wonder why.

e.g. optimizing rotations in the se(3) tangent. —> so(3) tangent lines 311-312

“Several details matter in practice. Rotations are stored as the 6D representation but optimized in the se(3) tangent, re-anchored after each accepted step, since the redundant storage would otherwise contribute spurious null directions to H.”

5 comments

We thank the reviewer for the thoughtful review. We appreciate the reviewer's recognition that our pseudo-labeling pipeline can improve generalization to unseen domains. We appreciate the insightful critique of the scope and usefulness of self-improvement.

A1: Self-improvement trend (W1, W3).

We sincerely thank the reviewer for this insightful question. We indeed observe a hill-climbing trend. In revised supplementary Table B.2, the approximate scene-weighted FastMap average RTA@3 for BA poses evaluated on all frames rises from 90.0% for Ours°+BA to 91.8% for Ours†+BA after transductive fine-tuning. Equivalently, the fraction of predictions outside the RTA@3 threshold falls from 10.0% to 8.2%, an approximately 18% relative reduction. In particular, fine-tuning substantially improves two examples: RTA@3 rises from 51.8% to 99.8% on Mill 19 building and from 69.4% to 76.8% on the Urban split. These examples illustrate particularly well how fine-tuning provides better initial poses, enabling BA to produce more accurate results.

However, this trend does not appear on the retained views used as pseudo ground truth: their average BA RTA@3 changes from 94.2% to 93.8% after transductive fine-tuning. Our outlier filtering removes views with unreliable geometric consistency to obtain more reliable supervision. Its thresholded decisions may prevent small improvements from translating smoothly into additional pseudo-labels. This is a limitation of our view-filtering strategy and an opportunity for improvement. BA also still improves the fine-tuned model, even in the transductive setting. Revised Figure 6 reports BA gains with both configurations evaluated on all frames. This suggests that transferring BA's benefits through fine-tuning is not straightforward. This may also limit further hill-climbing. Nevertheless, fine-tuning reduces reliance on BA and hence enables faster inference.

FastMap results (revised Table B.2). RTA@3 ↑

ConfigurationUrbanMill 19 buildingFastMap average
Ours° + BA, all frames69.451.890.0
Ours† + BA, all frames76.899.891.8
Ours° + BA, retained views96.599.994.2
Ours† + BA, retained views97.299.893.8

The FastMap averages are approximate scene-weighted averages computed from rounded split results. The retained-view rows can score different subsets.

A2: More than more pseudo-labeled data (W4).

We view self-improvement as seeking scalable supervision by encouraging consistency among model predictions. We believe that a self-improvement pipeline for 3D foundation models should encourage their predicted 3D structure to satisfy multi-view geometric constraints. This vision motivates our pipeline: bundle adjustment enforces geometric consistency to produce pose pseudo-labels, and fine-tuning transfers the refined poses back to the model.

Both our pipeline and classical SfM can produce camera pose pseudo-labels, but they differ in how model-predicted structure (depth or point maps) participates in the process. Classical SfM triangulates structure from image correspondences and optimizes it jointly with the cameras, without requiring a model-predicted structure prior. In contrast, model depths are central to our pipeline. We build on MBA's depth-anchored objective, initialize depth scales to accommodate errors in the depth prior, and filter unreliable views based on depth-based projective consistency. Throughout these steps, the model's depth maps remain unchanged except for a per-frame scale and shift. Although both approaches can produce pose labels, we view our method as a distinct self-improvement pipeline that seeks pseudo-labels by maximizing consistency among model predictions.

(Continued from last post)

A3: Limitations of frozen depth and correspondence models (W2, W4).

We thank the reviewer for this insightful critique. In 3D foundation models, camera poses have six degrees of freedom per view, whereas depth and correspondences are high-dimensional, pixel-wise predictions. Their labels also come from different sources: camera poses are commonly obtained through geometric optimization such as BA, while depth and correspondence supervision mostly comes from LiDAR and stereo camera measurements (with correspondences derived from calibrated geometry). Thus, while optimization commonly provides pose labels, downstream multi-view stereo is not necessarily the primary source of depth and correspondence supervision. These differences suggest that pose, depth, and correspondence may benefit from different learning strategies. This motivates our focus on pose self-improvement while keeping the depth and correspondence models frozen.

However, keeping the depth and correspondence models frozen is a critical limitation of our method. As the reviewer points out, errors in their predictions, including matcher failures on unfamiliar domains, limit the self-improvement pipeline. In our current loop, any hill-climbing benefit to BA can only come from improved camera initialization. Despite this, we observe gains on the FastMap examples above. We agree that extending self-improvement to depth and correspondence is important. Doing so requires developing suitable supervision and training procedures for both depth and correspondence models. We see this as an important direction for future work. Following the reviewer's critique, we have changed “close the generalization gap” to “alleviate the generalization gap” in Sec. 3 of the revised manuscript.

A4: What makes the BA effective? (W5)

Thank you for this suggestion. In the revised manuscript, we additionally ablate individual components of the BA system, including scale initialization, outlier view filtering, the BA objective, free-point refinement, and second-order optimization (Tables 5 and B.2), with a runtime comparison in Table B.1.

A5: Monocular depth and π³ (Q1).

Our pseudo-labeling pipeline consumes depth maps without requiring a specific depth estimator. We choose monocular depth for several reasons. First, unconstrained collections such as 1DSfM can contain unrelated images, making it nontrivial to adapt a π³-like multi-view model (for example, choosing and grouping related views). Second, π³ does not consistently outperform strong monocular depth models across datasets (Wang et al., 2026; Yuan et al., 2026). Depth Anything 3 also uses a monocular teacher model (Lin et al., 2026). Finally, π³ was not yet available when we started this research. Since the choice of depth estimator is not the focus of our work, we did not incorporate π³ into our data preparation pipeline.

A6: Weak texture, free points, and tractability (Q2).

We agree with the reviewer: our free-point refinement uses independent two-view correspondences rather than multi-view tracks, making it potentially brittle when matches are unreliable. We therefore release points only after depth-anchored optimization has converged and view filtering has been applied (Sec. 3.3). In revised Table 5, applying free-point refinement early (after scale initialization) lowers T&T advanced AUC@3 from 42.9 to 25.1, consistent with the reviewer's concern.

We included free-point refinement partly to explore whether our pipeline could also produce pseudo-labels for depth and correspondence. Although it improves pose accuracy, and we therefore retain it, our experiments did not show consistent benefits for depth and correspondence. This echoes the challenge discussed above of developing a reliable pseudo-labeling pipeline for these mostly sensor-supervised modalities.

Free-point refinement ablation (revised Table 5): T&T AUC@3 / RTA@3 ↑.

BA configurationAdvancedIntermediateTraining
Ours° + BA42.9 / 71.869.1 / 98.058.1 / 89.9
Without free points41.6 / 71.468.3 / 97.956.8 / 89.8
Early free points25.1 / 47.339.8 / 88.432.4 / 66.1

(Continued from last post)

A7: Robust CDF objective (Q3).

With good initialization, Cauchy loss can perform comparably to or better than the CDF-based loss. For example, in revised Table 5, Cauchy achieves 70.4 vs. 69.1 AUC@3 on T&T intermediate. However, when initialization is poor, Cauchy loss downweights large reprojection errors approximately in proportion to the inverse square of their magnitude. The CDF-based loss instead adapts to the observed error distribution. Large errors can therefore still guide optimization when they are common. This can help when initialization is poor. Refining poor initial predictions with bundle adjustment to produce pseudo-labels is central to our pipeline. For reference, on T&T advanced, CDF achieves 42.9 vs. 39.8 AUC@3 with Cauchy, illustrating the motivation for our choice.

Robust loss ablation (revised Table 5): T&T AUC@3 / RTA@3 ↑.

BA configurationAdvancedIntermediateTraining
Ours° + BA42.9 / 71.869.1 / 98.058.1 / 89.9
Cauchy loss39.8 / 68.370.4 / 98.555.6 / 86.5

A8: SE(3) vs. SO(3) (Q4).

Thank you for pointing out the imprecise wording. Our implementation stores the pose using the 6D rotation representation of Zhou et al. (2019) and a 3D translation vector. In second-order BA, the update is instead computed for the full pose in the se(3) tangent space, with a rotational component in so(3) and a translational component. The updated rotation is then stored again in the 6D representation. Our original wording conflated the rotation storage representation with the full-pose update; we have clarified this distinction in the revised manuscript.

References

Lin et al. (2026). Depth Anything 3: Recovering the Visual Space from Any Views. ICLR.

Wang et al. (2026). π³: Permutation-Equivariant Visual Geometry Learning. ICLR.

Yuan et al. (2026). SurroundNEXO: Ego-Centric Metric Bridging for Spatially Consistent Geometry in Autonomous Driving. arXiv:2606.16960.

Reviewer_uCJeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a self-improving feed-forward Structure-from-Motion (SfM) framework that improves camera pose estimation on unseen image collections without additional ground-truth labels. It refines initial feed-forward predictions via bundle adjustment (BA), selects reliable views, and uses the optimized parameters as pseudo-labels to fine-tune the pose estimator.

Evaluations across small-scale, large-scale, and unconstrained internet datasets show that pseudo-label fine-tuning significantly improves the base feed-forward model, while the BA-refined results achieve accuracy competitive with traditional optimization-based SfM.

Critical strengths

1. A well-motivated integration of geometric refinement and self-training for feed-forward SfM.

The paper presents a coherent framework that uses BA-refined camera estimates as pseudo-labels to improve the feed-forward pose estimator itself. Although geometric refinement and self-supervised learning have both been explored in prior work, the explicit transfer of globally optimized camera estimates back into the pose estimator offers a meaningful extension of existing approaches.

2. A scalable optimization and pseudo-label generation pipeline for unconstrained image collections.

Building upon existing marginalized bundle adjustment, the authors combine robust scale initialization, outlier view filtering, free-point refinement, and second-order optimization to generate reliable camera estimates from large, unordered image collections. The ability to apply the resulting pipeline to open-set image registration is practically valuable, particularly for constructing pseudo-labels without relying on external SfM reconstructions.

Critical weaknesses

1. The novelty relative to existing self-improving reconstruction methods could be better established.

The central idea of improving learned 3D reconstruction through model-generated geometric supervision has been explored in prior work, particularly Selfi. While the proposed method differs by using BA-refined camera poses to directly fine-tune the feed-forward pose estimator, the methodological distinction and its practical advantages over related approaches could be more clearly established. The paper provides extensive comparisons against existing SfM methods in terms of pose accuracy, but limited direct evidence isolating the benefits of its specific self-improving strategy relative to alternative approaches.

2. The contribution of several optimization-specific components remains insufficiently quantified.

Appendix C provides useful ablations of pseudo-label fine-tuning, free-point refinement, and test-time BA, as well as a leave-one-domain-out evaluation supporting inductive generalization. However, other components, including scale initialization, outlier view filtering, and the second-order optimizer, are not individually evaluated against simpler alternatives. Since these components are presented as important elements of the proposed pipeline, further analysis would help establish their respective contributions to pose accuracy, registration completeness, and computational efficiency. Ablation on a small-scale data partition would be sufficient.

Questions for authors

How does the proposed self-improving strategy compare with related approaches under a comparable setting?

In particular, Selfi also exploits model-generated geometric supervision, although it updates a feature adapter rather than the pose estimator itself. Could the authors further clarify the advantages of directly supervising camera poses with BA-refined pseudo-labels? If feasible, a comparison with an alternative self-improvement strategy or applying similar pseudo-label fine-tuning to an existing feed-forward SfM baseline would help substantiate the methodological contribution.

4 comments

We sincerely appreciate the reviewer's valuable comments. We are grateful that the reviewer recognizes our scalable optimize-then-fine-tune pipeline as a well-motivated extension of existing self-training methods. This reflects precisely our aim of scalable SfM learning that generalizes to unseen domains using unlabeled images.

A1: Distinction from Selfi (W1, Q).

We thank the reviewer for raising Selfi. Both methods turn a model's own predictions into training signal, but differ in the scale and quality of that signal. (1) Scale: Selfi harvests labels from neighboring frames of ordered videos (sequences of up to 100 frames), whereas ours come from entire unordered collections, including internet photos, and our pipeline scales to ~15,000 images. (2) Quality: Selfi trains on unchanged raw VGGT intrinsics and extrinsics predictions. It optimizes a photometric similarity loss (for Gaussian splatting). We instead include several quality-oriented design choices, including global bundle adjustment, a specialized correspondence estimator, a robust BA objective, a dedicated optimization scheme, and outlier view filtering.

The scale and quality of supervision are why we make pseudo-labeling our core design choice. Selfi and other self-supervised designs derive their training signal as part of the loss computation at each iteration. Our pseudo-labels, in contrast, are produced offline after many bundle adjustment updates. This enables us to prepare higher-quality supervision signals at a larger scale. We think this is key to realizing scalable 3D foundation-model learning. Following the reviewer's comments, we have revised the last two paragraphs of the Related Work section in the revised manuscript to clarify these distinctions.

A2: Ablation of optimization components (W2).

We thank the reviewer for this suggestion and have added a component ablation on Tanks & Temples (Table 5 in the revised manuscript). The extended results and runtime comparison are in its Tables B.2 and B.1, respectively. For convenience, we include a snapshot of the relevant results from the revised manuscript below.

(1) Scale initialization: Monocular depth scale ambiguity is a significant source of error. We estimate per-frame scales first to provide a better starting point for joint optimization. Optimizing all parameters jointly from the start instead lowers T&T advanced AUC@3 from 42.9 to 33.0.

T&T: AUC@3 / RTA@3 ↑, scored on all frames.

BA configurationAdvancedIntermediateTraining
Ours° + BA42.9 / 71.869.1 / 98.058.1 / 89.9
Without scale initialization33.0 / 61.260.1 / 95.446.3 / 79.6

(2) View filtering has two roles. First, it excludes poorly registered views to improve pseudo-label quality. On T&T advanced, BA's RTA@3 is 89.1 on retained views versus 71.8 over all views (Table B.2). To assess the effect on fine-tuning, we compare test-time optimization from Ours° using target-domain pseudo-labels generated with and without filtering. Without filtering, feed-forward accuracy generally decreases, with small exceptions. On Mill 19 building, RTA@3 drops from 75.3 to 26.0, suggesting that the excluded labels can impair fine-tuning (Table B.2). Second, filtering helps us leverage unconstrained collections such as Google Landmarks v2 and 1DSfM.

Ours° + BA on T&T: AUC@3 / RTA@3 ↑. Percentages are the retained fractions.

ViewsAdvancedIntermediateTraining
All frames42.9 / 71.869.1 / 98.058.1 / 89.9
Retained60.0 / 89.1 (72.1%)67.9 / 93.3 (70.7%)68.5 / 91.9 (70.3%)

Test-time pseudo-label fine-tuning: RTA@3 ↑. Adv., Interm., and Train. are T&T splits; Mill 19 is the building scene.

View filteringAdv.Interm.Train.Mill 19
With69.097.090.275.3
Without67.997.385.426.0

(3) Second-order optimization: We primarily improve runtime. First-order gradient descent achieves similar accuracy (43.5 vs. 42.9 AUC@3), but bundle adjustment takes about 2.61 vs. 0.71 hours on T&T advanced, approximately 3.7× longer on the same hardware (Table B.1).

T&T: AUC@3 / RTA@3 ↑, scored on all frames. Runtime is BA time in hours on the advanced split.

BA configurationAdvancedIntermediateTrainingTime ↓
Ours° + BA42.9 / 71.869.1 / 98.058.1 / 89.90.71
First-order gradient descent43.5 / 72.369.5 / 98.158.4 / 90.62.61

(Continued from last post)

A3: Comparison under a comparable setting (W1, Q).

We thank the reviewer for suggesting a comparison that demonstrates the benefit of self-improvement. We have added Pi3 results on ETH3D and Tanks & Temples (Tables 1 and 3 in the revised manuscript), alongside the existing IMC2021 comparison (Table 2). Across all three benchmarks, our initial model Ours° performs worse than Pi3, but Ours⊖ surpasses it after inductive pseudo-label fine-tuning: on ETH3D, RTA@5 rises from 60.4 to 84.3 versus Pi3's 73.9; on IMC2021, AUC@3 rises from 30.88 to 46.4 versus 43.34; and on Tanks & Temples, RTA@5 rises from 74.0 to 76.1 versus 75.6. This provides indirect evidence that self-improvement may benefit similarly structured feed-forward models. Pi3's inference code ran out of memory on FastMap (Table 4), so we could not include it there. We are working toward applying our pseudo-label fine-tuning to Pi3 in both inductive and transductive settings for the revised manuscript. Because of the engineering complexity, these results may not be available during the rebuttal period. A direct comparison with Selfi is difficult because its reported Tanks & Temples evaluation does not clearly specify the full protocol.

Thank you for the detailed rebuttal and additional experiments. The new results help clarify the contributions of individual components.

While I am not an expert in this field, I remain curious about the novelty of the proposed framework and share the other reviewers' concerns regarding the effectiveness of sustained iterative self-improvement, which has not yet been convincingly demonstrated.

Nevertheless, I appreciate the authors' efforts and will maintain my current rating.

We sincerely thank you again for the constructive feedback that has helped strengthen our work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers: Summary of Manuscript Revisions

We sincerely thank the reviewers for their careful reading and constructive suggestions. Their feedback helped us strengthen the evaluation, clarify the scope of self-improvement, and improve the presentation. Below we summarize the corresponding revisions.

Reviewer uCJe

  • Added a component ablation on Tanks & Temples covering scale initialization and second-order optimization, with a runtime comparison to first-order gradient descent (Sec. 4, Table 5 and Appendix B, Table B.1).
  • Added a comparison of pseudo-label fine-tuning with and without outlier view filtering (Sec. 4 and Table B.2).
  • Added π³ comparisons on ETH3D and Tanks & Temples alongside the existing IMC2021 comparison (Sec. 4, Tables 1–3).
  • Clarified the distinction between our offline BA-refined pose supervision and Selfi’s feature-adapter training (Sec. 2).

Reviewer Kw7X

  • Expanded the analysis of self-improvement to compare BA accuracy before and after fine-tuning, using both all-frame and retained-view evaluations (Sec. 4, Fig. 6 and Table B.2).
  • Added ablations of free-point refinement and the robust optimization objective, including early free-point refinement and a Cauchy-loss alternative (Sec. 4, Table 5).
  • Revised the limitations to state that depth and correspondence models remain frozen and that only one round of pseudo-label fine-tuning is evaluated (Sec. 5).

Reviewer DWWi

  • Added the leave-one-domain-out model Ours⊖ to the main benchmark tables and clarified the distinction between inductive transfer and transductive adaptation (Sec. 4, Tables 1–4).
  • Added test-time pseudo-label fine-tuning results and comparisons with Ours° and Ours† (Sec. 4 and Table B.2).
  • Reported registration rates in the main benchmark tables and all-frame and retained-view BA accuracy in the supplement (Sec. 4, Tables 1–4 and Table B.2).
  • Clarified the distinction between inference time and offline pseudo-labeling and training costs, and noted that runtime comparisons use different hardware (Sec. 4 and Appendix B, Table B.1).
  • Expanded the main-text implementation details, including training data, parameter updates, loss weighting, and the optimizer (Sec. 4 and Appendix C).
  • Corrected the reproducibility statement to distinguish RoMa features for covisibility from Doppelgangers++ correspondences for BA (Sec. 4, reproducibility statement, and Appendix D.2).

We also enlarged Figures 1–5 to improve readability.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.