acceptodds
Under review as a conference paper at ICLR 2027

Any3D-SLAM: Taming Any General 3D Priors for Robust In-the-Wild SLAM

Abstract

We introduce Any3D-SLAM, a robust visual SLAM system for jointly recovering camera poses and dense depth from streaming videos under unconstrained capture conditions. Conventional visual SLAM achieves accurate camera estimation by explicitly modeling multi-view geometric constraints, but its reliance on reliable correspondences makes it vulnerable to weak visual evidence and scene dynamics. Existing methods address these challenges with task-specific predictors, but their separate representations limit the use of complementary cues across tasks to support pose estimation. Recent 3D foundation models offer shared multi-frame representations and complementary priors, yet their feed-forward predictions may not fully satisfy multi-view geometric constraints. To harness these representations while enforcing geometric consistency, Any3D-SLAM integrates foundation-model features and complementary depth, motion, pose, and intrinsics priors into an end-to-end differential bundle adjustment (DBA) framework. We develop a warping-based updater for efficient correspondence estimation, substantially reducing memory overhead and enabling richer multi-frame representations and their associated priors to participate in online optimization. Among these priors, we introduce uncertainty-gated relative-pose constraints into online BA to support camera estimation when translational observability is weak or moving objects occlude the static background. Extensive experiments across six benchmarks spanning indoor, outdoor, dynamic, and egocentric environments demonstrate strong camera pose estimation performance in both calibrated and uncalibrated settings.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual SLAM, 3D Foundation Models, Bundle Adjustment

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_PXbhOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Any3D-SLAM, a DROID-style SLAM framework that integrates pretrained 3D foundation models into learned correspondence estimation and differentiable bundle adjustment. The main contributions include a memory-efficient warping-based updater and an uncertainty-gated pose prior. Experiments on six benchmarks demonstrate competitive tracking accuracy across different backbones.

Critical strengths
  • The framework achieves competitive results across several challenging benchmarks.

  • The design is compatible with multiple geometry foundation models.

Critical weaknesses
  1. The proposed framework shares substantial architectural similarities with WildPose, which already integrates pretrained 3D foundation model features, learned correspondence estimation, motion-aware weighting, depth priors, and differentiable BA. The main differences appear to be the warping-based updater and the uncertainty-gated pose regularization.

    Furthermore, the "Any3D" claim is not sufficiently justified. Although WildPose uses MASt3R for feature extraction, its architecture also appears conceptually adaptable to other 3D backbones. Supporting several pretrained models does not necessarily constitute a fundamental methodological contribution, especially when separate training is required for different backbones. The authors should discuss this overlap more explicitly and clarify what makes the proposed framework fundamentally different from WildPose.

  2. Insufficient experimental evidence to support the claimed improvements and isolate the proposed contributions. WildPose uses MASt3R features, while the proposed method mainly reports results using DAv3 and Pi3X. It is therefore difficult to determine whether the improvements come from the proposed architecture or simply stronger pretrained representations. Given that WildPose already employs MoGe-2 as its depth prior, in addition to MASt3R features, the proposed MoGe variant does not demonstrate a clear advantage over WildPose. In Table 2, it performs worse on several cases. Although the backbone usage and calibration settings differ, these results raise questions about the benefits of the proposed unified representation. More controlled experiments are needed to demonstrate that the proposed architecture provides improvements beyond the choice of pretrained models.

  3. Insufficient isolation of individual architectural contributions. The method introduces several changes simultaneously, including the warping-based updater, depth regularization, motion weighting, and pose priors. While the ablations are useful, they do not fully establish whether the improvements come from the proposed architecture or simply adding more priors to an existing DROID-style framework.

  4. The authors state that their results do not use loop closure or hierarchical global optimization, whereas other baselines may include loop closure and global BA. The comparison would benefit from explicitly marked optimization protocols, or an analysis of how these differences affect the reported results.

Questions for authors

See the weaknesses. Overall, I am positive to the strong engineering effort and competitive results. However, given the substantial architectural overlap with WildPose and the lack of sufficiently controlled comparisons, I am not yet convinced that the improvements can be attributed to the proposed methodological contributions. Will consider changing the rating if the concerns can be resolved.

Reviewer_pXc3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Any3D-SLAM, a visual SLAM framework that combines general 3D foundation models with iterative geometric optimization to improve robustness in challenging real-world videos. The method uses pretrained 3D representations for correspondence estimation and incorporates depth, motion, camera pose, and camera intrinsics priors into differentiable bundle adjustment. To reduce memory consumption, the authors introduce a warping-based correspondence updater that avoids expensive all-pairs correlation. They also propose an observability-based gating strategy that selectively applies translation priors when visual constraints are weak, such as in low-parallax or highly dynamic scenes. The framework supports different 3D backbones and both calibrated and uncalibrated settings. Experiments on six benchmarks demonstrate competitive camera tracking accuracy, substantial memory savings, and improvements in video depth estimation.

Critical strengths
  1. The system supports different backbones and achieves competitive results across six datasets, including dynamic scenes (Table 2).
  2. The warping-based updater has a clear practical benefit: inference memory drops from 44.3 to 17.7 GB for DA3-Single (Table 1), with similar accuracy.
  3. Pose priors improve tracking on challenging datasets such as KITTI and EPIC Fields (Table 5).
  4. The depth-estimation results show that the framework is also useful beyond camera tracking (Table 3).
Critical weaknesses
  1. It is still hard to tell how much improvement comes specifically from reusing foundation-model features. Some key components are closely related to existing SLAM and tracking methods. A more controlled comparison would help.
  2. The effect of pose-prior gating is not fully clear. Removing it does not change the EPIC result in Table 5, although it leads to tracking failure on KITTI. More direct evidence would strengthen this claim.
Questions for authors
  1. Could the authors show when the translation gate is activated and compare against a rotation-only prior?
  2. Can the benefit of using foundation-model features for matching be better isolated?
  3. Does the correspondence updater need to be retrained for each new 3D backbone?
  4. How well does the system work on much longer videos without loop closure?
1 comment

As the discussion period is nearing its end, we would be grateful for your thoughts on our updated responses and manuscript. If any concerns remain, we would welcome the opportunity to clarify them. Thank you again for your time.

Reviewer_iZydOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Any3D-SLAM, a visual SLAM framework that combines pretrained 3D foundation models with differentiable bundle adjustment (DBA), built on top of the DROID-SLAM style framework. The model uses a shared foundation-model features for correspondence, depth and motion priors, which claimed to the main novelty. The authors also introduce an uncertainty-gated pose regularization strategy to improve tracking when geometric constraints are unreliable. The framework supports different 3D backbones and is evaluated on six benchmarks, showing competitive camera tracking performance in both calibrated and uncalibrated settings.

Critical strengths
  1. The method achieves strong camera tracking results across six benchmarks, including challenging outdoor and dynamic scenes, and outperforms several competitive baselines.
  2. The warping-based updater significantly reduces GPU memory usage while maintaining comparable tracking accuracy. This is particularly useful for larger multi-frame backbones, where correlation-based approaches would otherwise run out of memory.
Critical weaknesses
  1. The benefit of shared 3D representations is not clearly demonstrated. One of the main claims is that using shared foundation-model representations for correspondence estimation and geometric priors improves SLAM performance. However, the experiments do not clearly show how much this helps. In particular, it is unclear whether the improvements over MegaSaM come from the shared representations or simply from using stronger pretrained models.
  2. Limited methodological novelty. The overall framework is quite similar to existing SLAM methods, especially MegaSaM and DROID-SLAM. Many components, including depth regularization, motion handling, and warping-based correspondence estimation, have already been explored in previous work. While combining them with 3D foundation models is useful, I find the methodological contribution somewhat incremental.
  3. The uncertainty-gated pose regularization depends on the quality of the pose priors. The proposed gating strategy is interesting and could help DROID-style SLAM in geometrically ill-posed scenarios. However, it only considers the reliability of the geometric constraints, not the predicted pose priors themselves. In challenging cases, both could be unreliable, and it is unclear how well the method would work in such situations.
Questions for authors
  1. My main concern is that the overall pipeline seems quite similar to MegaSaM, with the main difference being the use of 3D foundation models. Could the authors clarify what benefits the shared 3D representations bring over independent task-specific models? How does MegaSaM perform using the same 3D foundation priors?
  2. What are the main methodological differences from MegaSaM beyond the choice of backbone? It would be helpful to better understand which components are actually responsible for the performance improvements.
  3. A more fundamental question is whether a simpler approach could achieve similar results by directly applying geometric optimization (e.g., BA) to the feed-forward reconstructions from 3D foundation models. Could the authors discuss the advantages of the proposed SLAM framework over such an approach?
  4. How robust is the method when the underlying 3D foundation model produces inaccurate predictions? The authors could investigate cases where the predicted pose or depth priors are unreliable, since the proposed framework seems to depend heavily on their quality.
1 comment

Thanks for the further clarification! I think the additional experiments and discussion help clarify the contributions, especially the observability-gated pose constraints. While I still see the overall novelty as somewhat incremental, I appreciate the practical value of the proposed approach and am happy to raise my rating accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.