acceptodds
Under review as a conference paper at ICLR 2027

Map2SLAM: From Streaming Map Prediction to Real-Time SLAM

Abstract

Streaming 3D foundation models (3DFMs) estimate camera poses and dense geometry online, but they fall short of SLAM: their predictions are fixed once emitted, so loop closures cannot correct accumulated drift. With such models already providing strong geometric capability, a central question for SLAM becomes how to schedule their inference and how to keep its results as revisable state. We present Map2SLAM, which keeps LingBot-Map frozen, trains only a lightweight retrieval head, and builds SLAM on the model's inference interface, turning its predictions into observations of an external persistent SLAM state. The front-end tracks every incoming frame with the model's causal streaming inference and decides which frames are written to the persistent state as keyframes. The back-end asynchronously decides when and which past frames to re-observe, re-running the model's joint inference over them with fresh context; verified loop observations correct the trajectory online, and segment observations refine dense geometry after the sequence. All invocations share a single frame encoder pass per image, whose cached features also drive loop retrieval, so frequent re-invocation incurs no additional encoding cost. Map2SLAM achieves state-of-the-art performance on KITTI, VBR, and Oxford Spires, reducing mean ATE by over 40% compared with the best uncalibrated baselines, while maintaining real-time tracking at over 28 FPS at a resolution of on an NVIDIA RTX 5090D GPU.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Streaming 3D Foundation Models, Real-Time Visual SLAM, Large-Scale SLAM

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_XY31OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduce MAP2SLAM, that build a SLAM pipeline with existed components such as 3DFM, loop detection, iPGO, confidence-weighted 3D pointmap alignment and etc.

Critical strengths

The pipeline shows a good performance, but with existed compoments and overlapped scheduling from prior works.

Critical weaknesses
  1. The introdution is not clear, the motivation does not vary much with the baselines.

  2. The related work is fairly weak. Authors should introduce the baselines to motivate the contribution.

  3. Most of the components utilized are introduced in prior works, the scheduling is also handcrafted and overlapped with existed systems, leaving the methodological novelty of the proposed framework unclear.

  4. There is no clear evidence to show the improvement on mapping.

  5. No speed and memory comparison.

Questions for authors

Addressing my concerns on clarity, novelty and experiments.

1 comment
AuthorsOpenReview ↗

Response to Reviewer XY31

Thank you for your critical feedback. We take your concerns seriously and address each point below, with manuscript revisions marked in blue.

1. System design and motivation (W1, W2, W3)

Building a SLAM system around a 3DFM comes down to three questions: ① what to observe, ② when to observe, and ③ how to optimize. Although individual modules may look similar to existing ones at first glance, all three mechanisms are tightly and cleanly designed around a single frozen streaming 3DFM:

  • ① What to observe: Unlike pairwise two-view matching or chunk-based inference, we build on a frame-streaming 3DFM for continuous per-frame tracking in causal mode, while reusing cached encoder features in fresh bidirectional context to re-observe loop pairs and historical segments without re-encoding images.
  • ② When to observe: Without an external VPR model or external optical flow, the model's cached encoder tokens trigger loop closure and its predicted motion drives a commit-or-rollback KV probe for keyframe admission. Meanwhile, historical segment re-observations are scheduled into idle GPU intervals during online tracking, avoiding a large end-of-sequence backlog.
  • ③ How to optimize: Rather than optimizing only rigid poses or submaps, we optimize at the frame and dense-depth level: online pose graph optimization continuously corrects per-keyframe trajectory drift during tracking, and the resulting trajectory anchors dense map optimization to jointly refine keyframe poses and dense depth maps at sequence end.

The table below compares how existing 3DFM pipelines and Map2SLAM address these three questions:

Method① 3DFM Mode① Frame-wise② Loop Closure② No Ext. VPR② KF Gate② No Ext. Flow③ Global Pose Opt.③ Online Pose Opt.③ Dense Map Opt.
LingBot-MapFrame-Stream✓
LoGeRChunk-Stream
VGGT-LongChunk✓✓
SwiftVGGTChunk✓✓✓
HorizonStreamChunk-Stream✓✓
ABot-ReconFrame-Stream✓✓✓
VGGT-SLAM 2.0Chunk✓✓✓✓
MASt3R-SLAMPairwise✓✓✓✓✓✓✓
Map2SLAM(Ours)Frame-Stream✓✓✓✓✓✓✓✓

Following your suggestion, we have revised the Introduction to clarify this motivation and expanded Related Work to discuss the compared baselines.

2. Evaluation of dense reconstruction (W4)

The table below reports the mean 3D reconstruction metrics across the Oxford Spires sequences against LiDAR ground truth (added to Appendix A, Table 4). Even before dense refinement, Map2SLAM improves over LingBot-Map across all metrics, and dense refinement brings a further gain.

MethodAcc. (m) ↓Comp. (m) ↓CD (m) ↓F1@1 m (%) ↑F1@2 m (%) ↑F1@4 m (%) ↑
DROID-SLAM126.311.5663.9360.6574.3185.35
DROID-Calib365.840.90183.3757.2772.7784.87
VGGT-SLAM 2.08.795.487.1452.2766.6981.01
VGGT-Long3.894.314.1035.1752.8371.35
Swift-VGGT4.035.744.8834.1150.2368.25
LoGeR2.413.272.8448.5967.7884.12
LingBot-Map(S)2.373.192.7851.5869.9484.52
LingBot-Map(W)4.294.844.5633.5650.9970.18
Horizon-Stream4.794.354.5732.5150.0468.46
ABot-Recon2.663.182.9243.4962.9581.81
Map2SLAM (w/o ref.)2.122.782.4558.5376.1288.63
Map2SLAM1.972.642.3161.5678.7690.50

“w/o ref.” denotes the map before dense refinement.

3. Speed and memory comparison (W5)

The table below compares the end-to-end runtime and peak GPU memory of all methods on KITTI 05 (2,761 frames) on the same machine (added to Appendix C, Table 5).

MethodMap2SLAM (1 GPU)Map2SLAM (2 GPUs)DROID-SLAMDROID-CalibDPV-SLAM++ORB-SLAM3VGGT-SLAM2.0†VGGT-LongSwift-VGGTLoGeRLingBot-Map(S)LingBot-Map(W)Horizon-StreamABot-Recon
Runtime (s)196.38142.18281.50945.81163.09283.0282.16696.52284.14448.47206.38234.90567.28247.75
Memory (GiB)14.889.71 / 6.9619.1322.726.76—4.9813.7017.0716.4011.3111.3519.2210.11

With 2 GPUs, front-end tracking and back-end optimization run on separate GPUs; with 1 GPU, both run on the same GPU. — indicates CPU-only execution, and † denotes evaluation on native keyframes only.

Reviewer_2e4DOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Map2SLAM builds on frozen LingBot-Map predictions using bounded-window causal tracking, cached-feature loop retrieval and verification, incremental pose-graph optimization, and terminal pose/depth refinement. KITTI, VBR, and Oxford Spires experiments evaluate long-sequence operation and the roles of verification and refinement.

Critical strengths
  1. Separating model predictions from revisable SLAM state produces a coherent system for feature reuse, geometric verification, and trajectory updates, evaluated on three long-sequence datasets.

  2. The ablations expose consequential failures: unverified loops harm ATE, and free segment alignment can improve local RPE while worsening global accuracy. These results explain key design choices.

Critical weaknesses

Accuracy and speed do not clearly describe one output stage. The 28–29 FPS figure concerns frontend tracking on a dual-GPU system; terminal refinement is separate. The main KITTI mean of 7.56 m is closer to iPGO-only 7.568 m than fully refined 7.489 m. Different stages or runs could explain this, but the correspondence should be explicit.

Questions for authors
  1. Which output stage supplies each main table? Please reconcile the three KITTI means and report total time, terminal time, and backend backlog for representative existing runs.

  2. On which sequences did omitted baselines fail, and with what resolution, calibration, and settings? Please distinguish failure to complete from poor accuracy; a full benchmark rerun is unnecessary.

  3. Can the complete system run on one GPU? Please report throughput, peak VRAM, backend latency, and ATE on one representative sequence against the dual-GPU configuration, listing terminal refinement time separately. If single-GPU execution is unsupported, what technical constraint makes the second GPU necessary?

P.S. I encourage the authors to address these questions carefully.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 2e4D

Thank you for highlighting the coherence of our system design (S1) and how our ablations explain key design choices (S2). We hope our replies below address your questions.

1. Clarification of the reported accuracy (W1, Q1)

Tables 1–3 report Map2SLAM’s all-frame ATE after dense refinement (7.56 m on KITTI), where refined keyframe pose corrections are propagated to non-keyframes. Fig. 5(a) reports ATE on keyframes, yielding 7.568 m after iPGO and 7.489 m after dense refinement. We apologize for not making this explicit in the paper and have clarified it in the Fig. 5 caption.

2. Runtime and single- versus dual-GPU performance (W1, Q1, Q3)

Map2SLAM can run on a single GPU. We have added a single-GPU profile alongside the dual-GPU profile for KITTI 00 in Appendix B.

1 GPU2 GPUs
Tracking (FPS)19.6530.75
Total time (s)290.05208.25
Peak VRAM (GiB/GPU)14.919.73 / 10.22
Terminal refinement (s)19.4520.80
Final ATE (m)3.923.10

After tracking ended, the back end took another 4.57 s on one GPU and 4.05 s on two GPUs to finish before terminal refinement. With two GPUs, tracking has a dedicated GPU while the back end runs on the other, avoiding GPU contention and increasing tracking throughput.

3. Omitted baselines and failure conditions (Q2)

For all baselines, we used the official implementations and followed their default settings. ORB-SLAM3 used the native image resolution, and MASt3R-SLAM used images resized to 512 pixels on the longer side. The tables below report per-sequence ATE (m) for ORB-SLAM3 and MASt3R-SLAM using the provided intrinsics.

KITTI0001020304050607080910
ORB-SLAM37.77—25.180.850.726.0316.932.5764.556.768.48
MASt3R-SLAM†——————68.04————
VBRColosseo-0Campus-0Campus-1Pincio-0Spagna-0Diag-0Ciampino-1
ORB-SLAM3—62.5123.75———15.31
MASt3R-SLAM†———38.5960.9210.29—
Oxford SpiresKC-02KC-03KC-04KC-05OQ-01OQ-02CC-02CC-03CC-05BL-02
ORB-SLAM3————0.100.110.64———
MASt3R-SLAM†5.9315.881.68—0.830.9713.050.21—31.51

A dash (—) indicates failure to produce a complete sequence trajectory (e.g., due to tracking loss or unmerged submaps), whereas completed runs with large drift are still reported numerically. † denotes a method evaluated on native keyframes only and excluded from ranking.

We have revised the baseline comparison text and added the previously omitted results to the corresponding tables.

Thank you for the detailed rebuttal. The clarification of the evaluation stages, additional runtime profiling, and per-sequence baseline results address my main concerns.

However, the noticeable ATE difference between the single-GPU and dual-GPU configurations remains a concern. From a practical SLAM perspective, I do not consider reliance on a dual-GPU setup particularly reasonable, as resource efficiency and deployment feasibility are important considerations for SLAM systems.

Overall, I maintain my rating of Weak Accept.

Reviewer_f6ddOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Map2SLAM, a monocular SLAM method built on top of a frozen LingBot-Map model to handle large drift in long trajectories. The main idea is to combine a causal front-end process with an asynchronous back-end process derived from LingBot-Map, sharing a single frame-encoder pass per image. In the front-end, frames are tracked within bounded windows, admitting keyframes via a 3D motion check and rolling back KV-cache updates for rejected frames.In the back-end, a lightweight SALAD head trained on cached encoder tokens retrieves loop candidates, while fresh bidirectional-prefix inference verifies pairwise loop geometry for online Sim(3) incremental pose graph optimization by iSAM2. The proposed method outperforms uncalibrated baselines in ATE accuracy on large-scale SLAM datasets (KITTI, VBR, and Oxford Spires) while running at over 28 FPS on two desktop GPUs.

Critical strengths
  • The proposed framework, which uses a single frozen streaming 3D foundation model (LingBot-Map) with causal mode for front-end live tracking and bidirectional mode for back-end keyframe re-observation, is clean and novel. Its single-pass feature sharing is particularly effective; it reuses cached image-encoder tokens across a lightweight 1.8M-parameter retrieval head, two-view loop verification, and multi-view segment refinement, avoiding redundant image encoding. Paired with asynchronous two-GPU execution, this keeps online tracking above 28 FPS.
  • The proposed hybrid approach, combining bounded-window causal tracking with verified Sim(3) loop-closure optimization, is well-motivated. It effectively pairs immediate frame-wise tracking with global consistency, avoiding both the local jitter of continuous streaming models (like LingBot-Map(S)) and the lack of frame-wise tracking and rigid submap drift of chunk-based methods (like VGGT-Long and SwiftVGGT).
  • Extensive quantitative evaluations on KITTI, VBR, and Oxford Spires demonstrate strong and consistent trajectory accuracy on long sequences. The proposed method outperforms all competing uncalibrated methods across every benchmark, cutting mean ATE by over 40% relative to the second-best baseline, and even surpasses complete calibrated SLAM baselines (DROID-SLAM and DPV-SLAM++) on KITTI and VBR without using camera intrinsics. Back-end ablations also clearly validate the accuracy gains from geometric verification, outlier rejection, and pose-anchored refinement.
Critical weaknesses
  • The headline hardware claim is misleading. The abstract and introduction claim >28 FPS "on an NVIDIA RTX 5090D GPU", however, actual execution uses two RTX 5090D GPUs, and no single-GPU FPS or runtime comparison against single-GPU baselines is reported.
  • Dense geometry is never quantitatively evaluated, even though dense 3D reconstruction is emphasized throughout the paper, including a dedicated dense geometric refinement pipeline and a Dense Geo. feature column in Tables 1 to 3. Although Oxford Spires provides LiDAR ground truth, the paper reports no quantitative depth, point-cloud accuracy/completeness, or calibration metrics, relying only on qualitative visualizations.
  • The component-level novelty within the individual SLAM modules is somewhat limited, as most stages directly adopt established techniques: motion-based keyframing, SALAD on frozen ViT features, two-view depth-consistency verification, riSAM/iSAM2 pose graph optimization, and DUSt3R-style pointmap alignment.
Questions for authors
  • Can the authors report quantitative 3D reconstruction metrics, e.g., point-cloud accuracy and completeness against LiDAR ground truth on Oxford Spires?
  • Were all hyperparameters fixed across datasets, and were the evaluation sequences used to tune them?
  • How do runtime and peak memory compare with baselines on equal hardware? What is single-GPU performance, and how long does the final dense optimization take on the longest sequences?
  • How do cached-feature memory and pose-graph size grow with sequence length?
2 comments
AuthorsOpenReview ↗

Response to Reviewer f6dd

We are encouraged that you described our framework as clean and novel (S1), found our hybrid approach well motivated (S2), and highlighted our trajectory results and ablations (S3). We appreciate your questions and hope our point-by-point replies address your concerns. We have marked the revisions in blue in the manuscript.

1. GPU configuration, runtime, and peak memory (W1, Q3)

(1) GPU configuration. Following your comment, we have clarified in the abstract, introduction, and conclusion that the front end tracks at over 28 FPS on one GPU while the back end runs asynchronously on a second GPU.

(2) Runtime and peak GPU memory. On KITTI 05 (2,761 frames), Map2SLAM tracked at 19.83 FPS with one GPU and 30.14 FPS with two. We measured end-to-end runtime and peak GPU memory for all methods on the same machine, and have added this comparison to Appendix C (Table 5).

MethodMap2SLAM (1 GPU)Map2SLAM (2 GPUs)DROID-SLAMDROID-CalibDPV-SLAM++ORB-SLAM3VGGT-SLAM2.0†VGGT-LongSwift-VGGTLoGeRLingBot-Map(S)LingBot-Map(W)Horizon-StreamABot-Recon
Runtime (s)196.38142.18281.50945.81163.09283.0282.16696.52284.14448.47206.38234.90567.28247.75
Memory (GiB)14.889.71 / 6.9619.1322.726.76—4.9813.7017.0716.4011.3111.3519.2210.11

In the 1-GPU setting, tracking and optimization share the same GPU; in the 2-GPU setting, they are assigned to separate GPUs (9.71 / 6.96 GiB). — indicates CPU-only execution, and † denotes evaluation on native keyframes only.

(3) Final dense optimization. The final dense optimization solver took 56.78 s on VBR Ciampino (18,846 frames) and 37.68 s on VBR Spagna (14,141 frames), the two longest sequences.

2. Quantitative 3D reconstruction on Oxford Spires (W2, Q1)

Following your suggestion, we evaluated the reconstructed point clouds against LiDAR ground truth and report the mean results across Oxford Spires sequences below (added to Appendix A, Table 4).

MethodAcc. (m) ↓Comp. (m) ↓CD (m) ↓F1@1 m (%) ↑F1@2 m (%) ↑F1@4 m (%) ↑
DROID-SLAM126.311.5663.9360.6574.3185.35
DROID-Calib365.840.90183.3757.2772.7784.87
VGGT-SLAM 2.08.795.487.1452.2766.6981.01
VGGT-Long3.894.314.1035.1752.8371.35
Swift-VGGT4.035.744.8834.1150.2368.25
LoGeR2.413.272.8448.5967.7884.12
LingBot-Map(S)2.373.192.7851.5869.9484.52
LingBot-Map(W)4.294.844.5633.5650.9970.18
Horizon-Stream4.794.354.5732.5150.0468.46
ABot-Recon2.663.182.9243.4962.9581.81
Map2SLAM (w/o ref.)2.122.782.4558.5376.1288.63
Map2SLAM1.972.642.3161.5678.7690.50

“w/o ref.” denotes the map before dense refinement. Map2SLAM already outperforms the standalone LingBot-Map baselines before refinement, and dense refinement further improves all metrics on the same keyframes.

AuthorsOpenReview ↗

Response to Reviewer f6dd - Part 2

3. System contribution and use of established components (W3)

While individual stages build on established formulations, our contribution is not reinventing standalone SLAM primitives, but tightly and cleanly designing how a single frozen streaming 3DFM is turned into a complete SLAM system across three core questions:

  • ① What to observe: We build on a frame-streaming 3DFM for continuous per-frame tracking in causal mode, while reusing cached encoder features in fresh bidirectional context to re-observe loop pairs and historical segments without re-encoding images.
  • ② When to observe: Without external VPR or external optical flow, the model's cached encoder tokens trigger loop closure and its predicted motion drives a commit-or-rollback KV probe for keyframe admission. Meanwhile, historical segment re-observations are scheduled into idle GPU intervals during online tracking, avoiding a large end-of-sequence backlog.
  • ③ How to optimize: Rather than optimizing only rigid poses or submaps, we optimize at the frame and dense-depth level: online pose graph optimization continuously corrects per-keyframe trajectory drift during tracking, and the resulting trajectory anchors dense map optimization to jointly refine keyframe poses and dense depth maps at sequence end.

4. Hyperparameters and use of evaluation sequences (Q2)

We used the same SLAM hyperparameters across all sequences, with no dataset- or sequence-specific adjustments. We used KITTI 05 during early development. Difficult loop closures on VBR Diag motivated us to adopt a robust back end and informed our choice of the GNC-TLS threshold.

5. Growth of the feature cache and pose graph (Q4)

Both grow linearly with accepted keyframes rather than all input frames, and the feature cache is stored in host RAM (2.75 MiB per keyframe) rather than GPU memory. On VBR Ciampino (18,846 frames), Map2SLAM retains 3,656 keyframes, resulting in a 9.80 GiB host-memory cache and a pose graph with 3,656 nodes and 3,852 edges (including 197 loop closures).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Summary of Revisions

We thank the Area Chair for overseeing the review process and all reviewers for their thoughtful and constructive feedback. Based on the reviewers’ suggestions, we have conducted new experiments and revised the manuscript (with changes marked in blue). Our key additions and revisions are summarized below:

  • System design and motivation (f6dd, XY31): Clarified how Map2SLAM turns a single frozen streaming 3DFM into a complete SLAM system across ① what to observe, ② when to observe, and ③ how to optimize, compared existing 3DFM pipelines along these dimensions, and revised the Introduction and Related Work accordingly.
  • Quantitative 3D reconstruction (f6dd, XY31): Added point-cloud reconstruction comparisons against LiDAR ground truth across all Oxford Spires sequences (Appendix A, Table 4), showing clear gains both over standalone LingBot-Map and from dense refinement.
  • Runtime, GPU memory, and 1-GPU vs. 2-GPU profile (f6dd, 2e4D, XY31): Added an end-to-end runtime and peak GPU memory comparison across all baselines on KITTI 05 (Appendix C, Table 5), a single- versus dual-GPU profile on KITTI 00 (Appendix B), and clarified the GPU configuration in the Abstract, Introduction, and Conclusion.
  • Evaluation protocols and omitted baselines (2e4D): Clarified the all-frame versus keyframe ATE protocols in §4.2 and Fig. 5, and added the per-sequence results of ORB-SLAM3 and MASt3R-SLAM to Tables 1–3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.