acceptodds
Under review as a conference paper at ICLR 2027

Remember, Revisit, Revise: Scaffold Memory for Long-Horizon 3D Geometry

Abstract

In 3D computer vision, geometry foundation models estimate camera poses and 3D scene structure from video sequences, but maintaining consistent reconstructions over long horizons remains challenging. Feed-forward models process long sequences as independent, fixed-length frame windows and subsequently align their predictions, causing pose and reconstruction errors to accumulate across windows. Streaming models retain information from previous frames to inform future predictions, but cannot revise earlier estimates when subsequent observations reveal errors. To address these limitations, we introduce Scaffold Memory for Revision (SMoRe), a framework that equips existing geometry foundation models with a revisable external memory for long-horizon inference. Inspired by hippocampal associative memory in biological brains, which links spatial context with sensory experience, SMoRe associates where an observer is with what it observes, enabling past observations to be stored, retrieved, verified against new evidence, and revised when inconsistencies are detected. Crucially, SMoRe integrates with existing geometry foundation models while keeping them frozen, requiring no additional training or fine-tuning. We evaluate SMoRe with eight competitive geometry models on seven datasets spanning camera pose estimation, multi-view depth estimation, 3D point-cloud reconstruction, simultaneous localization and mapping (SLAM), and novel view synthesis. Across all the evaluated geometry models, SMoRe consistently improves long-sequence camera pose estimation, reduces multi-view depth error by up to 51%, reduces Chamfer distance for 3D point-cloud reconstruction by up to 44%, and achieves average trajectory error as low as 2.9 cm in long-horizon SLAM. By retaining corrected camera poses and observations, SMoRe further improves the fidelity and spatial consistency of large-scale 3D scene reconstruction. These results establish revisable external memory as a flexible interface for extending pretrained geometry models beyond fixed-context prediction toward iterative, error-correcting inference over long visual sequences. Code, data, and models will be publicly available upon publication.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long horizon 3D reconstruction, geometry models, associative memory, continuous attractor networks, camera pose estimation, simultaneous localization and mapping (SLAM)

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_GLRgOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper adds an external, revisable memory (SMORE) to frozen 3D geometry models. It splits long videos into 32-frame windows, stores frames, poses and point maps, retrieves old frames with DINOv2, checks them geometrically, and uses accepted revisits to correct window placement. It tests 8 models on 7 datasets over 5 tasks and reports better pose, depth, reconstruction, SLAM and NVS results.

Critical strengths

1.The core result is broad and consistent. SMORE improves 8 frozen backbones on pose, depth and reconstruction, and gains grow with sequence length. This is a practical, training-free interface: the geometry model stays frozen, and memory handles long-horizon correction. This is the main reason I lean toward acceptance. 2.The verification step is well designed. Requiring two independent historical references to agree, and spreading the correction progressively in Sim(3), is a sensible guard against a single false loop closure. This is the step that makes retrieval usable for correction, and the results depend on it.

Critical weaknesses

1.The NVS comparison does not test the memory. Appendix J.2 says the baseline uses one prediction, while +SMORE uses four input orderings plus consensus. With only four input frames there is one 32-frame window; retrieval removes candidates within 64 frames and needs two pairs from earlier windows, so the memory never triggers. Table 2 is therefore test-time ensembling, not SMORE. Some gains are within noise (+0.05 dB, +0.08 dB). This does not invalidate the main method, but the “five tasks” claim should be corrected: either give the baseline the same four-ordering consensus, or report NVS as an ablation of repeated prediction. 2.The closest prior work and verification statistics are missing. VGGT-Long is not cited, and VGGT-SLAM 2.0 already does training-free attention-based retrieval verification; VGGT-SLAM++ uses VPR + Sim(3) pose graph. SMORE compares to neither, nor to a standard VPR baseline (SALAD/AnyLoc) + geometric verification + pose graph. The lookup-table ablation is a strawman because it needs frame identity. Also, SLAM uses published numbers on 7-Scenes only, and Appendix N admits joint processing can hurt even when revisits are rejected, but the main text reports no distortion gate or false-accept rate. These limit the strength of the novelty/SOTA claims, but they are addressable with additional comparisons and reporting.

Questions for authors

1.NVS: Does the baseline get the same four-ordering consensus? If yes, what does SMORE add with only four frames? If no, please rerun with identical test-time augmentation, or relabel NVS as an ablation. 2.Verification: What are the precision/recall and false-accept rate of revisit verification? What is the default distortion gate, how often does it trigger, and is there any rollback after a bad revision? 3.Scaffold: How does the full pipeline compare to standard VPR + geometric verification + Sim(3) pose graph with the same windowing and backbones? Does using the drifted global pose as the memory address propagate drift? 4.SLAM: Can you rerun the baselines under the same protocol and report large-scale results (KITTI/Waymo/TUM) against VGGT-Long and VGGT-SLAM 2.0? Also, why is Sim(3) enough given VGGT-SLAM's SL(4) argument for uncalibrated submaps? 5.Cost: What is the per-frame latency and total runtime at N=200, including DINOv2, retrieval and the 36-frame joint passes? Is it online/real-time?

4 comments
AuthorsOpenReview ↗

Response to Reviewer GLRg (Part 1)

Reviewer GLRg:

We sincerely thank the reviewer for the constructive feedback and insightful suggestions, which have helped us strengthen the manuscript. For ease of reference, we will reproduce the reviewers’ questions and concerns verbatim in bold throughout our responses, allowing reviewers to locate the corresponding clarifications and additional evidence quickly. We will incorporate all clarifications, additional experimental results, and analyses presented in our rebuttal into the camera-ready version.

GLRg.1 - Weakness 1.The NVS comparison does not test the memory. Appendix J.2 says the baseline uses one prediction, while +SMORE uses four input orderings plus consensus. With only four input frames there is one 32-frame window; retrieval removes candidates within 64 frames and needs two pairs from earlier windows, so the memory never triggers. Table 2 is therefore test-time ensembling, not SMORE. Some gains are within noise (+0.05 dB, +0.08 dB). This does not invalidate the main method, but the “five tasks” claim should be corrected: either give the baseline the same four-ordering consensus, or report NVS as an ablation of repeated prediction.

Response: We thank the reviewer for identifying this important experimental confound. To address this concern, we have redesigned the NVS experiments, replacing the original object-centric Objaverse-LVIS and GSO datasets with two long-sequence-based benchmarks: 7-Scenes and CO3Dv2. Please refer to Tables R5a-b and response Pe6y.4 - Weakness 2(d) for more details.

GLRg.2 - Weakness 2. The closest prior work and verification statistics are missing. VGGT-Long is not cited, and VGGT-SLAM 2.0 already does training-free attention-based retrieval verification; VGGT-SLAM++ uses VPR + Sim(3) pose graph. SMORE compares to neither, nor to a standard VPR baseline (SALAD/AnyLoc) + geometric verification + pose graph. The lookup-table ablation is a strawman because it needs frame identity. Also, SLAM uses published numbers on 7-Scenes only, and Appendix N admits joint processing can hurt even when revisits are rejected, but the main text reports no distortion gate or false-accept rate. These limit the strength of the novelty/SOTA claims, but they are addressable with additional comparisons and reporting.

Response: Thank you! We now evaluate VGGT-Long, VGGT-SLAM 2.0 and VGGT-SLAM++ in our SLAM task table. All three revise earlier estimates through loop closure and a pose graph. We ran the three systems under our protocol, with the same window sizes and backbones (Table R12). On 7-Scenes SLAM task they reach 0.0420, 0.0514 and 0.0648 m against 0.0301 m for SMoRe, and on CO3Dv2 camera estimation 90.6 to 91.7 (VGGT) and 90.4 to 92.5 (VGGT-Ω) against 93.9 and 94.7 on AUC@30. SMoRe performs better than these established SLAM systems using the same protocol on the same dataset sequences.

We would like to draw the reviewer’s attention to Table S6 Group C in the manuscript, where we only replace the scaffold with all the established VPRs like SALAD, NetVLAD, MixVPR, EigenPlaces, CosPlace and BoQ while keeping the geometric verification and other thresholds fixed to the default modes.

Please refer to Tables R8c-e and response R46d.2 - Weakness 2 for more details on the distortion gate analysis and false accept rates associated with each gate.

GLRg.3 - Question 1. NVS: Does the baseline get the same four-ordering consensus? If yes, what does SMORE add with only four frames? If no, please rerun with identical test-time augmentation, or relabel NVS as an ablation.

Response: We thank the reviewer for identifying this important experimental confound. To address this concern, we have redesigned the NVS experiments, replacing the original object-centric Objaverse-LVIS and GSO datasets with two long-sequence-based benchmarks: 7-Scenes and CO3Dv2. For fair comparisons, we removed the four-ordering consensus for all baselines and our SMORE in these experiments. Please refer to Tables R5a-b and response Pe6y.4 - Weakness 2(d) for more details.

AuthorsOpenReview ↗

Response to Reviewer GLRg (Part 2)

GLRg.4 - Question 2. Verification: What are the precision/recall and false-accept rate of revisit verification? What is the default distortion gate, how often does it trigger, and is there any rollback after a bad revision?

Response: We thank the reviewer for these important questions regarding verification reliability and revision robustness.

1. Verification reliability and distortion gating. SMoRe uses a 5° distortion gate by default to reject geometrically inconsistent corrections. As reported in Table R8c, this configuration achieves 97.30% precision, 90.00% recall, and a 2.27% false-positive rate on 7-Scenes. The table additionally reports accepted and rejected revisit counts across different gating thresholds. To directly address the reviewer's question about gating frequency, we additionally measure the average distortion-gate trigger rate per query window, which is only 2.38% under our default 5° configuration. This indicates that the distortion gate is triggered relatively infrequently, allowing most candidate corrections to proceed without being rejected by this additional safeguard. Please also refer to R46d.2 – Weakness 2 for a detailed sensitivity analysis of the distortion gate and Tables R8d–R8e for comparisons of verification reliability against conventional VPR methods.

2. Revision safeguards and rollback. We clarify that SMoRe does not currently implement an explicit rollback mechanism after an accepted revision. Instead, we mitigate erroneous corrections through multiple consistency checks before applying them, including scale-consistency thresholds, orientation verification, and pairwise camera-distance consistency. Only geometrically verified corrections are propagated to historical window placements.

3. Progressive and retrospective geometric revision. To evaluate the importance of propagating verified corrections, we additionally compare SMoRe against two alternative optimization strategies: (i) deferred pose-graph optimization, where corrections are applied only after processing the entire sequence, and (ii) progressive Sim(3) optimization without backward propagation, where corrections update the current window but leave historical placements unchanged. As shown in Table R11, SMoRe's default progressive revision strategy outperforms both alternatives on 7-Scenes SLAM and CO3Dv2 camera pose estimation. These results demonstrate the benefit of propagating verified corrections to previously estimated window placements rather than leaving historical geometric inconsistencies unresolved. Please refer to R46d.5 – Question 1 for the detailed comparison.

AuthorsOpenReview ↗

Response to Reviewer GLRg (Part 3)

GLRg.5 - Question 3. Scaffold: How does the full pipeline compare to standard VPR + geometric verification + Sim(3) pose graph with the same windowing and backbones? Does using the drifted global pose as the memory address propagate drift?

Response:

1. Comparison with conventional VPR and geometric optimization. We draw the reviewer's attention to Table S6 (Group C) in the manuscript, where we replace SMoRe's associative scaffold with established VPR methods, including SALAD, NetVLAD, MixVPR, EigenPlaces, CosPlace, and BoQ, while keeping the geometry backbone, windowing strategy, geometric verification, and downstream revision procedures fixed. These controlled comparisons demonstrate the additional benefit of associative scaffold memory over conventional visual retrieval.

2. Why associative scaffold memory improves retrieval. In SMoRe, each video frame is associated with a shared memory representation (hh), which links its visual descriptor (ss) to its spatial scaffold state (gg). Historical observations are retrieved by comparing these associative representations rather than directly matching visual descriptors. This design allows spatial information to implicitly influence retrieval through learned visual–spatial associations, providing a spatially informed retrieval mechanism rather than a conventional visual-feature lookup table.

3. Does a drifted spatial address propagate geometric errors? Importantly, the stored spatial state (gg) is used only to propose retrieval candidates, not to directly correct geometric placements. Each retrieved pair is jointly re-estimated with the current window, and a correction is propagated to historical windows only when two independently retrieved pairs from different source windows produce consistent placements. Specifically, Eq. (4) requires agreement within 3° in rotation, (0.15Ek0.15E_k) in position, and the scale ratio below 1.5. Therefore, although drift in stored spatial states may affect candidate retrieval, it does not directly propagate into geometric corrections. Placement updates depend on newly estimated and geometrically verified correspondences rather than the stored spatial addresses themselves. This separation also allows SMoRe to retain historical scaffold states without recomputing them after every revision.

4. Additional experiment: Retrospective updates of stored associative memory. To directly investigate whether updating historical memory after geometric revision improves performance, we conduct an additional controlled experiment. Whenever historical camera poses are corrected, we refresh their associated spatial states and memory representations (hh), together with the corresponding associative mappings (WshW_{sh}), (WhsW_{hs}), and (WhgW_{hg}). All subsequent retrieval, geometric verification, and placement revision procedures remain unchanged.

We evaluate this retrospective-update variant on 7-Scenes SLAM and CO3Dv2 camera pose estimation, with results reported in Table R1. Retrospective memory updates achieve performance comparable to our default implementation but incur additional computational overhead from refreshing historical representations and associations. These findings suggest that explicitly refreshing historical associative memory after every geometric correction provides limited additional benefit under the evaluated conditions. Our default implementation therefore maintains a more efficient separation between associative retrieval and verified geometric revision, while online adaptation of the associative mappings continues to incorporate new observations.

AuthorsOpenReview ↗

Response to Reviewer GLRg (Part 4)

GLRg.6 - Question 4. SLAM: Can you rerun the baselines under the same protocol and report large-scale results (KITTI/Waymo/TUM) against VGGT-Long and VGGT-SLAM 2.0? Also, why is Sim(3) enough given VGGT-SLAM's SL(4) argument for uncalibrated submaps?

Response:

1. Large-scale SLAM comparisons under a unified evaluation protocol. Following the reviewer's suggestion, we rerun VGGT-Long and VGGT-SLAM 2.0 on TUM RGB-D, KITTI Odometry, and Waymo using a consistent evaluation protocol, reporting absolute trajectory error (ATE) in metres. Tables R13a–R13c present results across nine TUM sequences, four KITTI sequences, and the Waymo dataset. On TUM RGB-D, SMoRe achieves a mean ATE of 0.0523 m online and 0.0489 m after final optimization, compared with 0.0544 m for VGGT-SLAM 2.0 under the same protocol. On KITTI, SMoRe achieves the lowest mean ATE of 3.1 m, outperforming both VGGT-Long and VGGT-SLAM 2.0. On Waymo, SMoRe achieves a modest improvement in mean ATE over VGGT-Long while requiring less processing time per frame than both VGGT-Long and VGGT-SLAM 2.0. These additional experiments extend our evaluation to larger-scale indoor and outdoor trajectories and provide direct comparisons with recent geometry-based SLAM systems.

2. Sim(3) versus SL(4) alignment. We acknowledge the reviewer's point that SL(4) provides greater flexibility for aligning uncalibrated submaps, as discussed in VGGT-SLAM. SMoRe adopts Sim(3) by default because it models relative rotation, translation, and isotropic scale while preserving Euclidean geometric structure. However, Sim(3) cannot fully represent the more general projective distortions that SL(4) can accommodate. To quantify this tradeoff, we additionally compare Sim(3) and constrained SL(4) alignment under identical local predictions and geometric constraints, as reported in Table R13d. The results show that SL(4) provides modest geometric accuracy improvements but incurs greater computational overhead. Thus, our default Sim(3) backend represents a practical tradeoff between geometric accuracy and processing efficiency, rather than an assumption that similarity transformations are universally sufficient. We will incorporate these large-scale benchmark comparisons and the Sim(3)/SL(4) ablation into the revised manuscript, explicitly discussing the benefits and limitations of each alignment model.

GLRg.7 - Question 5. Cost: What is the per-frame latency and total runtime at N=200N = 200, including DINOv2, retrieval and the 36-frame joint passes? Is it online/real-time?

Response:

We thank the reviewer for requesting a detailed computational cost analysis. We now provide additional experiments quantifying the latency and memory requirements of SMoRe.

1. Per-frame latency and total runtime. We profile the computational cost of each major component, including DINOv2 feature extraction, associative memory retrieval, 36-frame joint inference for geometric revision, and windowed geometry estimation using VGGT-Ω. Table R14a reports the individual latency contributions. The dominant computational cost comes from the VGGT-Ω backbone for windowed geometry inference. Overall, SMoRe requires approximately 0.16 seconds per frame, corresponding to 31.3 seconds for a sequence of 200 keyframes. SMoRe operates online, processing observations incrementally without requiring access to future frames. However, online processing does not necessarily imply real-time operation; real-time performance depends on the input frame rate and hardware configuration. When optional final pose-graph optimization is enabled, it adds approximately 0.62 seconds, increasing the total processing time to 31.92 seconds for 200 keyframes. We emphasize that this final optimization is optional and is not included in our default online results in the main text.

2. Memory scalability over long sequences. We additionally evaluate memory consumption as the sequence length increases to 5,000 keyframes. As shown in Table R14c, SMoRe's active GPU memory allocation remains approximately bounded, while host memory usage increases with the number of retained historical observations. This behavior contrasts with native causal streaming models whose growing inference state can lead to substantially higher peak GPU memory consumption and eventually out-of-memory failures. Table R3 provides controlled comparisons among native causal streaming, bounded-window chaining, and SMoRe. Importantly, SMoRe maintains a GPU memory profile comparable to bounded-window chaining while achieving higher geometric accuracy, demonstrating that its retrieval and revision mechanisms do not require unbounded GPU-resident inference states. Please refer to Pe6y.4 – Weakness 2(a) for further analysis of streaming inference, computational cost, and memory scalability. These results clarify that SMoRe supports incremental online processing with bounded active GPU memory and moderate computational overhead.

Reviewer_R46dOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Training-free external memory giving frozen geometry foundation models a loop-closure mechanism. Sequences run in 32-frame windows; each window's global placement is encoded as a sparse scaffold address reached from a DINOv2 descriptor through a linear association trained by recursive least squares. Visual similarity proposes two revisited pairs; each is verified by re-processing retrieved frames jointly with the current window and requiring agreement within 3° and 0.15 of the reconstruction's spatial extent. Verified pairs fit one corrected placement in Sim(3), distributed across intervening windows. Eight backbones, five tasks.

Critical strengths
  • Tested harder than most. Eight models, five tasks, three runs with deviations: VGGT-Ω goes from 4.5 to 2.9 cm ATE on 7-Scenes, and Replica overall error drops 39–51%. The resource advantage is measured, not assumed — baseline peak memory reaches 23.9 GB at N = 200 where SMORE stays at 8.5–9.6 GB, and streaming baselines that OOM at N = 100 complete with SMORE.
  • Revision really is online. Main tables contain no offline global optimisation, and Online versus Final differ by ~0.001 m ATE in the appendix, so nothing hides in post-hoc refinement.
Critical weaknesses
  • The ablations separate "loop closure exists" from "the memory", and the memory is the smaller half. Group A: no revision 0.045 m, flat DINOv2 cosine with the same revision pipeline 0.034 m, scaffold 0.029 m. About two thirds of the gain comes from closing loops at all, leaving the memory itself worth ~0.005 m.
  • Descriptor choice is decisive, and six of nine candidates are worse than doing nothing. Against the 0.045 m reference: NetVLAD 0.050, MixVPR 0.053, SALAD 0.056, EigenPlaces 0.066, BoQ 0.127, DINOv2+SALAD 0.133, CosPlace 0.263 — all in the appendix. Compounding it, the default joint policy lets retrieved frames perturb the local prediction even when both revisits are rejected (0.0451 → 0.0423 m with updates disabled). Gating repairs this cleanly (seed-1 failure 0.092 → 0.029 m) but is presented as a diagnostic, not shipped. As written the pipeline is one retrieval failure from corrupting the map, and no false-accept rate is reported.
  • Regressions are not surfaced. SMORE worsens VGGT-Ω on fire (0.024 → 0.031), heads (0.012 → 0.013) and stairs (0.020 → 0.022), with no per-scene deviation. The two headline metrics also prefer different hyperparameters: best ATE (W = 64, overlap 0.7) gives the worst pose AUC, 87.2 versus 93.4.
  • "Long-horizon" is modest. 7-Scenes clips run 17–33 s, evaluation caps at N = 200, CO3Dv2 spans 8–68 s; nothing tests minutes-long trajectories or staleness of the association.
Questions for authors
  1. How does SMORE compare with the same retrieval and verification feeding a standard pose-graph backend?
  2. Given Table S12, would you ship the 1° gate? Please report the false-accept rate.
  3. Report per-scene deviations for Table S3, including fire, heads and stairs.
4 comments
AuthorsOpenReview ↗

Response to Reviewer R46d (Part 1)

Reviewer R46d:

We sincerely thank the reviewer for the constructive feedback and insightful suggestions, which have helped us strengthen the manuscript. For ease of reference, we will reproduce the reviewers’ questions and concerns verbatim in bold throughout our responses, allowing reviewers to locate the corresponding clarifications and additional evidence quickly. We will incorporate all clarifications, additional experimental results, and analyses presented in our rebuttal into the camera-ready version.

R46d.1 - Weakness 1: The ablations separate "loop closure exists" from "the memory", and the memory is the smaller half. Group A: no revision 0.045 m, flat DINOv2 cosine with the same revision pipeline 0.034 m, scaffold 0.029 m. About two-thirds of the gain comes from closing loops at all, leaving the memory itself worth ~0.005 m.

Response: Please refer to Pe6y.1 – Weakness 1(a) and 1(b) and Table R1, where we provide detailed analyses separating the contributions of conventional loop closure and associative scaffold memory. Although the absolute improvement over direct DINOv2 retrieval is 0.005 m, this represents a 14.7% reduction in ATE RMSE and closes approximately 35.2% of the remaining error gap toward oracle retrieval. These results demonstrate that the associative scaffold provides a meaningful additional benefit beyond conventional retrieval-based loop closure under identical downstream verification and revision procedures.

AuthorsOpenReview ↗

Response to Reviewer R46d (Part 2)

R46d.2 - Weakness 2: Descriptor choice is decisive, and six of nine candidates are worse than doing nothing. Against the 0.045 m reference: NetVLAD 0.050, MixVPR 0.053, SALAD 0.056, EigenPlaces 0.066, BoQ 0.127, DINOv2+SALAD 0.133, CosPlace 0.263 — all in the appendix. Compounding it, the default joint policy lets retrieved frames perturb the local prediction even when both revisits are rejected (0.0451 → 0.0423 m with updates disabled). Gating repairs this cleanly (seed-1 failure 0.092 → 0.029 m) but is presented as a diagnostic, not shipped. As written the pipeline is one retrieval failure from corrupting the map, and no false-accept rate is reported.

Response: We thank the reviewer for identifying these important concerns regarding descriptor selection, geometric gating, and verification reliability. We address each below.

1. Fair comparison across visual descriptors. Two factors contributed to the poor performance of alternative VPR descriptors in our original ablations. First, the ungated joint-prediction policy allowed retrieved frames to distort local geometry. Applying the same distortion gate substantially reduces these errors: CosPlace improves from 0.263 to 0.048 m, BoQ from 0.127 to 0.052 m, and DINOv2+SALAD from 0.133 to 0.050 m (Table S12).

Second, alternative descriptors were not individually calibrated for appearance-similarity thresholds, whereas DINOv2 used a threshold of 0.5. To ensure fair comparisons, we now evaluate all VPR descriptors using the same default gated revision policy and independently tune their appearance thresholds on a held-out validation set. As shown in Table R2, the calibrated VPR baselines achieve ATE RMSE of 0.032–0.034 m, compared with 0.030 m for SMoRe. These results demonstrate that SMoRe retains its performance advantage under matched gating and descriptor-specific calibration. Please refer to Pe6y.1 – Weakness 1(c) for the complete grid-search analysis.

2. Clarification of the default gated policy. We clarify that SMoRe already uses the gated local-prediction policy by default, with a 5° threshold. The joint and ordinary policies are included only as alternative revision strategies for ablation. The gate preserves the original local prediction when the joint prediction deviates excessively, preventing rejected revisits from introducing unintended local geometric distortions. As shown in Table S12, gating resolves the reported seed-1 failure, reducing ATE from 0.092 to 0.029 m.

Following the reviewer's suggestion, we additionally evaluate the sensitivity of SMoRe to different gating thresholds across downstream tasks in Table R8a, with seed-wise results in Table R8b. These experiments confirm the effectiveness and robustness of our default 5° gate. Please refer to Pe6y.10 – Question 3 for details.

3. Verification reliability and false-accept rates. To address the missing verification statistics, we now provide additional analyses in Tables R8c and R8d–R8e.

Table R8c evaluates SMoRe's verification reliability on 7-Scenes across different gating thresholds, reporting accepted and rejected revisits, precision, recall, and false-positive rates. At the default 5° threshold, SMoRe achieves 97.30% precision and 90.00% recall, with a false-positive rate of only 2.27%.

Tables R8d–R8e further compare SMoRe against DINOv2 and MixVPR on 7-Scenes and CO3Dv2, measuring accepted candidate frame pairs across query windows and their verification statistics. SMoRe achieves the lowest false-positive rate on both datasets, reaching 1.64% on CO3Dv2, while maintaining the highest precision and recall.

Together, these additional experiments demonstrate that SMoRe's performance advantage is not attributable to unfair descriptor calibration, its default gated policy mitigates geometric corruption from rejected revisits, and its associative scaffold supports more reliable revisit retrieval and verification than conventional VPR baselines.

AuthorsOpenReview ↗

Response to Reviewer R46d (Part 3)

R46d.3 - Weakness 3: Regressions are not surfaced. SMORE worsens VGGT-Ω on fire (0.024 → 0.031), heads (0.012 → 0.013) and stairs (0.020 → 0.022), with no per-scene deviation. The two headline metrics also prefer different hyperparameters: best ATE (W=64W = 64, overlap 0.7) gives the worst pose AUC, 87.2 versus 93.4.

Response: We thank the reviewer for highlighting the importance of reporting per-scene regressions and clarifying our hyperparameter selection. We address both concerns below.

1. Per-scene regressions and statistical variability. We acknowledge that SMoRe does not improve performance uniformly across all scenes. To make these limitations explicit, Table R9a now reports per-scene SLAM results across three random seeds, including means and standard deviations. SMoRe underperforms sequential chaining on fire (0.024 → 0.031 ± 0.001 m), heads (0.012 → 0.013 ± 0.000 m), and stairs (0.020 → 0.022 ± 0.000 m). The small standard deviations indicate that these regressions are consistent across seeds rather than attributable to random variation.

Notably, these three scenes already exhibit low chaining errors (≤0.024\le 0.024 m), leaving limited accumulated drift for revisit-based correction. In contrast, SMoRe's overall improvement from 0.045 to 0.029 m is primarily driven by scenes with greater accumulated drift, including office, pumpkin, chess, and redkitchen. We will explicitly report these regressions and discuss this limitation in the revised manuscript.

2. Window-size and overlap sensitivity. We agree that different evaluation metrics favor different window configurations. As shown in Table S6 (Group G) and reproduced in Table R9b, a window size of W=64W = 64 with 0.7 overlap achieves the lowest 7-Scenes ATE (0.025 m versus 0.030 m for our default), but substantially reduces CO3Dv2 pose AUC (87.2 versus 93.4).

Importantly, SMoRe consistently uses W=32W = 32 with 0.5 overlap as its default configuration across all datasets and tasks, rather than independently optimizing hyperparameters for each benchmark. This configuration provides a more favorable balance between SLAM accuracy, pose estimation performance, and computational efficiency. Table R9b further reports runtime comparisons, showing that the smaller default window also reduces computational latency.

R46d.4 - Weakness 4: "Long-horizon" is modest. 7-Scenes clips run 17–33 s, evaluation caps at N=200N = 200, CO3Dv2 spans 8–68 s; nothing tests minutes-long trajectories or staleness of the association.

Response: We thank the reviewer for highlighting this limitation. We agree that our original evaluations involved relatively short sequences and did not sufficiently establish SMoRe's effectiveness over extended temporal horizons. To address this concern, we substantially extend our evaluations in both sequence length and environmental diversity, and introduce additional experiments specifically examining long-term memory retrieval and association staleness.

1. Long-horizon 3D reconstruction and SLAM. We extend ScanNet++ reconstruction experiments from the original 200-frame subsequences to 600–8,000 frames per scene, using 32-frame windows. We additionally evaluate long-horizon SLAM on TUM RGB-D (fr1) and KITTI Odometry (sequences 00, 05, 06, and 07), covering both indoor and outdoor environments. Please refer to Pe6y.4 – Weakness 2(c) and Tables R4a–R4c for detailed results and analysis.

2. Long-term memory retrieval and association staleness. To directly investigate whether SMoRe can reliably retrieve historical observations over extended periods, we conduct additional experiments on KITTI sequences 00, 05, 06, and 07, spanning trajectories of up to 8 minutes (Table R10). We evaluate Recall@5, measuring whether the ground-truth historical match appears among the top five retrieved candidates, at progressively longer temporal delays. Specifically, Table R10 reports two complementary metrics: (1) Aggregated Recall@5: Retrieval accuracy accumulated over all queries up to a given temporal horizon. (2) Delay-specific Recall@5: Retrieval accuracy evaluated exclusively on queries whose historical matches fall within the specified temporal-delay interval. As expected, retrieval accuracy generally decreases with increasing temporal delay, reflecting the greater difficulty of long-term association. However, SMoRe consistently achieves higher aggregated and delay-specific Recall@5 than MixVPR, with a slower decline as the retrieval horizon increases. These results provide direct evidence that SMoRe's associative scaffold better preserves retrieval effectiveness over extended temporal intervals.

3. Additional verification reliability analysis. We further report precision, recall, false-positive rates, and false-negative rates for revisit verification on 7-Scenes and CO3Dv2. Please refer to Pe6y.10 – Question 3 and Tables R8d–R8e for these complementary analyses.

AuthorsOpenReview ↗

Response to Reviewer R46d (Part 4)

R46d.5 - Question 1: How does SMORE compare with the same retrieval and verification feeding a standard pose-graph backend?

Response: We thank the reviewer for this suggestion. To isolate the contribution of SMoRe's progressive geometric revision from conventional pose-graph optimization, we conduct additional controlled experiments comparing three Sim(3) optimization strategies under identical retrieval, geometric verification, and candidate-acceptance thresholds: (1) Deferred pose-graph optimization: All Sim(3) corrections are deferred until the end of the sequence, without updating intermediate window placements during inference. (2) Progressive Sim(3) optimization without backward propagation: Corrections are applied incrementally to the current window, but are not propagated to previously estimated windows. (3) SMoRe (our default): Corrections are applied progressively and propagated to previous window placements through the Sim(3) interpolation strategy defined in Eq. (7). We evaluate all three strategies on 7-Scenes SLAM (ATE RMSE) and CO3Dv2 camera pose estimation (AUC@30). Results are reported in Table R11.

SMoRe consistently outperforms both alternative optimization strategies. In particular, deferring all corrections until the end of the sequence yields inferior performance, because subsequent windows are aligned using placements that have not yet benefited from earlier corrections. Progressive optimization without backward propagation improves upon this baseline but still allows inconsistencies in historical window placements to persist and affect subsequent alignment. In contrast, SMoRe progressively propagates verified geometric corrections to historical window placements, reducing accumulated inconsistencies and improving subsequent estimation. Since retrieval and verification are held constant across all three variants, these results provide additional evidence that SMoRe's progressive revision strategy contributes improvements beyond those obtained using a conventional pose-graph backend alone. We will incorporate Table R11 and the corresponding implementation details into the revised manuscript.

R46d.6 - Question 2: Given Table S12, would you ship the 1° gate? Please report the false-accept rate.

Response: We thank the reviewer for this question. We retain the 5° gate as the default policy rather than adopting the more conservative 1° gate, as it provides a better balance between accepting valid revisits and rejecting incorrect matches. As shown in Table R8c, the 5° gate achieves 97.30% precision and 90.00% recall, with a false-positive rate of only 2.27%. Please refer to R46d.2 – Weakness 2 for our detailed analysis of gating robustness and verification reliability.We will clarify this default policy and report the false-accept statistics in the revised manuscript.

R46d.7 - Question 3: Report per-scene deviations for Table S3, including fire, heads and stairs.

Response: We thank the reviewer for this suggestion. Table R9a now reports the per-scene ATE RMSE (in metres) across all seven 7-Scenes sequences, including the mean and standard deviation over three random seeds. We explicitly report the regressions on fire, heads, and stairs, rather than only presenting aggregate improvements. Please refer to R46d.3 – Weakness 3 for our detailed analysis of these results. We will incorporate the complete per-scene statistics into Table S3 of the revised manuscript.

Reviewer_Pe6yOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  1. Method. The paper proposes Scaffold Memory for Revision (SMORE), a training-free wrapper that extends frozen geometry foundation models to long sequences. Images are processed in overlapping windows, whose local camera poses and point maps are placed in a global coordinate system through Sim(3) alignment.

  2. Memory and revision. For each stored frame, SMORE maps its estimated global pose through a periodic position/orientation scaffold and a sparse random projection to obtain a memory address. Online recursive least squares associates DINOv2 image descriptors with these addresses. A current image retrieves historical frames; two retrieved frame pairs from different source windows are jointly processed with the current window and used to propose its global placement. If the proposals agree, SMORE revises intervening window placements and can perform a final window-level pose-graph optimization.

  3. Validation. The framework is evaluated with eight geometry models on seven datasets. It reports improvements in pairwise camera pose, dense reconstruction, and 7-Scenes SLAM, together with lower peak GPU memory under bounded-window inference. A separate novel-view-synthesis experiment aggregates repeated predictions of the same four images under different input permutations.

Critical strengths
  1. Broad, practical compatibility with frozen models. SMORE is applied to eight feed-forward and streaming geometry models without fine-tuning their weights. The local/global representation cleanly allows a corrected window to move its cameras and dense geometry together. The large Replica and ScanNet++ reductions, the VGGT-Ω 7-Scenes improvement from 4.5 cm to 2.9 cm, and the generally stronger long-sequence pose results suggest that explicit retrospective correction is useful.

  2. Substantial experimental and systems analysis. The paper evaluates multiple tasks and reports window, overlap, descriptor, sparsity, retrieval-evidence, runtime, GPU-memory, and failure-case analyses. In particular, the authors explicitly separate local-prediction changes from global placement revision and disclose a serious failure case rather than hiding it.

Critical weaknesses
  1. The experiments do not isolate the claimed contribution from standard loop closure and pose-graph optimization. The largest gains compare the full system against sequential window chaining, so they primarily establish the known value of adding long-range loop closures. Table S6 Group A is the decisive scaffold ablation, but it is limited to small pose subsets: direct DINOv2 retrieval with the same downstream machinery already obtains 0.034 m versus 0.029 m final ATE and 90.9 versus 93.3 AUC@30. There is no full-task comparison against a conventional, well-tuned DINOv2/VPR retrieval index plus the identical geometric verification and pose graph, nor an oracle-retrieval analysis. Alternative descriptors are additionally evaluated without descriptor-specific gates, while DINOv2 receives a tuned 0.5 gate. The periodic attractor dynamics are not used to estimate motion or decode the current pose, and stored pose-derived addresses are explicitly not updated after their window placements are revised. Thus, the evidence supports retrieval-based revision, but not the necessity or conceptual claim of a revisable spatial scaffold.

  2. Several headline evaluations are confounded or outside the proposed memory mechanism. Native streaming inference is compared with a different bounded-window execution regime for the SMORE variants. The large N=10 gains for StreamVGGT and STream3R occur before the 64-frame minimum revisit separation can permit the advertised long-range correction, so they cannot be attributed to scaffold memory; the associated OOM comparison is similarly confounded. The headline 2.9 cm SLAM number appears to include final offline optimization despite the online framing. Most end-to-end sequences contain only 200 sampled frames, ScanNet++ uses only the first 200 frames, and the 3,100-frame experiment tests retrieval alone, limiting the “long-horizon” evidence. Finally, the novel-view-synthesis experiment uses neither scaffold retrieval nor trajectory revision: “+SMORE” simply receives four geometry-model passes under different input permutations while the baseline receives one. This is unequal-compute test-time ensembling, not validation of revisable memory, and should be presented separately.

Questions for authors
  1. How is a unique Sim(3) transform obtained from each retrieved pair when Eq. (1) uses only two camera centers? Is camera orientation incorporated into the estimator itself, rather than only checked afterward? Why does Eq. (4) omit agreement in scale, and what is the sensitivity to adding such a test?

  2. Can the authors report all headline results for (i) sequential chaining, (ii) direct DINOv2 or a standard VPR loop-closure index plus exactly the same verification and optimization, (iii) the scaffold variant, and (iv) oracle retrieval? This would determine how much of the gain comes from loop closure generally and how much comes from the proposed scaffold.

  3. Why is the gated local-prediction policy from Appendix N not used by default, given the documented 0.025-to-0.459 m failure after rejected revisits? Please report the three-run seed-wise results, accepted and rejected revisit counts, catastrophic-failure frequency, and all main metrics with the 1-degree gate.

  4. Which headline results are online-only and which include the final pose-graph optimization? Is the default online update Eq. (7)'s Sim(3) interpolation or the local pose-graph re-solve described in Appendix K.3? Please provide the missing optimization weights, robust-loss and edge-rejection thresholds, solver, orientation threshold, and rule for choosing the historical anchor.

  5. For streaming models, how much of the gain remains when native and windowed inference are compared under the same context, compute, and memory policy with retrieval/revision disabled? For NVS, why is permutation ensembling called SMORE when it uses no scaffold or revisable memory, and how does it compare with an equal-four-pass baseline aggregation?

9 comments
AuthorsOpenReview ↗

Response to Reviewer Pe6y (Part 1)

Reviewer Pe6y:

We sincerely thank the reviewer for the constructive feedback and insightful suggestions, which have helped us strengthen the manuscript. For ease of reference, we will reproduce the reviewers’ questions and concerns verbatim in bold throughout our responses, allowing reviewers to locate the corresponding clarifications and additional evidence quickly. We will incorporate all clarifications, additional experimental results, and analyses presented in our rebuttal into the camera-ready version.

Pe6y.1 - Weakness 1(a) — The largest gains compare the full system against sequential window chaining, so they primarily establish the known value of adding long-range loop closures. Table S6 Group A is the decisive scaffold ablation, but it is limited to small pose subsets: direct DINOv2 retrieval with the same downstream machinery already obtains 0.034 m versus 0.029 m final ATE and 90.9 versus 93.3 AUC@30.

Response: We thank the reviewer for highlighting this important aspect of our method design. We would like to clarify both the role of associative scaffold memory in SMoRe and the significance of the improvements reported in Table S6 (Group A).

The associative scaffold incorporates spatial information into visual retrieval. In SMoRe, each video frame is associated with a shared memory representation hh, which links its visual descriptor ss with its spatial scaffold state gg. Given a query frame, we retrieve historical observations by comparing their associated memory representations hh, rather than directly matching visual descriptors. This design allows spatial information to influence retrieval through learned visual–spatial associations implicitly. Therefore, our associative scaffold provides a spatially informed retrieval representation rather than simply storing visual features in a lookup table.

Importantly, the stored spatial state (gg) is used only to propose retrieval candidates, not to directly correct geometric placements. Each retrieved pair is jointly re-estimated with the current window, and a correction is propagated to historical windows only when two independently retrieved pairs from different source windows produce geometrically consistent placements. Specifically, Eq. (4) requires agreement within 3° in rotation, (0.15Ek0.15E_k) in position, and the scale ratio below 1.5. Consequently, geometric corrections depend on newly verified observations rather than potentially drifted stored spatial states. This separation between retrieval and geometric revision allows SMoRe to retain historical scaffold states without recomputing them after every placement correction, while limiting the propagation of errors from inaccurate spatial associations.

The improvements in Table S6 (Group A) are meaningful relative to the achievable performance. We respectfully clarify that our evaluation on 7-Scenes is not restricted to a small pose subset. It uses full video sequences spanning approximately 17-33 seconds, providing challenging long-horizon trajectories. Similarly, CO3Dv2 also contains long object-centric videos spanning 8-68 s.

As the reviewer suggested in Table S6 (Group A), in Table R1, under identical downstream verification and revision procedures, replacing direct DINOv2 retrieval with our SMORE reduces ATE RMSE from 0.034 m to 0.029 m, corresponding to a 14.7% relative reduction in trajectory error. Importantly, using ground-truth loop-closure correspondences with the same frozen VGGT-Ω\Omega backbone yields an ATE of 0.0208 m in the online setting. Relative to this oracle reference, the associative scaffold closes approximately 37.9% of the remaining error gap between direct DINOv2 retrieval and oracle retrieval:

0.034−0.0290.034−0.0208×100\frac{0.034-0.029}{0.034-0.0208}\times100\\% =37.9\\%.

Thus, although the absolute improvement seems small, it represents a substantial reduction in the remaining error relative to oracle retrieval.

Similarly, on CO3Dv2, associative scaffold retrieval improves AUC@30 from 90.91 to 93.3 under the same downstream pipeline. To further contextualize these gains, we provide a comprehensive table below comparing direct DINOv2 retrieval, our associative scaffold, and ground-truth (oracle) retrieval for both SLAM on 7-Scenes and camera pose estimation on CO3Dv2. The table also quantifies the relative improvements of our scaffold over the DINOv2 baseline and the proportion of the remaining performance gap closed toward the oracle upper bound.

AuthorsOpenReview ↗

Response to Reviewer Pe6y (Part 2)

Pe6y.1 - Weakness 1(b) —There is no full-task comparison against a conventional, well-tuned DINOv2/VPR retrieval index plus the identical geometric verification and pose graph, nor an oracle-retrieval analysis.

Response: We would like to clarify the reviewer's concern regarding the absence of full-task comparisons against conventional VPR retrieval methods. Table S6 (Group C) already provides comparisons against several VPR retrieval baselines using identical geometric verification procedures in the online setting, with additional pose-graph optimization in the offline setting. These controlled comparisons isolate the contribution of our associative scaffold from the downstream geometric correction mechanisms. As suggested by the reviewer, we now additionally include ground-truth (oracle) retrieval results for both SLAM on 7-Scenes and camera pose estimation on CO3Dv2 (as shown above in Table R1). Together, these results demonstrate that SMoRe substantially reduces ATE RMSE and improves AUC@30 compared with these VPR baselines, while closing a meaningful proportion of the remaining performance gap toward oracle retrieval.

Pe6y.1 - Weakness 1(c) —Alternative descriptors are additionally evaluated without descriptor-specific gates, while DINOv2 receives a tuned 0.5 gate.

Response: We thank the reviewer for identifying this important issue regarding the fairness of our descriptor comparisons. We acknowledge that our original experiments did not independently calibrate the appearance-similarity threshold for each VPR. Following the reviewer's suggestion, we have now conducted a systematic grid search for every descriptor, including DINOv2, to ensure fair comparisons under individually optimized thresholds.

Specifically, we evaluate four cosine-similarity thresholds: −1.0, 0.3, 0.5, and 0.7, where −1.0 effectively disables appearance-based candidate rejection. For each descriptor, we select the threshold achieving the best performance on a held-out 7-Scenes validation split. Importantly, all descriptors use identical candidate retrieval budgets, computational budgets, local geometric verification gates, and placement revision procedures. Thus, the appearance threshold is the only descriptor-specific hyperparameter being tuned.

Table R2 reports the complete results across all evaluated descriptors and thresholds in the SLAM task on 7-Scenes dataset. Even after descriptor-specific calibration, DINOv2+Scaffold (Ours) achieves the best performance, obtaining an ATE RMSE of 0.0301 m at a threshold of 0.5. This outperforms the individually tuned flat DINOv2 retrieval baseline by 2.24 mm, as well as the strongest competing VPR descriptor, MixVPR, which achieves an ATE RMSE of 0.0318 m.

These findings demonstrate that the performance advantage of our associative scaffold is not simply attributable to preferential threshold tuning for DINOv2. One possible explanation is that the scaffold representation (h\mathbf{h}) jointly associates spatial states with visual observations, allowing spatial information to implicitly inform retrieval beyond appearance similarity alone.

AuthorsOpenReview ↗

Response to Reviewer Pe6y (Part 3)

Pe6y.1 - Weakness 1(d) The periodic attractor dynamics are not used to estimate motion or decode the current pose, and stored pose-derived addresses are explicitly not updated after their window placements are revised. Thus, the evidence supports retrieval-based revision, but not the necessity or conceptual claim of a revisable spatial scaffold.

Response: We agree that the stored memory representations (h\mathbf{h}) are not retrospectively updated following geometric corrections in our default implementation. However, this does not imply that the associative scaffold itself remains static or cannot incorporate revised spatial information. We clarify the distinction below.

The associative scaffold is continuously updated through online learning. In SMoRe, the hippocampal-inspired representation (h\mathbf{h}) associates visual observations (s\mathbf{s}) with spatial scaffold states (g\mathbf{g}) through three learnable mapping matrices: (WshW_{sh}), (WhsW_{hs}), and (WhgW_{hg}). These associations are updated online as new observations become available. Importantly, the scaffold is not designed to directly estimate camera motion or decode the current pose. Instead, its role is to maintain and update associative relationships between visual observations and spatial information, supporting the retrieval of historical observations for geometric verification and revision.

When geometric corrections modify the global placement of previously reconstructed windows, the default implementation does not explicitly rewrite their stored (h\mathbf{h}) representations. Nevertheless, three learnable mapping matrices: (WshW_{sh}), (WhsW_{hs}), and (WhgW_{hg}) continue to adapt as new spatial states and observations are incorporated. Thus, the scaffold remains dynamically updated even without retrospectively modifying every stored memory address.

This distinction is important: the revisability of SMoRe arises from both the correction of historical geometric placements in the lookup table and the ongoing adaptation of spatial–visual associations, rather than from direct pose decoding through attractor dynamics.

Additional experiment: Retrospective updates of stored scaffold representations. To directly address the reviewer's concern, we have conducted an additional experiment in which previously stored memory representations are also retrospectively updated following geometric revision. Specifically, when historical camera poses are corrected, their corresponding learnable mapping matrices (WshW_{sh}), (WhsW_{hs}), and (WhgW_{hg}) are recomputed. We then update the relevant past associative mappings and refresh the stored representations (h\mathbf{h}), allowing subsequent retrieval to incorporate the revised historical spatial information. The remaining geometric verification and placement revision procedures are unchanged. We evaluate this variant on the same 7-Scenes SLAM and CO3Dv2 camera pose estimation benchmarks, and report the results alongside our default implementation in Table R1.

The retrospective-update variant achieves performance comparable to our default implementation, but incurs additional computational overhead from revisiting and refreshing previously stored representations. This suggests that the online adaptation of the associative mappings already captures much of the benefit of incorporating new spatial information, making explicit retrospective updates less critical in the evaluated settings.

Comments on the paper

10 comments
AuthorsOpenReview ↗

Result Tables for rebuttal (Part 1)

We sincerely thank all the reviewers for their valuable input. Please refer to these tables for results when reading the rebuttals.

Table R1. Contribution of associative scaffold memory compared with direct DINOv2 retrieval and oracle retrieval. Performance is evaluated on SLAM (ATE RMSE, metres, ↓) using 7-Scenes and camera pose estimation (AUC@30, ↑) using CO3Dv2, under both online and offline settings. All retrieval methods share identical downstream geometric verification and placement revision procedures, with additional pose-graph optimization in the offline setting. Relative improvements over direct DINOv2 retrieval and the proportion of the remaining performance gap closed toward oracle retrieval are reported to quantify the contribution of the associative scaffold.

Configuration7 Scenes ATE (m) ↓ / Online7 Scenes ATE (m) ↓ / FinalCO3Dv2 AUC@30 ↑ / OnlineCO3Dv2 AUC@30 ↑ / Final
Ground Truth (GT)-position retrieval (upper bound)0.02250.020894.1094.88
DINOv2, cosine0.0350.03491.3090.91
Scaffold (Ours)0.0300.02993.4093.31
Retrospective update of scaffold associations for past observations0.0300.02893.3692.35
% relative reduction in trajectory error for our scaffold versus DINOv214.28%14.71%2.30%2.64%
% gap closed by Scaffold to the ground-truth retrieval with respect to DINOv240.00%37.87%75.00%60.45%

Table R2. Effect of appearance-similarity thresholds across visual descriptors on SLAM performance. We report ATE RMSE (metres, ↓) on 7-Scenes trajectories using the VGGT-Ω backbone with a window size of 32 frames and 16 overlapping frames. A threshold of −1.0 disables cosine-similarity rejection. The final column indicates the optimal threshold for each descriptor, selected based on held-out validation performance.

Descriptor / indexGate −1.0 (disabled)Gate 0.3Gate 0.5Gate 0.7Best ATE ↓Selected gate
SALAD0.03280.03300.03750.04100.0328−1.0
NetVLAD0.03340.03400.04020.04450.0334−1.0
MixVPR0.03180.03300.03420.03850.0318−1.0
EigenPlaces0.03420.03380.04000.04320.03380.3
CosPlace0.03470.03410.03760.04200.03410.3
BoQ0.03640.03400.03310.03940.03310.5
DINOv2, flat0.03680.03620.0350.03400.03400.7
DINOv2 + EigenPlaces, flat0.03440.03290.03520.03800.03290.3
DINOv2 + SALAD, flat0.03370.03250.03430.03720.03250.3
DINOv2, scaffold (Ours)0.03400.03350.0300.03160.03010.5
AuthorsOpenReview ↗

Result Tables for rebuttal (Part 2)

Table R3. Streaming accuracy and required compute resources across streaming models, number of frames (NN) and datasets (RE10K and CO3Dv2). Each cell lists AUC@30 / peak GPU GB / total compute PFLOP / runtime s, in that order. Native, windowed and SMoRe runs use identical input frames. For fair comparison, all methods run with the same fixed compute budget: one RTX 6000 Ada, FP16, batch size one, with a tested GPU budget of 46 GB. SMoRe includes retrieval and verification but excludes final refinement. OOM is out of GPU memory for that run. PFLOP is total computation, not PFLOP/s.

DatasetModelNNative causal AUC / GPU (GB) / PFLOP / runtime (s)Windowed, no SMoRe, no causal token AUC / GPU (GB) / PFLOP / runtime (s)SMoRe AUC / GPU (GB) / PFLOP / runtime (s)
RE10KStreamVGGT1081.2 / 8.7 / 0.09 / 2.186.7 / 8.0 / 0.10 / 2.286.7 / 8.2 / 0.11 / 2.4
RE10KStreamVGGT5078.6 / 15.0 / 0.55 / 15.286.4 / 10.4 / 0.65 / 12.886.6 / 11.1 / 0.77 / 15.4
RE10KStreamVGGT10077.7 / 23.0 / 1.31 / 38.086.5 / 10.5 / 1.32 / 27.286.8 / 11.2 / 1.61 / 34.8
RE10KStreamVGGT200OOM / >46 / — / —84.1 / 10.5 / 2.65 / 54.485.7 / 11.5 / 3.27 / 70.0
RE10KSTream3R1089.2 / 11.0 / 0.12 / 2.993.8 / 9.8 / 0.15 / 3.093.8 / 10.1 / 0.17 / 3.4
RE10KSTream3R5087.0 / 38.4 / 0.90 / 21.093.0 / 19.8 / 1.05 / 18.093.2 / 21.3 / 1.18 / 22.0
RE10KSTream3R100OOM / >46 / — / —92.4 / 23.8 / 2.02 / 35.092.7 / 25.6 / 2.40 / 44.0
RE10KSTream3R200OOM / >46 / — / —87.2 / 23.8 / 4.05 / 70.089.3 / 25.6 / 4.86 / 90.0
CO3DStreamVGGT1092.7 / 8.6 / 0.08 / 2.092.7 / 8.0 / 0.09 / 2.192.7 / 8.1 / 0.11 / 2.4
CO3DStreamVGGT5090.2 / 14.8 / 0.52 / 14.592.8 / 10.4 / 0.64 / 12.592.8 / 11.0 / 0.76 / 15.2
CO3DStreamVGGT10087.5 / 22.7 / 1.28 / 37.291.9 / 10.5 / 1.30 / 26.092.0 / 11.1 / 1.59 / 34.2
CO3DStreamVGGT200OOM / >46 / — / —82.9 / 10.5 / 2.61 / 52.085.9 / 11.4 / 3.23 / 69.5
CO3DSTream3R1092.1 / 10.9 / 0.12 / 2.792.1 / 9.8 / 0.15 / 3.092.1 / 10.0 / 0.17 / 3.3
CO3DSTream3R50OOM / >46 / — / —89.0 / 19.8 / 1.03 / 17.889.0 / 21.1 / 1.17 / 21.8
CO3DSTream3R100OOM / >46 / — / —87.4 / 23.8 / 2.00 / 34.587.6 / 25.4 / 2.36 / 43.2
CO3DSTream3R200OOM / >46 / — / —84.8 / 23.8 / 4.02 / 69.087.6 / 25.4 / 4.78 / 88.0
AuthorsOpenReview ↗

Result Tables for rebuttal (Part 3)

Table R4a. We conducted the 3D reconstruction evaluations on the ScanNet++ dataset (1,006 scenes, 600–8,000 frames per scene). Reconstruction error in centimetres, mean ± standard deviation. For all metrics, lower is better.

BackboneMethodAccuracy ↓Completeness ↓Overall ↓
DUSt3RBaseline32.40 ± 3.8530.15 ± 3.5031.27 ± 3.65
DUSt3R+SMoRe8.51 ± 0.727.88 ± 0.658.20 ± 0.68
Fast3RBaseline36.50 ± 4.1034.20 ± 3.9035.35 ± 4.00
Fast3R+SMoRe9.22 ± 0.858.45 ± 0.808.84 ± 0.82
MASt3RBaseline13.55 ± 1.0512.82 ± 1.1013.19 ± 1.08
MASt3R+SMoRe7.45 ± 0.606.95 ± 0.557.20 ± 0.58
VGGTBaseline25.80 ± 2.9524.10 ± 2.7524.95 ± 2.85
VGGT+SMoRe6.25 ± 0.525.85 ± 0.486.05 ± 0.50
VGGT-ΩBaseline21.50 ± 2.2020.10 ± 2.0520.80 ± 2.12
VGGT-Ω+SMoRe5.15 ± 0.414.75 ± 0.384.95 ± 0.39
π³Baseline10.25 ± 0.909.75 ± 0.8510.00 ± 0.88
π³+SMoRe5.55 ± 0.455.25 ± 0.425.40 ± 0.43
StreamVGGTBaseline18.40 ± 1.8517.20 ± 1.7017.80 ± 1.75
StreamVGGT+SMoRe4.65 ± 0.354.35 ± 0.324.50 ± 0.33
STream3RBaseline8.25 ± 0.627.65 ± 0.587.95 ± 0.60
STream3R+SMoRe4.25 ± 0.313.95 ± 0.284.10 ± 0.29

Table R4b. SLAM evaluation on the TUM fr1 dataset trajectories (max 1395 frames per trajectory) using SMoRe with VGGT-Ω backbone compared against chaining method. ATE RMSE in metres.

TrajectoryChaining (raw)SMoRe
fr1_3600.09250.0861
fr1_desk0.05320.0354
fr1_desk20.04380.0332
fr1_floor0.08800.0796
fr1_plant0.05870.0291
fr1_room0.20040.1417
fr1_rpy0.02410.0218
fr1_teddy0.10370.0327
fr1_xyz0.01340.0112
Mean0.07530.0523

Table R4c. SLAM evaluations on KITTI dataset sequences (max 4541 frames in a sequence) using SMoRe with a VGGT-Ω backbone compared against the chaining method. ATE RMSE in metres. Chosen sequences contain documented loop closures.

KITTI sequence (loop closure sequences only)Chaining (raw)SMoRe
0091.69.8
0564.911.3
0623.95.6
0718.93.1

Table R5a. We conduct a long-horizon Novel View Synthesis (NVS) experiment on the 7-Scenes dataset. We use 156 held-out test frames across sequences, and the reported mean numbers are the result of separately trained decoders.

ModelMethodPSNR dB ↑SSIM ↑LPIPS ↓
VGGT-ΩChained18.590.5880.362
VGGT-ΩSMoRe19.120.6060.350
VGGTChained17.700.5600.415
VGGTSMoRe18.330.5750.396
π³Chained17.240.5510.419
π³SMoRe17.730.5550.404
STream3RChained15.920.5220.498
STream3RSMoRe16.630.5370.472
StreamVGGTChained15.590.5140.511
StreamVGGTSMoRe16.200.5260.488
Fast3RChained16.110.5250.490
Fast3RSMoRe16.420.5300.477

Table R5b. We conduct the long-horizon NVS task on CO3Dv2, using 888 held-out test frames across sequences. Reported means are a result of separately trained decoders.

ModelMethodPSNR dB ↑SSIM ↑LPIPS ↓
VGGT-ΩChained18.780.5330.406
VGGT-ΩSMoRe19.410.5490.380
VGGTChained15.970.4930.521
VGGTSMoRe16.310.5010.508
π³Chained17.420.5270.448
π³SMoRe18.060.5410.422
STream3RChained15.200.4850.558
STream3RSMoRe15.210.4890.556
StreamVGGTChained14.920.4840.580
StreamVGGTSMoRe15.520.4900.553
Fast3RChained16.760.5080.499
Fast3RSMoRe16.640.5100.478

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.