acceptodds
Under review as a conference paper at ICLR 2027

Mamba-VGGT: Explicit Spatial State for Cross-Window Video Geometry

Abstract

Recovering camera motion and scene geometry from long videos requires strong multi-view reasoning and effective reuse of past observations. Pretrained geometric transformers provide powerful multi-view priors, but their global attention becomes increasingly costly as the input grows. Windowed inference limits this cost while interrupting feature-level communication across windows, which output-level geometric stitching cannot restore. We introduce Mamba-VGGT, a recurrent adapter that extends a frozen Visual Geometry Grounded Transformer with fixed-capacity cross-window memory. A Mamba-based two-dimensional selective scan (SS2D) jointly processes the incoming state and current patch tokens, producing coupled residual feature updates and memory writes. This design retains pretrained within-window attention while allowing subsequent windows to use historical context. State decay and periodic segment resets control state accumulation during long-sequence inference. On KITTI-360 sequences with 2,000 sampled frames, Mamba-VGGT reduces mean absolute trajectory error by 42.3% and mean LiDAR-supported reconstruction distance by 56.3% relative to LoGeR with sliding-window attention under the evaluated inference configuration. It also achieves lower trajectory error on VBR inputs of 8,000 and 10,000 frames under matched window and reset schedules. Mixer-replacement experiments favor SS2D over a parameter-matched pointwise MLP under the tested training setup, while state-propagation ablations show that carried context further improves trajectory estimation. Together, these results support joint spatial adaptation and cross-window memory as an effective approach to extending frozen geometric models for long-sequence camera estimation and reconstruction.

Primary area:
applications to robotics, autonomy, planning
Keywords:
3D Reconstruction, Feed forward Reconstruction, VGGT

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xycbOpenReview ↗
Rating
2
Confidence
2(−2)
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Mamba-VGGT, which introduces Mamba-based adapters into a frozen VGGT backbone. It uses SS2D to integrate current visual features with cross-window memory for long-sequence camera pose estimation and 3D reconstruction. The method trains only approximately 7.9M additional parameters and employs state decay and periodic resets to manage long-sequence inference.

Critical strengths
  1. The method is relatively simple and enables cross-window feature propagation without retraining the VGGT backbone.

  2. The paper reports competitive results on KITTI-360 and VBR. In particular, on 2,000-frame KITTI-360 sequences, it reduces mean ATE by 42.3% compared with LoGeR.

  3. The paper includes ablations comparing SS2D with an MLP and enabling versus disabling cross-window memory, which help assess the contributions of individual components.

Critical weaknesses
  1. Limited methodological originality and insufficient differentiation from existing approaches. The proposed method mainly combines existing techniques, including SS2D, residual adapters, and fixed-capacity memory, within VGGT. Scal3R has already explored cross-window memory in VGGT, while LoGeR combines local attention with persistent memory. Furthermore, LongStream has introduced Periodic Cache Refresh to address long-term cache contamination and drift. Although the implementations differ, periodically refreshing historical information is not a new idea, and state decay is also a standard technique. Therefore, the paper lacks a clearly differentiated core technical contribution.

  2. Missing comparisons with more recent strong baselines and insufficient evaluation coverage. The paper lacks comparisons with more recent long-sequence reconstruction methods, including VGGT-Long, Scal3R, LongStream, LingBot-Map, and HorizonStream. Since these methods directly address long-term memory, drift mitigation, and efficient reconstruction, the current experiments are insufficient to establish the competitiveness of the proposed approach. Furthermore, the long-sequence KITTI-360 evaluation covers only sampled versions of two trajectories, while only one VBR sequence is evaluated using all available frames. This limited evaluation coverage does not adequately support broad claims about long-sequence performance and makes it difficult to rule out potential biases arising from dataset or sequence selection.

  3. Insufficient evidence for the contribution of the proposed memory mechanism. Table 6 shows that disabling cross-window memory preserves most of the improvement over VGGT-W16. On KITTI-360 test_2/test_3, enabling memory only reduces ATE from 43.53/26.21 to 41.15/22.52, suggesting that much of the improvement may come from the adapter itself rather than long-term memory. Moreover, SS2D is compared only against a pointwise MLP, which does not establish its advantage over other spatial feature-mixing modules. The additional 7.9M parameters are also not supported by a sufficiently clear analysis of the trade-offs among accuracy, parameter count, and computational cost.

Questions for authors

Q1. What fundamentally distinguishes the proposed SS2D-based state update mechanism from existing cross-window memory approaches such as Scal3R and LoGeR? In particular, what new technical contributions do state decay and periodic resets offer beyond existing strategies such as those used in LongStream?

Q2. Why are more recent methods, including VGGT-Long, Scal3R, LongStream, LingBot-Map, and HorizonStream, missing from the comparisons? Could the authors provide direct comparisons under consistent datasets, input lengths, and evaluation protocols?

Q3. Why does the long-sequence KITTI-360 evaluation include only sampled trajectories from test_2/test_3, while most VBR evaluations also use temporally sampled sequences? Could the authors evaluate more complete, continuous long videos to rule out potential biases introduced by dataset or sequence selection?

Q4. Table 6 shows that most of the performance improvement remains even when cross-window memory is disabled. How do the authors explain this observation? Can they provide stronger evidence that long-term memory, rather than local feature adaptation, is responsible for the improvements? Additionally, what concrete accuracy–efficiency advantages are achieved by introducing approximately 7.9M additional parameters?

18 comments

We thank the reviewer for the specific questions on technical positioning, comparison coverage, and the contribution of memory. Spatial-mixing and memory-state ablations retain their explicitly stated training branches, checkpoints, and state settings. All other Ours, host-integration, and resource results use the paper’s main model A2 (64-frame continuation step1800), including the additional full-route evaluations.

AuthorsOpenReview ↗

Response to Q1: Mechanistic differences from Scal3R, LoGeR, and LongStream, and the role of decay/reset

Thank you for raising this question. The main distinction from Scal3R and LoGeR concerns how historical information is represented, read, and updated, and how these operations are coupled with current-feature adaptation inside a fully frozen geometric network. We do not claim cross-window memory, state decay, or periodic resets themselves as new concepts.

First, we establish a cross-window feature-information path through a selective state-space model, rather than storing history by optimizing fast weights at inference time. Scal3R’s GCM/AMUs and LoGeR’s TTT modules encode context in updatable fast weights. Their updates can be summarized as

\qquad Y=f_{W^{+}}(Q).$$ Scal3R additionally aggregates gradients from different chunks to share global context. LoGeR combines TTT with sliding-window attention, using fast weights and explicit local features to handle information at different temporal scales. [SC, §4.2; LG, §3–4] Our method instead freezes the VGGT backbone, bidirectional within-window attention, and original prediction heads, training only the added adaptation modules. Each injection layer maintains a fixed-capacity two-dimensional activation state M and jointly processes it with the current window’s patch features X: $$[Y^M,Y^X]=\Phi_{\mathrm{SS2D}}([M_{m-1},X_m]),$$ $$\widetilde X_m=X_m+W_XY^X,\qquad M_m=\alpha_m M_{m-1} +W_M\operatorname{Pool}_{\mathrm{new}}(Y^M).$$ The current feature residual and the content written for the next window thus come from **the same joint state–feature representation**. Historical information directly conditions current geometry predictions as well as memory updates, while current observations enter the next window through write aggregation over newly appearing frames only. Compared with geometrically stitching the outputs of independently processed windows, this creates a feature-level communication path that frozen VGGT windows otherwise lack. The role of Mamba/SS2D here is **content-dependent spatial information mixing**. Its selective scan coefficients are determined by the current joint input, allowing historical state and current patch content to interact through four directional scans rather than independent pointwise mappings. Two states must be distinguished: the internal SS2D scan state h exists only within a single frame-grid scan, whereas the external activation state M is propagated across video windows. The design therefore goes beyond simply applying a Mamba model to a video sequence: it uses selective spatial mixing for joint feature adaptation and cross-window reading/writing within a frozen geometric network. **Second, this information path does not require online optimization.** Let B denote batch size, S the number of frames per window, Q the number of patches per frame, D the internal SS2D width, N the scan-state dimension, L the number of adaptation layers, and K the external-memory channel count. The four-direction scan has computational cost and persistent-state storage $$\mathcal C_{\mathrm{scan}}=O(LBSQDN), \qquad \mathcal M_{\mathrm{persistent}}=O(LBQK).$$ The computation expression describes the scan component. Input/output projections and residual/write mappings also incur costs, but these likewise scale linearly with the number of patches when channel dimensions are fixed. At a fixed resolution and window configuration, persistent state does not grow with total video length. “Fixed capacity” refers only to cross-window state, not to all intermediate tensors, outputs, or total system storage. TTT can also scale linearly with sequence length, so we do not claim a universal advantage in asymptotic complexity order. The distinction is that TTT computes gradients of a fast-weight objective, updates the weights, and applies the updated query mapping; Scal3R additionally aggregates updates across chunks. Our method performs a forward joint transformation with fixed parameters, without these test-time optimization steps. **This structure offers the potential to reduce update overhead, while actual efficiency must be established through measurements of concrete implementations.** Fused TTT kernels, parallel execution, and model width all affect practical cost; the formulas alone do not establish that SS2D is always faster.
AuthorsOpenReview ↗

Response to Q1: Mechanistic differences from Scal3R, LoGeR, and LongStream, and the role of decay/reset

The current resource table provides system-level evidence consistent with this mechanism:

Table 1a: Resources for native execution, frontend replacement, and additive injection

ConfigurationSampled device-memory peak (GiB)Effective FPS
VGGT-W167.30820.27
Ours7.30819.75
Native Scal3R8.1381.68
Scal3R + our frontend replacement12.6347.36
Scal3R + additive SS2D, TTT retained8.1611.58

Tables 1a and 8 use the same resource evaluation: standalone Ours and VGGT-W16 achieve 19.75 and 20.27 FPS with the same 7.308 GiB memory peak. Relative to native Scal3R at 1.68 FPS, frontend replacement achieves 7.36 FPS (approximately 4.38×), and additive SS2D with TTT retained achieves 1.58 FPS (approximately 5.95% lower). Their respective memory changes from native Scal3R are 8.138→12.634 GiB and 8.138→8.161 GiB. Table 8 gives the measurement settings.

Third, VGGT-Long frontend integration, Scal3R frontend replacement, and additive SS2D with TTT retained jointly evaluate transferability. Below, each Scal3R integration mode is compared with native Scal3R.

Table 1b: Pose results for native Scal3R, frontend replacement, and additive injection with TTT retained

KITTI-360 pose settingNative Scal3RFrontend replacementTTT retained + SS2D
ATE, 300 frames, paper sampling protocol26.5521.7521.60
ATE, 300 consecutive frames7.487.106.55
1000 frames, paper sampling protocol, test_259.1143.9349.50
1000 frames, paper sampling protocol, test_322.9511.7217.62
2000 frames, paper sampling protocol, test_278.1454.0963.74
2000 frames, paper sampling protocol, test_329.5213.2525.65

Relative to native Scal3R, frontend replacement reduces ATE from 29.52 to 13.25 on 2000-frame test_3 using the paper’s sampling protocol, approximately 55.12%; additive SS2D with TTT retained reduces ATE from 29.52 to 25.65 in the same setting, approximately 13.11%. On 300 frames using the paper’s sampling protocol, the two modes respectively reduce ATE from 26.55 to 21.75 and 21.60; on 300 consecutive frames, they respectively reduce ATE from 7.48 to 7.10 and 6.55. On the complete Spagna route, frontend replacement reduces ATE from 28.2137 to 24.0932 (approximately 14.60%), and additive SS2D with TTT retained reduces it from 28.2137 to 18.7146 (approximately 33.67%).

These results show that learned adaptation can be reused through a prediction frontend and can improve pose estimation while native TTT remains active. VGGT-Long frontend integration reduces mean ATE from 36.51 to 26.94 across the four long-sequence cases using the paper’s sampling protocol, approximately 26.2%, further supporting reuse across hosts.

Finally, state decay and periodic resets are not the main technical novelty of this paper. We adopt these established approaches to controlling historical state to limit accumulation and contamination in the proposed activation memory. LongStream likewise uses Periodic Cache Refresh to control long-horizon degradation of the sink/KV cache, and LoGeR also employs periodic state resets. [LS, §3.5; LG, §5.1] Our Leak5 attenuates old information in the external activation state, while reset300 clears state and restarts the corresponding window-execution schedule. They are deployment-stabilization choices, rather than newly introduced memory principles. Our main contribution remains coupling historical reading, current feature residuals, and new-state writing through selective spatial mixing inside a fully frozen VGGT, and evaluating its role through frontend replacement and additive integration that preserves the host’s memory mechanism.

Reviewer_BXK3OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Summary:

This paper deals with reconstructing 3D point maps and recovering camera pose from long video sequences. The global attention of prior approaches present a difficult challenge owing to its increasing computational cost with greater number of frames. Windowed solutions prevent this cost explosion but suffer from disruption of cross-window feature-level communication. Towards this end, the authors introduce Mamba-VGGT, which extends a frozen fixed capacity cross-windowed VGGT model. A Mamba-based 2D selective scan (SS2D) jointly process the incoming state and current patch tokens to produce coupled residual feature updates and memory writes. State decay and periodic segment resets control state accumulation during long sequence inference. Empirical evaluations are conducted on multiple datasets of long sequence videos.

Critical strengths

The key strengths of the paper are as follows:

  1. The authors identify an important problem - that of feed-forward 3D reconstruction and camera-pose recovery of long videos.
  2. The proposed method allows for joint processing of historical information and current patch tokens.
  3. The approach only needs training of a small set of additional parameters while the strong pre-trained backbone model is frozen.
  4. The model performance is impressive, especially on long videos.
Critical weaknesses

The principal weaknesses of the proposed method are as follows:

  1. The external memory is represented as a two-dimensional patch grid, with each slot corresponding to a position in the image feature grid. However, these slots do not have a one-to-one world-coordinate assignment.
  2. The persistent state is directly updated without explicitly considering whether there are substantial camera motions, occlusions, or scene transitions. This can lead to possible noisy updates of the state.
  3. Comparison with several established 3D reconstruction methods have not been shown [a, b].

References:

[a] Elflein, S., Li, R., Agostinho, S., Gojcic, Z., Leal-Taixé, L., Zhou, Q. and Osep, A., 2026. VGG-T 3^ 3: Offline Feed-Forward 3D Reconstruction at Scale. arXiv preprint arXiv:2602.23361.

[b] Deng, K., Ti, Z., Xu, J., Yang, J. and Xie, J., 2026, June. Vggt-long: Chunk it, loop it, align it–pushing vggt’s limits on kilometer-scale long rgb sequences. In 2026 IEEE International Conference on Robotics and Automation (ICRA) (pp. 5446-5453). IEEE.

Questions for authors

Questions/Comments:

  1. The released LoGeR checkpoint was developed with a W32/O3 configuration, but the main comparison evaluates it using W16/O4. Can this variant of the baseline be tried?
  2. Is SS2D necessary, or is conventional spatial mixing (through a standard attention) sufficient?
  3. Is the model flexible enough to permit the use of newer backbones, like VGGT - Omega?
5 comments

We thank the reviewer for recognizing the value of long-video geometric reconstruction, the joint processing of historical information and current features, and lightweight adaptation of a frozen pretrained backbone. We address the three questions on LoGeR’s native window configuration, spatial-attention controls, and compatibility with newer backbones below.

AuthorsOpenReview ↗

Response to Q1: LoGeR’s native W32/O3 configuration and additional chunk-based systems

Thank you for pointing out the LoGeR window-configuration issue. In the KITTI-360 300-, 1000-, and 2000-frame experiments and the NRGBD depth experiment, we mistakenly placed W32/O3 results in entries labeled W16/O4. In the revised manuscript, we will replace the W32/O3 results mistakenly entered under W16/O4 with the matched-window W16/O4 results, and update the affected tables, figure captions, text, and derived percentages in the abstract. W32/O3 results will be retained as separately labeled native-window comparisons.

Inputs follow the paper’s sampling protocol, and scoring matches the corresponding paper experiments. VGGT-Long and Scal3R use W16/O4 windows, SE(3) inter-chunk alignment, and loop closure; integrated configurations retain the host backends. VGGT-Long integrates the A2 frontend by replacing its chunk predictor. Scal3R uses two integration modes: “+ our frontend” replaces its prediction network and adapts reference-frame ordering and camera encoding; “+ additive SS2D” retains the native backbone, prediction heads, TTT/GCM/AMU, and loop-closure backend, transferring only the four A2 step1800 SS2D memory modules without retraining. In the additive configuration, native TTT remains active at layers 14/17/20/23, and SS2D runs after native attention at layers 3/10/16/22; SS2D Leak5/reset300 does not clear native TTT. Real-input checks confirmed bitwise reproduction of native cameras and intrinsics with SS2D disabled, and complete native TTT calls with nonzero SS2D states and residuals when enabled. PGO concurrency is 1 for additive integration, 4 for KITTI quality baselines, and 1 for VBR quality baselines. LoGeR retains native image preprocessing, and Omega retains its official balanced preprocessing. For NRGBD, Ours/VGGT-W16 follow the paper’s settings, whereas VGGT-Long/Scal3R/Omega use SE(3) stitching; each sequence receives one median depth-scale fit, and nominal300 contains 304 frames. A2 and its integrations use Leak5 and reset300 for long sequences, without an additional segment reset in the 300-frame setting. Methods retain their respective preprocessing, so resolution and compute budgets are not identical.

Table 1: Medium-length geometry at 300 frames using the paper’s sampling protocol

MethodKITTI-360 ATE↓ (m)NRGBD AbsRel↓KITTI-360 CD↓ (m)
Ours (A2, 16/4)21.220.03105.657
VGGT-W1627.560.03707.807
LoGeR + SWA (32/3)49.760.021510.090
LoGeR + SWA (16/4)57.290.019810.112
VGGT-Long (W16/O4, SE(3))26.740.21387.530
VGGT-Long + our frontend (W16/O4, SE(3))22.930.21415.945
Scal3R (W16/O4, SE(3))26.550.22426.676
Scal3R + our frontend (W16/O4, SE(3))21.750.22426.846
Scal3R + additive SS2D (TTT retained, W16/O4, SE(3))21.600.22055.144
VGGT-Omega (16/4)18.820.19854.303
VGGT-Omega + our mixer/memory (16/4)18.450.19374.133

On the 300-frame inputs following the paper’s sampling protocol, frontend integration reduces VGGT-Long ATE from 26.74 to 22.93 and CD from 7.530 to 5.945. For Scal3R, frontend replacement reduces ATE from 26.55 to 21.75, approximately 18.08%; retaining TTT and adding SS2D reduces ATE from 26.55 to 21.60, NRGBD AbsRel from 0.2242 to 0.2205, and CD from 6.676 to 5.144, approximately 18.64%, 1.65%, and 22.95%, respectively. These results show pose improvements across different baselines, with depth and reconstruction gains in some integration configurations, supporting reuse of the learned adaptation.

Table 2: KITTI-360 ATE on 1000/2000-frame inputs using the paper’s sampling protocol

Method1000f test_21000f test_32000f test_22000f test_3
Ours (A2, 16/4)41.1522.5246.6518.68
VGGT-W1651.3448.6748.5753.01
LoGeR + SWA (32/3)81.1658.1568.6244.64
LoGeR + SWA (16/4)131.0255.27104.70101.00
VGGT-Long (W16/O4, SE(3))48.7824.9351.9220.42
VGGT-Long + our frontend (W16/O4, SE(3))38.0215.0344.0010.71
Scal3R (W16/O4, SE(3))59.1122.9578.1429.52
Scal3R + our frontend (W16/O4, SE(3))43.9311.7254.0913.25
Scal3R + additive SS2D (TTT retained, W16/O4, SE(3))49.5017.6263.7425.65
VGGT-Omega (16/4)55.3123.1561.1620.64
VGGT-Omega + our mixer/memory (16/4)46.7024.8558.5820.20

Across the four 1000/2000-frame cases following the paper’s sampling protocol, VGGT-Long frontend integration, Scal3R frontend replacement, and additive SS2D with Scal3R TTT retained each reduce ATE relative to their respective baselines. The equally weighted mean ATE changes are 36.51→26.94 for VGGT-Long, 47.43→30.75 for Scal3R frontend replacement, and 47.43→39.13 for additive Scal3R, corresponding to reductions of approximately 26.2%, 35.2%, and 17.5%. This supports improved long-sequence pose estimation across hosts and integration modes.

AuthorsOpenReview ↗

Response to Q2: A standard spatial-attention control with a similar parameter count

To add a control with spatial mixing, we introduce a standard softmax spatial-attention alternative and compare it with the matched recurrent SS2D branch already reported in the paper. The table also retains the paper’s MLP, zero-history intervention, and main-model results.

Table 3: Paper ablations and spatial-mixing controls under the same training protocol

Branch / mixerMixer parameters per tapcheckpointTraining stateInference statetest_2test_3
Paper: pointwise MLP924,984step 1400OnOn47.6845.78
Paper: SS2D matched recurrent924,224Validation-selected (step not reported)OnOn36.1618.24
Paper: SS2D zero-history continuation924,224step 400ZeroZero41.0222.05
Paper: SS2D zero-history continuation924,224step 400ZeroOn40.7914.75
Paper: SS2D main model924,224step 1800OnZero43.5326.21
Paper: SS2D main model924,224step 1800OnOn41.1522.52
New: low-rank global attention r128 (checkpoint evaluation)971,584step 2000 (validation best = final step)OnOn41.1625.89

Standard attention provides spatial mixing. We concatenate the 1024-dimensional patch tokens and 64-dimensional historical state along channels. Input projection and a 3×3 depthwise convolution produce a 256-dimensional content representation, flattened into L=H×W positions per frame. We compute attention using 128-dimensional Q/K/V:

Q=ZWQ,K=ZWK,V=ZWV,U=softmax⁡(QK⊤/128)VWO.Q=ZW_Q,\quad K=ZW_K,\quad V=ZW_V,\qquad U=\operatorname{softmax}(QK^\top/\sqrt{128})VW_O.

Normalization, gating, and output projection then restore the joint representation, which is split into token-residual and state-write branches. The same external recurrent state and new-frame-only write rule are retained. Each position can aggregate all spatial positions, making this a recurrent adapter with global spatial interaction rather than a pointwise MLP. “Low-rank” refers only to the projection dimension; it does not imply that the softmax matrix is necessarily low-rank or that complexity becomes linear.

The results favor SS2D over this attention alternative. Attention and SS2D have 971,584 and 924,224 parameters per tap, respectively, a difference of approximately 5.12%. The paper’s matched recurrent SS2D result, 36.16/18.24, is lower than attention’s 41.16/25.89. Using the displayed values, ATE decreases by approximately 12.15%/29.55%, and mean ATE over the two sequences decreases by approximately 18.87%. Thus, the observed advantage does not arise solely from comparing against a module without spatial mixing.

Training settings and result provenance. The SS2D comparator is the matched recurrent branch already reported in the paper’s Table 6 and Appendix D.2 (36.16/18.24). This branch is reported separately from main model A2. The added attention branch uses the same ablation training settings as the paper: it starts from the 32-frame step3400 checkpoint, inherits compatible backbone, prediction-head, token-residual, and state-write parameters, reinitializes the mixer being compared, and uses the paper’s 64-frame continuation setup. The data split, 16/4 recurrent unrolling, losses, optimization rules, 2000-update limit, and independent-validation selection rule are retained. The paper does not specify the selected checkpoint step for matched recurrent SS2D, so we do not assign one here; attention uses validation-selected step2000.

Image-grid slots, state writing, and historical accumulation. The external state is not a map with fixed world-coordinate assignments. The same grid slot need not correspond to the same physical point across frames; it provides latent feature context on the image grid. SS2D mixes spatial positions through four-directional content-dependent scans and modulates the mixed output through internal content gating. Specifically, Eqs. (12) and (15) in Appendix D.1 use G=SiLU(Z) to gate the normalized scan output elementwise before projection into a joint representation that is split into feature-residual and state-write content. This internal content gate modulates the joint state–feature representation. The writer in Eqs. (6)–(7) aggregates only newly appearing frames and updates state once at each window boundary; Leak5 and reset300 further limit historical accumulation. The state-magnitude, local-geometry, and full-trajectory diagnostics in Appendices D.4–D.5 support stabilization in the evaluated scenes. Local spatial adaptation is an important source of gains; with the same trained weights and deployment schedule, cross-window state provides additional benefits on the evaluated routes, with a magnitude that varies by scene. The current mechanism does not explicitly implement geometric reprojection or visibility association to resolve state contamination under large motion, occlusion, or scene transitions.

Reviewer_vnLwOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper adds a recurrent cross-window adapter to a frozen VGGT backbone. SS2D jointly processes historical state and current features, producing feature residuals and state increments. Memory is updated at window boundaries, extending VGGT to long-sequence geometry inference.

Critical strengths
  1. Only about 0.63% of the parameters are trained, while the pretrained backbone and prediction heads remain frozen. Reusing the existing geometric capability for longer sequences has clear practical value.
  2. Coupling feature adaptation with memory writing through SS2D is a natural design. The long-sequence results and state ablations support the improvement and show that historical state is useful.
Critical weaknesses
  1. Sliding windows, recurrent memory, and SS2D are established ingredients. I see this mainly as an effective adaptation of VGGT. The added value of SS2D over a simpler spatial recurrent adapter needs a clearer account.
  2. Comparison with recent streaming methods is incomplete. LingBot-Map and ABot-Recon are neither compared nor specifically discussed. This leaves the accuracy and cost of the lightweight adaptation less clear relative to recent streaming systems.
Questions for authors
  1. Adding one representative recent streaming method would be useful, such as LingBot-Map. Keep the input and evaluation settings consistent, report accuracy, GPU memory, and runtime, and specify reset and loop-closure settings.
  2. The MLP control in Table 6 is useful, but a pointwise MLP also lacks spatial mixing. My question is whether SS2D still offers a clear advantage over a simple spatial recurrent module with a similar parameter and training budget.
  3. I would also like to see how errors evolve over a full long trajectory, and what late-sequence degradation looks like when it occurs.
3 comments
AuthorsOpenReview ↗

Response to Q1: LoGeR’s native W32/O3 configuration and additional chunk-based systems

We thank the reviewer for recognizing the value of long-video geometric reconstruction, the joint processing of historical information and current features, and lightweight adaptation of a frozen pretrained backbone. We address the three questions on LoGeR’s native window configuration, spatial-attention controls, and compatibility with newer backbones below.

Q1: LoGeR’s native W32/O3 configuration and additional chunk-based systems

In the KITTI-360 300-, 1000-, and 2000-frame experiments and the NRGBD depth experiment, we mistakenly placed W32/O3 results in entries labeled W16/O4. We will correct this in the revised manuscript. Ours outperforms both LoGeR configurations in these KITTI pose and 300-frame reconstruction comparisons, while LoGeR remains better on NRGBD depth. The correction therefore does not change the relative ranking between Ours and LoGeR on these metrics.

The measurement procedures below follow the corresponding tables in the paper. VGGT-Long and Scal3R use 16/4 windows, loop closure, and SE(3) inter-chunk alignment. Their integrated variants retain the host backend and reuse the trained frontend without host-specific retraining. LoGeR retains native image preprocessing, and Omega retains its official balanced preprocessing. For NRGBD depth, Ours/VGGT-W16 use the paper’s scale-aware robust_weak stitching, whereas VGGT-Long/Scal3R/Omega use SE(3) stitching. Each sequence receives one median depth-scale fit, without per-frame scale correction. The nominal300 depth input actually contains 304 frames. The standalone and integrated models retain Leak5/reset300 for long sequences; the 300-frame setting has no additional segment reset. With these preprocessing and stitching settings, the methods should not be interpreted as having identical resolution or compute budgets.

Table 1: Medium-length geometry at 300 frames

MethodKITTI-360 ATE↓ (m)NRGBD AbsRel↓KITTI-360 CD↓ (m)
Ours (A2, 16/4)21.220.03105.657
VGGT-W1627.560.03707.807
LoGeR + SWA (32/3)49.760.021510.090
LoGeR + SWA (16/4)57.290.019810.112
VGGT-Long (16/4)26.740.21387.530
VGGT-Long + our frontend (16/4)22.930.21415.945
Scal3R (16/4)26.550.22426.676
Scal3R + our frontend (16/4)21.750.22426.846
VGGT-Omega (16/4)18.820.19854.303
VGGT-Omega + our mixer/memory (16/4)18.450.19374.133

CD on sampled inputs uses fixed common valid LiDAR-ray support, disjoint 20% fitting and 80% scoring subsets, a shared 0.1m voxel grid, and no ICP. It measures reconstruction error on sparse observed support, rather than complete dense-scene coverage.

Table 2: KITTI-360 ATE on the paper’s sampled 1000/2000-frame inputs

Method1000f test_21000f test_32000f test_22000f test_3
Ours (A2, 16/4)41.1522.5246.6518.68
VGGT-W1651.3448.6748.5753.01
LoGeR + SWA (32/3)81.1658.1568.6244.64
LoGeR + SWA (16/4)131.0255.27104.70101.00
VGGT-Long (16/4)48.7824.9351.9220.42
VGGT-Long + our frontend (16/4)38.0215.0344.0010.71
Scal3R (16/4)59.1122.9578.1429.52
Scal3R + our frontend (16/4)43.9311.7254.0913.25
VGGT-Omega (16/4)55.3123.1561.1620.64
VGGT-Omega + our mixer/memory (16/4)46.7024.8558.5820.20

With 16/4 windows, integrating our frontend reduces 300-frame ATE from 26.74 to 22.93 for VGGT-Long and from 26.55 to 21.75 for Scal3R. Both hosts also improve in all four 1000/2000-frame ATE comparisons. This supports transfer gains within the same host and window setting, although not every geometric metric improves: Scal3R’s 300-frame CD changes from 6.676→6.846, and VGGT-Long’s NRGBD AbsRel changes from 0.2138→0.2141. Omega also has lower 300-frame ATE/CD than standalone Ours, so we do not claim the best performance across all systems.

AuthorsOpenReview ↗

Response to Q2: A standard spatial-attention control with a similar parameter count

To add a control with spatial mixing, we retrain SS2D and introduce a standard softmax spatial-attention alternative using the same starting checkpoint, initialization policy, and training protocol as the paper’s ablations. The table retains the paper’s MLP, zero-history intervention, and main-model results alongside the new controls.

Table 3: Paper ablations and spatial-mixing controls under the same training protocol

Branch / mixerMixer parameters per tapcheckpointTraining stateInference statetest_2test_3
Paper: pointwise MLP924,984step 1400OnOn47.6845.78
Paper: SS2D zero-history continuation924,224step 400ZeroZero41.0222.05
Paper: SS2D zero-history continuation924,224step 400ZeroOn40.7914.75
Paper: SS2D main model924,224step 1800OnZero43.5326.21
Paper: SS2D main model924,224step 1800OnOn41.1522.52
New: retrained SS2D (paper’s ablation protocol)924,224step 200 (validation best)OnOn38.9121.38
New: low-rank global attention r128 (checkpoint evaluation)971,584step 2000 (validation best = final step)OnOn41.1625.89

Standard attention provides spatial mixing. We concatenate the 1024-dimensional patch tokens and 64-dimensional historical state along channels. Input projection and a 3×3 depthwise convolution produce a 256-dimensional content representation, flattened into L=H×W positions per frame. We compute attention using 128-dimensional Q/K/V:

Q=ZWQ,K=ZWK,V=ZWV,U=softmax⁡(QK⊤/128)VWO.Q=ZW_Q,\quad K=ZW_K,\quad V=ZW_V,\qquad U=\operatorname{softmax}(QK^\top/\sqrt{128})VW_O.

Normalization, gating, and output projection then restore the joint representation, which is split into token-residual and state-write branches. The same external recurrent state and new-frame-only write rule are retained. Each position can aggregate all spatial positions, making this a recurrent adapter with global spatial interaction rather than a pointwise MLP. “Low-rank” refers only to the projection dimension; it does not imply that the softmax matrix is necessarily low-rank or that complexity becomes linear.

The results favor SS2D over this attention alternative. Attention and SS2D have 971,584 and 924,224 parameters per tap, respectively, a difference of approximately 5.12%. The new SS2D result, 38.91/21.38, is lower than attention’s 41.16/25.89. Using the displayed values, ATE decreases by approximately 5.47%/17.42%, and mean ATE over the two sequences decreases by approximately 10.08%. Thus, the observed advantage does not arise solely from comparing against a module without spatial mixing.

Consistency of training settings. The retrained SS2D and spatial-attention control follow the starting checkpoint, initialization policy, and training protocol of the paper’s ablations. Both start from the 32-frame step3400 checkpoint, inherit compatible backbone, prediction-head, token-residual, and state-write parameters, reinitialize the mixer being compared, and use the paper’s 64-frame continuation setup. The data split, 16/4 recurrent unrolling, losses, optimization rules, 2000-update limit, and independent-validation selection rule remain the same. SS2D and the spatial-attention control are retrained under these settings; the selected checkpoints are listed in the table.

Mechanistically, SS2D jointly processes state and features through four-directional content-dependent scans, producing residuals and write content from a shared representation. At fixed internal dimensions, its scan cost grows linearly with the number of patches, whereas spatial interaction in this explicit QKᵀ attention grows quadratically. Both alternatives already have fixed-capacity external memory, so fixed memory is not an advantage unique to SS2D. This provides evidence about quality and asymptotic computational structure, rather than a measured speed advantage over attention.

AuthorsOpenReview ↗

Response to Q3:Compatibility and transfer gains with VGGT-Omega and other hosts

We added a VGGT-Omega transfer experiment without retraining: the four SS2D memory blocks and three role tokens from main model A2 are transferred into the Omega aggregator, while retaining Omega’s backbone and prediction heads. We use 16/4, Leak5, reset300 for long sequences, and SE(3) stitching; the 304-frame NRGBD input is a single segment without an additional reset. After integration, 300-frame ATE/CD decrease from 18.82/4.303 to 18.45/4.133, and NRGBD AbsRel decreases from 0.1985 to 0.1937. Three of the four 1000/2000-frame ATE cases improve: their equally weighted mean decreases from 40.07 to 37.58, approximately 6.2%. The 1000-frame test_3 case still increases from 23.15 to 24.85, indicating variation in the gains. Overall, this initial transfer demonstrates benefits across backbones. We believe modest adapter fine-tuning on the target backbone may further improve transfer performance and stability, which we plan to investigate.

Frontend reuse produces larger pose gains in VGGT-Long and Scal3R. In Table 1, VGGT-Long’s mean 300-frame ATE decreases from 26.74 to 22.93, approximately 14.2%, and Scal3R’s decreases from 26.55 to 21.75, approximately 18.1%. In the four sampled 1000/2000-frame cases in Table 2, both hosts improve in every case. VGGT-Long’s equally weighted mean ATE decreases from 36.51 to 26.94, approximately 26.2%, and Scal3R’s from 47.43 to 30.75, approximately 35.2%. Improvements are particularly large on 2000-frame test_3: 20.42→10.71 and 29.52→13.25, approximately 47.6% and 55.1%, respectively. These results show that the learned adaptation can improve long-sequence pose estimation while retaining the host’s chunking, alignment, and loop-closure backend.

Integration replaces VGGT-Long’s chunk predictor and Scal3R’s TTT-fine-tuned prediction network while retaining their backends. These two experiments reuse the complete trained prediction frontend. The Omega experiment instead transfers mixer/memory modules while retaining the target backbone and prediction heads. Together, these integration modes demonstrate reusability of the learned adaptation. Further adapter fine-tuning, injection-stage adjustments, or feature-interface adaptation may improve transfer gains across backbones.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.