- Rating
- 2
- Confidence
- 2(−2)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes Mamba-VGGT, which introduces Mamba-based adapters into a frozen VGGT backbone. It uses SS2D to integrate current visual features with cross-window memory for long-sequence camera pose estimation and 3D reconstruction. The method trains only approximately 7.9M additional parameters and employs state decay and periodic resets to manage long-sequence inference.
Critical strengths
-
The method is relatively simple and enables cross-window feature propagation without retraining the VGGT backbone.
-
The paper reports competitive results on KITTI-360 and VBR. In particular, on 2,000-frame KITTI-360 sequences, it reduces mean ATE by 42.3% compared with LoGeR.
-
The paper includes ablations comparing SS2D with an MLP and enabling versus disabling cross-window memory, which help assess the contributions of individual components.
Critical weaknesses
-
Limited methodological originality and insufficient differentiation from existing approaches. The proposed method mainly combines existing techniques, including SS2D, residual adapters, and fixed-capacity memory, within VGGT. Scal3R has already explored cross-window memory in VGGT, while LoGeR combines local attention with persistent memory. Furthermore, LongStream has introduced Periodic Cache Refresh to address long-term cache contamination and drift. Although the implementations differ, periodically refreshing historical information is not a new idea, and state decay is also a standard technique. Therefore, the paper lacks a clearly differentiated core technical contribution.
-
Missing comparisons with more recent strong baselines and insufficient evaluation coverage. The paper lacks comparisons with more recent long-sequence reconstruction methods, including VGGT-Long, Scal3R, LongStream, LingBot-Map, and HorizonStream. Since these methods directly address long-term memory, drift mitigation, and efficient reconstruction, the current experiments are insufficient to establish the competitiveness of the proposed approach. Furthermore, the long-sequence KITTI-360 evaluation covers only sampled versions of two trajectories, while only one VBR sequence is evaluated using all available frames. This limited evaluation coverage does not adequately support broad claims about long-sequence performance and makes it difficult to rule out potential biases arising from dataset or sequence selection.
-
Insufficient evidence for the contribution of the proposed memory mechanism. Table 6 shows that disabling cross-window memory preserves most of the improvement over VGGT-W16. On KITTI-360 test_2/test_3, enabling memory only reduces ATE from 43.53/26.21 to 41.15/22.52, suggesting that much of the improvement may come from the adapter itself rather than long-term memory. Moreover, SS2D is compared only against a pointwise MLP, which does not establish its advantage over other spatial feature-mixing modules. The additional 7.9M parameters are also not supported by a sufficiently clear analysis of the trade-offs among accuracy, parameter count, and computational cost.
Questions for authors
Q1. What fundamentally distinguishes the proposed SS2D-based state update mechanism from existing cross-window memory approaches such as Scal3R and LoGeR? In particular, what new technical contributions do state decay and periodic resets offer beyond existing strategies such as those used in LongStream?
Q2. Why are more recent methods, including VGGT-Long, Scal3R, LongStream, LingBot-Map, and HorizonStream, missing from the comparisons? Could the authors provide direct comparisons under consistent datasets, input lengths, and evaluation protocols?
Q3. Why does the long-sequence KITTI-360 evaluation include only sampled trajectories from test_2/test_3, while most VBR evaluations also use temporally sampled sequences? Could the authors evaluate more complete, continuous long videos to rule out potential biases introduced by dataset or sequence selection?
Q4. Table 6 shows that most of the performance improvement remains even when cross-window memory is disabled. How do the authors explain this observation? Can they provide stronger evidence that long-term memory, rather than local feature adaptation, is responsible for the improvements? Additionally, what concrete accuracy–efficiency advantages are achieved by introducing approximately 7.9M additional parameters?
18 comments
Response to Q1: Mechanistic differences from Scal3R, LoGeR, and LongStream, and the role of decay/reset
Thank you for raising this question. The main distinction from Scal3R and LoGeR concerns how historical information is represented, read, and updated, and how these operations are coupled with current-feature adaptation inside a fully frozen geometric network. We do not claim cross-window memory, state decay, or periodic resets themselves as new concepts.
First, we establish a cross-window feature-information path through a selective state-space model, rather than storing history by optimizing fast weights at inference time. Scal3R’s GCM/AMUs and LoGeR’s TTT modules encode context in updatable fast weights. Their updates can be summarized as
\qquad Y=f_{W^{+}}(Q).$$ Scal3R additionally aggregates gradients from different chunks to share global context. LoGeR combines TTT with sliding-window attention, using fast weights and explicit local features to handle information at different temporal scales. [SC, §4.2; LG, §3–4] Our method instead freezes the VGGT backbone, bidirectional within-window attention, and original prediction heads, training only the added adaptation modules. Each injection layer maintains a fixed-capacity two-dimensional activation state M and jointly processes it with the current window’s patch features X: $$[Y^M,Y^X]=\Phi_{\mathrm{SS2D}}([M_{m-1},X_m]),$$ $$\widetilde X_m=X_m+W_XY^X,\qquad M_m=\alpha_m M_{m-1} +W_M\operatorname{Pool}_{\mathrm{new}}(Y^M).$$ The current feature residual and the content written for the next window thus come from **the same joint state–feature representation**. Historical information directly conditions current geometry predictions as well as memory updates, while current observations enter the next window through write aggregation over newly appearing frames only. Compared with geometrically stitching the outputs of independently processed windows, this creates a feature-level communication path that frozen VGGT windows otherwise lack. The role of Mamba/SS2D here is **content-dependent spatial information mixing**. Its selective scan coefficients are determined by the current joint input, allowing historical state and current patch content to interact through four directional scans rather than independent pointwise mappings. Two states must be distinguished: the internal SS2D scan state h exists only within a single frame-grid scan, whereas the external activation state M is propagated across video windows. The design therefore goes beyond simply applying a Mamba model to a video sequence: it uses selective spatial mixing for joint feature adaptation and cross-window reading/writing within a frozen geometric network. **Second, this information path does not require online optimization.** Let B denote batch size, S the number of frames per window, Q the number of patches per frame, D the internal SS2D width, N the scan-state dimension, L the number of adaptation layers, and K the external-memory channel count. The four-direction scan has computational cost and persistent-state storage $$\mathcal C_{\mathrm{scan}}=O(LBSQDN), \qquad \mathcal M_{\mathrm{persistent}}=O(LBQK).$$ The computation expression describes the scan component. Input/output projections and residual/write mappings also incur costs, but these likewise scale linearly with the number of patches when channel dimensions are fixed. At a fixed resolution and window configuration, persistent state does not grow with total video length. “Fixed capacity” refers only to cross-window state, not to all intermediate tensors, outputs, or total system storage. TTT can also scale linearly with sequence length, so we do not claim a universal advantage in asymptotic complexity order. The distinction is that TTT computes gradients of a fast-weight objective, updates the weights, and applies the updated query mapping; Scal3R additionally aggregates updates across chunks. Our method performs a forward joint transformation with fixed parameters, without these test-time optimization steps. **This structure offers the potential to reduce update overhead, while actual efficiency must be established through measurements of concrete implementations.** Fused TTT kernels, parallel execution, and model width all affect practical cost; the formulas alone do not establish that SS2D is always faster.Response to Q1: Mechanistic differences from Scal3R, LoGeR, and LongStream, and the role of decay/reset
The current resource table provides system-level evidence consistent with this mechanism:
Table 1a: Resources for native execution, frontend replacement, and additive injection
| Configuration | Sampled device-memory peak (GiB) | Effective FPS |
|---|---|---|
| VGGT-W16 | 7.308 | 20.27 |
| Ours | 7.308 | 19.75 |
| Native Scal3R | 8.138 | 1.68 |
| Scal3R + our frontend replacement | 12.634 | 7.36 |
| Scal3R + additive SS2D, TTT retained | 8.161 | 1.58 |
Tables 1a and 8 use the same resource evaluation: standalone Ours and VGGT-W16 achieve 19.75 and 20.27 FPS with the same 7.308 GiB memory peak. Relative to native Scal3R at 1.68 FPS, frontend replacement achieves 7.36 FPS (approximately 4.38×), and additive SS2D with TTT retained achieves 1.58 FPS (approximately 5.95% lower). Their respective memory changes from native Scal3R are 8.138→12.634 GiB and 8.138→8.161 GiB. Table 8 gives the measurement settings.
Third, VGGT-Long frontend integration, Scal3R frontend replacement, and additive SS2D with TTT retained jointly evaluate transferability. Below, each Scal3R integration mode is compared with native Scal3R.
Table 1b: Pose results for native Scal3R, frontend replacement, and additive injection with TTT retained
| KITTI-360 pose setting | Native Scal3R | Frontend replacement | TTT retained + SS2D |
|---|---|---|---|
| ATE, 300 frames, paper sampling protocol | 26.55 | 21.75 | 21.60 |
| ATE, 300 consecutive frames | 7.48 | 7.10 | 6.55 |
| 1000 frames, paper sampling protocol, test_2 | 59.11 | 43.93 | 49.50 |
| 1000 frames, paper sampling protocol, test_3 | 22.95 | 11.72 | 17.62 |
| 2000 frames, paper sampling protocol, test_2 | 78.14 | 54.09 | 63.74 |
| 2000 frames, paper sampling protocol, test_3 | 29.52 | 13.25 | 25.65 |
Relative to native Scal3R, frontend replacement reduces ATE from 29.52 to 13.25 on 2000-frame test_3 using the paper’s sampling protocol, approximately 55.12%; additive SS2D with TTT retained reduces ATE from 29.52 to 25.65 in the same setting, approximately 13.11%. On 300 frames using the paper’s sampling protocol, the two modes respectively reduce ATE from 26.55 to 21.75 and 21.60; on 300 consecutive frames, they respectively reduce ATE from 7.48 to 7.10 and 6.55. On the complete Spagna route, frontend replacement reduces ATE from 28.2137 to 24.0932 (approximately 14.60%), and additive SS2D with TTT retained reduces it from 28.2137 to 18.7146 (approximately 33.67%).
These results show that learned adaptation can be reused through a prediction frontend and can improve pose estimation while native TTT remains active. VGGT-Long frontend integration reduces mean ATE from 36.51 to 26.94 across the four long-sequence cases using the paper’s sampling protocol, approximately 26.2%, further supporting reuse across hosts.
Finally, state decay and periodic resets are not the main technical novelty of this paper. We adopt these established approaches to controlling historical state to limit accumulation and contamination in the proposed activation memory. LongStream likewise uses Periodic Cache Refresh to control long-horizon degradation of the sink/KV cache, and LoGeR also employs periodic state resets. [LS, §3.5; LG, §5.1] Our Leak5 attenuates old information in the external activation state, while reset300 clears state and restarts the corresponding window-execution schedule. They are deployment-stabilization choices, rather than newly introduced memory principles. Our main contribution remains coupling historical reading, current feature residuals, and new-state writing through selective spatial mixing inside a fully frozen VGGT, and evaluating its role through frontend replacement and additive integration that preserves the host’s memory mechanism.
We thank the reviewer for the specific questions on technical positioning, comparison coverage, and the contribution of memory. Spatial-mixing and memory-state ablations retain their explicitly stated training branches, checkpoints, and state settings. All other Ours, host-integration, and resource results use the paper’s main model A2 (64-frame continuation step1800), including the additional full-route evaluations.