acceptodds
Under review as a conference paper at ICLR 2027

Fast and stable long-context 3D reconstruction using Recon-Aware Memory Bank

Abstract

Feed-forward 3D reconstruction models such as VGGT provide high-quality geometry and camera estimation without per-scene optimization, but scaling them to long visual sequences remains challenging. Dense models incur quadratic attention computation as context grows, while streaming variants still accumulate increasingly large historical context. We argue that efficient context reduction should be both reconstruction-aware and architecture-decoupled: selected frames should preserve downstream reconstruction quality without relying on backbone-specific cache or attention structures. To this end, we introduce Reconstruction-Aware Memory Bank (RAMB), a plug-in framework that uses frame-associated scene tokens as a common representation across VGGT-style backbones. RAMB combines a lightweight Rank Model trained with downstream reconstruction outcomes to identify high-utility frames with scene-token similarity to retain complementary observations and remove redundant context. We evaluate RAMB on VGGT- and StreamVGGT across ScanNet, ETH3D, and KITTI-VO. The selected frames are highly informative: reconstructing from only about 3% of a 400-frame sequence, VGGT- matches or exceeds full-sequence reconstruction accuracy (ACC). For chunked inference, RAMB carries informative context across chunks, enabling 540-frame reconstruction on a single 24 GB GPU, where full-sequence inference requires over 60 GB, with comparable quality. For StreamVGGT, RAMB bounds the historical context to 20.9 of 240 frames on average while maintaining comparable quality, and reduces forward latency and peak memory by .

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Reconstruction, Reinforce Learning, Memory Bank

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_DQKpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RAMB is a pluggable frame memory buffer designed for long-sequence reconstruction tasks with VGGT-family models. It operates under a specific frame selection rule: it reserves the very first frame of the sequence, plus one frame picked out by a 20M-parameter ranking model trained via the REINFORCE algorithm, and then iteratively adds the least similar frames from the register-token space into the buffer, until all remaining frames have a similarity score above the 0.97 threshold.

This module serves two purposes: it curates input frames for the VGGT-Ω model, and caps the length of the KV cache history for StreamVGGT. According to the paper, this approach achieves higher accuracy than full-sequence inference while using only around 3% of the total frames on the ScanNet, ETH3D and KITTI-VO benchmarks. It also delivers 2.25× faster inference on 540-frame sequences under a 24 GB memory constraint.

Critical strengths

S1. The core practical advantage of RAMB is that it caps memory usage without any modifications to the backbone architecture. StreamVGGT keeps memory consumption steady at 12–15 GB even for sequences up to 560 frames. By comparison, the native model already uses roughly 60 GB of memory at just 240 frames, and runs into out-of-memory failures beyond that length (Fig. 3).

S2. Figure 4 demonstrates that the accuracy of VGGT-family models does not scale monotonically with input sequence length. For VGGT-Ω in particular, it reaches an accuracy of around 0.107 with 60 uniformly sampled frames, and 0.122 when all 400 frames are fed into the model. This means frame selection can deliver accuracy gains, rather than only functioning as a memory-saving measure.

Critical weaknesses

W1. The paper’s central claim—that using roughly 3% of total frames can match or outperform full-sequence inference—relies on an evaluation protocol that is biased toward the selected frame subset. In the frame-selection-only mode, the memory bank is derived from a complete 400-frame forward pass (as outlined in Algorithm 1), meaning this mode delivers no computational savings at all. Since the ~13-frame point cloud is evaluated against the entire scene, accuracy (ACC) improves while completeness (COMP) degrades (on ScanNet: ACC shifts from 0.126 to 0.082, and COMP shifts from 0.141 to 0.181), leaving the average of the two metrics nearly unchanged. Furthermore, Figure 6 reports a full-sequence ACC of 0.0944 rather than the 0.126 referenced earlier, which calls into question the actual accuracy gain of the method.

W2. The work does not include ablation studies to isolate the contributions of its individual components. With only vanilla backbones and uniform sampling serving as performance baselines, the reported 2.25× speedup could stem entirely from the chunking mechanism alone. The Rank Model only contributes one additional frame during inference, yet removing it causes the ACC error to jump from 0.082 to 0.187 (Table 4)—a result worse than both uniform sampling and full-sequence inference. Such a pronounced performance impact requires further justification. Additionally, the similarity-based selection step is essentially farthest-point keyframe selection, a technique already used by MUSt3R (CVPR 2025); however, neither MUSt3R nor StreamVGGT is cited in the paper.

Questions for authors

Q1 (clarification). Do the training sequences used for the Rank Model have no overlap with the test splits of the ScanNet and KITTI-VO benchmarks, and how many test sequences are included in the evaluation?

Q2 (clarification). How are chunk outputs fused across separate inference passes that may have scale discrepancies, and are the scores reported in Table 3 calculated on these per-pass chunk outputs, or on a final re-inference run over the assembled memory bank (as illustrated in Fig. 1)?

Q3 (clarification). Figure 3 suggests a memory footprint of roughly 49 GB for 540-frame sequences, whereas the abstract states memory consumption exceeds 60 GB under the same setting. Which of these two values is accurate?

8 comments

We thank the reviewer for the detailed and constructive assessment. In particular, we appreciate the concerns regarding the evaluation protocol, the contributions of individual components, and the consistency of the reported results. We address these issues below and provide additional experiments and clarifications. Below, we respond to the Critical Weaknesses W1 and W2, and to Questions Q1, Q2, and Q3, respectively.

W1.1 - Memory bank frames vs. full frames

We agree that evaluating a reconstruction from substantially fewer frames requires careful interpretation, particularly because accuracy (ACC) and completeness (COMP) capture different aspects of reconstruction quality.

To separate the effects of selecting a frame subset from those of reducing the backbone's inference context, we conduct an additional controlled experiment using VGGT-Ω on ScanNet.

We consider four configurations:

  1. Full sequence (full_seq_res): VGGT-Ω jointly processes all 400 frames, and the resulting point cloud is evaluated.
  2. Selected frames from full-context predictions (full_mb_res): We extract the predictions corresponding to RAMB-selected frames from the original full-sequence forward pass. No additional reconstruction inference is performed.
  3. Uniformly sampled frames (unif_seq_res): We independently reconstruct an equally sized set of uniformly sampled frames using short-context inference.
  4. RAMB-selected frames (mb_seq_res): We independently reconstruct the RAMB-selected frames using short-context inference.

The last three configurations use the same average number of frames. In particular, full_mb_res and mb_seq_res contain predictions for identical frame indices, allowing us to examine the effect of inference context while holding frame selection fixed.

Table 1: Reconstruction quality with different frame sources and context lengths (VGGT-Ω, ScanNet)

Method# framesACC ↓COMP ↓NC ↑
full_seq_res4000.1260.1410.801
full_mb_res12.60.1040.1960.820
unif_seq_res12.60.1730.3340.786
mb_seq_res12.60.0820.1810.822

First, we acknowledge the effect of evaluating a selected subset. Extracting only the RAMB-selected predictions from the original full-sequence reconstruction improves ACC from 0.126 to 0.104, even without changing the backbone inference context. This confirms that frame-subset selection itself contributes to the observed accuracy improvement. It also reduces scene coverage, as reflected by the increase in COMP from 0.141 to 0.196.

Second, reduced-context inference provides an additional improvement. When we reconstruct exactly the same selected frames independently, ACC further decreases from 0.104 to 0.082. Because the selected frame indices remain unchanged, this improvement cannot be attributed to evaluating a different subset of frames. Rather, it demonstrates that the predictions for those frames improve under the shorter inference context.

Third, the identity of the selected frames matters. Under the same short-context setting and frame budget, RAMB substantially outperforms uniform sampling across the reported reconstruction metrics. This supports the contribution of reconstruction-aware frame selection beyond simply reducing the input length.

These results distinguish two sources of improvement: selecting informative observations and obtaining improved predictions for those observations under a reduced context. We do not claim that the reduced-frame reconstruction provides the same geometric coverage as full-sequence reconstruction.

W1.2 - Computational savings of the selection-only mode

We agree that the selection-only mode yields no computational savings, since it requires a full-sequence forward pass. However, the selection-only mode is an analysis setting intended to isolate frame-selection quality, not to demonstrate deployment efficiency rather than a deployment mode.

The efficiency results in the paper instead concern two other settings:

  • Chunked VGGT-Ω inference: The sequence is processed incrementally, with a compact memory bank retaining informative observations from earlier chunks.
  • Streaming StreamVGGT inference: Each incoming frame is processed with a bounded historical KV cache, avoiding the unbounded growth of retained context.

In these settings, scene-token representations are obtained during the backbone inference already required to process incoming observations, rather than through a separate full-sequence selection pass.

The additional controlled experiment in Table 1 evaluates the quality of frame selection. The efficiency benefits must be assessed separately through the complete chunked or streaming inference pipelines.

We have revised the abstract and introduction to more clearly distinguish the approximately 30-fold frame reduction observed in selection-only evaluation from the end-to-end latency and memory improvements.

We also recognize that the chunked VGGT-Ω pipeline must be described precisely in terms of which frames receive final reconstruction predictions, as clarified in our response to Q2.

W2.1 - Contributions of individual components

We agree that the contributions of the Rank Model (RM) and the Minimum-Similarity Frame Selection strategy should be evaluated separately.

Our memory-bank construction combines two complementary mechanisms:

  • Reconstruction-aware initialization: The RM selects a leading frame based on its predicted reconstruction utility.
  • Coverage-aware expansion: Minimum-Similarity Frame Selection adds observations that are least redundant with the frames already retained.

The leading frame is important not only as an additional observation but also because it determines the initial state of the subsequent greedy selection procedure. Consequently, changing the leading frame can alter the entire selected subset

In Section 4.3.1 of the original paper, we demonstrate the importance of the leading frame selected by RM. It provides reconstruction-aware initialization, while the similarity-based strategy explicitly considers redundancy relative to the current memory bank. Here, we show the contribution of the Minimum-Similarity Frame Selection strategy and further explain the pronounced performance changes observed.

Since RM is designed to select frames that are highly complementary to the anchor frame, it tends to assign similar scores to frames with similar content. Consequently, directly selecting frames based on RM's ranking results in a set of highly redundant frames. Such a selection fails to adequately represent the entire sequence and, in turn, cannot achieve reconstruction quality comparable to that obtained using the full sequence, as shown in Table 2. The Minimum-Similarity Frame Selection strategy is specifically designed to address this limitation by promoting diversity among the selected frames.

In this experiment, RM selection uses the same number of input frames for each sequence as Minimum-Similarity Selection. The number of frames retained by Minimum-Similarity Selection is determined by the stopping criteria.

Table 2: Reconstruction quality when frames are selected directly by the RM ranking, without Minimum-Similarity Frame Selection

MethodACC ↓COMP ↓NC ↑
VGGT-Ω0.1850.4880.735
StreamVGGT0.2020.4310.654

The similarity-only variant performs substantially worse than full RAMB, supporting the importance of reconstruction-aware initialization. Likewise, the ranking-only variant yields substantially worse completeness, indicating that selecting frames solely from a fixed RM ranking does not provide sufficient scene coverage under the evaluated setting.

RM selection and Minimum-Similarity Frame Selection serve different purposes. Without either of them, the performance degrades dramatically. Conceptually, our method can be viewed as processing a rope: the anchor frame fixes one end, while RM selection pulls the rope from the other. Minimum-Similarity Selection then divides the rope into semantically balanced segments, ensuring diversity across the selected frames.

This distinction also explains why the RM's full-ranking training objective does not imply that the final memory bank should be constructed directly from its top-ranked frames. Ranking-prefix supervision provides reconstruction-quality signals across multiple context sizes, whereas memory-bank expansion requires decisions conditioned on previously selected observations.

Together, the ablations support the complementary roles of the two components. We have clarified this distinction and provide the experimental details necessary for interpreting the comparisons fairly in the new version of the paper.

Reviewer_rzkDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RAMB is a memory bank framework for long-context 3D reconstruction. It uses a model trained to identify which frames are most useful for reconstruction, then selects additional frames that provide new visual information. The method is evaluated on both VGGT-Ω and StreamVGGT, showing that it can substantially reduce the number of retained frames while maintaining competitive reconstruction quality and improving inference efficiency.

Critical strengths
  1. The paper makes an interesting observation that using more frames does not always lead to better reconstruction quality. The authors build on this observation with a frame selection strategy that combines reconstruction-aware ranking and similarity-based selection, which is a clever and well-motivated design.
  2. The results are promising on both VGGT-Ω and StreamVGGT. RAMB substantially reduces the retained context while maintaining competitive reconstruction quality, and the long-sequence experiments show practical benefits in latency and memory usage.
Critical weaknesses
  1. The comparison with existing methods could be more comprehensive. The results show that RAMB improves over the original backbones and uniform sampling, but it is still difficult to assess how much it offers over existing approaches to frame selection and context management.
  2. The overhead of the proposed method could be explained more clearly. In particular, VGGT-Ω requires an initial forward pass over the full sequence to obtain scene tokens before selecting frames. A clearer breakdown of this cost would help readers better understand the overall efficiency gains in practice.
Questions for authors
  1. Could the authors provide a more direct comparison with existing frame selection or context management methods? This would help clarify the advantages of RAMB over prior approaches.
  2. Could the authors clarify the overhead of RAMB, particularly the initial full-sequence forward pass required to extract scene tokens for VGGT-Ω? A breakdown of this cost would help readers better understand the overall efficiency gains.
3 comments

We thank the reviewer for the constructive feedback. We address the concerns regarding comparisons with existing methods and the computational overhead of RAMB below.

W1 - Context length for memory bank

We agree that a clearer comparison is needed to distinguish the benefits of reconstruction-aware frame selection from those of reducing the input context alone.

The original paper compares RAMB against uniform frame sampling (Fig. 4) and reports an efficiency comparison with IncVGGT (Fig. 3). However, these experiments do not provide a comprehensive, matched-budget comparison with existing context-management methods.

To further isolate the sources of RAMB's reconstruction-quality improvements, we conduct an additional controlled experiment that separates frame-selection quality from the effect of reconstruction context length.

We consider four configurations using VGGT-Ω on ScanNet:

  1. Full-sequence inference (full_seq_res): The backbone processes all 400 frames jointly and reconstructs the full sequence.
  2. Selected predictions from full context (full_mb_res): We select frames using RAMB but directly extract their predictions from the original 400-frame inference. Thus, the evaluated frames were reconstructed using the full context.
  3. Uniformly sampled short-context inference (unif_seq_res): We uniformly sample the same number of frames as RAMB and reconstruct them using a separate, shorter forward pass.
  4. RAMB-selected short-context inference (mb_seq_res): We reconstruct the RAMB-selected frames using a separate forward pass containing only those frames.

The last three configurations evaluate the same number of frames per sequence. In particular, comparing full_mb_res with mb_seq_res holds the selected frame identities constant while changing the context used for prediction. Comparing unif_seq_res with mb_seq_res instead holds the input-frame budget and short-context inference setting constant while changing the selected observations.

Table 1: Reconstruction quality with different frame sources and context lengths (VGGT-Ω, ScanNet)

Method# framesACC ↓COMP ↓NC ↑
full_seq_res4000.1260.1410.801
full_mb_res12.60.1040.1960.820
unif_seq_res12.60.1730.3340.786
mb_seq_res12.60.0820.1810.822

First, frame selection matters beyond input reduction. Under the same short-context inference setting and frame budget, RAMB-selected frames outperform uniformly sampled frames across all three metrics. This indicates that reducing the input length alone does not explain the observed improvements.

Second, the inference context affects reconstruction quality even when the selected frames are fixed. Compared with predictions extracted from full-sequence inference (full_mb_res), independently reconstructing the same RAMB-selected frames (mb_seq_res) improves ACC and NC, as well as COMP. This suggests that shorter-context re-inference can produce more accurate predictions for the selected observations.

Together, these results indicate that RAMB benefits from both informative frame selection and reconstruction under a reduced effective context.

We emphasize that this experiment isolates these two effects within the same backbone. It complements, rather than replaces, direct comparisons with existing frame-selection and context-management methods.

Q1 - Comparing with frame selection method

MUSt3R (CVPR 2025) introduces different memory-management mechanisms for online and offline reconstruction. In its online setting, MUSt3R uses predicted 3D pointmaps and a KD-tree-based spatial discovery measure to determine whether an incoming frame provides sufficiently novel geometric observations. In contrast, RAMB combines reconstruction-aware ranking with scene-token similarity to select complementary frames.

To compare the selection policies under a common reconstruction backbone, we adapt the online KD-tree-based keyframe-selection policy of MUSt3R to StreamVGGT, replacing RAMB's selection mechanism while retaining the same StreamVGGT backbone.

This comparison evaluates the keyframe-selection policies within StreamVGGT; it is not a comparison against the complete native MUSt3R reconstruction system.

We conduct two additional experiments using 240-frame sequences.

Exp 1: Frame-selection overhead

We measure the inference latency of the two selection mechanisms. Accroding to MUSt3R's codes, the KD-tree selection runs on CPU. For RAMB, RM runs on GPU and the minimum-similarity selection runs on CPU. All latencies are tested on AMD EPYC 7763 + NVIDIA A100 in eager mode.

Table 2. Measured frame-selection latency.

Selection policyLatency
MUSt3R-inspired KD-tree selection35.56 ms
RAMB selection3.32 ms

Comparing the average processing time per forward pass, RAMB's frame selection is faster than that of the adapted KD-tree-based policy.

Exp 2: Reconstruction quality

We next evaluate the reconstruction quality obtained with each selection policy using StreamVGGT.

Table 3. Reconstruction quality with RAMB and MUSt3R-inspired KD-tree selection (StreamVGGT, ScanNet, 240-frame input).

Selection policySelected # FramesACC ↓COMP ↓NC ↑
RAMB20.90.1730.1880.725
KD-tree selection22.90.1820.2010.693

With slightly few selected frames, RAMB achieves better ACC, COMP, and NC than the adapted KD-tree selection policy.

These results provide an additional comparison with a geometry-based keyframe-selection heuristic. They suggest that RAMB can provide favorable reconstruction quality and selection overhead when both policies are applied to StreamVGGT.

We emphasize that this comparison does not establish overall superiority over MUSt3R, which uses a different reconstruction backbone and memory architecture. Rather, it isolates the effect of the adapted online keyframe-selection policy under a common backbone.

W2 & Q2 - Overhead of RAMB

We report the latency breakdown of StreamVGGT and VGGT-Ω. All latencies are measured on an NVIDIA A100 in eager mode. Latency of RAMB is included in latency of backbone.

Table 4: Latency breakdown of the backbones and RAMB (NVIDIA A100)

Backbone# inputsLatency of backboneLatency of RAMBPercentage of RAMB
VGGT-Ω (Selection-only)40097.94 s110+506 ms*0.63%
VGGT-Ω (End-to-end)18022.01 s95 ms0.43%
StreamVGGT24087.16 s668+2800 ms**3.98%

* The RAMB inference time is 110 ms, and re-running VGGT-Ω on the selected frames takes 506 ms.

** The RAMB inference time is 668 ms, and manipulating the KV cache takes 2800 ms.

Dear reviwer,

We hope our previous responses have adequately addressed most of your concerns. If you have any further questions, we would be happy to clarify them. Since today is the final day of the rebuttal period, if we do not have sufficient time to obtain the experimental results, we will include them in the supplementary appendix in a subsequent revision.

Best regards, The Authors

Reviewer_WC5NOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Reconstruction-Aware Memory Bank (RAMB) for long-context 3D reconstruction. A Rank Model selects a high-utility leading frame, while scene-token similarity adds complementary frames. The method is evaluated with VGGT-Ω and StreamVGGT on ScanNet, ETH3D, and KITTI-VO.

Critical strengths
  1. The paper addresses an important scalability problem for feed-forward and streaming reconstruction.
  2. Learning frame utility from reconstruction outcomes is a useful contribution, and the frame-level strategy works with two VGGT-style backbones.
  3. Uniform-sampling and leading-frame ablations support the value of informed selection; the chunked experiment demonstrates a practical memory–quality trade-off.
Critical weaknesses
  1. In the VGGT-Ω selection-only experiment, obtaining scene tokens requires a full-sequence forward pass. The reported frame reduction should therefore be distinguished from end-to-end savings.
  2. Generalization remains insufficiently tested: KITTI-VO’s training status is unclear, there is no independent outdoor zero-shot or dynamic-scene evaluation, and completeness sometimes worsens despite improved accuracy.
  3. Important design choices lack ablation. These include frame-level versus token-level memory, the contribution of different scene-token layers, and training a full ranking when inference directly uses only its top-ranked frame.
Questions for authors
  1. Which datasets and sequences are used to train the Rank Model, and is KITTI-VO excluded from both its training and backbone pretraining?
  2. Could the authors report end-to-end cost, including token extraction and memory updates, and compare RAMB with equal-budget uniform sampling and contiguous short windows?
  3. Could the authors ablate individual scene-token layers and layer combinations? Prior VGGT analyses suggest that different layers provide complementary geometric cues. It would also help to test whether supervising the final memory bank, rather than ranking prefixes, improves selection.
8 comments

We thank the reviewer for the kind review. Below, we respond to the Critical Weaknesses W1, W2, and W3, and to Questions Q1, Q2, and Q3, respectively.

W1 - Frame reduction in the end-to-end mode

We understand your concern and agree that RAMB's application to VGGT-Ω should primarily be demonstrated through the end-to-end mode, while the selection-only mode serves mainly as an analytical setting. Therefore, the performance achieved in the end-to-end setting should be considered as the basis for reporting the main contributions. We have revised the corresponding claims in the paper accordingly.

In the paper, we report a memory bank (mb) compression ratio of 12.6/400 for VGGT-Ω under the selection-only setting. We further report the compression ratios of mb1, mb2, and mb2+seq3 under the end-to-end mode, which are 11.5/180, 21.8/360, and 201.8/540 respectively. Thus, for VGGT-Ω, RAMB brings the context reduce as 2.67×2.67 \times and speed up as 2.45×2.45 \times.

Q2.1 - RAMB vs. equal-budget uniform sampling and contiguous short windows

Added experiments shows the advantage of RAMB comapring with equal-budget uniform sampling and contiguous short windows.

In the original design, memory bank selection is performed only during the first two forward passes. In the third forward pass, the selected second memory bank (mb2) is concatenated with the final 180 frames, and the model performs a single forward pass to obtain the reconstruction results without further frame selection. We designed three sets of experiments to further demonstrate the efficacy of the end-to-end mode.

All reconstruction results are evaluated against the ground-truth point cloud corresponding to the full set of 540 frames.

Exp1: Quality of the retained historical context

We first evaluate the contribution of the selected memory banks: we compare the reconstruction results obtained using mb2 as input against those obtained using an equally sized set of frames uniformly sampled (unif2) from the first two chunks (360 frames). The dataset is ScanNet. The reconstruction quality is shown below:

Table 1: Reconstruction quality of the memory bank (mb2) vs. equal-budget uniform sampling (unif2) on ScanNet

MethodACC ↓COMP ↓NC ↑
mb20.1010.1510.826
unif20.1130.1770.806

Under the same input budget, the memory bank outperforms uniformly sampled frames.

Exp2: Historical-context selection in chunked inference

We then evaluate the end-to-end RAMB method against equal-budget uniform sampling followed by contiguous short windows, that is, mb2+seq3 vs. unif2+seq3. The dataset is ScanNet. The reconstruction results are shown below:

Table 2: End-to-end reconstruction quality of mb2+seq3 vs. unif2+seq3 on ScanNet

MethodACC ↓COMP ↓NC ↑
mb2+seq3 (RAMB)0.1070.1150.821
unif2+seq3 (Uniform)0.1100.1250.819

RAMB achieves better results across all three metrics, although the differences are smaller than in Experiment 1. This is consistent with the final 180-frame chunk providing substantial shared geometric information to both methods, reducing the relative influence of their historical-context selection policies. Importantly, this experiment controls for the final chunk and the historical-context budget, allowing us to isolate the benefit of the memory-selection strategy.

Once seq3 is added, the performance gap between the memory-bank-based and uniform-sampling-based methods becomes smaller. In other words, incorporating the final long sequence (180 frames) narrows this gap. To further demonstrate the advantages of our method, we conduct exp3 on longer sequences.

Exp3: Longer sequences on KITTI-VO

We sampled 900 frames from KITTI-VO and performed 5 forward passes, each processing 180 frames. The end-to-end results (mb4+seq5) are compared against those obtained using uniform sampling with the same frame budget followed by contiguous-window processing (unif4+seq5), as shown below.

Table 3: End-to-end reconstruction quality of mb4+seq5 vs. unif4+seq5 on 900-frame KITTI-VO sequences

MethodACC ↓COMP ↓NC ↑
mb4+seq515.1142.490.83
unif4+seq518.0140.520.82

The results demonstrate that our method achieves better ACC and NC on long sequences. Our method also outperforms the baseline when comparing the average of ACC and COMP.

Q2.2 - End-to-end computational overhead

We additionally profile the full three-pass pipeline and separate backbone inference time from the remaining memory-management operations.

Table 4. Latency breakdown of the 540-frame chunked inference pipeline on an NVIDIA A100, using eager execution.

ComponentPass 1Pass 2Pass 3Total
Pipeline latency22.01 s25.01 s30.28 s77.30 s*
Non-backbone processing95 ms95 ms20 ms210 ms

* This value differs from the one reported in the original paper because we noticed that, except for the final forward pass, the preceding N−1N-1 forward passes do not require executing the full backbone, as only scene tokens are needed. Specifically, the camera, depth, and point heads can be skipped, reducing the inference time by ~10% compared to a full forward pass. We have therefore updated this record accordingly.

The non-backbone processing includes scene-token extraction and memory-bank management. Memory-bank selection is performed after the first two passes, while the final pass directly processes the last chunk together with the retained memory bank.

Across the three passes, the measured non-backbone overhead is 210 ms, accounting for only 0.27% of the total measured pipeline latency.

Thus, the computational cost of the memory-management operations is small relative to backbone inference. The primary efficiency benefit comes from limiting the number of frames jointly processed in each forward pass, rather than accelerating the backbone itself.

In this cell, we focus on the concerns about the datasets and KITTI-VO.

W2.1 & Q1 - Training recipe for KITTI-VO

We clarify the training and evaluation data used for the Rank Model (RM), particularly for KITTI-VO.

The reconstruction backbones remain frozen throughout our experiments. Only the RM is trained or fine-tuned for frame selection.

For KITTI-VO, we use sequence-level separation between RM fine-tuning and evaluation. Of the 11 sequences with ground-truth annotations, sequences 00 and 02 are reserved for testing, as they are the two longest sequences. The remaining nine sequences are used for RM fine-tuning. Therefore, the KITTI-VO evaluation sequences are not included in RM fine-tuning.

For both VGGT-Ω and StreamVGGT, we initialize the RM using weights pretrained on ScanNet and fine-tune it for an additional 200 training steps using the corresponding backbone. We use ScanNet_v1_train as the training set and ScanNet_v1_test as the test set.

Regarding backbone pretraining, KITTI-VO is not listed among the publicly documented pretraining sources of the backbones. Since we use pretrained backbone weights without modification, our claim of training-set separation specifically refers to the RM training and fine-tuning procedure.

We use domain-specific RM fine-tuning because the visual characteristics and camera trajectories of indoor and outdoor scenes differ substantially. Accordingly, the KITTI-VO results evaluate generalization to held-out sequences within the fine-tuned outdoor domain, rather than zero-shot transfer from ScanNet to KITTI-VO.

To assess generalization beyond the RM training datasets, we additionally conduct a zero-shot evaluation on OmniWorld-Game, as described below.

W2.2 - Generalization evaluation on outdoor zero-shot and dynamic scenes

We extend the evaluation to OmniWorld-Game, an independent synthetic outdoor dataset containing dynamic objects. Neither backbone nor RM is adapted using this dataset. We use the KITTI-VO-fine-tuned RM weights without additional training.

Table 5: Zero-shot reconstruction quality on OmniWorld-Game

ModelInputACC ↓COMP ↓NC ↑
VGGT-Ωfull-seq1.7379.250.766
VGGT-Ωmb-seq1.29111.360.773
StreamVGGTfull-seq1.86526.390.730
StreamVGGTmb-seq1.30124.670.746

For VGGT-Ω, RAMB reduces ACC by approximately 25.7% and improves normal consistency, although completeness decreases. For StreamVGGT, RAMB improves all three metrics, reducing ACC by approximately 30.2%.

These results provide additional evidence that the proposed frame-selection strategy transfers to an unseen outdoor dataset with different visual characteristics and dynamic content.

We emphasize that these are aggregate reconstruction metrics on scenes containing dynamic objects; they do not separately measure reconstruction quality in dynamic regions. Nevertheless, the improvements indicate that RAMB's benefits are not limited to the ScanNet and KITTI-VO evaluation sequences.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Paper Revision Statement

We sincerely thank all reviewers for their time, careful reading, and constructive feedback. Their comments have substantially improved the clarity and rigor of our paper. We summarize the main revisions below.

1. End-to-end mode as the primary setting for VGGT-Ω. Most of the reviewers' concerns centered on the end-to-end mode of VGGT-Ω. We agree that the end-to-end mode, rather than the selection-only mode, should serve as the primary basis for evaluating RAMB on VGGT-Ω, since it is the method we ultimately deploy on this backbone. The selection-only mode requires a full-sequence forward pass and is therefore intended only as an analytical setting for isolating frame-selection quality. Accordingly, we have revised our claims throughout the paper, including the abstract and introduction: for 540-frame sequences, VGGT-Ω retains 37.3% of the frames, achieves a 2.45× speedup over full-sequence inference with comparable reconstruction quality, and fits on a single 24 GB GPU.

2. Faster end-to-end inference. In follow-up work, we realized that in the end-to-end mode, the first N−1 forward passes only need to produce scene tokens for memory-bank selection, not reconstruction predictions. The camera, depth, and point heads can therefore be skipped in these passes, which makes the end-to-end mode about 10% faster overall (from 2.25× to 2.45× speedup). We have updated the corresponding results in the paper.

3. Additional experiments. Prompted by the reviewers' insightful questions, we conducted a substantial number of additional experiments. Those that fit naturally into the main narrative have been incorporated into the ablation study (Sec. 4.3.1, contributions of the individual components; Sec. 4.3.2, disentangling frame selection from context-length reduction), while the remaining analyses, such as the contribution of different scene-token layers, are provided in the appendix (Appendix E).

Unless otherwise stated, all results reported in the original submission remain unchanged in the revised version. The exceptions are the end-to-end latency results described above and Fig. 6 (Appendix D), which we corrected after fixing a sequence-sampling bug. The hyperparameter choices are unaffected.

All revised or newly added content is highlighted in blue in the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.