- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
RAMB is a pluggable frame memory buffer designed for long-sequence reconstruction tasks with VGGT-family models. It operates under a specific frame selection rule: it reserves the very first frame of the sequence, plus one frame picked out by a 20M-parameter ranking model trained via the REINFORCE algorithm, and then iteratively adds the least similar frames from the register-token space into the buffer, until all remaining frames have a similarity score above the 0.97 threshold.
This module serves two purposes: it curates input frames for the VGGT-Ω model, and caps the length of the KV cache history for StreamVGGT. According to the paper, this approach achieves higher accuracy than full-sequence inference while using only around 3% of the total frames on the ScanNet, ETH3D and KITTI-VO benchmarks. It also delivers 2.25× faster inference on 540-frame sequences under a 24 GB memory constraint.
Critical strengths
S1. The core practical advantage of RAMB is that it caps memory usage without any modifications to the backbone architecture. StreamVGGT keeps memory consumption steady at 12–15 GB even for sequences up to 560 frames. By comparison, the native model already uses roughly 60 GB of memory at just 240 frames, and runs into out-of-memory failures beyond that length (Fig. 3).
S2. Figure 4 demonstrates that the accuracy of VGGT-family models does not scale monotonically with input sequence length. For VGGT-Ω in particular, it reaches an accuracy of around 0.107 with 60 uniformly sampled frames, and 0.122 when all 400 frames are fed into the model. This means frame selection can deliver accuracy gains, rather than only functioning as a memory-saving measure.
Critical weaknesses
W1. The paper’s central claim—that using roughly 3% of total frames can match or outperform full-sequence inference—relies on an evaluation protocol that is biased toward the selected frame subset. In the frame-selection-only mode, the memory bank is derived from a complete 400-frame forward pass (as outlined in Algorithm 1), meaning this mode delivers no computational savings at all. Since the ~13-frame point cloud is evaluated against the entire scene, accuracy (ACC) improves while completeness (COMP) degrades (on ScanNet: ACC shifts from 0.126 to 0.082, and COMP shifts from 0.141 to 0.181), leaving the average of the two metrics nearly unchanged. Furthermore, Figure 6 reports a full-sequence ACC of 0.0944 rather than the 0.126 referenced earlier, which calls into question the actual accuracy gain of the method.
W2. The work does not include ablation studies to isolate the contributions of its individual components. With only vanilla backbones and uniform sampling serving as performance baselines, the reported 2.25× speedup could stem entirely from the chunking mechanism alone. The Rank Model only contributes one additional frame during inference, yet removing it causes the ACC error to jump from 0.082 to 0.187 (Table 4)—a result worse than both uniform sampling and full-sequence inference. Such a pronounced performance impact requires further justification. Additionally, the similarity-based selection step is essentially farthest-point keyframe selection, a technique already used by MUSt3R (CVPR 2025); however, neither MUSt3R nor StreamVGGT is cited in the paper.
Questions for authors
Q1 (clarification). Do the training sequences used for the Rank Model have no overlap with the test splits of the ScanNet and KITTI-VO benchmarks, and how many test sequences are included in the evaluation?
Q2 (clarification). How are chunk outputs fused across separate inference passes that may have scale discrepancies, and are the scores reported in Table 3 calculated on these per-pass chunk outputs, or on a final re-inference run over the assembled memory bank (as illustrated in Fig. 1)?
Q3 (clarification). Figure 3 suggests a memory footprint of roughly 49 GB for 540-frame sequences, whereas the abstract states memory consumption exceeds 60 GB under the same setting. Which of these two values is accurate?
8 comments
W1.2 - Computational savings of the selection-only mode
We agree that the selection-only mode yields no computational savings, since it requires a full-sequence forward pass. However, the selection-only mode is an analysis setting intended to isolate frame-selection quality, not to demonstrate deployment efficiency rather than a deployment mode.
The efficiency results in the paper instead concern two other settings:
- Chunked VGGT-Ω inference: The sequence is processed incrementally, with a compact memory bank retaining informative observations from earlier chunks.
- Streaming StreamVGGT inference: Each incoming frame is processed with a bounded historical KV cache, avoiding the unbounded growth of retained context.
In these settings, scene-token representations are obtained during the backbone inference already required to process incoming observations, rather than through a separate full-sequence selection pass.
The additional controlled experiment in Table 1 evaluates the quality of frame selection. The efficiency benefits must be assessed separately through the complete chunked or streaming inference pipelines.
We have revised the abstract and introduction to more clearly distinguish the approximately 30-fold frame reduction observed in selection-only evaluation from the end-to-end latency and memory improvements.
We also recognize that the chunked VGGT-Ω pipeline must be described precisely in terms of which frames receive final reconstruction predictions, as clarified in our response to Q2.
W2.1 - Contributions of individual components
We agree that the contributions of the Rank Model (RM) and the Minimum-Similarity Frame Selection strategy should be evaluated separately.
Our memory-bank construction combines two complementary mechanisms:
- Reconstruction-aware initialization: The RM selects a leading frame based on its predicted reconstruction utility.
- Coverage-aware expansion: Minimum-Similarity Frame Selection adds observations that are least redundant with the frames already retained.
The leading frame is important not only as an additional observation but also because it determines the initial state of the subsequent greedy selection procedure. Consequently, changing the leading frame can alter the entire selected subset
In Section 4.3.1 of the original paper, we demonstrate the importance of the leading frame selected by RM. It provides reconstruction-aware initialization, while the similarity-based strategy explicitly considers redundancy relative to the current memory bank. Here, we show the contribution of the Minimum-Similarity Frame Selection strategy and further explain the pronounced performance changes observed.
Since RM is designed to select frames that are highly complementary to the anchor frame, it tends to assign similar scores to frames with similar content. Consequently, directly selecting frames based on RM's ranking results in a set of highly redundant frames. Such a selection fails to adequately represent the entire sequence and, in turn, cannot achieve reconstruction quality comparable to that obtained using the full sequence, as shown in Table 2. The Minimum-Similarity Frame Selection strategy is specifically designed to address this limitation by promoting diversity among the selected frames.
In this experiment, RM selection uses the same number of input frames for each sequence as Minimum-Similarity Selection. The number of frames retained by Minimum-Similarity Selection is determined by the stopping criteria.
Table 2: Reconstruction quality when frames are selected directly by the RM ranking, without Minimum-Similarity Frame Selection
| Method | ACC ↓ | COMP ↓ | NC ↑ |
|---|---|---|---|
| VGGT-Ω | 0.185 | 0.488 | 0.735 |
| StreamVGGT | 0.202 | 0.431 | 0.654 |
The similarity-only variant performs substantially worse than full RAMB, supporting the importance of reconstruction-aware initialization. Likewise, the ranking-only variant yields substantially worse completeness, indicating that selecting frames solely from a fixed RM ranking does not provide sufficient scene coverage under the evaluated setting.
RM selection and Minimum-Similarity Frame Selection serve different purposes. Without either of them, the performance degrades dramatically. Conceptually, our method can be viewed as processing a rope: the anchor frame fixes one end, while RM selection pulls the rope from the other. Minimum-Similarity Selection then divides the rope into semantically balanced segments, ensuring diversity across the selected frames.
This distinction also explains why the RM's full-ranking training objective does not imply that the final memory bank should be constructed directly from its top-ranked frames. Ranking-prefix supervision provides reconstruction-quality signals across multiple context sizes, whereas memory-bank expansion requires decisions conditioned on previously selected observations.
Together, the ablations support the complementary roles of the two components. We have clarified this distinction and provide the experimental details necessary for interpreting the comparisons fairly in the new version of the paper.
W1.1 - Memory bank frames vs. full frames
We agree that evaluating a reconstruction from substantially fewer frames requires careful interpretation, particularly because accuracy (ACC) and completeness (COMP) capture different aspects of reconstruction quality.
To separate the effects of selecting a frame subset from those of reducing the backbone's inference context, we conduct an additional controlled experiment using VGGT-Ω on ScanNet.
We consider four configurations:
full_seq_res): VGGT-Ω jointly processes all 400 frames, and the resulting point cloud is evaluated.full_mb_res): We extract the predictions corresponding to RAMB-selected frames from the original full-sequence forward pass. No additional reconstruction inference is performed.unif_seq_res): We independently reconstruct an equally sized set of uniformly sampled frames using short-context inference.mb_seq_res): We independently reconstruct the RAMB-selected frames using short-context inference.The last three configurations use the same average number of frames. In particular,
full_mb_resandmb_seq_rescontain predictions for identical frame indices, allowing us to examine the effect of inference context while holding frame selection fixed.Table 1: Reconstruction quality with different frame sources and context lengths (VGGT-Ω, ScanNet)
First, we acknowledge the effect of evaluating a selected subset. Extracting only the RAMB-selected predictions from the original full-sequence reconstruction improves ACC from 0.126 to 0.104, even without changing the backbone inference context. This confirms that frame-subset selection itself contributes to the observed accuracy improvement. It also reduces scene coverage, as reflected by the increase in COMP from 0.141 to 0.196.
Second, reduced-context inference provides an additional improvement. When we reconstruct exactly the same selected frames independently, ACC further decreases from 0.104 to 0.082. Because the selected frame indices remain unchanged, this improvement cannot be attributed to evaluating a different subset of frames. Rather, it demonstrates that the predictions for those frames improve under the shorter inference context.
Third, the identity of the selected frames matters. Under the same short-context setting and frame budget, RAMB substantially outperforms uniform sampling across the reported reconstruction metrics. This supports the contribution of reconstruction-aware frame selection beyond simply reducing the input length.
These results distinguish two sources of improvement: selecting informative observations and obtaining improved predictions for those observations under a reduced context. We do not claim that the reduced-frame reconstruction provides the same geometric coverage as full-sequence reconstruction.