- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper starts with a clear issue, which I paraphrase as follows: When VLMs analyze video to answer spatial questions (like counting items, measuring distances, or estimating room sizes), they usually record video frames in plain chronological order. This clutters the model's memory with irrelevant or repetitive views, making long-term visual reasoning harder.
…And a proposed solution: a memory framework that filters video frames based on the question being asked and the camera's 3D movement. Instead of saving every frame, it evaluates incoming visual evidence using two simple questions: (1) Is it relevant to the question? (2) Does it show something new (novelty), or is it repetitive?
The proposed solution is delivered via 2 main contributions: (i) A short-term Fine-Grained Context Bank (FGCB) that keeps a rolling memory of recent frames and camera position shifts to track smooth viewpoint changes; (ii) A longer-term Semantic-Geometric Evidence Bank (SGEB) that retains a small, curated set of pooled features that are both question-relevant and non-redundant.
Three interesting results are worth noting:
- The filtering method introduced by the paper beats raw capacity: a small, high-value evidence set (just 8 select frames) performs better than hoarding all 32 sampled frames.
- SOTA Performance: The method achieves top results on VSI-Bench and VSTI-Bench.
- Adaptable: The method also generalizes well to out-of-distribution tasks, including 3D scene understanding and VQA
Critical strengths
Two strengths (as relayed in my summary)
- The filtering method introduced by the paper is sound and is demonstrated to outperform raw capacity.
- SOTA in spatial understanding, + the method's Semantic-Geometric Evidence Bank (SGEB) also generalizes well to out-of-distribution tasks, including 3D scene understanding and VQA. This is good to see and could be picked up by folks working on general video QA or long-form video QA.
Critical weaknesses
I have no doubts about the effectiveness of the approach. My main concern is the incremental nature of the method.
The core building blocks draw directly from prior work, which is not an issue by itself; however, I don't see much readaptation or reengineering. For instance, a) the camera-guided feature gating is explicitly adapted from SpaceMind; b) maintaining short-term dense context alongside long-term sparse memory mirrors dual-memory architectures like VLM^2; c) Scoring entries via a product of question relevance and novelty is a variation of classic Maximal Marginal Relevance (MMR) and Q-Former query matching.
Questions for authors
- Could the authors address the novelty issues I raised in "Critical Weaknesses"? The answer to this question could affect my assessment of the paper.
- As shown in the paper's capacity ablation (Table 6b), performance drops non-monotonically when memory capacity increases beyond . This indicates that the attention and fusion mechanisms get cluttered when presented with larger evidence sets. Could the authors explain why that is? My intuition is that long videos, at least, should support longer context. Also, why is K not adaptive?
2 comments
Response - 2
2. Why Performance Drops Beyond (Including Long Videos) & Adaptive- (Q2)
(a) Why degrades, and why Long Videos benefit more from compact :
Every capacity in Table 6(b) was independently trained under the same protocol, and all videos (short, mid, and long) use standard 32-frame uniform sampling (VLM-3R, ). Through the residual path in Eq. (14) (), the fused representations of all 32 sampled frames still enter the MLLM, while capacity governs only the selective cross-temporal auxiliary bank (Response to Reviewer 1W3s, Sec. 2).
Within this architecture, compact SGEB eviction matters even more on longer indoor scans:
- Each SGEB slot contributes spatial tokens to cross-attention in Eq. (12) (up to keys at vs. curated keys at ). At (no eviction), accuracy (
67.9, Table 6(b)) drops back to thew/o SGEBbaseline (67.92, Table 4). Consistent with Table 7, where soft read biasing alone (69.2) underperforms hard eviction (70.2), we hypothesize that retaining all uncurated views dilutes cross-attention readout despite the soft utility bias . - Longer VSI-Bench scans (
>120s) sweep slowly across broader room layouts, close-up surfaces, and non-queried furniture, whereas a specific query (e.g., "distance between the table and the bed") still depends on a small subset of views. Across length splits, the gain of over no-eviction grows monotonically with video duration: +2.0 on Short (72.3vs.70.3), +2.7 on Mid (72.7vs.70.0), and +3.9 on Long (70.1vs.66.2). (Across intermediate capacities, the1.1-point difference between separately trained (68.4) and (69.5) likely reflects run-to-run optimization variation in the learned readout and gate , while every selective capacity consistently outperforms , as detailed in Response to Reviewer 1W3s, Sec. 3.)
(b) Why was fixed & Adaptive- Extension:
We fixed () to maintain a deterministic memory and compute bound during batched training without threshold hyperparameters. At the same time, because relevance is calibrated per video to form , SGEB naturally supports an inference-time Adaptive- mode on the same trained checkpoint. With upper bound and floor , SGEB retains entries meeting (keeping the top- entries when fewer than 4 exceed ), where a single fixed is calibrated offline on training scenes to match a mean final size of without accessing VSI-Bench test data:
| Capacity Strategy | Final Retained Slots (Avg / Range) | VSI Avg. | VSI Short | VSI Mid | VSI Long | Room Size / Obj. Cnt. |
|---|---|---|---|---|---|---|
| Fixed Full Capacity () | 32.0 | 67.9 | 70.3 | 70.0 | 66.2 | 73.2 / 72.8 |
| Fixed Compact Capacity (, Paper) | 8.0 | 71.2 | 72.3 | 72.7 | 70.1 | 75.3 / 74.8 |
| Adaptive- (, floor ) | 8.1 (4 to 15) | 71.1 | 71.9 | 72.6 | 70.3 | 75.8 / 75.2 |
Without retraining the readout gates or per-query manual tuning, Adaptive- remains close to fixed overall (71.1 vs. 71.2, with a minor trade-off on short clips) while retaining more final entries on average for scene-level categories (Room Size: 9.8, Obj. Cnt.: 9.3) than for two-object distance queries (Abs. Dist.: 7.3), slightly improving VSI Long (70.3 vs. 70.1), Room Size (+0.5), and Obj. Cnt. (+0.4). We will include this analysis and Adaptive- variant in Sec. 4.4.
Response - 1
Thank you for the clear summary of our work and the thoughtful questions regarding architectural novelty and memory capacity scaling. We address both questions in detail below.
1. Technical Novelty Beyond Prior Building Blocks (W1 & Q1)
While Q-GeoMem builds on established primitives (credited in Sec. 2 and Sec. 3), directly combining frame-level fusion (SpaceMind), question-agnostic dual memory (VLM2), and classic offline MMR does not address question-guided sequential spatial memory retention. Our contribution lies in coupling camera-conditioned geometry, question relevance, and sequential memory updates:
Sequential Active-Set Re-Evaluation vs. Classic Offline MMR / Write-Once Utility (Eqs. 10 to 13): Classic greedy MMR selects a subset offline from a static, fully observed pool without eviction, whereas VLM2 updates episodic memory using question-agnostic saliency and similarity-based updating without conditioning on the queried spatial relation. During sequential SGEB updates (t=1…T), redundancy is non-stationary: a distant, partially occluded view of a bed at t=3 is initially novel, yet becomes redundant once a clear close-up arrives at t=18, while an earlier view of a TV (needed for a bed-to-TV distance query) must remain protected. Rather than freezing a write-once utility score wt upon arrival, Q-GeoMem recomputes bank-relative novelty νu(Ct) jointly across Ct=Kt−1∪{t} at every replacement step and across Kt−1 at every read step. Replacing this dynamic active-set recomputation with a write-once MMR utility (freezing wt at write time) drops VSI-Bench accuracy from 71.2 to 69.5 (and 70.1 to 68.4 on VSI Long).
Camera-Conditioned Geometry Across Storage, Readout, and Selection: SpaceMind performs frame-local fusion without temporal memory. In Q-GeoMem, camera-conditioned geometry (CGGF,
64.83to67.28in Table 4) shapes long-range memory at two levels. First, rather than collapsing frames to single vectors, each SGEB entry eu∈R49×d preserves a 7×7 grid of spatially grounded tokens for cross-view token attention in Eq. (12). Second, in a controlled test holding stored eu, relevance ru, readout, and downstream weights fixed while replacing only write-selection novelty in Eqs. (10) and (13) with pre-CGGF 2D visual features (Response to Reviewer 1W3s, Sec. 1), geometry-enriched novelty increases the 3D camera-pose diversity of retained views (0.341to0.373) and consistently aids counting and ordering (74.3to 74.8 onObj. Cnt.,74.0to 74.6 onAppr. Order, and71.0to 71.2 overall, consistent with the modest pure-selection delta under global 1×d pooling in Eq. (10) and motivating our refined wording in Sec. 3.3.2 and Sec. 5).Write-and-Read Utility Co-Design & Camera-Δ Local Readout: Unlike decoupled write and read rules in prior dual-memory models, Q-GeoMem uses the same active utility wu to jointly control hard eviction (Eq. 13) and soft per-head attention-logit biasing βwu (Eq. 12), outperforming either mechanism alone (
68.7FIFO,69.2Read-only,70.2Write-only, and 71.2 Co-designed in Table 7). Similarly, FGCB modulates local readout logits and values using explicit inter-frame camera shifts Δfc,t,u (Eq. 6), improving over unmodulated local readout by +1.35 on VSI-Bench (Table 5) and contributing to Q-GeoMem's +7.2 margin over VLM2 on Camera Movement Direction (84.0vs.76.8, Table 2).† Controlled inference-time swap under identical Q-GeoMem weights and stored eu, isolating pure memory-update policy effects.