acceptodds
Under review as a conference paper at ICLR 2027

Q-GeoMem: Question-Guided Geometric Memory for Video Spatial Reasoning

Abstract

Video spatial reasoning requires accumulating viewpoint-dependent evidence over time while retaining information useful to the question being asked. Existing spatial video-language models improve geometric perception and long-range context modeling, but often treat memory as a generic temporal cache, which can introduce redundant or irrelevant evidence and weaken long-horizon reasoning. We propose Q-GeoMem, a question-guided geometric memory framework for video spatial reasoning. Q-GeoMem injects camera-conditioned geometry into visual tokens and maintains a Fine-Grained Context Bank for recent dense context and a Semantic-Geometric Evidence Bank for compact long-range evidence. Each candidate frame is scored by calibrated question relevance and novelty with respect to the active bank, and the resulting utility drives both memory replacement and reading. Experiments on two in-domain and five out-of-distribution benchmarks show that Q-GeoMem achieves state-of-the-art performance in the evaluated settings, and controlled analyses validate question-guided geometric evidence selection.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
VLM spatial reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_kWDQOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper starts with a clear issue, which I paraphrase as follows: When VLMs analyze video to answer spatial questions (like counting items, measuring distances, or estimating room sizes), they usually record video frames in plain chronological order. This clutters the model's memory with irrelevant or repetitive views, making long-term visual reasoning harder.

…And a proposed solution: a memory framework that filters video frames based on the question being asked and the camera's 3D movement. Instead of saving every frame, it evaluates incoming visual evidence using two simple questions: (1) Is it relevant to the question? (2) Does it show something new (novelty), or is it repetitive?

The proposed solution is delivered via 2 main contributions: (i) A short-term Fine-Grained Context Bank (FGCB) that keeps a rolling memory of recent frames and camera position shifts to track smooth viewpoint changes; (ii) A longer-term Semantic-Geometric Evidence Bank (SGEB) that retains a small, curated set of pooled features that are both question-relevant and non-redundant.

Three interesting results are worth noting:

  • The filtering method introduced by the paper beats raw capacity: a small, high-value evidence set (just 8 select frames) performs better than hoarding all 32 sampled frames.
  • SOTA Performance: The method achieves top results on VSI-Bench and VSTI-Bench.
  • Adaptable: The method also generalizes well to out-of-distribution tasks, including 3D scene understanding and VQA
Critical strengths

Two strengths (as relayed in my summary)

  1. The filtering method introduced by the paper is sound and is demonstrated to outperform raw capacity.
  2. SOTA in spatial understanding, + the method's Semantic-Geometric Evidence Bank (SGEB) also generalizes well to out-of-distribution tasks, including 3D scene understanding and VQA. This is good to see and could be picked up by folks working on general video QA or long-form video QA.
Critical weaknesses

I have no doubts about the effectiveness of the approach. My main concern is the incremental nature of the method.

The core building blocks draw directly from prior work, which is not an issue by itself; however, I don't see much readaptation or reengineering. For instance, a) the camera-guided feature gating is explicitly adapted from SpaceMind; b) maintaining short-term dense context alongside long-term sparse memory mirrors dual-memory architectures like VLM^2; c) Scoring entries via a product of question relevance and novelty is a variation of classic Maximal Marginal Relevance (MMR) and Q-Former query matching.

Questions for authors
  1. Could the authors address the novelty issues I raised in "Critical Weaknesses"? The answer to this question could affect my assessment of the paper.
  2. As shown in the paper's capacity ablation (Table 6b), performance drops non-monotonically when memory capacity increases beyond K=8K=8. This indicates that the attention and fusion mechanisms get cluttered when presented with larger evidence sets. Could the authors explain why that is? My intuition is that long videos, at least, should support longer context. Also, why is K not adaptive?
2 comments
AuthorsOpenReview ↗

Response - 1

Thank you for the clear summary of our work and the thoughtful questions regarding architectural novelty and memory capacity scaling. We address both questions in detail below.

1. Technical Novelty Beyond Prior Building Blocks (W1 & Q1)

While Q-GeoMem builds on established primitives (credited in Sec. 2 and Sec. 3), directly combining frame-level fusion (SpaceMind), question-agnostic dual memory (VLM2\text{VLM}^2), and classic offline MMR does not address question-guided sequential spatial memory retention. Our contribution lies in coupling camera-conditioned geometry, question relevance, and sequential memory updates:

  1. Sequential Active-Set Re-Evaluation vs. Classic Offline MMR / Write-Once Utility (Eqs. 10 to 13): Classic greedy MMR selects a subset offline from a static, fully observed pool without eviction, whereas VLM2\text{VLM}^2 updates episodic memory using question-agnostic saliency and similarity-based updating without conditioning on the queried spatial relation. During sequential SGEB updates (t=1…Tt = 1 \dots T), redundancy is non-stationary: a distant, partially occluded view of a bed at t=3t=3 is initially novel, yet becomes redundant once a clear close-up arrives at t=18t=18, while an earlier view of a TV (needed for a bed-to-TV distance query) must remain protected. Rather than freezing a write-once utility score wtw_t upon arrival, Q-GeoMem recomputes bank-relative novelty νu(Ct)\nu _ u(\mathcal{C} _ t) jointly across Ct=Kt−1∪{t}\mathcal{C} _ t = \mathcal{K} _ {t-1} \cup \lbrace t \rbrace at every replacement step and across Kt−1\mathcal{K} _ {t-1} at every read step. Replacing this dynamic active-set recomputation with a write-once MMR utility (freezing wtw_t at write time) drops VSI-Bench accuracy from 71.2 to 69.5 (and 70.1 to 68.4 on VSI Long).

  2. Camera-Conditioned Geometry Across Storage, Readout, and Selection: SpaceMind performs frame-local fusion without temporal memory. In Q-GeoMem, camera-conditioned geometry (CGGF, 64.83 to 67.28 in Table 4) shapes long-range memory at two levels. First, rather than collapsing frames to single vectors, each SGEB entry eu∈R49×de_u \in \mathbb{R}^{49 \times d} preserves a 7×77 \times 7 grid of spatially grounded tokens for cross-view token attention in Eq. (12). Second, in a controlled test holding stored eue_u, relevance rur_u, readout, and downstream weights fixed while replacing only write-selection novelty in Eqs. (10) and (13) with pre-CGGF 2D visual features (Response to Reviewer 1W3s, Sec. 1), geometry-enriched novelty increases the 3D camera-pose diversity of retained views (0.341 to 0.373) and consistently aids counting and ordering (74.3 to 74.8 on Obj. Cnt., 74.0 to 74.6 on Appr. Order, and 71.0 to 71.2 overall, consistent with the modest pure-selection delta under global 1×d1 \times d pooling in Eq. (10) and motivating our refined wording in Sec. 3.3.2 and Sec. 5).

  3. Write-and-Read Utility Co-Design & Camera-Δ\Delta Local Readout: Unlike decoupled write and read rules in prior dual-memory models, Q-GeoMem uses the same active utility wuw_u to jointly control hard eviction (Eq. 13) and soft per-head attention-logit biasing βwu\beta w_u (Eq. 12), outperforming either mechanism alone (68.7 FIFO, 69.2 Read-only, 70.2 Write-only, and 71.2 Co-designed in Table 7). Similarly, FGCB modulates local readout logits and values using explicit inter-frame camera shifts Δfc,t,u\Delta f _ {c,t,u} (Eq. 6), improving over unmodulated local readout by +1.35 on VSI-Bench (Table 5) and contributing to Q-GeoMem's +7.2 margin over VLM2\text{VLM}^2 on Camera Movement Direction (84.0 vs. 76.8, Table 2).

SGEB Coupling Mechanism (with CGGF + Camera-Δ\Delta FGCB fixed)VSI Avg.Obj. Cnt.Appr. OrderVSI Longvs. Q-GeoMem
Question-Agnostic FIFO Dual Memory (FIFO SGEB, Table 6(a))68.771.868.966.3-2.5
Write-Once MMR Utility$^\dagger$ (Freeze wtw_t at arrival, no active-set recompute)69.573.072.468.4-1.7
Write-Only Eviction (Without co-designed read-time logit bias βwu\beta w_u, Table 7)70.273.773.068.9-1.0
2D Visual-Only MMR Selection$^\dagger$ (fv,uf _ {v,u} for write novelty, geometry-fused eue_u stored)71.074.374.069.8-0.2
Q-GeoMem (Dynamic Active-Set + 3D-Grounded Memory + Write-and-Read Co-Design)71.274.874.670.1Ref.

†^\dagger Controlled inference-time swap under identical Q-GeoMem weights and stored eue_u, isolating pure memory-update policy effects.

AuthorsOpenReview ↗

Response - 2

2. Why Performance Drops Beyond K=8K=8 (Including Long Videos) & Adaptive-KK (Q2)

(a) Why K>8K > 8 degrades, and why Long Videos benefit more from compact KK:

Every capacity K∈{4,8,16,24,32}K \in \lbrace 4, 8, 16, 24, 32 \rbrace in Table 6(b) was independently trained under the same protocol, and all videos (short, mid, and long) use standard 32-frame uniform sampling (VLM-3R, VLM2\text{VLM}^2). Through the residual path in Eq. (14) (f~h,t=fh,t+gtF⊙RtF+gtS⊙RtS\tilde{f} _ {h,t} = f _ {h,t} + \mathbf{g} _ t^{\mathcal{F}} \odot R _ t^{\mathcal{F}} + \mathbf{g} _ t^{\mathcal{S}} \odot R _ t^{\mathcal{S}}), the fused representations of all 32 sampled frames still enter the MLLM, while capacity KK governs only the selective cross-temporal auxiliary bank St−1\mathcal{S} _ {t-1} (Response to Reviewer 1W3s, Sec. 2).

Within this architecture, compact SGEB eviction matters even more on longer indoor scans:

  • Each SGEB slot contributes M=7×7=49M = 7 \times 7 = 49 spatial tokens to cross-attention in Eq. (12) (up to 31×49=1,51931 \times 49 = 1,519 keys at K=32K=32 vs. 392392 curated keys at K=8K=8). At K=32K=32 (no eviction), accuracy (67.9, Table 6(b)) drops back to the w/o SGEB baseline (67.92, Table 4). Consistent with Table 7, where soft read biasing alone (69.2) underperforms hard eviction (70.2), we hypothesize that retaining all uncurated views dilutes cross-attention readout RtSR _ t^{\mathcal{S}} despite the soft utility bias βwu\beta w_u.
  • Longer VSI-Bench scans (>120s) sweep slowly across broader room layouts, close-up surfaces, and non-queried furniture, whereas a specific query (e.g., "distance between the table and the bed") still depends on a small subset of views. Across length splits, the gain of K=8K=8 over no-eviction K=32K=32 grows monotonically with video duration: +2.0 on Short (72.3 vs. 70.3), +2.7 on Mid (72.7 vs. 70.0), and +3.9 on Long (70.1 vs. 66.2). (Across intermediate capacities, the 1.1-point difference between separately trained K=16K=16 (68.4) and K=24K=24 (69.5) likely reflects run-to-run optimization variation in the learned readout and gate gtS\mathbf{g} _ t^{\mathcal{S}}, while every selective capacity K∈{4,8,16,24}K \in \lbrace 4, 8, 16, 24 \rbrace consistently outperforms K=32K=32, as detailed in Response to Reviewer 1W3s, Sec. 3.)

(b) Why KK was fixed & Adaptive-KK Extension:

We fixed K=8K=8 (T/4=8T/4 = 8) to maintain a deterministic memory and compute bound during batched training without threshold hyperparameters. At the same time, because relevance ru∈[0,1]r_u \in [0, 1] is calibrated per video to form wu=ru⋅νu∈[0,1]w _ u = r _ u \cdot \nu _ u \in [0, 1], SGEB naturally supports an inference-time Adaptive-KK mode on the same trained checkpoint. With upper bound Kmax⁡=32K _ {\max}=32 and floor Kmin⁡=4K _ {\min}=4, SGEB retains entries meeting wu≥τww _ u \ge \tau _ w (keeping the top-Kmin⁡K _ {\min} entries when fewer than 4 exceed τw\tau _ w), where a single fixed τw\tau _ w is calibrated offline on training scenes to match a mean final size ∣ST∣\lvert \mathcal{S} _ T \rvert of 88 without accessing VSI-Bench test data:

Capacity StrategyFinal Retained Slots (Avg / Range)VSI Avg.VSI ShortVSI MidVSI LongRoom Size / Obj. Cnt.
Fixed Full Capacity (K=32K = 32)32.067.970.370.066.273.2 / 72.8
Fixed Compact Capacity (K=8K = 8, Paper)8.071.272.372.770.175.3 / 74.8
Adaptive-KK (wu≥τww _ u \ge \tau _ w, floor Kmin⁡=4K _ {\min}=4)8.1 (4 to 15)71.171.972.670.375.8 / 75.2

Without retraining the K=8K=8 readout gates or per-query manual KK tuning, Adaptive-KK remains close to fixed K=8K=8 overall (71.1 vs. 71.2, with a minor trade-off on short clips) while retaining more final entries on average for scene-level categories (Room Size: 9.8, Obj. Cnt.: 9.3) than for two-object distance queries (Abs. Dist.: 7.3), slightly improving VSI Long (70.3 vs. 70.1), Room Size (+0.5), and Obj. Cnt. (+0.4). We will include this analysis and Adaptive-KK variant in Sec. 4.4.

Reviewer_1W3sOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Q-GeoMem proposes a question-guided memory framework for video spatial reasoning. It combines camera-guided geometry fusion, a Fine-Grained Context Bank for recent observations, and a Semantic-Geometric Evidence Bank (SGEB) for longer-range evidence. SGEB combines question relevance with bank-relative feature novelty to determine which entries to retain and how to bias memory retrieval. The resulting memory readouts are fused with the current frame representation before being passed to the multimodal language model. The main contribution is the relevance-novelty policy for managing geometry-enriched memory, rather than camera-guided fusion or the dual-memory structure alone. The paper reports improvements on VSI-Bench and VSTI-Bench, consistent benefits from SGEB on five transfer benchmarks, and ablations of the memory components, selection policies, and capacity.

Critical strengths
  1. The policy ablations provide meaningful evidence for the proposed utility. At fixed capacity, Table 6 reports 71.2 on VSI-Bench for the combined relevance-novelty policy, compared with 68.7 for FIFO, 69.3 for relevance-only, and 68.9 for novelty-only. This is a more informative comparison than simply adding or removing SGEB: it supports the usefulness of combining the two criteria within the proposed architecture.
  2. The improvements are reasonably broad. SGEB improves all five transfer benchmarks in Table 3, while the main results show gains on both static and temporally evolving spatial reasoning tasks. These findings support the practical relevance of the design, although the transfer comparisons establish improvements over the authors’ ablated model rather than state of the art on all five benchmarks.
Critical weaknesses
  1. The paper does not sufficiently establish the specifically geometric contribution of its evidence-selection mechanism. Equation (10) defines novelty through feature similarity between memory entries. Although these entries contain geometry-enriched features, feature dissimilarity is not necessarily equivalent to complementary spatial evidence: different views of the same object can be dissimilar, while different instances of similar-looking objects can be similar. This distinction is particularly important for the paper’s motivating counting and cross-view reasoning tasks. Table 4 establishes that geometry fusion is useful to the overall model, and Table 6 establishes that relevance and novelty work better together. Neither comparison isolates whether geometry improves the selection decisions themselves. In particular, the paper does not compare novelty computed from visual-only versus geometry-enriched representations while keeping the stored tokens and downstream reasoning model fixed. Consequently, the evidence supports an effective relevance-diversity heuristic applied to geometry-enriched features more directly than it supports the claimed geometric evidence-selection mechanism. This limits the demonstrated conceptual advance beyond combining existing representation and memory components.
  2. The capacity experiments do not establish the claimed compact evidence bottleneck. Equation (14) retains the current frame representation through a residual connection, and Section 3.4 states that the fused frame representations are supplied to the language model. As described, exclusion from SGEB therefore does not imply exclusion of that frame’s information from the language model’s overall input. The statement that K=8 retains only 25% of the 32 sampled frames applies to the auxiliary bank, not necessarily to the information available for answering. This distinction changes the interpretation of Table 6(b). Better performance with eight entries than with 32 could reflect more selective intermediate retrieval or reduced interference, rather than preservation of sufficient evidence under an actual information bottleneck. Moreover, the manuscript does not clearly specify whether each capacity is separately trained or whether capacity is changed only at evaluation. These alternatives have different implications for explaining the non-monotonic results. The experiment supports a preferred auxiliary-memory size in the evaluated setting, but the stronger explanation concerning compact long-range evidence retention remains insufficiently established.
Questions for authors
  1. Contribution of geometry to selection: With the stored memory tokens, question-relevance scores, and retrieval/fusion architecture unchanged, how does performance change when Equation (10) computes novelty from visual-only features instead of geometry-enriched features? This targeted comparison would determine whether geometry contributes specifically to deciding what to retain, rather than only to the representations used for answering.
  2. Information available to the answer model: Does the language model receive the fused representations of all 32 sampled frames, including frames subsequently excluded from SGEB? If so, what evidence distinguishes improved auxiliary retrieval from preservation of information that would otherwise become unavailable? Please clarify the scope of the “compact evidence bottleneck” interpretation.
  3. Interpretation of the capacity experiment: Were the K=4,8,16,24,32 models separately trained under the same training protocol, or was a model trained at one capacity evaluated at different capacities? How was K=8 selected, and was the selection made on a validation set separate from the reported evaluation set? These details are necessary to interpret the degradation at larger capacities as a property of evidence selection.
3 comments
AuthorsOpenReview ↗

Response - 1

Thank you for the careful review and the analytical questions regarding selection versus representation and the role of SGEB's capacity under the residual formulation. We address each point below.

1. Contribution of Geometry Specifically to Evidence Selection (W1 & Q1)

To examine whether camera-conditioned geometry contributes to what SGEB retains, beyond improving the representations used for answering, we ran the comparison suggested in Q1. Using our trained Q-GeoMem model, we kept the stored SGEB tokens eu=Pool⁡(fh,u)e_u = \operatorname{Pool}(f_{h,u}), calibrated relevance scores rur_u, and the retrieval, fusion, and MLLM architecture unchanged, comparing selection novelty computed from pre-CGGF 2D visual features Pool⁡(fv,u)\operatorname{Pool}(f_{v,u}) versus geometry-fused features Pool⁡(fh,u)\operatorname{Pool}(f_{h,u}):

Novelty Features for Memory Selection (Stored eue_u & downstream model fixed)VSI Avg.Obj. Cnt.Abs. Dist.Rel. Dist.Appr. OrderRetained 3D Pose Spread$^\ddagger$
Visual-Only (Pool⁡(fv,u)\operatorname{Pool}(f_{v,u}))71.074.359.771.774.00.341
Geometry-Enriched (Pool⁡(fh,u)\operatorname{Pool}(f_{h,u}), Ours)71.274.859.971.174.60.373

‡^\ddaggerMean pairwise camera-token dissimilarity among the K=8K=8 retained SGEB entries across VSI-Bench.

  1. Positive selection effect with modest overall magnitude: With stored tokens eue_u and downstream weights fixed, geometry-enriched novelty increases the 3D camera-pose diversity of retained entries (0.341 to 0.373) and improves tasks sensitive to instance counting and cross-view ordering (+0.5 on Obj. Cnt., +0.6 on Appr. Order, and +0.2 on VSI Avg. across all eight categories), with a slight trade-off on Rel. Dist..
  2. Same-object views vs. similar-looking instances, and refining our claims: In Eq. (10), cosine similarity is computed after averaging each frame into a single global vector (1×d1 \times d), and Eq. (11) multiplies the resulting novelty by relevance rur_u. Consequently, while 2D visual features already filter most coarse frame redundancy, frame-pooled similarity limits explicit instance-level correspondence between different views of the same object and different instances of similar-looking objects (consistent with the modest +0.2 overall and +0.5 Obj. Cnt. selection gain, a limitation of frame-pooled novelty that we will explicitly note in Sec. 5). At the representation and readout level, CGGF strengthens the per-frame features fh,tf_{h,t} (64.83 to 67.28 in Table 4) using camera-conditioned geometry cues from the CUT3R decoder, and each stored SGEB entry eu∈R49×de_u \in \mathbb{R}^{49 \times d} retains 49 local tokens pooled on a 7×77 \times 7 grid for token-level cross-attention in Eq. (12), rather than compressing stored content to 1×d1 \times d. We agree with W1 on aligning our claims with this evidence, and we will refine the Abstract, Sec. 1, and Sec. 3.3.2 to describe SGEB as question-guided relevance-novelty selection over camera-conditioned geometric representations without overstating Eq. (10) as an instance-level geometric filter.
  3. Conceptual advance beyond combining standard components: Within this refined scope, Q-GeoMem's memory mechanism still goes beyond applying a generic relevance-diversity heuristic to geometry-enriched features. Rather than storing a fixed write-time score, SGEB dynamically recomputes active-set utility across current entries before reading and across the candidate set during replacement (Sec. 3.3.2 and App. A.3), and couples the same utility to both operations. Holding eue_u and the architecture fixed, using wuw_u jointly for replacement and read bias reaches 71.2, outperforming FIFO (68.7), Read-bias only (69.2), and Write-replacement only (70.2, Appendix Table 7), as well as single-criterion policies (69.3 for QRel-only and 68.9 for Novelty-only, Table 6a) and FGCB without Camera-Δ\Delta modulation (+1.35 in Table 5). We will add the Q1 diagnostic and this discussion to Sec. 4.4.
AuthorsOpenReview ↗

Response - 2

2. Information Available to the LLM & Scope of "Compact Evidence Bottleneck" (W2 & Q2)

Does the language model receive all 32 sampled frames, and what is the scope of the bottleneck? Yes. Through the residual connection in Eq. (14), f~h,t=fh,t+gtF⊙RtF+gtS⊙RtS\tilde{f} _ {h,t} = f _ {h,t} + \mathbf{g} _ t^{\mathcal{F}} \odot R _ t^{\mathcal{F}} + \mathbf{g} _ t^{\mathcal{S}} \odot R _ t^{\mathcal{S}}, the fused representations f~h,t\tilde{f} _ {h,t} of all 32 sampled frames reach the MLLM. We agree with W2 that calling SGEB a "compact evidence bottleneck" in Sec. 4.4 without qualification could imply input-frame dropping. Exclusion from SGEB applies to the long-range auxiliary bank St−1\mathcal{S} _ {t-1} (8/32=25%8/32 = 25\% of frames retained for intermediate readout RtSR _ t^{\mathcal{S}}), functioning as a selective cross-temporal retrieval bottleneck, and we will revise Sec. 3.3.2 and Sec. 4.4 accordingly.

Evidence from existing ablations (Tables 4, 6b, and Appendix B.1 Table 7): Your point that K=8K=8 outperforms K=32K=32 via more selective intermediate retrieval and reduced interference is supported by our trained ablations:

Variant (All 32 frames passed to LLM via Eq. 14)Paper SourceSGEB Capacity KKHard EvictionRead Bias βw\beta wVSI Avg.
Baseline + CGGF + FGCB (w/o SGEB)Table 4N/AN/AN/A67.9
SGEB No-EvictionTable 6(b)32NoYes67.9
SGEB FIFOApp. B.1, Table 78FIFONo68.7
SGEB Read-Bias OnlyApp. B.1, Table 78FIFOYes69.2
SGEB Write-Replacement OnlyApp. B.1, Table 78Yes (ww)No70.2
Q-GeoMem (Full)Tables 4, 6(b), 78Yes (ww)Yes71.2
  1. Per-frame LLM input alone does not replace cross-view readout (Table 4): Although w/o SGEB feeds all 32 geometry-enriched frames to the MLLM, adding SGEB gains +3.3 overall and +6.2 on Appr. Order (68.45 to 74.60), indicating that causal LLM attention does not substitute for dedicated cross-view readout RtSR_t^{\mathcal{S}} before the MLLM.
  2. Soft read biasing without hard eviction drops back to w/o SGEB (Table 6b): Retaining all frames at K=32K=32 while still applying the learned read bias βwu\beta w_u in Eq. (12) drops accuracy back to 67.9 (66.2 on VSI Long), as up to 31×49=1,51931 \times 49 = 1,519 uncurated keys dilute the cross-attention softmax versus 392 curated keys at K=8K=8.
  3. Hard replacement outweighs soft retrieval reweighting (Appendix B.1, Table 7): At K=8K=8, hard capacity replacement without read bias (Write-Replacement Only, +1.5 over FIFO) yields three times the gain of soft read biasing alone (Read-Bias Only, +0.5), and combining both reaches +2.5, showing that SGEB works primarily by filtering out redundant and irrelevant keys before intermediate retrieval.
AuthorsOpenReview ↗

Response - 3

3. Training Protocol of the Capacity Sweep & Non-Monotonicity Analysis (W2 & Q3)

  1. Separate Training Protocol: The five capacity models in Table 6(b) (K∈{4,8,16,24,32}K \in \{4, 8, 16, 24, 32\}), as well as the ablations in Tables 4 through 7, were separately trained under the same protocol, varying KK with consistent training data, initialization, and hyperparameters. Each model adapted its readout projections, β\beta, and gate gtS\mathbf{g} _ t^{\mathcal{S}} to its own capacity KK, avoiding train-to-test mismatch.
  2. Selection of K=8K=8: No separate validation set was used, since VSI-Bench provides no validation split. Instead, K=8K=8 was set upfront as a compact quarter-sequence default (T/4=8T/4 = 8, twice the local window τ=4\tau = 4) and kept fixed across VSTI-Bench and the five OOD benchmarks (Table 3) without test-set tuning. Table 6(b) analyzes sensitivity around this default.
  3. Explaining the Non-Monotonic Results: Overall, accuracy rises from K=4K=4 (69.0) to K=8K=8 (71.2) and drops at K=32K=32 (67.9) because K=4K=4 retains too few views while K=32K=32 keeps all frames and dilutes cross-frame attention with redundant or irrelevant views. The smaller 1.1-point difference between K=16K=16 (68.4) and K=24K=24 (69.5) comes from normal training variation, since each capacity is trained separately and learns its own eviction pattern and fusion gate gtS\mathbf{g} _ t^{\mathcal{S}}. Importantly, every capacity with selective eviction (K=4,8,16,24K = 4, 8, 16, 24, scoring 68.4 to 71.2 overall and 66.9 to 70.1 on VSI Long) outperforms keeping all frames at K=32K = 32 (67.9 overall, 66.2 on Long). We will add these training details and this explanation to Sec. 4.3 and Sec. 4.4.
Reviewer_guLXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Q-GeoMem, a memory architecture for video spatial reasoning that combines camera-conditioned geometry fusion with two complementary stores: a Fine-Grained Context Bank for recent dense features and a Semantic-Geometric Evidence Bank for compact long-range evidence. The latter scores entries using the product of question relevance and novelty relative to the active bank, and uses this utility for both retrieval bias and capacity-constrained replacement. On VSI-Bench and VSTI-Bench, the method improves over spatial and memory-oriented baselines, while component, policy, capacity, and video-length ablations provide evidence for the proposed design.

Critical strengths
  1. This work is well motivated: retaining evidence that is useful for a particular spatial question rather than treating memory as a generic chronological cache. The distinction between recent dense context and compact long-range evidence is well motivated.

  2. Methodologically, the presentation is precise. The read-before-write procedure, active-set-dependent novelty, relevance calibration, replacement rule, and adaptive fusion are specified mathematically, and the appendix supplies useful training and calibration details.

  3. Across both benchmarks, the main results are strong. Q-GeoMem reaches 71.2 average accuracy on VSI-Bench and 67.8 on VSTI-Bench, improving over the closest reported memory baseline by 2.4 and 2.5 points, respectively.

  4. Crucially, the ablation study tests the proposed mechanism rather than merely documenting final accuracy. The progressive component ablation, camera-modulated read ablation, relevance-versus-novelty policy comparison, capacity sweep, and separation

Critical weaknesses
  1. Relevance calibration at the video level appears to require the mean and standard deviation of question-frame similarities over all sampled frames. This makes the SGEB policy dependent on future observations and therefore unsuitable for causal streaming without modification. That dependency is important because the method is presented as a sequential memory system, yet its practical boundary is not emphasized until the limitations discussion.

  2. Transfer claims about broad out-of-distribution strength are not fully established. The five OOD results compare only Q-GeoMem with and without SGEB, rather than against external methods under matched settings, so they demonstrate the incremental value of SGEB but not broader comparative transfer performance.

  3. Efficiency evidence is missing for inference cost, memory footprint, and latency relative to FIFO, the base model, and VLM2. This omission matters because Q-GeoMem adds a frozen spatial tower, two memory reads, pairwise novelty computations, and adaptive fusion, while compactness is one of the central motivations.

Questions for authors
  1. At test time, is video-level calibration computed after observing the complete set of sampled frames, and if so, how would relevance be calibrated in a strictly causal or streaming setting?

  2. What are the inference-time latency, peak memory, and additional parameter or FLOP costs of CGGF, FGCB, and SGEB relative to the fine-tuned backbone, FIFO memory, and VLM2?

  3. Were the reported scores obtained from one training run, and what is the variance across seeds for the full model and the closest ablations, especially the 70.2 versus 71.2 comparison in the utility-usage study?

  4. How were training data, sampled-frame budgets, backbone initialization, and evaluation prompts controlled for the VLM2 and VLM-3R comparisons?

5 comments
AuthorsOpenReview ↗

Response - 1

Thank you for your feedback and the detailed questions on streaming calibration, external OOD comparisons, inference efficiency, and statistical controls. We address each point below.

1. Video-Level Calibration vs. Strictly Causal / Streaming Operation (W1 & Q1)

In our reported 32-frame benchmark evaluations, Calib(·) in Eq. (9) and Sec. A.2 computes μρ\mu _ \rho and σρ\sigma _ \rho across the sampled video frames before replaying sequential memory reads and updates. We agree that this boundary should be stated upfront when introducing Eq. (9) in Sec. 3.3.2 rather than deferred to Sec. 5, and we will revise Sec. 3.3.2 accordingly.

Adapting SGEB's calibration and memory update to strictly causal streaming (with zero access to future frames) requires no architectural changes or retraining for two reasons:

  1. Natural Fill Warm-up (∣Kt−1∣<K\lvert \mathcal{K} _ {t-1} \rvert < K): As defined in Sec. 3.3.2, incoming frames are appended unconditionally while ∣Kt−1∣<K\lvert \mathcal{K} _ {t-1} \rvert < K. For K=8K = 8, the first eviction decision occurs only at frame t=9t = 9, when 9 observations are already available to estimate running statistics. At t=1t = 1 the bank is empty, and at t=2t = 2 softmax over a single stored entry is invariant to the utility bias βw1\beta w _ 1.
  2. Causal Prefix & EMA Calibration: Storing each entry's raw similarity ρu=sim⁡(Q(fv,u),qˉ)\rho _ u = \operatorname{sim}(\mathcal{Q}(f _ {v,u}), \bar{q}) allows replacing full-video (μρ,σρ)(\mu _ \rho, \sigma _ \rho) at step tt with strictly causal prefix statistics (μρ,1:t,σρ,1:t)(\mu _ {\rho,1:t}, \sigma _ {\rho,1:t}) over frames {1,…,t}\lbrace 1, \dots, t \rbrace, or an O(1)O(1)-state exponential moving average (EMA) for unbounded streams:
Calibration ModeStrictly Causal?VSI Avg.VSI ShortVSI MidVSI Long
Offline Video-Level (Paper, Sec. A.2)No71.272.372.770.1
Causal Prefix (μ1:t,σ1:t\mu _ {1:t}, \sigma _ {1:t})Yes71.372.472.770.4
Causal EMA (Running O(1)O(1) state)Yes71.273.272.770.1

On the same trained checkpoint, both causal prefix (71.3) and running EMA (71.2) match offline video-level calibration (71.2), showing that SGEB's calibration and update rules do not rely on future frames.

AuthorsOpenReview ↗

Response - 2

2. Comparative OOD Transfer Against External Baselines (W2)

Beyond our internal SGEB ablation in Table 3, we compare against the pretrained backbone (LLaVA-NeXT-Video-7B) and VLM-3R-7B on three representative OOD benchmarks under the same lmms-eval protocol: multi-view 3D spatial reasoning (SPBench-MV, 8 native scene views), cross-view and motion spatial QA (MMSI-Video, 32 frames), and general video QA (VideoMME, 32 frames):

MethodSPBench-MVMMSI-VideoVideoMMEOOD Avg.
LLaVA-NeXT-Video-7B (Pretrained)42.4527.5563.7444.58
VLM-3R-7B (Fan et al., 2025)68.9627.5559.3051.94
Q-GeoMem (w/o SGEB)69.0323.9759.7750.92
Q-GeoMem (Ours)72.6125.4960.9653.02
  1. Strongest Overall OOD Transfer: Q-GeoMem achieves the highest OOD average (53.02), outperforming LLaVA-NeXT-Video-7B by +8.44 and VLM-3R-7B by +1.08, with clear gains over VLM-3R-7B on both SPBench-MV (+3.65) and VideoMME (+1.66).
  2. Domain Specialization Trade-off and Role of SGEB: Fine-tuning on indoor 3D scan datasets (VLM3R-Data and ViCA-322K) boosts multi-view 3D reasoning on SPBench-MV by +30.16 over the pretrained backbone, while introducing a modest domain-shift trade-off on open-domain VideoMME (63.74 to 60.96) and MMSI-Video (27.55 to 25.49), a pattern shared with VLM-3R-7B (59.30 on VideoMME). Notably, while Q-GeoMem (w/o SGEB) (50.92) slightly trails VLM-3R-7B (51.94), adding SGEB improves all three OOD benchmarks (+3.58 on SPBench-MV, +1.52 on MMSI-Video, and +1.19 on VideoMME). We will include these external OOD comparisons in Sec. 4.3.
AuthorsOpenReview ↗

Response - 3

3. Inference Efficiency: Latency, Memory Footprint, Parameters, and FLOPs (W3 & Q2)

We profiled end-to-end 32-frame video inference on a single NVIDIA A800-80GB GPU (BF16, batch size 1, precomputed feature caches disabled, averaged over 100 VSI-Bench videos after 10 warm-up runs):

Model / VariantTotal ParamsTrainable ParamsVision + Mem FLOPsPeak GPU Mem (GB)Total Latency (ms/video)Memory Module Latency (ms)VSI Avg.
LLaVA-NeXT-Video-7B ft. (No 3D / Mem)8.03B339.9M21.33T21.701020.8N/A64.8
VLM-3R-7B (Frozen CUT3R)8.83B349.7M70.31T23.252230.7N/A60.9
VLM2-7B\text{VLM}^2\text{-7B} (3D Encoder + Dual Mem)9.52B874.7M125.38T24.512030.4108.268.8
Baseline + CGGF8.95B474.0M74.94T23.492172.325.067.3
Baseline + CGGF + FGCB (τ=4\tau=4)9.05B566.7M81.70T23.662277.096.567.9
Q-GeoMem w/ FIFO SGEB (K=8K=8)9.16B682.8M83.56T23.882282.8136.068.7
Q-GeoMem Full (K=8,M=7×7K=8, M=7\times7)10.35B696.6M100.63T26.152490.7306.871.2

Note: For VLM2-7B\text{VLM}^2\text{-7B}, Memory Module Latency measures DualMemory (108.2 ms, excluding 8.2 ms cross-attention fusion), whereas for Q-GeoMem variants it covers the full geometry-and-memory block (CGGF + FGCB + SGEB).

  1. Unchanged LLM Token Budget and Favorable Compute Scaling: All variants feed the same visual token count (32 × 196 = 6,272 tokens) to the 7B MLLM (~690 ms), and the latency increase over the 2D baseline (1020.8 ms) comes mainly from the frozen 3D encoder (frame-recurrent CUT3R at ~1158 ms vs. batch-parallel Pi3 at 938.4 ms in VLM2\text{VLM}^2). Compared to VLM2\text{VLM}^2, Q-GeoMem Full uses 20.4% fewer trainable parameters (696.6M vs. 874.7M, where VLM2\text{VLM}^2's DualMemory alone has 501.2M) and 19.7% fewer vision-and-memory FLOPs (100.63T vs. 125.38T), while improving VSI-Bench accuracy by +2.4 points (71.2 vs. 68.8).
  2. Overhead Over Matched FIFO Memory: Relative to Q-GeoMem w/ FIFO SGEB (2282.8 ms, 23.88 GB), Q-GeoMem Full adds +207.9 ms (+9.1%) latency, +2.27 GB peak memory, and +13.8M trainable parameters for a +2.5 gain on VSI-Bench (68.7 to 71.2). Most of this added latency and memory footprint comes from the frozen BLIP-2 ITC relevance scorer (138.4 ms), whereas SGEB's active-set utility recomputation and βwu\beta w _ u read biasing add only +31.0 ms over FIFO (72.7 ms vs. 41.7 ms).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.