- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies spatial understanding from long videos under a fixed memory budget. The paper claims that a frame can be redundant for geometry estimation yet still hold content that later questions may need. GeoMem therefore keeps two separate KV memories. One stores geometry-encoder states and keeps frames by geometric coverage. The other stores language model states and drops a token only when it matches a retained token geometrically and visually. It reports state-of-the-art results on VSI-Bench and VSI-SUPER, with stable memory and per-frame time on a four-hour video.
Critical strengths
-
The paper separates the history a geometry encoder needs from the history a language model needs, and it keeps them under independent budgets. The design is intuitively reasonable.
-
The method adds only small peak memory and time per frame, and both stay flat over a four-hour video. This supports long-video understanding under a fixed budget.
Critical weaknesses
-
The paper does not convincingly show that geometry is what makes the memory work. The method is presented as geometry-aware, but the evidence is confusing. On VSI-Bench, GeoMem-7B beats the strongest baselines by only about 1-2 points, and the models use different backbones. GeoMem's clearest advantages are on counting, object size, and appearance order, which are largely semantic or temporal tasks. It does not lead on absolute distance, relative direction, or route planning, where geometry should help the most.
-
GeoMem's accuracy barely drops as videos get longer. But the episodic cache is capped at 16,384 tokens, while a four-hour video sampled at 1 FPS has 14,400 frames, so the cache can keep only a small fraction of what the model has seen. The paper does not show what is actually kept at these lengths or why that is enough to answer the questions. One possible explanation is simply recency. If the frames needed to answer happen to be near the end of the video, a small cache of recent content would give the same flat curve without any long-term memory. The benchmarks are also small, with 60 questions per recall duration and 50 per counting duration, so gaps of a few points amount to only a few questions. The headline long-horizon results are difficult to attribute to the proposed design.
Questions for authors
For the long VSI-SUPER videos, how much of the video does the episodic cache actually cover at 120 and 240 minutes (for example, tokens per frame and the time span kept)? Where in the video does the evidence needed for each question lie, and how does accuracy change when it is split by that position?
The gains are strongest on counting, object size and appearance order, and weaker on relative direction and route planning. Is this expected? What does it imply about where geometry helps?
2 comments
Response to Reviewer YwmM (2/2)
YwmM-Q1-1
For the long VSI-SUPER videos, how much of the video does the episodic cache actually cover at 120 and 240 minutes (for example, tokens per frame and the time span kept)?
At both durations, EM retains an average of 182.04 visual tokens per source frame represented in the cache, counting only frames with retained entries. The mean earliest and latest source timestamps are 118.50 and 119.97 minutes for the 120-minute setting, and 238.50 and 239.97 minutes for the 240-minute setting.
These timestamps span approximately 1.47 minutes on average at both durations, indicating that the retained entries originate mainly from the recent portion of each video.
YwmM-Q1-2
Where in the video does the evidence needed for each question lie, and how does accuracy change when it is split by that position?
The available VSR annotations do not include event timestamps, and the questions concern the order of multiple appearances. We therefore assess the full system’s dependence on earlier observations through controlled input windows.
Empirically, we compare three input windows on the same 60 VSR-240 questions, reusing the Full 240 min and Last 10 min results from W2-1. Each setting starts with empty memories and reset streaming states, while keeping the checkpoint, sampling rate, memory budgets, reading policy, and decoding settings fixed.
| Video Input | VSR-240 Acc. |
|---|---|
| Full 240 min | 43.3 |
| Last 120 min only | 31.7 |
| Last 10 min only | 26.7 |
Accuracy decreases when the first 120 minutes are omitted and falls further when only the final 10 minutes remain. These results support the contribution of earlier input to the full system’s answers. This comparison does not identify individual questions’ evidence positions or provide accuracy grouped by those positions.
YwmM-Q2
The gains are strongest on counting, object size and appearance order, and weaker on relative direction and route planning. Is this expected? What does it imply about where geometry helps?
GeoMem uses geometry to improve the encoding and retention of historical evidence. Geometric correspondence helps distinguish repeated observations from visually similar content at different locations, which can benefit counting and appearance order. Object size also involves geometric estimation. Our memory updates do not introduce a dedicated procedure for direction reasoning or route planning, so gains need not be uniform across tasks.
Empirically, the controlled comparison in W1 shows larger gains for counting and appearance order, while direction and planning also improve. Both settings retain geometry fusion and VSM Coverage, so this comparison specifically isolates the contribution of geometric correspondence to EM retention.
We have clarified this interpretation in §4.4.
Thank you again for your constructive feedback. We have incorporated the indicated revisions into the paper and provided further analyses in this rebuttal.
Response to Reviewer YwmM (1/2)
We appreciate your recognition of our memory design, efficiency, and stable resource usage over long streams. We address your comments point by point below.
YwmM-W1
Thank you for asking us to clarify geometry’s contribution. We agree that comparisons across backbones alone do not isolate this contribution.
Empirically, we additionally evaluate the Feature only setting from Tab. 4 (original) on VSI-Bench. We fix the GeoMem-7B checkpoint, VSM Coverage, geometry fusion, memory budgets, and reading policy, isolating the contribution of geometric correspondence to EM deduplication.
Within GeoMem, geometric correspondence improves all eight subtasks, including absolute distance, relative direction, and route planning. These gains characterize its contribution to EM retention within the same model. Rankings against other systems also involve differences in backbone, training, and components. Fig. 3 (original) and Tab. 3 (original) further support the benefits of VSM Coverage.
We have added this comparison to §4.4 of the revised paper.
YwmM-W2-1
Good question. Empirically, we additionally evaluate GeoMem-7B using only the final 10 minutes of each VSR-240 video. We reset both memories and all other streaming states, then answer the same 60 questions. All settings below share the checkpoint, geometry fusion, VSM Coverage, memory budgets, and reading policy.
The input window comparison supports the contribution of earlier observations to the full system. The comparison with EM FIFO separately demonstrates the benefit of our EM update strategy with the VSM policy fixed. These experiments do not isolate the respective contributions of EM and VSM to the benefit of earlier input. We report the actual cache coverage in Q1-1 and extend the same input window comparison in Q1-2.
We have added the comparison to §4.3.
YwmM-W2-2
We agree that the limited number of questions warrants caution when interpreting small score differences. For example, on VSR-240, our method answers 26 of 60 questions correctly, compared with 21 of 60 for EM FIFO.
The controlled comparisons in W1 and W2-1 support the contributions of geometric correspondence and the EM update policy, respectively. Q1-2 examines the full system’s dependence on earlier input. Together with the component ablations in Tab. 3 (original), these analyses help explain the observed gains, but the uncertainty associated with the small VSI-SUPER evaluation sets remains.