acceptodds
Withdrawn from ICLR 2027

Remembering by Place: Persistent Visual Memory for Streaming Video Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Streaming video models promise always-on assistants that can later recall what they have seen, yet their memory is bounded by context: observations evicted from the KV cache are discarded rather than preserved as a queryable record. Existing memory agents for long video typically fill this gap by captioning incoming observations and storing text. We argue that this caption-first design is mismatched to visual-spatial experience: it commits visual evidence to language before the question is known, scales writer-LLM call with the observation stream, and does not link revisits to the same place across time. We introduce PlaceMem, a query-blind, training-free persistent memory agent that serves as the backend of a streaming video model. Taking the surprise-selected observations of Cambrian-S as its front end, PlaceMem organizes them into persistent place records using visual place and semantic features, and derives language from accumulated place evidence rather than from each observation. A tool-using reader retrieves across both the textual and visual records. On VSI-Super Recall, PlaceMem improves average accuracy on 10–240-minute streams by 31.7 points over Cambrian-S alone, even with a lightweight 4B reader, and writing language per place rather than per observation cuts writer calls by 38% at competitive accuracy. On VSI-Super Count, it improves Cambrian-S's MRA from 40.7 to 48.2. These results suggest that caption-first memory need not be the default for visual-spatial memory: captions remain strong for semantic recall, but visual features and place organization supply grounding and cross-observation association that text alone does not encode.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long-Video Understanding, Agentic Memory, Visuospatial Memory, Visual Retrieval, Visual Place Recognition

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CFgzOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the challenge of building persistent visual-spatial memory for streaming video agents. Recognizing that "caption-first" text memories fail to encode cross-temporal spatial associations and incur high LLM generation costs, the authors propose PLACE-MEM, a training-free, query-blind memory backend. Using Cambrian-S as a front-end to select surprising keyframes/segments, PLACE-MEM organizes observations into persistent "place" records using visual place recognition (SALAD) and region features (SigLIP2). Language summaries are generated per-place rather than per-observation, reducing writer LLM calls. At query time, a tool-using reader retrieves visual and textual evidence. Evaluated on the VSI-Super benchmark (10–240 minute streams), PLACE-MEM improves Recall accuracy from 41.3% to 73.0% over the front-end alone, outperforms caption-first baselines, and reduces writer calls by 38%. On the Count task, place-based deduplication improves Mean Relative Accuracy (MRA) from 40.7 to 48.2 by preventing double-counting of revisited scenes. The evaluation is conducted via simulated streaming (sequential processing of pre-recorded videos), without real-time hardware latency measurements.

Critical strengths

This paper introduces a Place-Organized Persistent Visual-Spatial Memory Paradigm. It overcomes the limitations of the "caption-first" flat text memory prevalent in traditional long-video agents. Drawing on the concept of "cognitive maps" from cognitive science, this paper leverages visual place recognition (SALAD) and regional semantic features (SigLIP2) to organize streaming observations across time into persistent "place" records. This structured memory representation not only preserves visual grounding information but also endows the system with the ability to associate the same physical space across time, effectively addressing spatial reasoning challenges—such as deduplicating repeated observations (e.g., in Counting tasks)—that are intractable for purely text-based memories.

Critical weaknesses

W1. Missing wall-clock latency and throughput measurements. The paper frames PLACE-MEM as a "streaming video agent" and "always-on assistant" (Abstract, Sec 1). However, Appendix B explicitly notes that writer cost only counts LLM generation calls, "not tokens, FLOPs, or latency." There are no wall-clock measurements of processing throughput (e.g., milliseconds per packet) to prove the backend can keep pace with an incoming video stream, leaving the "real-time" capability claim unsubstantiated.

W2. Unbounded memory growth of the Region Prototype Bank. Section 3.3 describes the per-place region prototype bank PpP_p growing with the place's visual content, but does not specify if its size is bounded or if an eviction/merging policy exists. If PpP_p grows unboundedly over a 240-minute stream, nearest-neighbor retrieval latency and memory footprint will degrade continuously, contradicting the core premise of a bounded, persistent memory system.

W3. Front-end bottleneck and lack of error attribution. PLACE-MEM relies entirely on Cambrian-S's prediction surprise to emit packets (Sec 3.2). If the front-end fails to select a keyframe containing a subtle but critical object (due to low prediction error), the backend cannot recover it. The paper lacks an error analysis distinguishing "front-end missed the event" from "backend failed to retrieve the event," making it difficult to assess the true headroom of the proposed memory mechanism.

W4. Simulated streaming vs. deployed real-time capability. The evaluation processes pre-recorded videos sequentially (simulated streaming) rather than deploying the system on a live camera feed. Without measuring the end-to-end pipeline latency—including SALAD encoding, SigLIP2 extraction, bank updates, and LLM writer calls—it is impossible to verify whether the system can operate as a genuine "always-on" agent without dropping frames or lagging behind the video stream.

Questions for authors
  1. Can you report the average wall-clock time to process a single Cambrian-S observation packet (including SALAD/SigLIP2 encoding, bank updates, and writer calls)? Additionally, what is the peak memory footprint (RAM/VRAM) of the place bank and region prototype banks after processing a 240-minute stream?

  2. In Section 3.3, is the size of the per-place region prototype bank PpP_p bounded? If a place is observed repeatedly over hours, does PpP_p grow indefinitely, and if so, how does this affect nearest-neighbor retrieval latency? If there is a merging or eviction strategy, please describe it.

  3. What percentage of PLACE-MEM's incorrect answers in the Recall task are due to the Cambrian-S front-end failing to emit the relevant keyframe (i.e., the evidence was never sent to the backend), versus the backend failing to retrieve it despite it being in the bank?

  4. In Section 3.5, does the place-matching or segment-filtering process use the query (the object category) in any way, or is the segment aggregation strictly query-blind until the final counting prompt?

  5. Figure 3a shows that Place (SALAD) matching is more robust to threshold variations than Semantic (SigLIP2) matching. Could you clarify if the "Semantic" baseline in Fig 3a uses global CLS tokens or region-level features, and whether combining them (the OR-configuration mentioned in Appendix D.3) yields any statistically significant improvement over Place-only on the full dataset?

Reviewer_8L7WOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces PlaceMem, a training-free memory system for streaming video agents that addresses the limitations of context-bounded KV caches and caption-based memory. Instead of independently captioning every observation, PlaceMem organizes surprise-selected video frames into persistent place records using visual place recognition and semantic features, enabling cross-observation association and retrieval from both visual and textual memory. A tool-using reader then answers queries based on the retrieved evidence. Experiments on VSI-Super show the benefits over Cambrian-S. The key insight is that persistent visual-spatial memory complements textual summaries, providing stronger grounding and cross-time associations for long-term video understanding.

Critical strengths
  1. This paper organizes streaming observations into persistent place records using visual place recognition and semantic features. This is the first of its kind for streaming video understanding.

  2. The experiments on VSI-Super demonstrate promising gains in both performance and computational efficiency

Critical weaknesses

W1. The contribution of place organization itself is not isolated, and the recall evidence for it is statistically weak.

(a) Most of the headline gain comes from adding any external memory. The abstract leads with +31.7 points over Cambrian-S. However, Caption-only on the same frames already reaches 67.7%, so roughly 26 of those 31.7 points come from adding any external memory plus a reader. The authors acknowledge this in the Limitations section. The gain attributable to the paper’s thesis is PlaceMem K=1 vs. Caption+Image: 73.0 vs. 70.0 on average.

(b) The ablations point to region features rather than place structure. PlaceMem differs from Caption+Image simultaneously in prompt and output format (summary plus object inventory), region prototypes, and retrieval tools.

  • Adding place tools to the caption readout gives no gain (71.7 vs. 71.7).
  • Reranking caption–image candidates with 6×6 region features and no places matches full PlaceMem (78.3).

The missing control is PlaceMem with place association disabled, i.e., every observation as its own place with the same writer prompt and tools.

(c) The efficiency claim conflicts with streaming use. K=∞K=∞ defers all language until the stream has ended, so it is effectively offline post-processing and does not fit the always-on streaming setting. With the 4B reader, K=∞K=∞ also falls below the Caption+Image baseline on the longest streams (63.3/60.0 vs. 63.3/65.0 at 120/240 min).

W2. The experimental validation is too narrow and not rigorous enough to support general claims about visual-spatial memory.

(a) Single benchmark whose structure does not stress the proposed mechanism. All results are on VSI-Super. In Recall, the target object’s four occurrences cluster in a short window (Fig. 6: 82 s within a 240-min stream), so the task is largely needle retrieval with temporal ordering, and genuine revisits across long gaps rarely matter. A global SigLIP2 frame index already reaches 71.7% (Table 3b). Evaluating on benchmarks where revisits and long-range spatial association are essential (e.g., real continuous egocentric streams such as EgoLife or HourVideo, or other streaming suites) is necessary before concluding that place organization should replace caption-first memory.

(b) No statistical rigor and test-set tuning.With 60 questions per Recall duration and 50 per Count duration, most reported margins correspond to a handful of questions (e.g., the 3.0-point average gap over Caption+Image is 9 of 300 questions), yet no seeds, confidence intervals, or paired tests are reported. Matching thresholds were selected on the 10-minute split, which is also evaluated and used for all ablations, with no held-out validation.

(c) Efficiency is claimed but not measured. Cost is counted only as writer calls. No token counts, wall-clock latency, encoder overhead, memory growth over 240-minute streams, or per-query reader cost are reported. The only setting that reduces calls (K=∞K=∞) defers writing until the stream ends, so the real-time feasibility of the system in a streaming setting is not demonstrated.

Questions for authors
  1. Evaluate on another front end and on real continuous egocentric streams (e.g., RVS-Ego, UCS-Bench tasks). In VSI-Super, recall targets are inserted objects whose occurrences cluster in a short window (Fig. 6: 82 s within a 240-min stream). The task is therefore largely needle retrieval, where place structure is less essential.

  2. Please add the ablation “PlaceMem without place association” (each observation as its own place, with the same summary/inventory prompt, region prototypes, and tools). Does the 73.0 vs. 70.0 gap survive?

  3. Please report writer tokens, wall-clock time for memory construction (including SALAD/SigLIP2), memory size growth on 240-minute streams, and reader tool calls and latency per query. Is K=1 feasible in real time alongside Cambrian-S?

Reviewer_rBFiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces PLACEMEM, a training-free, query-blind persistent memory backend for streaming video models. Cambrian-S selects observations by prediction surprise; SALAD associates observations with place records, while SigLIP2 region features preserve visual evidence. A writer maintains place summaries and object inventories, and a tool-using reader retrieves textual and visual evidence. On VSI-Super Recall, the Qwen3-VL-4B configuration outperforms Cambrian-S and the matched Caption + Image backend.

Critical strengths
  1. The memory design is useful. PLACEMEM groups observations of the same place and keeps the original frames, timestamps, summaries and object lists together. With the same selected observations and writer, it improves recall over Caption + Image in Table 1. Place association also improves counting when segment inputs and counts are held fixed in Table 2.

  2. Informative design ablations. The experiments reveal limitations of place tools alone and prototype-only retrieval, as well as writer–reader interactions.

Critical weaknesses
  1. The advantage inherent to the location-based mechanism itself has not yet been isolated. In the 10-minute setting, the "caption–image + 6×6 reranking" approach performs on par with the full PLACEMEM system; however, this control condition is absent for the 120-minute and 240-minute settings.

  2. The contributions of count aggregation and non-adjacent revisits require clarification. The rules for multiple matching, bidirectional inclusion, and transitive grouping remain incomplete.

Questions for authors
  1. What differs between full PLACEMEM and caption–image +6×6 reranking in stored text, retrieved evidence and reader budget?

  2. When a segment matches multiple groups, how does the method decide which counts to keep in the final total?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.