acceptodds
Under review as a conference paper at ICLR 2027

Visual Agentic Memory: Enabling Online Long Video Understanding via Online Indexing, Hierarchical Memory, and Agentic Retrieval

10 views

Abstract

Long video understanding requires more than large context windows. It also needs a memory mechanism that decides what visual evidence to retain, keeps it searchable over long horizons, and grounds later reasoning in recoverable observations rather than compressed latent state alone. We propose Visual Agentic Memory (VAM), a training-free framework with three components. Online Indexing supports selective evidence retention under streaming constraints. Hierarchical Memory organises retained evidence in a parallel temporal-spatial representation, with temporal event summaries for long-horizon localisation and spatial observations for direct visual verification. Agentic Retrieval searches, inspects, and verifies candidate evidence before producing a grounded answer. On OVO-Bench, VAM reaches a competitive RT+BT average of 68.41, improving over the 67.46 end-to-end Gemini 3 Flash baseline and ranking fourth in the updated comparison. On the month-scale split of MM-Lifelong train@month (105.6 hours over 51 days), VAM reaches 17.11%, second only to ReMA with GPT-5 (17.62%). These results suggest that long-horizon video understanding benefits from treating visual memory as an explicit, inspectable, and queryable substrate. Code will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long-horizon Video Understanding, Visual Agentic Memory, Hierarchical Memory, Agentic Retrieval, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VgmVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Visual Agentic Memory (VAM), a training-free framework for online long-video understanding. VAM consists of three components: Online Indexing, which selectively retains frames using visual filtering and adaptive similarity thresholds; Hierarchical Memory, which organizes temporal event summaries and retrievable visual observations; and Agentic Retrieval, which iteratively searches and inspects relevant evidence to answer questions.

Critical strengths
  • The paper addresses an important challenge in online long-video understanding, particularly maintaining searchable visual evidence over extended temporal horizons.
  • The combination of online indexing, hierarchical memory, and agentic retrieval is intuitive and reasonably well presented.
  • The experiments include both online video understanding and month-scale memory retrieval, demonstrating the potential practical applicability of the proposed system.
Critical weaknesses
  • The proposed framework primarily combines existing techniques for visual filtering, hierarchical memory construction, and agentic evidence retrieval. In particular, VideoAgent (ECCV 2024) already explores structured multimodal memory and tool-based video inspection; WorldMM (CVPR 2026) combines textual and visual memories with adaptive retrieval; and VideoARM (CVPR 2026) performs agentic reasoning over hierarchical video memory.
  • VAM's adaptive thresholding is closely related to the Otsu-based mechanisms explored in FluxMem (CVPR 2026). Although VAM integrates these components for online indexing and evidence preservation, the paper does not sufficiently establish a distinctive algorithmic contribution beyond their combination.
  • Despite introducing three major components, the paper does not provide sufficient controlled ablations to establish their individual contributions. Without these experiments, it is difficult to determine whether the reported performance originates from the proposed memory architecture or simply from the underlying multimodal models and repeated inference.
  • On MM-Lifelong, VAM achieves 17.11% accuracy, slightly below ReMA's 17.62%, while its Ref@300 is considerably lower (3.65% versus 9.91%). This raises concerns about the effectiveness of the proposed temporal localization mechanism.
  • The system introduces multiple processing stages, including visual embeddings, event summarization, hierarchical indexing, and iterative multimodal inference. However, the paper provides limited quantitative analysis of online indexing throughput, end-to-end query latency, model calls, storage overhead, and computational costs. Since efficient continuous operation is a central motivation, these measurements are important for establishing practical scalability
Questions for authors
  • What is the fundamental algorithmic contribution of VAM beyond combining established online filtering, hierarchical memory, and agentic retrieval techniques?
  • Could the authors report controlled ablations for Online Indexing, Hierarchical Memory, and Agentic Retrieval?
  • Why does VAM improve the RT+BT average while decreasing both Backward Tracing and overall accuracy relative to Gemini 3 Flash?
Reviewer_HYijOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Visual Agentic Memory (VAM), a training-free framework designed to handle long-video understanding and streaming scenarios. VAM consists of three main components. (1) Online Indexing, which performs frame filtering and adaptive moment/event boundary detection using an Otsu-style partition method on embedding distances. (2) Hierarchical Memory, which organizes retained evidence into parallel temporal and spatial representations with age-aware storage. (3) Agentic Retrieval, a multi-turn reasoning loop where a multimodal model plans, searches through both temporal and spatial representations, visually inspects candidate frames, and synthesizes grounded answers. The method is evaluated on OVO-Bench and the month-scale split of MM-Lifelong.

Critical strengths
  1. The paper is well-written, with clear pipeline descriptions, detailed mathematical formulations, and a comprehensive appendix covering prompt structures and implementation parameters.

  2. The proposed memory mechanism demonstrates impressive compression rates, retaining only a tiny fraction of the raw video frames in the month-long MM-Lifelong evaluation while maintaining competitive performance.

  3. The dual temporal-spatial representation and the explicit visual inspection step help mitigate hallucination and improve answer provenance by linking responses to verifiable raw frames.

Critical weaknesses
  1. The core technical components used in VAM, such as embedding-based indexing, adaptive thresholding, hierarchical memory tiers, and agentic multi-turn search-and-inspect loops, are largely built upon existing, well-established techniques. The paper lacks a fundamentally new algorithmic mechanism or insight, making the framework feel more like an engineering integration of prior paradigms rather than a conceptual breakthrough.

  2. The empirical evaluation is relatively thin, relying heavily on just two benchmarks (OVO-Bench and MM-Lifelong). Given the rapid expansion of long-video understanding and streaming literature, the evaluation would benefit significantly from a broader suite of diverse benchmarks (such as Video-MME, MLVU, or LongVideoBench) to robustly validate the generalizability and robustness of the proposed agentic memory across different tasks and domains.

  3. The paper lacks a thorough runtime efficiency or cost-vs-accuracy trade-off analysis compared to other recent streaming/memory baselines.

Questions for authors
  1. To elevate the contribution, the authors should clearly isolate what fundamental limitation of existing memory architectures is uniquely solved by VAM, rather than combining standard filtering, clustering, and agentic loops. A more distinct theoretical or architectural advancement is needed.

  2. Include comprehensive evaluations on a wider array of established long-video and streaming benchmarks to demonstrate that VAM’s performance advantages are general and not over-fitted to OVO-Bench and MM-Lifelong.

  3. Include detailed runtime, latency, and API/token cost comparisons against end-to-end and compressed memory baselines to better understand the practical trade-offs of the agentic retrieval loop.

Reviewer_hG96OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents Visual Agentic Memory (VAM), a training-free architecture that treats memory for long-video understanding as an online, evidence-preserving interface. Online indexing filters blurry and redundant frames and forms temporally bounded events; a hierarchical memory stores event summaries together with retained frames; and a bounded agentic loop searches, inspects, summarizes, or answers while preserving references to supporting evidence. Experiments on OVO-Bench and a month-scale MM-Lifelong stream show informative system-level gains while retaining only a small fraction of raw frames. However, the current evidence leaves open how much each component contributes under a matched serving budget and whether retrieved references are correct and cost-effective in deployment.

Critical strengths
  1. VAM defines a coherent online memory abstraction: frame filtering, temporal event summaries, spatial evidence retention, and bounded agentic retrieval are connected through an explicit schema. This makes the compression-to-evidence path inspectable.

  2. The paper evaluates both an online benchmark and a month-scale stream, and the appendix exposes retention tiers, schema constraints, and model-call defaults. These details make the proposed system easier to reproduce and analyze than an unconstrained agent prompt.

  3. The method is presented as training-free and strictly online, which is a relevant design point for continual video streams where future frames are unavailable at indexing time.

Critical weaknesses
  1. The evaluation does not yet isolate component contributions or establish a matched comparison (pp. 7–9, Tables 1–2). Table 1 includes an end-to-end Gemini 3 Flash baseline. Under that comparison, VAM changes RT+BT from 67.46 to 68.41, while Overall changes from 64.02 to 61.95 and BT from 62.47 to 57.89. Without component-level controls under the same serving and call budget—such as summary-only memory, retained-frame-only memory, temporal-only retrieval, and fixed-rate indexing—the effects of filtering, hierarchical storage, agentic inspection, the additional embedding/planner/inspector stack, and serving configuration remain entangled. On MM-Lifelong, VAM's Accuracy 17.11 is compared with ReMA+GPT-5 at 17.62, and ReMA's Ref@300 is 9.91 versus VAM's 3.65; the model and serving protocols are not matched. The visible Table 1 values also do not explain the abstract's description of 68.41 as “fourth” in an “updated comparison,” because the complete ranking set is not shown. These protocol and ranking details are needed to interpret the headline comparison.

  2. The grounded-retrieval and deployment claims need retrieval-level and systems-level measurements (pp. 6–9, Appendix B). The paper reports final QA results and the benchmark-level Ref@300 metric, but it does not report event/frame retrieval precision or recall, reference or citation correctness, direct temporal-localization measures beyond the benchmark metric, hallucination or unsupported-answer rates, or verification success. When best_ref is invalid, the fallback to the highest-scoring candidate produces a reference but does not by itself establish that the reference supports the answer. The reported 0.06% retained-frame fraction is a frame-count ratio, not a storage or serving-cost measurement. The submission does not report bytes after age-aware compression, embedding storage, indexing lag, measured per-query latency, actual API calls or token usage, throughput, or monetary cost. The limitations section acknowledges serving cost; quantitative measurements would show whether the evidence-preserving design also meets the paper's practical deployment objective.

Questions for authors
  1. What is the complete comparison set and ranking rule behind the abstract's statement that RT+BT 68.41 is “fourth”? Please reconcile this with the values visible in Table 1 and report RT, BT, Forward, RT+BT, and Overall for a matched Gemini baseline.

  2. Under the same Gemini backbone and serving setup, what are the results for no memory/no agent, summary-only memory, retained-frame-only memory, temporal-only retrieval, and fixed-rate indexing? Please report the same OVO sub-scores and MM-Lifelong metrics, together with the model-call budget for each condition.

  3. Can the authors report retrieval-level evidence quality: event/frame recall and precision, best_ref correctness, temporal localization (including Ref@300), unsupported-answer or hallucination rates, and the fraction of queries requiring fallback or forced termination? How should Ref@300=3.65 be interpreted alongside the grounded-memory claim?

  4. What are the actual retained storage bytes after the recent/mid/long retention tiers and JPEG compression, embedding storage, indexing lag, per-video and per-query API calls, input/output tokens, frame counts, latency, throughput, and cost? The 6,876-frame count alone does not establish a deployment advantage.

  5. The paper states that Stage-1 filtering, adaptive Otsu thresholding, event-boundary detection, and buffering are strictly online with no future-frame access. What is the actual buffering/commit delay in the implementation, and can the authors provide evidence from the code/configuration commit, together with model versions, temperatures, prompts, data split, and repeated-run uncertainty intervals?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.