acceptodds
Under review as a conference paper at ICLR 2027

VideoMem-Bench: Benchmarking Query-Agnostic Memory in Video Understanding

Abstract

While recent Video Large Language Models (Video-LLMs) show strong performance in processing long context windows, it remains unclear whether they genuinely memorize information or merely retrieve it from a large context cache. Existing benchmarks often overestimate actual memory capabilities because they rely on query-guided joint encoding, allowing models to retrieve video frames conditioned on the question. Furthermore, these benchmarks lack a structured cognitive hierarchy. To address these limitations, we introduce VideoMem-Bench, a benchmark designed to systematically evaluate genuine memory in the field of long video understanding. We establish a four-level hierarchical framework inspired by cognitive neuroscience, ranging from basic semantic perception to predictive causal simulation. Crucially, we propose a query-agnostic, decoupled evaluation paradigm. By enforcing an information bottleneck, this paradigm requires models to compress continuous video streams into a capacity-constrained memory state without prior knowledge of the queries. Models must then answer subsequent questions relying solely on this compressed memory representation. Comprising 3,433 videos and 5,360 annotated QA pairs, VideoMem-Bench prevents retrieval shortcuts and provides a rigorous standard for assessing autonomous long-term memory in dynamic environments.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Large Language Models, Long Video Understanding, Memory Evaluation, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YkVuOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes VideoMem-Bench, a benchmark designed to evaluate genuine memory capabilities in long-video understanding. Inspired by cognitive neuroscience, the benchmark evaluates long-video understanding models across four levels of memory. It introduces decoupled, question-agnostic evaluation settings in which models compress visual information from videos into memory without access to the query set and subsequently answer questions using only the constructed memory.

Critical strengths
  • The paper points out that existing long-video understanding benchmarks allow query-dependent memory construction, making it difficult to distinguish whether a model genuinely memorizes video content or merely constructs a query-dependent cache.
  • The benchmark is structured around four cognitive levels inspired by neuroscience and focuses specifically on memory capabilities. A semi-automatic construction pipeline also enables the creation of a large-scale benchmark.
Critical weaknesses
  • The query dependency of video memory construction and the formulation in Eq. (1) appear to be issues that should be controlled by the evaluation protocol rather than by the benchmark itself. Even with existing benchmarks, it should be possible to prevent models from accessing queries during memory construction and allow query access only at answer generation time. Therefore, I question the central claim that VideoMem-Bench uniquely enables decoupled evaluation.
  • The models used in the construction pipeline and the evaluation appear outdated. In particular, the pipeline relies on models from 2024. While Memory Banks (BIMBA) achieves the highest performance in Table 3, more recent approaches, such as FlexMem [1] and Flash-VStream [2], should also be evaluated to better characterize the limitations of current methods.
  • I find the hallucination example in Figure 3 difficult to interpret. Based on the figure, actors and objects appear to be visible, so it is unclear why option C should be considered the correct answer. A more interpretable example would help support the claim that hallucination evaluation is one of distinctive features of the proposed benchmark.

[1] Chen, Tao, et al., "Scaling the long video understanding of multimodal large language models via visual memory mechanism," CVPR 2026

[2] Zhang, Haoji, et al., "Flash-VStream: Efficient real-time understanding for long video streams," ICCV 2025

Questions for authors

I would appreciate clarification on the points raised in the Critical Weaknesses section, particularly regarding the necessity of a new benchmark for decoupled evaluation, the coverage of recent video memory models, and the interpretation of the hallucination example in Figure 3.

Reviewer_3hx2OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces VideoMem-Bench, a benchmark for evaluating long-video memory through a query-agnostic, decoupled evaluation protocol that separates memory construction from question answering. It comprises 3,433 videos and 5,360 QA pairs organized into four cognitive levels, ranging from basic perception to spatiotemporal reconstruction and causal reasoning.

Critical strengths

The cognitive science-inspired taxonomy is an interesting aspect of the benchmark, providing a structured framework for evaluating different levels of video memory and reasoning capabilities.

Critical weaknesses

Weakness 1. Limited Novelty and Insufficient Differentiation from Existing Benchmarks

The authors claim that “To the best of our knowledge, there lacks a benchmark specifically designed to systematically evaluate the memory capability of long-video models.” However, this claim appears to overlook relevant prior work. EgoLifeQA [1] already evaluates long-term memory and reasoning over past events and behavioral patterns using week-long video recordings, while EgoMemReason [2] introduces a benchmark specifically designed to systematically assess long-horizon video memory across entity, event, and behavior memory. Furthermore, MERIT [3] explicitly describes query-agnostic memory construction followed by retrieval-based inference as an established paradigm for ultra-long video understanding.

Importantly, this paper does not propose a new memory construction or reasoning method, but rather positions the benchmark itself as its primary contribution. Therefore, a clear distinction from existing benchmarks and a compelling justification for the proposed evaluation setting are particularly important. However, the main difference in the proposed decoupled evaluation appears to be an additional restriction that prohibits access to the original video during inference. The paper does not sufficiently establish why this restriction constitutes a fundamentally new evaluation paradigm, nor does it clearly demonstrate what memory capabilities the proposed benchmark can assess that existing benchmarks cannot.

Weakness 2. Video Duration Distribution Misaligned with the Goal of Long-Term Memory Evaluation

Although the paper aims to evaluate long-term video memory, 53.3% of the videos are shorter than one minute, and 60.3% are shorter than five minutes, whereas only 6.7% exceed one hour (Section 4.3). This contrasts with benchmarks such as EgoLifeQA [1], which leverage week-long video recordings to evaluate the ability to retain and connect information across temporally distant events, as well as EgoMemReason [2], whch explicitly targets long-horizon memory.

While memory-based reasoning can certainly be evaluated on shorter videos, it is questionable whether the current duration distribution adequately supports the paper's stated objective of distinguishing short-term visual information retention from genuine long-term memory. More importantly, the paper lacks an analysis of the temporal distance between the evidence segments required to answer each question, which is arguably more informative than the overall video duration for assessing long-term memory dependencies. Consequently, it remains unclear whether the proposed dataset sufficiently evaluates the retention, compression, and reconstruction of information over extended temporal horizons.

[1] Yang, Jingkang, et al. "Egolife: Towards egocentric life assistant." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.

[2] Wang, Ziyang, et al. "EgoMemReason: A Memory-Driven Reasoning Benchmark for Long-Horizon Egocentric Video Understanding." arXiv preprint arXiv:2605.09874 (2026).

[3] Choi, Yeeun, et al. "Keep it simple: Multi-key episodic memory retrieval for ultra-long video understanding." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2026.

Questions for authors

Could the authors clarify how the proposed query-agnostic, decoupled evaluation paradigm fundamentally differs from the established “query-agnostic memory construction followed by retrieval-based inference” paradigm described in MERIT [3]?

Reviewer_4oGaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VideoMem-Bench, containing 3,433 videos and 5,360 QA pairs, to evaluate video memory through a four-level cognitive taxonomy. Its evaluation separates query-agnostic memory construction from question answering, when it prohibits access to the original video. The paper evaluates several agent-based method and token-compression approaches under this eval setup.

Critical strengths
  1. The benchmark covers a broad range of video scenarios and tasks. Its progression from semantic perception and short-term events to spatiotemporal reconstruction and predictive reasoning provides a useful hierarchy for analyzing different capabilities.

  2. Query-agnostic memory construction is a meaningful evaluation direction, particularly for systems that must process video before knowing future user requests.

Critical weaknesses
  1. The paper insufficiently justifies prohibiting query-time video retrieval as a criterion for “genuine” video memory. Existing agents could reasonably combine query-agnostic memory with targeted retrieval. Disabling retrieval without documenting or evaluating prompt-level adaptation to the no-revisit setting may disadvantage these baselines. Controlled comparisons are needed to isolate this effect, for example, if the agent can access the video frame of fixed number would be meaningful to see.

  2. Baseline-specific sampling rates, frame limits, resolutions, and temporal coverage are unclear.

  3. The paper lacks a clear breakdown of videos by source, including additional unlabeled videos and synthetically generated QA pairs. The relationship between Figure 2’s QA-based distribution and the 3,433 unique videos also needs clarification.

Questions for authors
  1. Can the authors compare memory-only recall against the same query-agnostic memory with budgeted query-time video retrieval, using the same backbone and initial observations?

  2. “Beyond the per-query statistics in Table 4, what are the sampling rate, maximum frame count, resolution, and memory budget for each baseline? Can a comparison with matched visual input and memory budgets be provided, alongside construction and answering costs?

  3. How much do performance and architectural rankings change when representative methods use a stronger recent backbone (e.g. Qwen 3.5)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.