- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes VideoMem-Bench, a benchmark designed to evaluate genuine memory capabilities in long-video understanding. Inspired by cognitive neuroscience, the benchmark evaluates long-video understanding models across four levels of memory. It introduces decoupled, question-agnostic evaluation settings in which models compress visual information from videos into memory without access to the query set and subsequently answer questions using only the constructed memory.
Critical strengths
- The paper points out that existing long-video understanding benchmarks allow query-dependent memory construction, making it difficult to distinguish whether a model genuinely memorizes video content or merely constructs a query-dependent cache.
- The benchmark is structured around four cognitive levels inspired by neuroscience and focuses specifically on memory capabilities. A semi-automatic construction pipeline also enables the creation of a large-scale benchmark.
Critical weaknesses
- The query dependency of video memory construction and the formulation in Eq. (1) appear to be issues that should be controlled by the evaluation protocol rather than by the benchmark itself. Even with existing benchmarks, it should be possible to prevent models from accessing queries during memory construction and allow query access only at answer generation time. Therefore, I question the central claim that VideoMem-Bench uniquely enables decoupled evaluation.
- The models used in the construction pipeline and the evaluation appear outdated. In particular, the pipeline relies on models from 2024. While Memory Banks (BIMBA) achieves the highest performance in Table 3, more recent approaches, such as FlexMem [1] and Flash-VStream [2], should also be evaluated to better characterize the limitations of current methods.
- I find the hallucination example in Figure 3 difficult to interpret. Based on the figure, actors and objects appear to be visible, so it is unclear why option C should be considered the correct answer. A more interpretable example would help support the claim that hallucination evaluation is one of distinctive features of the proposed benchmark.
[1] Chen, Tao, et al., "Scaling the long video understanding of multimodal large language models via visual memory mechanism," CVPR 2026
[2] Zhang, Haoji, et al., "Flash-VStream: Efficient real-time understanding for long video streams," ICCV 2025
Questions for authors
I would appreciate clarification on the points raised in the Critical Weaknesses section, particularly regarding the necessity of a new benchmark for decoupled evaluation, the coverage of recent video memory models, and the interpretation of the hallucination example in Figure 3.