- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces EOV-Bench, an evidence-oriented benchmark for long-horizon streaming video understanding. It constructs questions from temporally grounded evidence and evaluates current perception, historical memory, and cross-temporal reasoning. The benchmark contains 4,337 open-ended questions and supports retention evaluation at different query delays. The authors also propose AsyncMemFlow, a training-free framework that asynchronously builds structured video memory for online question answering. Experiments show improvements over several existing streaming video methods.
Critical strengths
-
The evidence-first benchmark design is meaningful. By explicitly controlling evidence locality and distribution, EOV-Bench provides a more targeted evaluation of long-term memory and cross-temporal reasoning than conventional streaming video benchmarks.
-
The experimental evaluation is fairly comprehensive, covering several recent baselines, different query delays, and multiple memory configurations. The results reveal interesting differences between current perception and historical reasoning, and the ablations support the importance of memory description quality and organization.
Critical weaknesses
-
Insufficient validation of benchmark quality. The dataset relies heavily on automatically generated video descriptions and questions. Although the authors perform consistency checks and manually review answers, they do not report quantitative evaluations of question validity, evidence correctness, or human–judge agreement. Since the benchmark is the main contribution, these missing evaluations raise concerns about its reliability.
-
Limited novelty and evaluation of AsyncMemFlow. Asynchronous memory construction and hierarchical textual memory have been explored in prior works. The proposed framework mainly combines these existing ideas. Moreover, the reported response latency does not fully account for the cost of background memory construction, making the practical efficiency less clear.
Questions for authors
-
Were the supporting evidence intervals independently verified against the original videos? Can the authors provide a quantitative human evaluation of question validity and evidence correctness?
-
How well does the automatic judge agree with human evaluations, particularly for questions requiring cross-event comparisons or temporal reasoning?
-
Can the authors report the computational cost of background memory construction, including latency and memory consumption? How does this cost scale with video length?
Official Comment by Reviewer jNNj
Thanks for the detailed response. While some of my concerns are addressed, I still keep my score because the paper's main claims still rest on a benchmark whose validity is not confirmed.