acceptodds
Withdrawn from ICLR 2027

EOV-Bench: An Evidence-Oriented Benchmark for Long-Horizon Streaming Video Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Streaming video benchmarks are typically organized around predefined tasks, offering limited control over the evidence structure of questions. We introduce EOV-Bench, an evidence-oriented benchmark that constructs questions bottom-up from reusable, multi-scale video annotations. Evidence units are selected and composed before question formulation, controlling question type, temporal span, evidence distribution, and compositional demand—factors that shape difficulty. Reusing the annotation pool makes fuller use of long videos and yields diverse, explicitly grounded questions spanning current perception, historical retention, and cross-temporal reasoning. EOV-Bench contains 4,337 open-ended questions and supports matched retention analysis under a strict causal streaming protocol. Across evaluated strategies, high-fidelity recent frames favor current perception, whereas textual memory better preserves historical information over long time horizons. Ablations further show that long-term retention depends substantially on how observations are described and organized, not only on retrieval. These findings motivate AsyncMemFlow, a training-free framework that decouples memory construction from latency-sensitive answering. An independently configurable pipeline generates detailed descriptions and asynchronously consolidates them into structured memory. At query time, the answerer adaptively uses recent frames or the latest available memory, supplemented with observations not yet incorporated into it. This enables stronger memory construction without blocking online answering. Experiments demonstrate improved historical and joint reasoning while preserving strong current perception, with further gains on OVO-Bench.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Evidence-Oriented Benchmarking, Online Video Understanding, Video-Language Models, Long-Horizon Reasoning, Memory-Augmented Agents, Multimodal Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EBrqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EOV-Bench, an evidence-oriented benchmark for long-horizon streaming video understanding. It constructs questions from temporally grounded evidence and evaluates current perception, historical memory, and cross-temporal reasoning. The benchmark contains 4,337 open-ended questions and supports retention evaluation at different query delays. The authors also propose AsyncMemFlow, a training-free framework that asynchronously builds structured video memory for online question answering. Experiments show improvements over several existing streaming video methods.

Critical strengths
  1. The evidence-first benchmark design is meaningful. By explicitly controlling evidence locality and distribution, EOV-Bench provides a more targeted evaluation of long-term memory and cross-temporal reasoning than conventional streaming video benchmarks.

  2. The experimental evaluation is fairly comprehensive, covering several recent baselines, different query delays, and multiple memory configurations. The results reveal interesting differences between current perception and historical reasoning, and the ablations support the importance of memory description quality and organization.

Critical weaknesses
  1. Insufficient validation of benchmark quality. The dataset relies heavily on automatically generated video descriptions and questions. Although the authors perform consistency checks and manually review answers, they do not report quantitative evaluations of question validity, evidence correctness, or human–judge agreement. Since the benchmark is the main contribution, these missing evaluations raise concerns about its reliability.

  2. Limited novelty and evaluation of AsyncMemFlow. Asynchronous memory construction and hierarchical textual memory have been explored in prior works. The proposed framework mainly combines these existing ideas. Moreover, the reported response latency does not fully account for the cost of background memory construction, making the practical efficiency less clear.

Questions for authors
  1. Were the supporting evidence intervals independently verified against the original videos? Can the authors provide a quantitative human evaluation of question validity and evidence correctness?

  2. How well does the automatic judge agree with human evaluations, particularly for questions requiring cross-event comparisons or temporal reasoning?

  3. Can the authors report the computational cost of background memory construction, including latency and memory consumption? How does this cost scale with video length?

Reviewer_rXMaOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Most streaming-video benchmarks are organized by question type, but SimpleStream showed that looking at only the last four frames can match or beat systems designed to remember earlier video. That raises a concern: these benchmarks may not really test long-term memory. EOV-Bench addresses this by building questions around specific evidence from 49 videos, covering 54 hours in total. Questions test what is visible now, what happened earlier, or how the past relates to the present; the evidence may come from one stretch of video or several separated moments. The benchmark also tests how answers change as the relevant moment gets farther into the past, under a strictly causal streaming setup. The paper’s second contribution is AsyncMemFlow, a training-free system for handling that stream. While video arrives, a describer creates summaries of each completed minute, and an organizer groups them into a structured “Memory Map” of scenes and events. When a question comes in, the answer model decides whether it needs the latest raw frames, the Memory Map, or both. Memory is built in the background, so answering a question does not have to wait for it.

Critical strengths

The paper tackles a real problem in streaming video evaluation: high benchmark scores don’t show that a model actually keeps and uses information from earlier in the video. To address this, the authors propose EOV-Bench, a benchmark of 4,337 open-ended questions built from annotated evidence, and AsyncMemFlow, a training-free method that builds memory in the background while answering questions.

I like that every question is built from specific, annotated evidence intervals. The split into Current, Historical, and Joint questions (and Localized vs. Distributed evidence) also makes it easy to compare very different memory designs on equal terms. The evaluation protocol is realistic, since the video keeps playing while memory is being updated. Asking the same 1,217 historical questions at increasing delays is a clean way to measure retention. The results are interesting in their own right. Method rankings change depending on the question type, and just giving OASIS’s existing memory directly to the answer model improves its historical score by almost 8 points. The ablations are well designed. The authors change one component at a time (the description model, the Memory Map, the answer model size), which clearly shows that description quality and memory organization both matter. AsyncMemFlow beats all baselines on historical questions at every delay while staying close to raw frames on current questions, and this holds even with a fully local 8B setup. The paper is also very transparent about its prompts, scoring, and evaluation details.

Critical weaknesses

The paper's main ideas are useful, but several issues make it hard to tell how much of the results reflect real memory ability and how much reflect choices in how the benchmark and method were built. The most important concerns are below.

  1. The benchmark questions are built from VLM-generated video descriptions, but the paper doesn't say which model wrote these descriptions; it only names Codex as the agent that turns them into questions. Since AsyncMemFlow also relies on text descriptions of 1-minute blocks, questions made from captions may favor the kind of facts that captioning models tend to write down. The paper doesn't measure this possible bias, and a set of questions written independently from the video would be a good way to test it.

  2. The paper notes that recent benchmarks annotate supporting evidence, but it misses several benchmarks that are directly relevant to its "evidence-first" claim. Suitable subsets of these could also be adapted to the same causal streaming setup:

  • HERBench builds every question to require at least three separate pieces of evidence from different parts of the video, which directly challenges the claim that controlling evidence before writing the question is new.
  • NExT-GQA adds start and end times for the evidence behind each answer and studies cases where models answer correctly without relying on the right evidence, which is a basic reference for evidence-grounded evaluation.
  • ReXTime builds questions where the event asked about and the answer sit in different parts of the video, which is closely related to the paper's questions that combine evidence across separate time intervals.
  • CG-Bench (ICLR 2025) has over 12,000 questions on 1,219 long videos with human-annotated clue intervals, and its human-written questions would help address the concern about questions coming from generated captions.
  1. The same paper introduces both the benchmark and the method, and the method is motivated by the benchmark's findings. Testing outside EOV-Bench is limited to two parts of OVO-Bench (Perception and Backward) with one fully local 8B configuration. More external evaluation, for example on StreamArena or RIVER, would show how well the method's gains carry over to other settings.

  2. The paper reviews every answer by hand and calibrates the judge on a separate batch, but it doesn't report a measured error rate for the questions or how often the judge agrees with humans. Categories are weighted equally in the averages, and no confidence intervals are given. Per-category question counts are also not reported. The authors should report these counts so readers can judge how reliable small differences are.

  3. The router classifies questions very accurately (99.5% for the 8B model), and the example questions in Table 6 contain explicit time cues such as "right now" and "the preceding five minutes". The paper doesn't test whether routing still works when questions are reworded without these cues. So it's unclear how well the routing, and the preserved current-perception scores, would hold up with differently phrased questions.

  4. For historical and combined questions, the full Memory Map is placed in the model's context. The paper doesn't report how large the Map grows or how the method behaves on streams longer than the evaluated videos, which average about 66 minutes. Response times were also measured in isolation, not while background description and organization were running.

  5. Almost every model in the experiments comes from the Qwen family: the answer models, both describers, the judge, and most baseline backbones. The paper varies model size but not model family, so it’s unclear whether AsyncMemFlow’s gains, and the benchmark’s own findings, hold for other models such as InternVL, LLaVA-Video or Gemini. Using a Qwen judge to score mostly Qwen-based systems also adds a risk of same-family bias that isn’t checked; a second judge from a different family on a subset would address this.

Questions for authors
  1. Which model generated the 5-second, 1-minute and 5-minute descriptions used to build the benchmark? Is it the same model, or the same family, as the Qwen-VL-Max describer used in AsyncMemFlow?
  2. In the human review of canonical and alternative answers, did reviewers check answers against the video itself, or against the generated descriptions? Do you have any estimate of the error rate in the final questions?
Reviewer_jNNjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents EOV-Bench, a streaming video QA benchmark with 4.3k open-ended questions over 49 videos (about 54 hours). Questions are built "evidence-first":

  • Videos are described at three time scales.
  • An LLM agent picks evidence intervals and writes a question from them.
  • Questions are labeled Current, Historical or Joint, and Localized or Distributed.

Evaluation is strictly causal. Historical questions are re-asked at delays from 30 seconds to 32 minutes. The authors also propose AsyncMemFlow, a training-free system:

  • A background model (Qwen-VL-Max) writes descriptions of each 1-minute block and organizes them into a hierarchical "Memory Map".
  • A small answer model routes Current questions to recent frames, and Historical or Joint questions to the Map.
Critical strengths
  • The matched-retention protocol is a nice idea: the same question, evidence and answer are asked at increasing delays. The strict causal evaluation is also more realistic than most prior setups.
  • The ablations on description quality and memory organization are useful. Table 3 clearly shows the Map helps, especially with the weaker local describer (26 → 41 at 8 minutes).
Critical weaknesses
  1. Benchmark correctness is not validated. Every evidence comes from model-generated descriptions, and every question is written by an LLM agent (Codex). The only human review covers the canonical and alternative answer strings, not whether the question and answer are actually correct for the video. The authors say themselves that no final-question error rate and no human–judge agreement are reported. On top of that, scoring uses a text-only Qwen3.5-9B judge that never sees the video. For a benchmark paper this is a major issue: without a human-verified sample, we can't tell how much of the reported difference between methods reflects real ability and how much is annotation or judging noise.

  2. Possible bias toward text-memory methods, including the proposed one. The questions are written from dense textual descriptions of the video, which is exactly the kind of memory AsyncMemFlow stores. The paper doesn't clearly say which model produced the benchmark annotations. If it is Qwen-VL-Max or a closely related model, there is a circularity: the method's describer produces text in the same style as the source of the questions. There is also no blind or text-only check of whether questions can be answered without the video. This matters because the paper's main empirical claims rest on this benchmark: textual memory beats visual memory over long horizons, and AsyncMemFlow beats all baselines.

  3. The method comparison is unfair, and the novelty is limited. AsyncMemFlow's headline results use a large API model (Qwen-VL-Max) to build memory, while the baselines run 7–8B models end to end. The reported 0.9–1.5 s latency also leaves out the cost of background description. With the fair local 8B describer the gains shrink. The design itself (asynchronous memory writing, hierarchical event memory) is already present in OASIS, StreamMind, M3-Agent, and StreamChat, as the related-work section acknowledges. So the method contribution is mostly a stronger describer plus a router, which weakens the second half of the paper.

Questions for authors

Clarifications (answers could change my score)

  1. Which model(s) generated the three-scale benchmark descriptions? Is it Qwen-VL-Max, or the same family as the AsyncMemFlow describer? If so, how do you rule out an advantage for methods that store text in the same style?
  2. Could you report a human-verified error rate on a random sample of final questions (say 200–300), checked against the video rather than against the annotations?
  3. How well does the text-only Qwen3.5-9B judge agree with human scores on held-out outputs from methods other than Qwen2.5-VL raw frames, e.g. AsyncMemFlow and OASIS? Is the partial-credit (0.5) scoring consistent across judges?
  4. Have you run a blind or text-only baseline, where the answerer sees only the question with no video or memory? This would show how many questions can be answered from priors or wording cues.
  5. What are the rejection (REJECT) rates per method and per question type? Several baselines score near zero on Historical, and it would help to know if that comes from memory failure or from the 5-word answer format and refusals.
  6. What does memory construction cost for AsyncMemFlow (GPU-hours or API calls per video hour, throughput)? How does it compare with the baselines under a matched compute budget?

Suggestions (would not change my score)

  1. Report results with a describer from a different model family (e.g. InternVL or Gemini) to show the conclusions don't depend on Qwen.
  2. Report how much questions from the same video overlap, and how many questions come from each video, given there are only 49 videos.
1 comment
Reviewer_jNNjOpenReview ↗

Official Comment by Reviewer jNNj

Thanks for the detailed response. While some of my concerns are addressed, I still keep my score because the paper's main claims still rest on a benchmark whose validity is not confirmed.

  • There is no human-verified error rate on the final questions, and judge agreement is based on aggregate counts from a single video, so it is unclear whether score differences between methods exceed label noise.
  • Additionally, the evaluation has an untested circular dependency on Qwen-VL. The rebuttal confirms that Qwen-VL-Max produced all benchmark annotations and is also AsyncMemFlow's strongest describer, while the judge and most answerers are also Qwen models. AsyncMemFlow's gains is also measured on questions built from Qwen-VL-Max text descriptions. Without a validated question set, a describer from a different family, or a judge from a different family, it is unclear how much of the advantage of text memory reflects real retention rather than alignment with how the benchmark was built.
  • Finally, precomputing memory independently of queries is already used in prior streaming methods such as OASIS, StreamChat etc. AsyncMemFlow's design follows the benchmark's findings directly, so it mainly confirms them rather than adding a separate methodological contribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.