acceptodds
Withdrawn from ICLR 2027

STORM-Bench: Evaluating Online Video QA under Evolving and Incomplete Evidence

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reliable online video question answering requires tracking state transitions while selectively abstaining when visual evidence is insufficient. Existing benchmarks focus on static recognition or long-range retrieval, rarely evaluating these coupled capabilities under evolving and incomplete evidence. We present STORM-Bench, comprising 5,736 questions across 630 compact, change-dense episodes spanning five egocentric domains (STORM-Real) and two controlled simulation subsets (STORM-Sim) at 1 FPS. Questions are stratified by a proxy for accumulated change intensity (Low, Medium, High) and query-time answerability (Known, Uncertain). To measure reliability, we introduce STORM-BR, a harmonic metric over joint answer-status correctness that exposes abstention failures masked by aggregate accuracy, alongside STORM-BR-ATTR for uncertainty attribution. Across 14 video LLMs, online accuracy peaks at 60.3% (mean 51.7%), whereas STORM-BR ranges from 5.7% to 35.6% (mean 18.8%), driven by pervasive overconfidence on uncertain queries. STORM-Bench shows that task accuracy masks these gaps in epistemic reliability and state tracking. Evaluation code is available at https://anonymous.4open.science/r/STORM-Bench/.

Primary area:
datasets and benchmarks
Keywords:
video understanding, benchmark, multimodal large language model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zyLVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces STORM-Bench, an online video QA benchmark built from short, change-dense egocentric and simulated episodes, where each question is labeled by accumulated scene change and by whether it is answerable from the frames observed so far. The authors propose a decoupled answerability probe and a balanced harmonic metric, STORM-BR, to jointly score answer correctness and abstention behavior. Experiments on open-source video LLMs suggest that standard accuracy overstates reliability, with models being frequently overconfident on unanswerable queries.

Critical strengths

The paper is extensively documented. It provides a large number of figures, tables, and appendix sections covering data construction, prompts, evaluation settings, and supplementary analyses across multiple domains and models.

Critical weaknesses
  1. Unclear motivation and hard to read. The introduction does not clearly explain why this task is needed. Several motivations are mixed together, and the claimed interaction between state tracking and abstention is never demonstrated. The main text is dense and heavy on terminology, and key definitions are placed in the appendix.

  2. Overly complex method with design flaws. One evaluated model (Qwen3.5-9B) was used to refine the benchmark questions. The answer and the answerability status come from two independent calls, so joint correctness may reflect cross-prompt consistency rather than genuine awareness. The metrics are numerous and inconsistently defined.

  3. Several models score a higher STORM-BR without video input, which questions whether the metric measures visual reliability. In addition, agentic settings are not considered: because the abstention option appears only on Uncertain items, an agent that knows the construction could exploit this cue to score well without watching the video.

Figure 3 overflows into the text and covers Eq. 1. Please check the formatting before submission.

Questions for authors
  1. Which real-world application does this benchmark represent? The episodes are compact clips spliced from repeated visits with black transitions, and much of the missing evidence is introduced by this construction.

  2. Current agent systems (e.g., Codex with GPT) can use skills and tools. If a user knows how this benchmark is constructed, they could write a skill that undoes the decoupled probe: for example, choose the abstention option whenever it appears and report “Uncertain” accordingly, then use tools to handle the remaining questions. Would such an agent obtain a high score without genuinely understanding the video?

Reviewer_j4buOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces STORM-Bench, an online Video-QA benchmark designed to evaluate state tracking and selective abstention under evolving and incomplete evidence. The benchmark comprises 5,736 questions across 630 episodes spanning five real-world egocentric domains (STORM-Real) and two controlled simulation subsets (STORM-Sim). Questions are categorized along two dimensions: accumulated state-change intensity (Low, Medium, High) and query-time answerability (Known, Uncertain). The authors further introduce a decoupled evidence sufficiency probing protocol and STORM-BR, a balanced reliability metric that aggregates joint answer and status correctness across six evaluation conditions using a harmonic mean. Experiments on 14 VideoLLMs reveal widespread overconfidence under insufficient evidence and demonstrate that conventional QA accuracy does not fully capture these reliability gaps.

Critical strengths
  1. By jointly evaluating state tracking and selective abstention under evolving and incomplete evidence, STORM-Bench complements existing benchmarks that primarily focus on static recognition or long-range retrieval. This setting provides a useful perspective on the challenges of maintaining accurate state information and recognizing insufficient visual evidence.
  2. STORM-Bench covers both real-world egocentric domains and controlled simulation subsets, with experiments across 14 video LLMs. The authors analyze performance across semantic question types, accumulated change intensity, query-time answerability, and input controls, revealing limitations in history aggregation and overconfidence on Uncertain queries.
Critical weaknesses
  1. Each figure and table in the main paper appears to have excessively compressed spacing, potentially due to the use of negative \vspace adjustments. In particular, Figure 3 contains severe overlaps between subfigures, captions, and the main text, which substantially affects readability. Given ICLR's strict formatting and page-limit requirements, the authors should clarify whether the submission complies with the official template.
  2. STORM-Real assembles temporally separated clips from repeated visits to shared functional regions, introducing artificial observation gaps that may not fully reflect natural online video streams. Moreover, accumulated change intensity is defined by revisit or event intervals rather than actual physical state transitions. It remains unclear whether this measure accurately reflects the difficulty of state tracking. Although the authors conduct human review, quantitative inter-annotator agreement for Known/Uncertain labels is not reported.
Questions for authors
  1. The paper uses the number of revisit or event intervals as a measure of accumulated change intensity. How well does this measure correlate with the actual number of physical state transitions?
  2. Can the authors report inter-annotator agreement for the Known/Uncertain labels and the proportion of cases requiring adjudication? Are the same criteria for query-time answerability consistently applied across STORM-Real and STORM-Sim?
  3. In STORM-Real, the explicit abstention option appears only in Uncertain questions, potentially revealing answerability through option structure. Although decoupled probing reduces this issue, the primary QA task may still exploit this cue. Can the authors evaluate a setting where all questions include an abstention option, and report how this affects task accuracy, joint answer–status correctness, and STORM-BR?
Reviewer_A1HqOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces STORM-Bench, a 5736 question multiple-choice benchmark built from stitched HD-EPIC/Ego-Exo4D episodes (around 45s) plus small AI2-THOR/VirtualHome subsets. Each question is labeled Known or Uncertain at query time, with six annotated causes for uncertainty, and stratified by accumulated change intensity. The authors propose a joint answer/status metric (J), a smoothed harmonic-mean score over intensity/answerability cells (STORM-BR), and an overconfidence rate. The authors evaluate 14 open-weight video LLMs zero-shot on causal prefixes.

Critical strengths

Separating “can the model answer” from “does the model know it can answer” is an important aspect in deployed assistants, and the cause taxonomy (missing vs. partial vs. ambiguous evidence, etc.) is also an useful diagnostic. Asking for the status before showing the task options is an interesting design choice that reduces some of the option-format leakage.

Critical weaknesses
  1. Novelty and positioning: It is hard to tell from the contribution list what is actually new. The core idea, whether enough evidence has been observed at query time, is close to the readiness formulation in StreamReady. StreamReady also evaluates offline models on truncated prefixes and explicitly raises unanswerability, yet the paper describes it only as “response timing.” The joint answer/abstain scoring also has a direct precedent in selective VQA, specifically Effective Reliability and risk-coverage evaluation [R1,R2]. Neither is discussed or compared against. Prior work also reports that video LLMs fail to refuse unanswerable questions and that scaling does not fix this [R3, R4], which weakens the novelty of the main empirical finding.

  2. The experiments don’t support the “evolving evidence/history” framing: The full-video control changes accuracy by at most about 2 points. Most evidence lies within about 2 s of the query. Keeping only the last 16 frames improves STORM-BR for 9 of 14 models. So the benchmark mainly tests recent-frame perception plus abstention, not history tracking. It seems that the answer format also leaks the status label.

  3. No actual streaming models [R5-R7] or proprietary models are evaluated. If the history and memory claims stay, memory-based long-video models and state-tracking/object-permanence benchmarks should be discussed [R8-R10].

  4. Presentation. Several figures are unreadable because of tiny fonts and dense panels. In Fig. 3, leftover panel text overlaps the caption and body text.

Missing References:

[R1] S. Whitehead, S. Petryk, V. Shakib, J. Gonzalez, T. Darrell, A. Rohrbach, M. Rohrbach. Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectly. ECCV 2022.

[R2] C. Dancette, S. Whitehead, R. Maheshwary, R. Vedantam, S. Scherer, X. Chen, M. Cord, M. Rohrbach. Improving Selective Visual Question Answering by Learning From Your Peers. CVPR 2023.

[R3] E. Yoon, H. S. Yoon, M. Hasegawa-Johnson, C. D. Yoo. Can Video LLMs Refuse to Answer? Alignment for Answerability in Video Large Language Models. ICLR 2025.

[R4] P. Kirichenko, M. Ibrahim, K. Chaudhuri, S. Bell. AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions. NeurIPS 2025.

[R5] J. Chen et al. VideoLLM-online: Online Video Large Language Model for Streaming Video. CVPR 2024.

[R6] R. Qian et al. Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction. CVPR 2025.

[R7] H. Wang et al. StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant. NeurIPS 2025.

[R8] B. He et al. MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding. CVPR 2024.

[R9] S. Azad, V. Vineet, Y. S. Rawat. HierarQ: Task-Aware Hierarchical Q-Former for Enhanced Video Understanding. CVPR 2025.

[R10] C. Plizzari et al. Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videos. CVPR 2025.

Questions for authors

Clarifications:

  1. How does STORM-BR compare with Effective Reliability or risk-coverage on the same predictions? Does it rank models differently?

  2. What do the results look like if abstention options are added to Known items as well (or removed everywhere), so format can’t leak the label?

  3. Can you provide human agreement on the Known/Uncertain and cause labels for a sample?

  4. Can you report a subset where evidence is >10 s old, and explain why the recent-16-frame variant scores higher?

  5. It will be good to evaluate at least some streaming models (e.g., Dispider, StreamBridge) and some proprietary models, and consider longer, non-stitched streams.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.