- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces STORM-Bench, an online video QA benchmark built from short, change-dense egocentric and simulated episodes, where each question is labeled by accumulated scene change and by whether it is answerable from the frames observed so far. The authors propose a decoupled answerability probe and a balanced harmonic metric, STORM-BR, to jointly score answer correctness and abstention behavior. Experiments on open-source video LLMs suggest that standard accuracy overstates reliability, with models being frequently overconfident on unanswerable queries.
Critical strengths
The paper is extensively documented. It provides a large number of figures, tables, and appendix sections covering data construction, prompts, evaluation settings, and supplementary analyses across multiple domains and models.
Critical weaknesses
-
Unclear motivation and hard to read. The introduction does not clearly explain why this task is needed. Several motivations are mixed together, and the claimed interaction between state tracking and abstention is never demonstrated. The main text is dense and heavy on terminology, and key definitions are placed in the appendix.
-
Overly complex method with design flaws. One evaluated model (Qwen3.5-9B) was used to refine the benchmark questions. The answer and the answerability status come from two independent calls, so joint correctness may reflect cross-prompt consistency rather than genuine awareness. The metrics are numerous and inconsistently defined.
-
Several models score a higher STORM-BR without video input, which questions whether the metric measures visual reliability. In addition, agentic settings are not considered: because the abstention option appears only on Uncertain items, an agent that knows the construction could exploit this cue to score well without watching the video.
Figure 3 overflows into the text and covers Eq. 1. Please check the formatting before submission.
Questions for authors
-
Which real-world application does this benchmark represent? The episodes are compact clips spliced from repeated visits with black transitions, and much of the missing evidence is introduced by this construction.
-
Current agent systems (e.g., Codex with GPT) can use skills and tools. If a user knows how this benchmark is constructed, they could write a skill that undoes the decoupled probe: for example, choose the abstention option whenever it appears and report “Uncertain” accordingly, then use tools to handle the remaining questions. Would such an agent obtain a high score without genuinely understanding the video?