- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces OpenVA-Bench, a benchmark for evaluating open-world video anomaly understanding in Video-LLMs. It integrates videos from three datasets and evaluates many models across prediction accuracy, confidence reliability, and reasoning quality. The experiments reveal limitations in unknown anomaly rejection, confidence calibration, and explanation consistency, highlighting challenges in open-world anomaly understanding.
Critical strengths
The paper addresses a meaningful problem by investigating the ability of Video-LLMs to recognize anomalies beyond predefined categories. The benchmark provides a unified evaluation framework covering prediction, confidence, and reasoning, offering a more comprehensive assessment than classification accuracy alone.
Critical weaknesses
1. Insufficient discussion of task definition and scope. The paper frames its task as open-world video anomaly understanding, but the evaluation is limited to video-level classification under predefined Known–Unknown category splits. Open-world VAD can involve broader challenges, such as temporal anomaly localization, explainability [1], and dynamic categories [2]. While different interpretations of open-world settings are reasonable, the authors should better discuss these distinctions and clarify the scope and limitations of their task formulation.
2. Insufficient and potentially biased evaluation. The evaluation exclusively considers general-purpose MLLMs, overlooking existing MLLM-based methods specifically developed for VAD/VAU. Since the proposed task does not appear fundamentally incompatible with these methods, including domain-specific baselines is important for a comprehensive evaluation. Furthermore, the fixed uniform frame sampling strategy may miss short anomalies in long videos, potentially underestimating model performance and making it difficult to distinguish sampling failures from limitations in anomaly understanding.
[1] Hawk: learning to understand open-world video anomalies. NeurIPS 2024.
[2] Language-guided open-world video anomaly detection under weak supervision. ICLR 2026.
Questions for authors
-
Could the authors clarify the scope of their open-world video anomaly understanding task and discuss its relationship to other methods that have different definitions of open-world VAD/VAU?
-
Why does the evaluation exclude existing MLLM-based VAD/VAU methods, given that the proposed task does not appear fundamentally incompatible with them?
-
How do the authors ensure that fixed uniform frame sampling does not miss short anomalies in long videos and consequently underestimate model performance?
-
Is confidence reliability a distinct capability required for open-world VAD, or is it a general property of classification models that is equally relevant in closed-set settings?
-
The boundary between known and unknown categories is not always well-defined, as different anomaly categories may share overlapping visual characteristics. For example, if a video depicts law enforcement officers arresting an individual, but the known categories include Abuse while excluding Arrest, classifying the event as Abuse could still be reasonable based on the observed physical interactions. However, under the proposed evaluation protocol, such a prediction would be considered incorrect. More generally, how do the authors determine whether an observed event sufficiently matches a known category? Since there may always exist a more specific or appropriate category outside the predefined set.
10 comments
Response to Reviewer 2CDp (2/5)
Thank you for raising this point. We agree that existing MLLM-based VAD/VAU methods are relevant baselines for the proposed task. We have therefore expanded the evaluation to include VAD-R1, HAWK-7B, and Holmes-VAU, covering all 1,330 benchmark videos, and assessed both their category predictions and generated explanations. VAD-R1 achieves 39.77% OS*, 26.79% UNK, and 32.01% HOS. HAWK-7B achieves 24.92% OS*, 41.34% UNK, and 31.09% HOS. Holmes-VAU achieves 35.23% OS*, 2.77% UNK, and 5.14% HOS. Their Macro-HOS scores are 25.17%, 29.07%, and 3.94%, respectively. These results reveal different recognition and rejection behaviors: VAD-R1 has the highest known-class accuracy among the three, whereas HAWK-7B has the highest unknown rejection. Holmes-VAU identifies some known categories but rarely rejects anomalies outside the candidate set. We also evaluated the explanations using DeepSeek, GPT, and Claude. Under the original DeepSeek judge, the paired EDC/LC scores are 57.98/94.88 for VAD-R1, 16.29/72.35 for HAWK-7B, and 42.02/63.17 for Holmes-VAU, each calculated on 1,329 records with available references. The three judges produce consistent model orderings for both explanation metrics. These additions demonstrate that OpenVA-Bench can evaluate specialized VAD/VAU systems alongside general-purpose Video-LLMs. Table R2. Specialized VAD/VAU baselines on OpenVA-Bench. All scores are percentages; higher is better.
\hline \text{Model} & \mathrm{OS}^{*} & \mathrm{UNK} & \mathrm{HOS} & \text{Macro-HOS} & \mathrm{EDC} & \mathrm{LC} \\\\ \hline \text{VAD-R1} & 39.77 & 26.79 & 32.01 & 25.17 & 57.98 & 94.88 \\\\ \text{HAWK-7B} & 24.92 & 41.34 & 31.09 & 29.07 & 16.29 & 72.35 \\\\ \text{Holmes-VAU} & 35.23 & 2.77 & 5.14 & 3.94 & 42.02 & 63.17 \\\\ \hline \end{array}$$Response to Reviewer 2CDp (3/5)
Thank you for highlighting the potential effect of temporal sampling. To examine this issue, we added a reference-interval-guided sampling experiment while retaining a total budget of 32 frames per video. For anomalous videos with usable reference intervals, we sample 16 frames uniformly across the full video and allocate another 16 frames across the event intervals. The additional frames are distributed approximately equally across multiple intervals and sampled uniformly within each interval. The selected frames are then presented in chronological order. Normal videos and videos without usable intervals retain uniform 32-frame sampling. This produces event-guided inputs for 741 anomalous videos; the remaining 589 videos comprise 570 normal videos and 19 anomalous videos without usable intervals. We evaluated Doubao-Seed-2.1-Turbo, Kimi-K2.6, and Qwen3-VL-8B-Instruct on the full 1,330-video benchmark. Compared with the original uniform-sampling results, unknown-anomaly rejection increases from 75.06% to 77.37% for Doubao, 58.66% to 62.20% for Kimi, and 39.72% to 43.96% for Qwen. Their HOS scores increase from 69.51 to 71.97, 64.60 to 68.68, and 42.10 to 46.66, respectively. These results are consistent with improved event coverage benefiting classification and rejection. The principal recognition–rejection pattern also remains: under event-guided sampling, Kimi achieves higher known-class accuracy than Doubao (73.82% versus 67.27%), whereas Doubao achieves higher unknown rejection (77.37% versus 62.20%). The HOS ordering of the three models is unchanged. Event-guided sampling raises HOS by at most 4.6 points (Doubao +2.46, Kimi +4.08, Qwen +4.56) and unknown rejection by at most 4.2 points, with the model ordering unchanged. Uniform 32-frame sampling therefore underestimates performance by a bounded amount — at most 4.6 HOS points — and does not alter any model ranking or the recognition–rejection pattern. Table R3. Uniform versus event-guided sampling on all 1,330 videos, using 32 frames per video.
\hline \text{Model} & \text{Sampling} & \mathrm{UNK} & \Delta\mathrm{UNK} & \mathrm{HOS} & \Delta\mathrm{HOS} \\\\ \hline \text{Doubao-Seed-2.1-Turbo} & \text{Uniform} & 75.06 & \text{—} & 69.51 & \text{—} \\\\ & \text{Event-guided} & 77.37 & +2.31 & 71.97 & +2.46 \\\\ \hline \text{Kimi-K2.6} & \text{Uniform} & 58.66 & \text{—} & 64.60 & \text{—} \\\\ & \text{Event-guided} & 62.20 & +3.54 & 68.68 & +4.08 \\\\ \hline \text{Qwen3-VL-8B-Instruct} & \text{Uniform} & 39.72 & \text{—} & 42.10 & \text{—} \\\\ & \text{Event-guided} & 43.96 & +4.24 & 46.66 & +4.56 \\\\ \hline \end{array}$$
Response to Reviewer 2CDp (1/5)
Thank you for requesting this clarification. OpenVA-Bench studies a specific aspect of open-world video anomaly understanding: whether a Video-LLM can recognize events covered by a supplied category set and reject anomalous events whose categories are absent from that set, together with the confidence and explanations accompanying its decisions. We operationalize this question through a controlled open-set protocol. Each model receives video frames and a candidate-category set with textual definitions, and outputs a category decision, self-reported confidence scores, and an explanation. The output space includes the supplied anomaly categories, Normal, and Unknown_Anomaly. Known/unknown status is determined by whether the harmonized event category is included in the candidate set. Thus, “unknown” is defined relative to the available output categories; it does not imply that the underlying event concept was absent from the model’s pretraining data. This scope complements several existing formulations. UBnormal evaluates detection of anomaly types held out from the benchmark’s training set. Open-vocabulary VAD, such as OVVAD, studies detecting and categorizing both seen and unseen anomalies. Language-guided open-world VAD, exemplified by LaGoVAD, studies adapting anomaly detection to user-provided definitions at inference time. These formulations respectively emphasize generalization across anomaly types, recognition with an expanded semantic vocabulary, and changes in what should count as anomalous. OpenVA-Bench focuses on a complementary decision problem: given the available category definitions, does an anomalous event warrant one of those labels, or should the model reject the available anomaly categories? The benchmark measures known-class recognition and unknown-anomaly rejection, and examines confidence reliability, event-description consistency, and logical coherence. Its video-level protocol therefore addresses category decisions and associated explanations, rather than temporal anomaly localization or incremental learning of newly discovered classes. We will clarify this distinction in the task definition and related work, explicitly describing OpenVA-Bench as a controlled open-set evaluation of Video-LLMs motivated by open-world anomaly understanding, and tie our conclusions to this evaluated setting.